Because the excess entropy production rate is defined in terms of the $L^2$-Wasserstein distance $\sigma_{t}^{\rm ex} = [\mathcal{W}_2 (P_{\tau}, P_{\tau+dt})]^2/( \mu T dt^2)$ up to $O(dt)$, Eq. (<ref>) implies a relation between the $L^2$-Wasserstein distance and the Kullback–Leibler divergence, and Eq. (<ref>) implies a relation between the $L^2$-Wasserstein distance and the Fisher metric, respectively. Information-geometrically, the square of the line element for path probability densities can be defined as $ds^2_{\rm path} = g_{\theta (\boldsymbol{\nu}_{\tau} - \boldsymbol{\nu}^*_{\tau})\theta (\boldsymbol{\nu}_{\tau} - \boldsymbol{\nu}^*_{\tau})} d\theta^2 =(1/2) \sigma^{\rm ex}_{\tau}  dt d\theta^2$ for the interpolation parameter $\theta$ in $\mathbb{P}^{\theta}_{\boldsymbol{\nu}_{\tau} - \boldsymbol{\nu}^*_{\tau}}$. Thus, the $L^2$-Wasserstein distance can be information-geometrically interpreted as $ [\mathcal{W}_2 (P_{\tau}, P_{\tau+dt})]^2 = (2\mu T dt) (ds^2_{\rm path} / d\theta^2)$.

§ CONCLUSION AND DISCUSSION

We discuss stochastic thermodynamic links between information geometry and optimal transport theory via the excess entropy production rate. We can discuss a link between information geometry in the space of probability densities and optimal transport theory and a link between information geometry in the space of path probability densities and optimal transport theory because the excess entropy production rate is related to the $L^2$-Wasserstein distance, the time derivative of the Kullback–Leibler divergence between probability densities, and the Kullback–Leibler divergence between the path probability densities. These links are useful for studying the mathematical properties of the entropy production rate in stochastic thermodynamics. For example, thermodynamic trade-off relations, namely the thermodynamic uncertainty relations and the  thermodynamic speed limit, can be obtained from geometric inequalities such as the Cauchy–Schwartz inequality and the triangle inequality. The optimal protocol to minimize the thermodynamic cost can also be discussed in terms of the geodesic.

We also remark on possible generalizations of the results in this paper. In this paper, we only focus on the stochastic dynamics described by the Fokker–Planck equation. Because stochastic thermodynamics has been discussed for the general Markov process described by the master equation, the generalization of the proposed results for the general Markov process is interesting. For example, generalizations of the results in this paper for the master equation have been seen in Refs. [22, 25, 76, 60, 28, 38, 41, 43, 44]. Because generalizations are not unique, rather different approaches of optimal transport theory for stochastic thermodynamics in the Markov jump process have also been seen in Refs. [77, 78, 79, 80]. Unlike these generalizations, our generalizations [43, 44] are related to the gradient flow expression and information-geometric projection discussed in this paper. The generalization for the deterministic chemical rate equation is also interesting to consider information geometry and optimal transport theory for chemical thermodynamics, which was proposed in Refs. [29, 81, 42, 82, 43, 83, 44, 80]. For the deterministic chemical rate equation, we do not need to use stochasticity to obtain generalized results, and geometric properties play a crucial role in the derivation of generalized results, similarly as in the stochastic case. This is the reason why we call our framework geometric thermodynamics instead of stochastic thermodynamics.

Finally, we point out that geometric thermodynamics is related to several fascinating topics, and has the potential to clarify the geometric properties of these topics. The classical correspondence of the control protocol called shortcuts to adiabatically for the stochastic process [84, 85, 86, 87, 88] is related to the geometry of the probability distribution. Remarkably, the link between shortcuts and information geometry has been proposed in Ref. [89]. Indeed, a generalization of our framework for general Markov jump processes [43] is related to the stochastic correspondence of shortcuts called shortcuts in stochastic systems [86].
An application of shortcuts or our framework to a low-power electronic circuits called adiabatic circuits [90] is promising.
A connection between geometric thermodynamics and a geometrical interpretation of another excess entropy production rate proposed in Ref. [91] in terms of the Berry phase [92], which is related to the geometry of the cyclic path, is interesting. The cyclic path in information geometry and optimal transport theory was discussed in the optimal heat engine [27, 31, 32, 93, 94] and the geometric pump [95]. A geometric interpretation of the restricted path may also be interesting in the context of optimal limited control [96, 97, 98, 99]. The dual coordinate systems in stochastic thermodynamics provide the duality in stochastic thermodynamics [100, 41, 42, 82, 83, 44, 101], which is related to the variational calculus such as the maximum caliber principle [102, 103, 44, 101] and the Schrödinger bridge [104, 105]. Our results may also useful for a machine learning technique based on non-equilibrium thermodynamics called the diffusion model [106] or the score-based generative modeling, because our result provides a link between machine learning technique based on optimal transport and non-equilibrium thermodynamics for diffusion dynamics described by the Fokker–Planck equation. Because the denoising matching [107, 108] or the reversed stochastic differential equation [109] can be described by proposed interpolated dynamics, our framework may be helpful in learning process by the diffusion-based generative model. Because our framework also can provides a link between the reversed stochastic differential equation and optimal transport theory, which is well used in generative models such as the Wasserstein generative adversarial networks [110], it might be interesting to consider a link between the diffusion model and learning based on the $L^2$-Wasserstein distance in our framework.
A connection to information thermodynamics [111] is also interesting. For example, the information-thermodynamic quantities, called the partial entropy production and the transfer entropy [112, 113, 114, 115, 116] can be information-geometrically treated by the projection theorem [117, 25], and the optimal transport for the subsystem is related to the problem of the finite bit erasure [118, 119, 120, 121] and the problem of the minimum partial entropy production in the subsystem [31, 122]. Applications to the evolutionary process [123, 124, 125, 45] is also interesting because information geometry provides a geometric interpretation of the Price equation [126, 127].
As a generalization of the gradient flow expression, an approach based on the general equation for non-equilibrium reversible-irreversible coupling (GENERIC) [128, 129, 130, 131] or the transport Hessian Hamiltonian flow [132] might be promising, The experimental application of geometric thermodynamics to biological dynamics is interesting [133, 134, 37], and the quantitative discussion on the design principle of the complex biological system from the viewpoint of geometric thermodynamics could be a significant topic in the near future.

S. I. gratefully thank Andreas Dechant, Kohei Yoshimura, Naruo Ohga, Artemy Kolchinsky, Shin-ichi Sasa, Muka Nakazato, Takahiro Sagawa, Shun Otsubo, Takuya Kamijima, Yuma Fujimoto, Ryuna Nagayama, Masahiro Hoshino, Jumpei F. Yamagishi, Masafumi Oizumi and Shun-ich Amari for valuable discussions on geometric aspects in stochastic thermodynamics and chemical thermodynamics. S. I. appreciate Artemy Kolchinsky, Ryuna Nagayama, Kohei Yoshimura and Naruo Ohga for careful reading of this manuscript. S. I. is supported by JST Presto Grant No. JPMJPR18M2 and JSPS KAKENHI Grants No. 19H05796, No. 21H01560, and No. 22H01141.
Data Availability
Data sharing is not applicable to this article as no data sets were generated or analyzed during the current study.
On behalf of all authors, the corresponding author states that there is no conflict of interest.

#1ISBN #1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<https://doi.org/#1>et al.#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1#1<><#>1#1#1#1#1#1#1#1#1#1#1#1#1#1PreBibitemsHook

[1]

Callen, H.B.:
Thermodynamics and an Introduction to Thermostatistics.
American Association of Physics Teachers

[2]

Einstein, A.:
Investigations on the Theory of the Brownian Movement.
Courier Corporation

[3]

Onsager, L.:
Reciprocal relations in irreversible processes. i.
Physical review

[4]

Onsager, L.:
Reciprocal relations in irreversible processes. ii.
Physical review

[5]

Glansdorff, P.,
Nicolis, G.,
Prigogine, I.:
The thermodynamic stability theory of non-equilibrium states.
Proceedings of the National Academy of Sciences

[6]

Schnakenberg, J.:
Network theory of microscopic and macroscopic behavior of master
equation systems.
Reviews of Modern physics

[7]

Weinhold, F.:
Metric geometry of equilibrium thermodynamics.
The Journal of Chemical Physics

[8]

Ruppeiner, G.:
Thermodynamics: A riemannian geometric model.
Physical Review A

[9]

Salamon, P.,
Berry, R.S.:
Thermodynamic length and dissipated availability.
Physical Review Letters

[10]

Ruppeiner, G.:
Riemannian geometry in thermodynamic fluctuation theory.
Reviews of Modern Physics

[11]

Crooks, G.E.:
Measuring thermodynamic length.
Physical Review Letters

[12]

Amari, S.-i.,
Nagaoka, H.:
Methods of information geometry.
American Mathematical Soc.

[13]

Amari, S.-i.:
Information geometry and its applications.

[14]

Rao, C.R.:
Information and the accuracy attainable in the estimation of
statistical parameters.
Bull. Calcutta Math. Soc.

[15]

Sekimoto, K.:
Stochastic energetics.

[16]

Seifert, U.:
Stochastic thermodynamics, fluctuation theorems and molecular
Reports on progress in physics

[17]

Ge, H.,
Qian, H.:
Nonequilibrium thermodynamic formalism of nonlinear chemical reaction
systems with waage–guldberg’s law of mass action.
Chemical Physics

[18]

Rao, R.,
Esposito, M.:
Nonequilibrium thermodynamics of chemical reaction networks: wisdom
from stochastic thermodynamics.
Physical Review X

[19]

Aurell, E.,
Mejía-Monasterio, C.,
Muratore-Ginanneschi, P.:
Optimal protocols and optimal transport in stochastic thermodynamics.
Physical review letters

[20]

Sivak, D.A.,
Crooks, G.E.:
Thermodynamic metrics and optimal paths.
Physical review letters

[21]

Mandal, D.,
Jarzynski, C.:
Analysis of slow transitions between nonequilibrium steady states.
Journal of Statistical Mechanics: Theory and Experiment

[22]

Ito, S.:
Stochastic thermodynamic interpretation of information geometry.
Physical review letters

[23]

Chen, Y.,
Georgiou, T.T.,
Tannenbaum, A.:
Stochastic control and nonequilibrium thermodynamics: Fundamental
IEEE transactions on automatic control

[24]

Ito, S.,
Dechant, A.:
Stochastic time evolution, information geometry, and the
cramér-rao bound.
Physical Review X

[25]

Ito, S.,
Oizumi, M.,
Amari, S.-i.:
Unified framework for the entropy production and the stochastic
interaction based on information geometry.
Physical Review Research

[26]

Nicholson, S.B.,
García-Pintos, L.P.,
del Campo, A.,
Green, J.R.:
Time–information uncertainty relations in thermodynamics.
Nature Physics

[27]

Brandner, K.,
Saito, K.:
Thermodynamic geometry of microscopic heat engines.
Physical review letters

[28]

Van Vu, T.,
Hasegawa, Y.:
Geometrical bounds of the irreversibility in markovian systems.
Physical Review Letters

[29]

Yoshimura, K.,
Ito, S.:
Information geometric inequalities of chemical thermodynamics.
Physical Review Research

[30]

Kolchinsky, A.,
Wolpert, D.H.:
Work, entropy production, and thermodynamics of information under
protocol constraints.
Physical Review X

[31]

Nakazato, M.,
Ito, S.:
Geometrical aspects of entropy production in stochastic thermodynamics
based on wasserstein distance.
Physical Review Research

[32]

Fu, R.,
Taghvaei, A.,
Chen, Y.,
Georgiou, T.T.:
Maximal power output of a stochastic thermodynamic engine.

[33]

Van Kampen, N.G.:
Stochastic processes in physics and chemistry.

[34]

Van den Broeck, C.,
Esposito, M.:
Three faces of the second law. ii. fokker-planck formulation.
Physical Review E

[35]

Villani, C., et al.:
Optimal transport: old and new.

[36]

Villani, C.:
Topics in optimal transportation.
American Mathematical Soc.

[37]

Ashida, K.,
Aoki, K.,
Ito, S.:
Experimental evaluation of thermodynamic cost and speed limit in living cells
via information geometry.

[38]

Ito, S.:
Information geometry, trade-off relations, and generalized
glansdorff–prigogine criterion for stability.
Journal of Physics A: Mathematical and Theoretical

[39]

Dechant, A.,
Sasa, S.-i.,
Ito, S.:
Geometric decomposition of entropy production in out-of-equilibrium
Physical Review Research

[40]

Dechant, A.,
Sasa, S.-i.,
Ito, S.:
Geometric decomposition of entropy production into excess,
housekeeping, and coupling parts.
Physical Review E

[41]

Ohga, N.,
Ito, S.:
Inferring nonequilibrium thermodynamics from tilted equilibrium using
information-geometric legendre transform.
arXiv preprint arXiv:2112.11008

[42]

Ohga, N.,
Ito, S.:
Information-geometric structure for chemical thermodynamics: An
explicit construction of dual affine coordinates.
Physical Review E

[43]

Yoshimura, K.,
Kolchinsky, A.,
Dechant, A.,
Ito, S.:
Housekeeping and excess entropy production for general nonlinear
Physical Review Research

[44]

Kolchinsky, A.,
Dechant, A.,
Yoshimura, K.,
Ito, S.:
Information geometry of excess and housekeeping entropy production.
arXiv preprint arXiv:2206.14599

[45]

Hoshino, M.,
Nagayama, R.,
Yoshimura, K.,
Yamagishi, J.F.,
Ito, S.:
A geometric speed limit for acceleration by natural selection in evolutionary
arXiv preprint arXiv:2207.04640

[46]

Amari, S.-i.,
Karakida, R.,
Oizumi, M.:
Information geometry connecting wasserstein distance and
kullback–leibler divergence via the entropy-relaxed transportation problem.
Information Geometry

[47]

Li, W.:
Transport information geometry: Riemannian calculus on probability
Information Geometry

[48]

Khan, G.,
Zhang, J.:
When optimal transport meets information geometry.
Information Geometry

[49]

Wong, T.-K.L.,
Yang, J.:
Pseudo-riemannian geometry encodes information geometry in optimal
Information Geometry

[50]

Cover, T.M.:
Elements of information theory.
John Wiley & Sons

[51]

Crooks, G.E.:
Entropy production fluctuation theorem and the nonequilibrium work
relation for free energy differences.
Physical Review E

[52]

Chernyak, V.Y.,
Chertkov, M.,
Jarzynski, C.:
Path-integral analysis of fluctuation theorems for general langevin
Journal of Statistical Mechanics: Theory and Experiment

[53]

Kawai, R.,
Parrondo, J.M.,
Van den Broeck, C.:
Dissipation: The phase-space perspective.
Physical review letters

[54]

Dechant, A.,
Sasa, S.-i.:
Continuous time reversal and equality in the thermodynamic uncertainty
Physical Review Research

[55]

Beghi, A.:
On the relative entropy of discrete-time markov processes with given
end-point densities.
IEEE Transactions on Information Theory

[56]

Barato, A.C.,
Seifert, U.:
Thermodynamic uncertainty relation for biomolecular processes.
Physical review letters

[57]

Gingrich, T.R.,
Horowitz, J.M.,
Perunov, N.,
England, J.L.:
Dissipation bounds all steady-state current fluctuations.
Physical review letters

[58]

Dechant, A.:
Multidimensional thermodynamic uncertainty relations.
Journal of Physics A: Mathematical and Theoretical

[59]

Hasegawa, Y.,
Van Vu, T.:
Uncertainty relations in stochastic processes: An information
inequality approach.
Physical Review E

[60]

Liu, K.,
Gong, Z.,
Ueda, M.:
Thermodynamic uncertainty relation for arbitrary initial states.
Physical Review Letters

[61]

Benamou, J.-D.,
Brenier, Y.:
A computational fluid mechanics solution to the monge-kantorovich mass
transfer problem.
Numerische Mathematik

[62]

Jordan, R.,
Kinderlehrer, D.,
Otto, F.:
The variational formulation of the fokker–planck equation.
SIAM journal on mathematical analysis

[63]

Aurell, E.,
Gawȩdzki, K.,
Mejía-Monasterio, C.,
Mohayaee, R.,
Muratore-Ginanneschi, P.:
Refined second law of thermodynamics for fast random processes.
Journal of statistical physics

[64]

Dechant, A.,
Sakurai, Y.:
Thermodynamic interpretation of wasserstein distance.
arXiv preprint arXiv:1912.08405

[65]

Hatano, T.,
Sasa, S.-i.:
Steady-state thermodynamics of langevin systems.
Physical review letters

[66]

Maes, C.,
Netočnỳ, K.:
A nonequilibrium extension of the clausius heat theorem.
Journal of Statistical Physics

[67]

Dechant, A.,
Sasa, S.-i.:
Current fluctuations and transport efficiency for general langevin
Journal of Statistical Mechanics: Theory and Experiment

[68]

Li, W.,
Zhao, J.:
Wasserstein information matrix.
Information Geometry,

[69]

Maas, J.:
Gradient flows of the entropy for finite markov chains.
Journal of Functional Analysis

[70]

Nakamura, T.,
Hasegawa, H.,
Driebe, D.:
Reconsideration of the generalized second law based on information
Journal of Physics Communications

[71]

Shiraishi, N.,
Saito, K.:
Information-theoretical bound of the irreversibility in thermal
relaxation processes.
Physical review letters

[72]

Maes, C.,
Netočnỳ, K.:
Revisiting the glansdorff–prigogine criterion for stability within
irreversible thermodynamics.
Journal of Statistical Physics

[73]

Qian, H.:
Entropy production and excess entropy in a nonequilibrium steady-state
of single macromolecules.
Physical Review E

[74]

Rotskoff, G.M.,
Crooks, G.E.,
Vanden-Eijnden, E.:
Geometric approach to optimal nonequilibrium control: Minimizing
dissipation in nanomagnetic spin systems.
Physical Review E

[75]

Kamijima, T.,
Ito, S.,
Dechant, A.,
Sagawa, T.:
Thermodynamic uncertainty relations for steady-state thermodynamics.
arXiv preprint arXiv:2207.05332

[76]

Otsubo, S.,
Ito, S.,
Dechant, A.,
Sagawa, T.:
Estimating entropy production by machine learning of short-time
fluctuating currents.
Physical Review E

[77]

Dechant, A.:
Minimum entropy production, detailed balance and wasserstein distance
for continuous-time markov processes.
Journal of Physics A: Mathematical and Theoretical

[78]

Hamazaki, R.:
Speed limits for macroscopic transitions.
PRX Quantum

[79]

Van Vu, T.,
Saito, K.:
Thermodynamic unification of optimal transport: thermodynamic
uncertainty relation, minimum dissipation, and thermodynamic speed limits.
Physical Review X

[80]

Van Vu, T.,
Saito, K.:
Topological speed limit.
Physical Review Letters

[81]

Yoshimura, K.,
Ito, S.:
Thermodynamic uncertainty relation and thermodynamic speed limit in
deterministic chemical reaction networks.
Physical review letters

[82]

Sughiyama, Y.,
Loutchko, D.,
Kamimura, A.,
Kobayashi, T.J.:
Hessian geometric structure of chemical thermodynamic systems with
stoichiometric constraints.
Physical Review Research

[83]

Kobayashi, T.J.,
Loutchko, D.,
Kamimura, A.,
Sughiyama, Y.:
Hessian geometry of nonequilibrium chemical reaction networks and
entropy production decompositions.
Physical Review Research

[84]

Patra, A.,
Jarzynski, C.:
Shortcuts to adiabaticity using flow fields.
New Journal of Physics

[85]

Li, G.,
Chen, J.-F.,
Sun, C.,
Dong, H.:
Geodesic path for the minimal energy cost in shortcuts to
Physical Review Letters

[86]

Ilker, E.,
Güngör, Ö.,
Kuznets-Speck, B.,
Chiel, J.,
Deffner, S.,
Hinczewski, M.:
Shortcuts in stochastic systems and control of biophysical processes.
Physical Review X

[87]

Guéry-Odelin, D.,
Jarzynski, C.,
Plata, C.A.,
Prados, A.,
Trizac, E.:
Driving rapidly while remaining in control: classical shortcuts from
hamiltonian to stochastic dynamics.
Reports on Progress in Physics

[88]

Patrón, A.,
Prados, A.,
Plata, C.A.:
Thermal brachistochrone for harmonically confined brownian particles.
The European Physical Journal Plus

[89]

Takahashi, K.:
Shortcuts to adiabaticity applied to nonequilibrium entropy
production: an information geometry viewpoint.
New Journal of Physics

[90]

Koller, J.G.,
Athas, W.C.:
Adiabatic switching, low energy computing, and the physics of storing
and erasing information.
In: Workshop on Physics and Computation,
pp. 267–270

[91]

Komatsu, T.S.,
Nakagawa, N.,
Sasa, S.-i.,
Tasaki, H.:
Steady-state thermodynamics for heat conduction: microscopic
Physical review letters

[92]

Sagawa, T.,
Hayakawa, H.:
Geometrical expression of excess entropy production.
Physical Review E

[93]

Miangolarra, O.M.,
Taghvaei, A.,
Chen, Y.,
Georgiou, T.T.:
Geometry of finite-time thermodynamic cycles with anisotropic thermal
IEEE Control Systems Letters

[94]

Frim, A.G.,
DeWeese, M.R.:
Geometric bound on the efficiency of irreversible thermodynamic
Physical Review Letters

[95]

Takahashi, K.,
Fujii, K.,
Hino, Y.,
Hayakawa, H.:
Nonadiabatic control of geometric pumping.
Physical Review Letters

[96]

Blaber, S.,
Louwerse, M.D.,
Sivak, D.A.:
Steps minimize dissipation in rapidly driven stochastic systems.
Physical Review E

[97]

Zhong, A.,
DeWeese, M.R.:
Limited-control optimal protocols arbitrarily far from equilibrium.
Physical Review E

[98]

Blaber, S.,
Sivak, D.A.:
Optimal control with a strong harmonic trap.
Physical Review E

[99]

Chennakesavalu, S.,
Rotskoff, G.M.:
A unified, geometric framework for nonequilibrium protocol optimization.
arXiv preprint arXiv:2205.01205

[100]

Lu, Z.,
Qian, H.:
Emergence and breaking of duality symmetry in generalized fundamental
thermodynamic relations.
Physical Review Letters

[101]

Yang, Y.-J.,
Qian, H.:
Statistical thermodynamics and data infinitum: Conjugate variables as forces,
and their statistical variations.
arXiv preprint arXiv:2205.09321

[102]

Pressé, S.,
Ghosh, K.,
Lee, J.,
Dill, K.A.:
Principles of maximum entropy and maximum caliber in statistical
Reviews of Modern Physics

[103]

Léonard, C.:
A survey of the schrödinger problem and some of its connections
with optimal transport.
Discrete & Continuous Dynamical Systems

[104]

Chen, Y.,
Georgiou, T.T.,
Pavon, M.:
On the relation between optimal transport and schrödinger bridges:
A stochastic control viewpoint.
Journal of Optimization Theory and Applications

[105]

Kamiya, S.,
Kawakita, G.,
Sasai, S.,
Kitazono, J.,
Oizumi, M.:
Optimal control costs of brain state transitions in linear stochastic
Journal of Neuroscience

[106]

Sohl-Dickstein, J.,
Weiss, E.,
Maheswaranathan, N.,
Ganguli, S.:
Deep unsupervised learning using nonequilibrium thermodynamics.
In: International Conference on Machine Learning,
pp. 2256–2265

[107]

Ho, J.,
Jain, A.,
Abbeel, P.:
Denoising diffusion probabilistic models.
Advances in Neural Information Processing Systems

[108]

Kingma, D.,
Salimans, T.,
Poole, B.,
Ho, J.:
Variational diffusion models.
Advances in neural information processing systems

[109]

Song, Y.,
Sohl-Dickstein, J.,
Kingma, D.P.,
Kumar, A.,
Ermon, S.,
Poole, B.:
Score-based generative modeling through stochastic differential equations.
International Conference on Learning Representations

[110]

Arjovsky, M.,
Chintala, S.,
Bottou, L.:
Wasserstein generative adversarial networks.
In: International Conference on Machine Learning,
pp. 214–223

[111]

Parrondo, J.M.,
Horowitz, J.M.,
Sagawa, T.:
Thermodynamics of information.
Nature physics

[112]

Ito, S.,
Sagawa, T.:
Information thermodynamics on causal networks.
Physical review letters

[113]

Hartich, D.,
Barato, A.C.,
Seifert, U.:
Stochastic thermodynamics of bipartite systems: transfer entropy
inequalities and a maxwell’s demon interpretation.
Journal of Statistical Mechanics: Theory and Experiment

[114]

Horowitz, J.M.,
Esposito, M.:
Thermodynamics with continuous information flow.
Physical Review X

[115]

Ito, S.:
Backward transfer entropy: Informational measure for detecting hidden
markov models and its interpretations in thermodynamics, gambling and
Scientific reports

[116]

Schreiber, T.:
Measuring information transfer.
Physical review letters

[117]

Oizumi, M.,
Tsuchiya, N.,
Amari, S.-i.:
Unified framework for information integration based on information
Proceedings of the National Academy of Sciences

[118]

Proesmans, K.,
Ehrich, J.,
Bechhoefer, J.:
Finite-time landauer principle.
Physical Review Letters

[119]

Proesmans, K.,
Ehrich, J.,
Bechhoefer, J.:
Optimal finite-time bit erasure under full control.
Physical Review E

[120]

Zhen, Y.-Z.,
Egloff, D.,
Modi, K.,
Dahlsten, O.:
Universal bound on energy cost of bit reset in finite time.
Physical Review Letters

[121]

Lee, J.S.,
Lee, S.,
Kwon, H.,
Park, H.:
Speed limit for a highly irreversible process and tight finite-time
landauer’s bound.
Physical review letters

[122]

Fujimoto, Y.,
Ito, S.:
Game-theoretical approach to minimum entropy productions in information
arXiv preprint arXiv:2112.14035

[123]

Zhang, Z.,
Guan, S.,
Shi, H.:
Information geometry in the population dynamics of bacteria.
Journal of Statistical Mechanics: Theory and Experiment

[124]

Adachi, K.,
Iritani, R.,
Hamazaki, R.:
Universal constraint on nonlinear population dynamics.
Communications Physics

[125]

García-Pintos, L.P.:
Diversity and fitness uncertainty allow for faster evolutionary rates.
arXiv preprint arXiv:2202.07533

[126]

Frank, S.A.:
The price equation program: simple invariances unify population
dynamics, thermodynamics, probability, information and inference.

[127]

Frank, S.A.,
Bruggeman, F.J.:
The fundamental equations of change in statistical ensembles and
biological populations.

[128]

Grmela, M.,
Öttinger, H.C.:
Dynamics and thermodynamics of complex fluids. i. development of a
general formalism.
Physical Review E

[129]

Öttinger, H.C.,
Grmela, M.:
Dynamics and thermodynamics of complex fluids. ii. illustrations of a
general formalism.
Physical Review E

[130]

Esen, O.,
Grmela, M.,
Pavelka, M.:
On the role of geometry in statistical mechanics and thermodynamics.
i. geometric perspective.
Journal of Mathematical Physics

[131]

Esen, O.,
Grmela, M.,
Pavelka, M.:
On the role of geometry in statistical mechanics and thermodynamics.
ii. thermodynamic perspective.
Journal of Mathematical Physics

[132]

Li, W.:
Hessian metric via transport information geometry.
Journal of Mathematical Physics

[133]

Tafoya, S.,
Large, S.J.,
Liu, S.,
Bustamante, C.,
Sivak, D.A.:
Using a system’s equilibrium behavior to reduce its energy
dissipation in nonequilibrium processes.
Proceedings of the National Academy of Sciences

[134]

Brown, A.I.,
Sivak, D.A.:
Theory of nonequilibrium free energy transduction by molecular
Chemical reviews


# Realism in Action: Anomaly-Aware Diagnosis of Brain Tumors from Medical
Images Using YOLOv8 and DeiT

Seyed Mohammad Hossein Hashemi, Leila Safari*, Amirhossein Dadashzade Taromi
S.M.H. H. is a B.Sc. graduate in Computer Engineering from University of
Zanjan. He is currently a prospective graduate student. (e-mail:
<EMAIL_ADDRESS>L. S. is a faculty of Computer
Engineering Department at University of Zanjan, Zanjan, Iran. (e-mail:
lsafari@znu.ac.ir).A. D.T. is a B.Sc. graduate in Computer Engineering from
University of Zanjan. He is currently a M.Sc. student in Artificial
Intelligence at Institute for Advanced Studies in Basic Sciences, Zanjan,
Iran. (e-mail: dadashzadeh@iasbs.ac.ir).

###### Abstract

In the field of medical sciences, reliable detection and classification of
brain tumors from images remains a formidable challenge due to the rarity of
tumors within the population of patients. Therefore, the ability to detect
tumors in anomaly scenarios is paramount for ensuring timely interventions and
improved patient outcomes. This study addresses the issue by leveraging deep
learning (DL) techniques to detect and classify brain tumors in challenging
situations. The curated data set from the National Brain Mapping Lab (NBML)
comprises 81 patients, including 30 Tumor cases and 51 Normal cases. The
detection and classification pipelines are separated into two consecutive
tasks. The detection phase involved comprehensive data analysis and pre-
processing to modify the number of image samples and the number of patients of
each class to anomaly distribution (9 Normal per 1 Tumor) to comply with real
world scenarios. Next, in addition to common evaluation metrics for the
testing, we employed a novel performance evaluation method called Patient to
Patient (PTP), focusing on the realistic evaluation of the model. In the
detection phase, we fine-tuned a YOLOv8n detection model to detect the tumor
region. Subsequent testing and evaluation yielded competitive performance both
in common evaluation metrics and PTP metrics. Furthermore, using the Data
Efficient Image Transformer (DeiT) module, we distilled a Vision Transformer
(ViT) model from a fine-tuned ResNet152 as a teacher in the classification
phase. This approach demonstrates promising strides in reliable tumor
detection and classification, offering potential advancements in tumor
diagnosis for real-world medical imaging scenarios.

###### Index Terms:

Anomaly distribution, Brain tumors, Tumor diagnosis, Medical imaging,
ResNet152, Vision Transformer, YOLOv8n

##  I Introduction

Reliability and precision are pivotal factors in the context of brain tumor
diagnosis. The brain, a vital organ situated within the human body that
oversees the entire nervous system [1]. Consequently, any deviations within
the brain can significantly impact human health. Among these anomalies, brain
tumors stand out as particularly severe. Brain tumors involve the uncontrolled
and aberrant growth of cells within the brain. These tumors can be categorized
into two groups: primary tumors and secondary tumors. Primary tumors emerge
within the brain tissue itself, whereas secondary tumors stem from other areas
of the body, migrating to the brain tissue through the bloodstream [2]. Glioma
and Meningioma are two severe types of brain tumors among primary tumors. If
not identified in their early stages, these tumors can lead to fatal outcomes
for patients [3]. As the World Health Organization (WHO) outlined, brain
tumors are categorized into four grades. Grade 1 and grade 2 tumors correspond
to less aggressive forms (such as Meningioma), whereas grade 3 and grade 4
tumors encompass more aggressive varieties (like Glioma)[1].

In the context of the detection and monitoring of brain tumors, nowadays, with
huge advancements in the medical imaging field, there are various imaging
technologies used by radiologists and doctors to observe internal human body
organs, such as computed tomography (CT), positron emission tomography (PET),
and magnetic resonance imaging (MRI). Among the array of modalities available,
MRI emerges as the foremost selection for non-invasive brain tumor detection
and evaluation. This preference is owed to its remarkable resolution and
superior ability to provide contrasting details of soft tissues [4]. Manually
scrutinizing these images is a laborious and demanding endeavor. Moreover, it
is susceptible to errors, particularly given the surge in patient volumes and
the relatively low incidence rate of brain tumors [5, 6]. This combination
makes detecting and categorizing these tumors a formidable challenge. In
response, our objective is to formulate a resilient, anomaly-conscious,
computer-aided, automated solution. This approach aims to enhance the accuracy
of clinical brain tumor diagnosis.

The initial challenge in the study of tumor diagnosis lies in the significant
variability of tumors in terms of their shapes, textures, and contrasts, both
within and between cases [7]. Within the realm of tumor diagnosis, scientists
have harnessed a diverse array of machine learning (ML) techniques. These
include Support Vector Machines (SVMs), K-Nearest Neighbor (KNN), Decision
Trees, and Naive Bayes. Furthermore, in the context of deep learning
algorithms, they employed a diverse array of methods, including Convolutional
Neural Networks (CNNs), VGGNets [8], GoogleNet [9], and ResNets [10]. These
advanced algorithms have been instrumental in assisting with tumor diagnosis.

Nevertheless, in the context of brain tumor diagnosis, a significant challenge
persists, stemming from the inherent rarity of brain tumor occurrences within
a larger population. Based on the most recent statistical data released by
Johns Hopkins Medicine, the incidence rate of brain and nervous system tumors
in the United States is approximately 30 adults per 100,000 individuals [6].
Indeed, the incidence rate can differ significantly based on age, gender,
location, and other population factors. Nevertheless, a thorough review of
recent data suggests that this number remains very low. Although the rare
occurrence of this illness is comforting, difficulties arise when doctors need
to identify these uncommon cases in a large and diverse population [11].

So far, several significant contributions have been made to offer robust
solutions for this task. Often, these models, regardless of their
architecture, have been trained on semi-balanced data sets where potential
class imbalances have been resolved by data techniques such as Random Over-
Sampling, Synthetic Minority Over-Sampling Technique (SMOTE), and Weighted
Loss Functions. However, in this study, we aimed to raise the realism leverage
of the problem. We proposed a DL-based solution that has been trained and
evaluated on close-to-clinical diagnosis scenarios. This unique perspective on
the problem could enable us to close the gap between academic research
outcomes and the practical usage of this study.

To resolve the mentioned issues, our solution utilizes two key factors.
Firstly, (1) we harness a substantial quantity of distinct image data from the
data set we acquired from NBML. This data set encompasses records from 81
patients who underwent monitoring using various imaging technologies, such as
CT, PET, and MRI. Secondly, (2) we implement a meticulous data pipeline. This
pipeline not only refines the data set to skew its distribution towards non-
tumor samples but also preserves category proportions while augmenting data
within each class, thus enhancing overall sample diversity.

In the context of the tumor detection phase, our approach involves leveraging
the robust and adaptable capabilities of the YOLOv8n detection model and fine-
tuning it to classify Tumor cases from Normal ones. We meticulously trained
the YOLOv8n model on our data set to yield accurate yet anomaly-resistant
detections. Subsequently, we proceed to the evaluation phase, where we gauge
the model’s efficacy in tumor detection. This assessment employs a novel PTP
evaluation function, providing a pragmatic understanding of the model’s
performance during execution.

Lastly, we utilized a distilled ViT using the DeiT architecture with a fine-
tuned ResNet152 as the teacher model [12]. We incorporated this model for its
attention mechanism and the lightness, features we exploit to our advantage.
This model contributes to classifying brain tumor types into three distinct
classes: Meningioma, Pituitary, and Glioma.

The following paragraphs are structured as follows: Section II is a literature
review on related works in this field, Section III explains the materials and
methods we utilized, Section IV presents the results we achieved, and Section
V concludes the article.

##  II Related Works

Following the advancements of ML algorithms, many valuable contributions have
been made to offer robust and accurate solutions for the brain tumor
classification problem [13, 14, 15]. The article [7], authored by J. Kang and
his colleagues, proposed a novel pipeline to classify brain tumors. They
initially pre-processed the input data and then utilized an array of CNN pre-
trained models (e.g., ResNet, DenseNet, VGG, AlexNet, MnasNet, etc.) to
extract features from them. Next, they evaluated those deep features using a
range of ML classifiers (e.g., Random Forest, SVM, KNN, etc.). Based on their
evaluation outcome, they cherry-picked and combined their top three feature
sets to form a highly distinctive feature vector for each image, making it
easier for the model to learn and classify the image. Lastly, using the
concept of bagging, they created a classifier comprising nine ML classifiers
that predict the label based on the input feature.

During the last decade, computer hardware advancements, especially Graphical
Processing Units (GPU), have caused DL solutions to be an undeniable and
consistent solution for many tasks. In this regard, various CNN-based
architectures (e.g., DenseNets, Xception, VGG-Nets, etc.) have been introduced
for image processing tasks, and many of them have been applied to studying
brain tumor classification[16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].

The paper [28] proposes a framework based on unsupervised deep generative
neural networks that combine Variational Auto Encoders (VAEs) and Generative
Adversarial Networks (GANs) to generate realistic brain tumor MRIs. The
proposed method significantly improves the performance of the ResNet50
classifier, achieving an average accuracy improvement from %72.63 to %96.25.
The framework shows potential as a valuable clinical tool for medical experts.

The article [29] utilizes the DenseNet201 pre-trained DL model which is fine-
tuned and trained using a deep transfer of imbalanced data learning. The
features of the trained model are extracted from the average pool layer, which
captures deep information about each type of tumor. However, the author claims
that the layer’s characteristics alone are insufficient for a reliable
classification of brain tumors. Therefore, two techniques for feature
selection are proposed: Entropy-Kurtosis based High Feature Values (EKbHFV)
and a Modified Genetic Algorithm (MGA) based on meta-heuristics. The selected
features from the genetic algorithm are further refined using a new threshold
function. The features obtained from EKbHFV and MGA are fused using a non-
redundant serial-based approach and classified using a multi-class SVM cubic
classifier. Their evaluation results indicate a significant accuracy of %95.

This paper [30] proposed a model called FT-ViT, a fine-tuned vision
transformer model that uses image processing, including image patching, to
further extract learnable features from them. Their testing results suggest a
significant accuracy score of %98.13.

The article [31] discusses using an ensemble of pre-trained ViT (vision
transformer) models to classify brain tumors from MR images. They trained and
evaluated an array of ViT models, namely B/16, B/32, L/16, and L/32, and
analyzed their performance on brain tumor classification tasks. Based on their
final results, they concluded that the best singular model among the pre-
trained ViT models is L/32, with an overall test accuracy of %98.2 at a
resolution of 384 × 384. However, they added that when all four ViT models are
combined as a stack of classifiers ensembled together, the overall accuracy
improves to %98.7.

In [32], the authors focused on the YOLOv7 model and aimed to improve its
performance through transfer learning. First, they employed image enhancement
techniques to enhance the visual representation of brain tissue in MRI scans
and also utilized data augmentation technique to increase their samples. To
enhance the training phase of the model, the authors incorporated the
Convolutional Block Attention Module (CBAM) into YOLOv7, enhancing its feature
extraction capabilities for salient regions associated with brain
malignancies. Also, they added a Spatial Pyramid Pooling Fast+ (SPPF+) layer
to improve the model’s sensitivity. Their proposed model achieved a remarkable
%99.5 accuracy in tumor detection task based on their testing results.

##  III Materials and Methods

In this section, we explain the steps we took for data preparation and then
elaborate our proposed framework for tumor detection and classification. Our
general approach here is to break down the complex task of tumor diagnosis
into smaller sub-problems. Accurate brain tumor diagnosis involves two
distinct goals: 1.Detecting tumors within a predominantly Normal dataset, and
2. Identifying unusual brain tissues and their type from their unique
characteristics in highly noisy scenes (e.g., shape and suspicious tissue
placement).

The first phase of our framework, tumor detection, focuses on addressing the
first goal, which is training a model that is resilient to anomaly-distributed
populations and can accurately detect brain tumors in various imaging
modalities.

The second phase of our framework, tumor classification, involves through data
collection and preparation steps to merge multiple publicly available datasets
into a single source of data and convert them into a suitable format for the
classifier. Next, we created a custom DeiT model and trained it to classify
brain tumors in three classes of Meningioma, Pituitary, and Glioma.

###  III-A Brain Tumor Detection

This phase includes comprehensive data collection from private data sources
and unique pre-processing steps to mimic realistic tumor diagnosis scenarios.
In this regard, we collected a relatively small image dataset of Normal and
Tumor classes with samples belonging to 81 patients, with 30 being Tumor cases
and 51 Normal. Next, we subjected the dataset to a custom method of data pre-
processing that included data distribution modification techniques and data
augmentation, to transform the dataset into a suitable format.

Coming to the next step, we trained and fine-tuned the YOLOv8n model for the
tumor detection task. Furthermore, during the evaluation step, in addition to
common evaluation metrics (i.e, F1-Score, Recall and Precision) we employed a
novel PTP evaluation function to facilitate a pragmatic assessment of the
model’s performance.

Figure 1 depicts the proposed brain tumor detection procedure. As its shown in
the figure, the raw data passes through a data pre-processing pipeline and
transformed into a suitable format for the detection model, the YOLOv8n. Based
on the model’s prediction, the tumor-detected samples are fed into the
classifier for tumor type classification. The details of the sub-components of
Figure 1 is explained in the following subsections.

Figure 1:  Brain Tumor Detection Process.

###  III-B Data Collection

During the training phase of our brain tumor detection model, we procured a
relatively small dataset from NBML, including 81 patients, with 30 cases
designated as tumor-positive and the remaining categorized as Normal. Notably,
each patient’s folder featured various images and encompassed a spectrum of
imaging modalities, including PET, CT, and MRI scans. Consequently, we assert
that the data set inherently incorporates a form of data augmentation. In
simpler terms, the inclusion of diverse imaging modalities for each patient
mitigates the potential for bias towards any particular modality, such as MRI.
Importantly, it is imperative to emphasize that the data set obtained from
NBML has been exclusively employed for the detection phase and it is held
privately, with all associated rights and credits attributed to NBML.

###  III-C Data Pre-processing

The essence of our data preparation pipeline centers on the careful pre-
processing of our data set to closely mirror real-world scenarios. In this
section, we will explain the first critical aspect of our methodology, which
is distributing the patient data to ensure it reflects real-world scenarios
for testing. In section D, we will outline the specific steps we took to
determine the threshold for classifying patients into their corresponding
classes.

In the context of brain tumors, the United States typically reports [33]
incidence rates ranging from 0.03 to 0.06. We recognize that other nations may
experience different rates due to their distinct factors. Given the
limitations stemming from our limited dataset for the detection phase and the
absence of comprehensive external data sources, we exercised caution by
adopting a conservative estimate of a 0.1 incidence rate. The decision was
made to simulate the anomalous scenario of detecting a Tumor case from Normal
cases. Subsequently, following our presumed brain tumor distribution in the
general population, we segmented the data set into two sets: one for training
and the other for testing, with a particular stipulation. Specifically, in the
training data set, we ensured that there were nine randomly selected Normal
images for every tumor image. This approach was rooted in our hypothesis that,
throughout both the training and evaluation phases, the model’s focus should
not lie in discerning each individual’s situation but rather in acquiring
knowledge from a diverse spectrum of patients exhibiting various scenarios and
learning more robust and distinctive features.

Moving on to the testing set, we specifically selected 30 patients,
encompassing 27 cases categorized as Normal and 3 cases as Tumor, with their
corresponding folders housing all associated images. We aimed to maintain a
distribution of Tumor and Normal patients that closely aligns with real-world
scenarios while upholding the distribution pattern established during
training. Notably, this step presented a primary challenge as each patient
possessed a varying number of images, and the initial data set distribution
significantly diverged from our desired goal. Additionally, we had to address
the issue of particular images from tumor patients not exhibiting any signs of
brain tumors, necessitating their removal and cleansing from the training data
set. To address the challenges at hand, we followed a systematic approach as
below:

1\. Utilizing a third-party software (MicroDicom) to convert DICOM-format
patient folders and their associated images into “.png” format, standardizing
their resolution to 540x540 pixels. Our initial data examination revealed
approximately 18,000 Normal images distributed among 51 patients, while around
30 Tumor patients contributed roughly 12,000 image samples.

2\. Resizing all images to a uniform size allowed us to compress each
patient’s folder into a ZIP archive, providing a precise count of images per
patient and facilitating the selection process for each data set section.

3\. Prioritizing data preservation and efficiency for the testing set, opting
for the 27 normal patient folders with the smallest ZIP file sizes, resulting
in 27 patients.

4\. Choosing the three files with the smallest ZIP sizes from the uncleaned
tumor patient folders. Ending up with a total of 30 patients for the testing
set, achieving a tumor case distribution rate of 10%.

5\. selectively picking the remaining cleaned tumor case folders to construct
our training set, and including approximately 1.4k tumor-indicative images,
supplemented by 12,000 normal samples. This created an intentionally skewed
training data set with a tumor-to-normal sample distribution of 10%, meaning
we assigned nine normal images for each tumor image.

6\. To enhance the diversity of our training samples and proactively mitigate
the risk of overfitting while having more control over the data preparation
process, we developed a tailored data augmentation class utilizing Python’s
built-in libraries. This augmentation class encompassed modifications such as
brightness intensity adjustments, rotations, as well as vertical and
horizontal flips, all while ensuring the corresponding bounding boxes (the
rectangular area indicating the tumor tissue) were appropriately adjusted. The
execution of the data augmentation pipeline has led to a substantial expansion
in the overall data set volume, all while preserving its initial distribution.
This augmentation procedure has contributed to enhancing and diversifying our
training data, resulting in an overall improvement in data quality.

Figure 2: Outputs of the augmentation module for a random MRI sample. (a)
Rotated sample; (b) Horizontally flipped sample; (c) Vertically flipped
sample; (d) Brightened sample; (e) Dimmed sample, (f) Normal sample.

###  III-D Proposed PTP Evaluation Metrics

In our research, our primary goal is to improve the model performance in real-
world situations compared to academic assessments. To achieve this, we have
introduced a new evaluation method called PTP. In this methodology, instead of
indiscriminately feeding MRI images into the model without considering their
patient origins, we adopt a more pragmatic strategy: feed in all the MRI scans
from each patient. The model then carefully examines each image for any signs
of abnormalities related to brain tumors. It keeps track of the number of
images showing these signs, and if this count goes above a certain threshold,
the model classifies the case as indicative of a tumor. So, instead of relying
solely on standard evaluation metrics like F1-Score, Recall, Precision, and
Accuracy, we have adopted a PTP-based evaluation metrics.

The PTP-based evaluation metrics are implemented as a straightforward Python
function that systematically iterates through each patient’s directory within
the testing data set. Within this process, it feeds the images specific to
each patient into the model for analysis. While the model processes each image
individually and generates corresponding predictions, the PTP function
maintains a record of these predictions. After completing this iterative
procedure and analyzing all images within a patient’s folder, the PTP function
calculates a patient-specific tumor threshold. This threshold represents the
proportion of images indicative of tumors within the entire collection of a
patient images. If this computed threshold surpasses the predefined General
Tumor Threshold (GTT), the PTP function classifies the patient as a Tumor
case. This process is repeated individually for each patient, with the PTP
function iterating through all patients in the dataset. Ultimately, the PTP
function computes additional metrics, encompassing PTP-ACC, PTP Recall, PTP-
Precision, and PTP-F1. It is noteworthy that this function is designed and
deployed only as an evaluation metric for the Tumor detection model; hence,
during the classification phase, we utilized common evaluation metrics such as
accuracy to assess the model.

Here is a concise explanation of the mentioned metrics:

• PTP-ACC: This metric assesses the model’s accuracy in classifying patients
as having tumors or not based on their individual tumor thresholds. This
metric quantifies the proportion of correctly identified patient cases in the
total patient population. In essence, it measures how well the model can
distinguish patients with tumors from those without, providing a valuable
indicator of its overall accuracy in patient classification.

• PTP-Recall: This metric evaluates the model’s ability to correctly identify
patients with tumors among all individuals who genuinely have tumors. It
quantifies the ratio of true positive patient cases (those correctly
identified as having tumors) to the total number of patients with tumors in
the data set. PTP-Recall is a crucial measure of the model’s sensitivity,
highlighting its effectiveness in capturing all patients with tumors and
minimizing the risk of missing any cases.

• PTP-Precision: This metric gauges the precision of the model in labeling
patients as having tumors, considering the instances it has identified as
positive cases. It calculates the ratio of true positive patient cases to the
total number of patient cases labeled as having tumors by the model. This
metric provides insights into the model’s precision in patient classification,
emphasizing its ability to minimize false positive identifications while
maintaining accuracy.

• PTP-F1: This metric combines the metrics of PTP-Precision and PTP-Recall
into a single score to offer a balanced evaluation of the model’s performance
in identifying patients with tumors. Calculated as the harmonic mean of these
two metrics, PTP-F1 takes into account both false positives and false
negatives in patient classification. This is particularly valuable when there
is an imbalance between the number of patients with and without tumors, as it
provides a comprehensive assessment of the model’s overall performance.

####  III-D1 General Tumor Threshold

An essential preliminary step in deploying the PTP function is the
determination of the GTT applied uniformly across the patient population. The
GTT is defined as the threshold value representing the proportion of images
the model must classify as indicative of tumors within all the images
belonging to a patient to designate it as a tumor case. To establish this
threshold, we adopted an approach wherein we provided the entire patient data
set, comprising patients from the training and validation sets, in its
original format and fed it into the trained model. Subsequently, we allowed
the PTP function to calculate the number of tumor-indicative images per
patient across the entire training and validation sets. Based on those values,
we picked a suitable yet reliable GTT threshold value. The details of the GTT
calculation further explained in the results section.

###  III-E Detection Model

Within the domain of computer vision, the YOLO series of algorithms, developed
by Ultralytics, has gained significant prominence and widespread usage,
primarily due to their remarkable ability to achieve high accuracy while
maintaining a compact model size. This attribute makes YOLO accessible to many
developers, as it can be effectively trained on a single GPU. The latest
advancement in the YOLO framework, known as YOLOv8, exhibits versatility
across various applications such as object identification, image
categorization, and segmentation. Over time, YOLO models have undergone
multiple iterations, each building upon its predecessor to address prior
limitations and enhance overall performance.

Figure 3: YOLO V8 Model Architecture [34].

The architectural modifications introduced in YOLOv8 have significantly
improved its object detection capabilities, especially in complex and noisy
scenes, surpassing the performance of earlier YOLO versions. As shown in
Figure 5, this updated architecture incorporates a backbone comprising a
sequence of convolutional layers responsible for extracting features at
different resolutions. These extracted features are subsequently processed
through a neck module for consolidation before channeling into the detection
head [35].

One of the distinctive characteristics of the YOLOv8 architecture lies in its
anchor-free design. This method eliminates the necessity for predefined
reference anchor points, hence making the model way more efficient. In
addition, the anchor-free brings a higher level of adaptability in scales and
aspect ratios for YOLOv8.

Although almost all the loss functions are intact in YOLOv8 compared to the
previous version, the V8 design deviates from conventional objectness loss and
utilizes distributional focal loss instead. The distributional focal loss
presents a novel approach to object detection by treating the continuous
distribution of box locations as a discretized probability distribution.
Instead of regarding box locations as exact coordinates, this perspective
views them as probability distributions [36].

Figure 4: YOLO Models Evaluation results compared based on COCO mAP (higher is
better) [34].

In our approach, based on what we observed from YOLOv8 for each version
performance, as illustrated in Figure 4, we picked the YOLOv8n as the backbone
model for the tumor detection phase. The decision was primarily influenced by
its relatively compact model size and faster processing speed compared to the
other variations. Moreover, YOLOv8n exhibited substantial performance
enhancements when contrasted with its predecessors from versions ”v5” to ”v7”.
To achieve this, we harnessed the formidable computational capabilities of the
YOLOv8n object detection model, leveraging its pre-trained weights to train it
specifically for tumor tissue and region detection and fine-tuning it on our
purposefully skewed training data set.

###  III-F Brain Tumor Classification

This section covers the second phase of the proposed pipeline, the brain tumor
classification step. In this step, the Knowledge Distillation (KD) technique
is deployed to train a lightweight ViT from the ResNet152 model on our
classification dataset (Figure 7).

KD [37], is a training method in which a less powerful student model learns
from the guidance of a more capable teacher network. Unlike traditional
training, where only the teacher’s highest-scoring outputs (hard labels) are
considered, KD uses the complete output vector generated by the teacher’s
Softmax function. This approach not only enhances the performance of the
student model but can also be seen as a way of compressing the knowledge
contained in the larger teacher model into a smaller, more efficient student
model. In the context of KD, following up on the further advancement of
computer vision, DeiT [12] marks a significant stride by initially applying
the KD technique to ViT, aimed at distilling valuable insights from CNNs to
enhance training efficiency.

Figure 5: The proposed framework for brain tumor classification.

The inherent design of CNNs, characterized by translation-invariant
convolution operations, makes them particularly well-suited for image-related
tasks, owing to their local inductive bias, which proves advantageous in
vision tasks. Therefore, the application of DeiT has the potential to
facilitate a swifter and more effective convergence of ViT when deployed in
vision-related tasks. In this regard, we employed the KD process, harnessed
the computational capacity of ResNet152, and fine-tuned a reliable CNN as the
backbone of our classification.

###  III-G ResNet-152

ResNet-152, Residual Network with 152 layers, is a significant deep CNN
architecture primarily tailored for image classification and feature
extraction tasks. It belongs to the ResNet model family, is widely recognized
for its outstanding performance in computer vision applications, and
represents an improved iteration of the original ResNet-152. A notable
innovation within ResNet models is the inclusion of residual blocks, which
effectively address the challenge of vanishing gradients when training deep
neural networks (DNN). The ”vanishing gradients” problem in DNN occurs when
gradients become too small during training, hindering the learning process of
the model.

###  III-H Vision Transformer

One of the main components of our pipeline is ViT. This model [38] works by
treating pieces of an image like words in a sentence, trying to mimic how the
original transformer model was used for understanding language [39]. Unlike
the original transformer, which had both an encoder and a decoder, ViT keeps
things more straightforward with just an encoder in its design. In ViT, the
input image has dimensions $\mathbb{R}^{H\times W\times C}$. It’s then split
into $N$ smaller pieces called patches, each sized at $P\times P\times C$ ,
where $N=\frac{{HW}}{{p^{2}}}$(H: height, W: width, C: number of
channels)[30].

Next, the model creates a linear representation for these patches and adds
position information to these representations to know where each patch is
located. Additionally, an additional patch is included in the embedding that
can be adjusted through learning. This embedding is used for the final
classification step and is processed by a multi-layer Perceptron (MLP) head.
Moreover, the combined information from the patches and their position
embeddings are taken and passed through a transformer encoder model. This
encoder model consists of multiple layers that alternate between multi-headed
self-attention and MLP blocks (as illustrated in Figure 4).

Figure 6: The vision transformer encoder with multi-head self-attention[32].

###  III-I Evaluation Metrics

Upon completing the training and testing phases, it is imperative to employ
standardized assessment criteria to evaluate the model’s effectiveness. In
recent studies, the researchers utilized a range of evaluation metrics,
including Precision, Recall, Sensitivity, Specificity, Accuracy, F1-score.
These metrics are derived by applying the model to the data set and tallying
the occurrences of True Positives (TP), True Negatives (TN), False Positives
(FP), and False Negatives (FN). TP denotes instances where the model
accurately identified and labeled tumor cases, while FP refers to non-tumor
cases incorrectly classified as tumors. FN represents tumors that went
unrecognized during the diagnostic process. TN signifies true negatives, where
the model’s predictions are aligned with the actual negative cases. As for the
classification phase, we utilized a diverse range of evaluation metrics to
clearly observe the model’s performance. It is critical to mention that the
PTP evaluation metrics are only employed in the tumor detection phase
exclusively.

$Precision=TP/(TP+FP)$ (1) $Recall=TP/(TP+FN)$ (2)

Notably, the F1-score, which computes the harmonic mean between Precision and
Recall, is frequently regarded as a primary metric for evaluating model
performance in situations where data sets exhibit an imbalance in class
distribution.

$F1=2\times(Precision\times Recall)/(Precision+Recall)$ (3)

###  III-J Classification Dataset

During the classification phase, we accurately combined two datasets
(www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset, accessed on
August 2023 and Figshare dataset [40]) for our training and validation sets
and also divided a set for benchmarking the model. To diversify and increase
the number of samples per class, we took similar to detection phase data pre-
processing steps.

###  III-K Computational Resources

We primarily utilized a local system with a single CUDA-enabled Nvidia GeForce
GTX 1650 GPU. On occasion, we also used Google Colab, which provided 12 GB of
RAM and a T4 GPU. For model training, validation, and testing, we employed
PyTorch version 2.0.1+cu117 with Python 3.9.7.

Furthermore, we deployed the ViT model using the P Wang GitHub repository [41]
implementation, while the ResNet152 model was trained and evaluated using the
Torch Vision models module. Custom Python scripts were developed as needed in
data cleaning and bounding box augmentation modules to support our workflow.

##  IV Results

In this section, we will elaborate on the experiments and the results we
achieved from deploying the proposed pipeline. First, we are going to explain
the detection phase results, and then we will transition to the next stage,
which is classification.

###  IV-A Detection Results

The initial step in this phase was data pre-processing. After conducting a
comprehensive data prepration and transforming the data set into a suitable
format, we loaded the YOLOv8n model pre-trained weights, tailored its hyper-
parameters and fine-tuned it on our detection dataset. For this phase, we used
the dataset we acquired from NBML for both the training and validation steps.

In first training attempt we ended up with a precise model with low recall
score as we were inducing unwanted misinformation into the model’s learning
path. The bounding boxes that we specified for tumor regions were accurate in
the majority of the augmentations including the vertical and horizontal flips.

Figure 7: The rotation of bounding boxes has a notable impact on our models, affecting their ability to accurately outline objects by introducing misleading information within the boxed area. The extent of this inaccuracy depends on the tumor’s shape, size, and how much it is rotated. TABLE I: YOLOv8n Model Validation Box Results over 50 Epochs Data | Precision | Recall | mAP50
---|---|---|---
With-Rotated Samples | 0.74 | 0.58 | 0.68
Without-Rotated Samples | 0.87 | 0.71 | 0.80

However, the rotation of the image and the corresponding object of interest
(Tumor) was injecting false information into the Tumor class. This happened
when we attempted to rotate the bounding box around its initial center using
our custom module, and some non-tumor regions were injected in the new
bounding box region. Therefore, we decided to stay with more reliable
augmentation effects such as vertical, horizontal flips and brightness
intensity.

Furthermore, in consideration of our constrained access to computational
resources, we opted for the most compact iteration of YOLOv8, denoted as Nano.
We employed larger batch sizes to expedite the training duration per epoch.
The consideration of employing advanced versions of YOLOv8, such as YOLOv8m or
YOLOv8L, suggests the possibility of achieving improved results. Nevertheless,
it is important to acknowledge that this choice comes with the trade-off of
extended training periods and increased demands on computational resources.

TABLE II: YOLOv8n Model Configuration

Opt | Sched | lr0 | lrf | AMP | Epochs
---|---|---|---|---|---
SGD | CosLR | 0.01 | 0.00001 | False | 50

Opt: Optimizer, Sched: Scheduler, lr0: Initial learning rate, lrf: Final
learning rate, AMP: Automatic Mixed Precision.

TABLE III: YOLOv8n Model Evaluation Results Class | Precision | Recall | F1 | Support
---|---|---|---|---
Tumor | 0.99 | 0.96 | 0.97 | 1905
Normal | 0.99 | 0.99 | 0.99 | 20750
AVG | 0.99 | 0.975 | 0.98 | 22655

We trained the mentioned model for 50 epochs, and the evaluation results
indicated (Table III) that the model is highly accurate in detecting tumor-
confiscated images, and despite being agile and super lightweight with only
3.2M parameters, it does have a reliable performance.

TABLE IV: YOLOv8n Model PTP Evaluation Results Support | Accuracy | F1 | Precision | Recall
---|---|---|---|---
3 Tumor Cases | 1.0 | 1.0 | 1.0 | 1.0
27 Normal Cases | 1.0 | 1.0 | 1.0 | 1.0
Figure 8: Patient Specific Tumor Threshold, computed for each patient,
represents the proportion of images depicting tumors within the entire image
set for that patient.

Furthermore, the model manages to achieve significant scores in our PTP
evaluation, as detailed in Table IV. The value of GTT is the output of a
meticulous data analysis step in our training and evaluation sets. We
incorporated the training and validation sets in their original format into
the model and calculated the value of GTT for each patient of these sets.
After a exploratory analysis of the patient specific tumor threshold (Figure
8) values among the both Normal and Tumor cases, we estimated the value of the
GTT to be at least be 0.04%. This value is calculated from the average of the
first quartile and the median values of tumor-indicative distribution.

###  IV-B Classification Results

TABLE V: DeiT hyper-parameters tuning experiments No. | Hard Distillation | Temperature | Depth | Patch Size | Dimension | Attention Head | MLP Dim | Val-Accuracy
---|---|---|---|---|---|---|---|---
1 | False (Default) | 2 (Default) | 4 (Default) | 24 (Default) | 256 (Default) | 16 (Default) | 128 (Default) | 81.91
2 | True | 2 (Default) | 4 (Default) | 24 (Default) | 256 (Default) | 16 (Default) | 128 (Default) | 84.74
3 | True | 1 | 4 (Default) | 24 (Default) | 256 (Default) | 16 (Default) | 128 (Default) | 83.22
4 | True | 9 | 4 (Default) | 24 (Default) | 256 (Default) | 16 (Default) | 128 (Default) | 81.69
5 | True | 3 | 6 | 24 (Default) | 256 (Default) | 16 (Default) | 128 (Default) | 82.35
6 | True | 3 | 2 | 32 | 256 (Default) | 16 (Default) | 128 (Default) | 85.40
7 | True | 3 | 2 | 24 | 256 (Default) | 16 (Default) | 128 (Default) | 86.05
8 | True | 3 | 2 | 24 | 1024 | 16 (Default) | 128 (Default) | 68.19
9 | True | 3 | 2 | 24 | 128 | 16 (Default) | 128 (Default) | 85.40
10 | True | 3 | 2 | 24 | 512 | 16 (Default) | 128 (Default) | 74.29
11 | True | 3 | 2 | 24 | 128 | 64 | 128 (Default) | 88.67
12 | True | 3 | 2 | 24 | 128 | 64 | 256 | 88.45
13 | True | 3 | 2 | 24 | 128 | 64 | 2048 | 87.58
14 | True | 3 | 2 | 24 | 128 | 64 | 512 | 89.76

The approach we took for this phase is knowledge distilation based on DeiT
architecture. Using the custom data augmentation module, we intially created a
relatively large dataset from training set which is obtained from Figshare
dataset. Then, we fine-tuned a heavy weight yet strong teacher model using
ResNet152 architecture.

During the process of distillation, having an access to a strong teacher model
is a necessity. By utilizing the DeiT architecture, the teacher model
(ResNet152) expertise in classifying the object of interest using its rich CNN
kernels would flow into the student model and eventually trains an efficient
student model from scratch. The obvious benefit of DeiT architecture is that
it enables us to train compact ViT models from rich and stronger teachers.
This is specifically useful when we have limited samples of data.

Proceeding to the subsequent stage, we initiated the hyper-parameters tuning
process for the DeiT model. We explored 14 different architectures to
determine optimal values for hyper-parameters. The specifics of these
experiments are delineated in table V.

Based on our observations, we settled on a DeiT model and fine-tuned it using
our dataset. We allocated 15% of the original dataset specifically for
benchmarking the classifier. It’s noteworthy that we utilized the same set of
data for both the teacher and student models. This measure was implemented to
prevent any potential bias or misleading results during testing, ensuring the
integrity of our results and guarding against any form of unwanted data
manipulation.

TABLE VI: Teacher Classifier Test Results Tumor Class | Precision | Recall | F1 | Support
---|---|---|---|---
Meningioma | 0.92 | 0.91 | 0.91 | 107
Glioma | 0.99 | 0.97 | 0.98 | 214
Pituitary | 0.94 | 0.97 | 0.96 | 140
Weighted AVG | 0.97 | 0.97 | 0.97 | 461
TABLE VII: Distilled Student Classifier Test Results Tumor Class | Precision | Recall | F1 | Support
---|---|---|---|---
Meningioma | 0.82 | 0.82 | 0.82 | 107
Glioma | 0.95 | 0.92 | 0.93 | 214
Pituitary | 0.95 | 0.99 | 0.97 | 140
Weighted AVG | 0.92 | 0.92 | 0.92 | 461

##  V Discussion and Conclusion

In conclusion, this article delves into the comprehensive evaluation of state-
of-the-art models, specifically YOLOv8 and DeiT, for the task of tumor
detection and classification. The distinctive contribution lies in the
introduction of novel performance metrics, notably PTP, into the evaluation
framework. These metrics not only assess the models’ proficiency in accurately
detecting and classifying brain tumors but also gauge their ability to make
informed decisions regarding the overall patient condition.

To rigorously assess the models, we curated a new dataset comprising Tumor and
Normal cases, maintaining a conservative ratio of 10% for tumor cases in the
entire population. The detection results, evaluated using common metrics,
demonstrated a commendable performance, with an F1-Score of 0.98. Notably, the
model showcased robust accuracy even in anomalous scenarios, achieving a
perfect score of 1.0 as measured by the PTP-F1 metric.

In the final phase, we adopted the DeiT architecture and fine-tuned a
lightweight student ViT using a ResNet152-based teacher model. Despite
training the model for a limited number of epochs, the process of distilling
knowledge from a complex CNN into a more compact student model proved to be a
promising direction for our task. The student model achieved a test accuracy
of 0.92 within 20 training epochs.

It is essential to note that the results presented in classification section
rely solely on common evaluation metrics, and PTP metrics were not applied.
This decision stems from the belief that the design logic of PTP is currently
tailored for the detection phase. However, exploring the integration of PTP-
like metrics in classification remains an intriguing avenue for future
research endeavors. We acknowledge that the current value represented by GTT
may not be perfectly precise for practical deployment, and there is potential
for refinement within a broader population of patients.

##  VI References

## References

  * [1] D. N. Louis, A. Perry, G. Reifenberger, A. Von Deimling, D. Figarella-Branger, W. K. Cavenee, H. Ohgaki, O. D. Wiestler, P. Kleihues, and D. W. Ellison, “The 2016 world health organization classification of tumors of the central nervous system: a summary,” _Acta neuropathologica_ , vol. 131, pp. 803–820, 2016.
  * [2] G. S. Tandel, M. Biswas, O. G. Kakde, A. Tiwari, H. S. Suri, M. Turk, J. R. Laird, C. K. Asare, A. A. Ankrah, N. Khanna _et al._ , “A review on a deep learning perspective in brain cancer classification,” _Cancers_ , vol. 11, no. 1, p. 111, 2019.
  * [3] A. K. Anaraki, M. Ayati, and F. Kazemi, “Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms,” _biocybernetics and biomedical engineering_ , vol. 39, no. 1, pp. 63–74, 2019.
  * [4] R. Augustine, A. Al Mamun, A. Hasan, S. A. Salam, R. Chandrasekaran, R. Ahmed, and A. S. Thakor, “Imaging cancer cells with nanostructures: Prospects of nanotechnology driven non-invasive cancer diagnosis,” _Advances in Colloid and Interface Science_ , vol. 294, p. 102457, 2021.
  * [5] K. Popuri, D. Cobzas, A. Murtha, and M. Jägersand, “3d variational brain tumor segmentation using dirichlet priors on a clustered feature set,” _International journal of computer assisted radiology and surgery_ , vol. 7, pp. 493–506, 2012.
  * [6] “Brain Tumors and Brain Cancer,” 2023, [Online; accessed 31. Aug. 2023]. [Online]. Available: https://www.hopkinsmedicine.org/health/conditions-and-diseases/brain-tumor
  * [7] J. Kang, Z. Ullah, and J. Gwak, “Mri-based brain tumor classification using ensemble of deep features and machine learning classifiers,” _Sensors_ , vol. 21, no. 6, p. 2222, 2021.
  * [8] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [9] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [11] “Brain Tumor - Statistics,” 2023, [Online; accessed 31. Aug. 2023]. [Online]. Available: https://www.cancer.net/cancer-types/brain-tumor/statistics
  * [12] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International conference on machine learning_. PMLR, 2021, pp. 10 347–10 357.
  * [13] M. Devi, S. Maheswaran _et al._ , “An efficient method for brain tumor detection using texture features and svm classifier in mr images,” _Asian Pacific journal of cancer prevention: APJCP_ , vol. 19, no. 10, p. 2789, 2018.
  * [14] E. I. Zacharaki, S. Wang, S. Chawla, D. Soo Yoo, R. Wolf, E. R. Melhem, and C. Davatzikos, “Classification of brain tumor type and grade using mri texture and shape in a machine learning scheme,” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 62, no. 6, pp. 1609–1618, 2009.
  * [15] S. Shrot, M. Salhov, N. Dvorski, E. Konen, A. Averbuch, and C. Hoffmann, “Application of mr morphologic, diffusion tensor, and perfusion imaging in the classification of brain tumors using machine learning scheme,” _Neuroradiology_ , vol. 61, pp. 757–765, 2019.
  * [16] S. Deepak and P. Ameer, “Retrieval of brain mri with tumor using contrastive loss based similarity on googlenet encodings,” _Computers in biology and medicine_ , vol. 125, p. 103993, 2020.
  * [17] Z. N. K. Swati, Q. Zhao, M. Kabir, F. Ali, Z. Ali, S. Ahmed, and J. Lu, “Brain tumor classification for mr images using transfer learning and fine-tuning,” _Computerized Medical Imaging and Graphics_ , vol. 75, pp. 34–46, 2019.
  * [18] Y. Zhuge, H. Ning, P. Mathen, J. Y. Cheng, A. V. Krauze, K. Camphausen, and R. W. Miller, “Automated glioma grading on conventional mri images using deep convolutional neural networks,” _Medical physics_ , vol. 47, no. 7, pp. 3044–3053, 2020.
  * [19] R. Pomponio, G. Erus, M. Habes, J. Doshi, D. Srinivasan, E. Mamourian, V. Bashyam, I. M. Nasrallah, T. D. Satterthwaite, Y. Fan _et al._ , “Harmonization of large mri datasets for the analysis of brain imaging patterns throughout the lifespan,” _NeuroImage_ , vol. 208, p. 116450, 2020.
  * [20] M. A. Naser and M. J. Deen, “Brain tumor segmentation and grading of lower-grade glioma using deep learning in mri images,” _Computers in biology and medicine_ , vol. 121, p. 103758, 2020.
  * [21] Ö. Polat and C. Güngen, “Classification of brain tumors from mr images using deep transfer learning,” _The Journal of Supercomputing_ , vol. 77, no. 7, pp. 7236–7252, 2021.
  * [22] H. A. Khan, W. Jue, M. Mushtaq, and M. U. Mushtaq, “Brain tumor classification in mri image using convolutional neural network,” _Mathematical Biosciences and Engineering_ , 2021.
  * [23] M. M. Badža and M. Č. Barjaktarović, “Classification of brain tumors from mri images using a convolutional neural network,” _Applied Sciences_ , vol. 10, no. 6, p. 1999, 2020.
  * [24] E. U. Haq, H. Jianjun, K. Li, H. U. Haq, and T. Zhang, “An mri-based deep learning approach for efficient classification of brain tumors,” _Journal of Ambient Intelligence and Humanized Computing_ , pp. 1–22, 2021.
  * [25] A. Sekhar, S. Biswas, R. Hazra, A. K. Sunaniya, A. Mukherjee, and L. Yang, “Brain tumor classification using fine-tuned googlenet features and machine learning algorithms: Iomt enabled cad system,” _IEEE journal of biomedical and health informatics_ , vol. 26, no. 3, pp. 983–991, 2021.
  * [26] N. S. Shaik and T. K. Cherukuri, “Multi-level attention network: application to brain tumor classification,” _Signal, Image and Video Processing_ , vol. 16, no. 3, pp. 817–824, 2022.
  * [27] M. F. Alanazi, M. U. Ali, S. J. Hussain, A. Zafar, M. Mohatram, M. Irfan, R. AlRuwaili, M. Alruwaili, N. H. Ali, and A. M. Albarrak, “Brain tumor/mass classification framework using magnetic-resonance-imaging-based isolated and developed transfer deep-learning model,” _Sensors_ , vol. 22, no. 1, p. 372, 2022.
  * [28] B. Ahmad, J. Sun, Q. You, V. Palade, and Z. Mao, “Brain tumor classification using a combination of variational autoencoders and generative adversarial networks,” _Biomedicines_ , vol. 10, no. 2, p. 223, 2022.
  * [29] M. I. Sharif, M. A. Khan, M. Alhussein, K. Aurangzeb, and M. Raza, “A decision support system for multimodal brain tumor classification using deep learning,” _Complex & Intelligent Systems_, pp. 1–14, 2021.
  * [30] A. A. Asiri, A. Shaf, T. Ali, U. Shakeel, M. Irfan, K. M. Mehdar, H. T. Halawani, A. H. Alghamdi, A. F. A. Alshamrani, and S. M. Alqhtani, “Exploring the power of deep learning: Fine-tuned vision transformer for accurate and efficient brain tumor detection in mri scans,” _Diagnostics_ , vol. 13, no. 12, p. 2094, 2023.
  * [31] S. Tummala, S. Kadry, S. A. C. Bukhari, and H. T. Rauf, “Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling,” _Current Oncology_ , vol. 29, no. 10, pp. 7498–7511, 2022.
  * [32] A. B. Abdusalomov, M. Mukhiddinov, and T. K. Whangbo, “Brain Tumor Detection Based on Deep Learning Approaches and Magnetic Resonance Imaging,” _Cancers_ , vol. 15, no. 16, August 2023.
  * [33] “Cancer of the Brain and Other Nervous System - Cancer Stat Facts,” 2023, [Online; accessed 31. Aug. 2023]. [Online]. Available: https://seer.cancer.gov/statfacts/html/brain.html
  * [34] “Brief summary of YOLOv8 model structure $\cdot$ Issue #189 $\cdot$ ultralytics/ultralytics,” august 2023, [Online; accessed 13. Aug. 2023]. [Online]. Available: https://github.com/ultralytics/ultralytics/issues/189
  * [35] J. Solawetz, “What is YOLOv8? The Ultimate Guide.” _Roboflow Blog_ , December 2023. [Online]. Available: https://blog.roboflow.com/whats-new-in-yolov8
  * [36] W. M. Elmessery, J. Gutiérrez, G. G. Abd El-Wahhab, I. A. Elkhaiat, I. S. El-Soaly, S. K. Alhag, L. A. Al-Shuraym, M. A. Akela, F. S. Moghanm, and M. F. Abdelshafie, “Yolo-based model for automatic detection of broiler pathological phenomena through visual and thermal images in intensive poultry houses,” _Agriculture_ , vol. 13, no. 8, p. 1527, 2023.
  * [37] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [38] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [40] J. Cheng, “Brain tumor dataset,” _figshare. Dataset_ , vol. 1512427, no. 5, 2017.
  * [41] “vit-pytorch,” 2023, [Online; accessed 31. Aug. 2023]. [Online]. Available: https://github.com/lucidrains/vit-pytorch