<p>We are <strong>Prometheus version 2.26.0</strong> and <strong>kubernetes verion 1.21.7 in Azure</strong>. We mount the data in <strong>Azure storage NFS</strong> and it was working fine. From last few days Prometheus container is crash-looping and below are the logs</p>
<pre><code>level=info ts=2022-01-26T08:04:14.375Z caller=main.go:418 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)&quot;
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:423 build_context=&quot;(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)&quot;
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:424 host_details=&quot;(Linux 5.4.0-1065-azure #68~18.04.1-Ubuntu SMP Fri Dec 3 14:08:44 UTC 2021 x86_64 prometheus-6b9d9d54f4-nc45x (none))&quot;
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:425 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:426 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=info ts=2022-01-26T08:04:14.503Z caller=web.go:540 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
level=info ts=2022-01-26T08:04:14.507Z caller=main.go:795 msg=&quot;Starting TSDB ...&quot;
level=info ts=2022-01-26T08:04:14.509Z caller=tls_config.go:191 component=web msg=&quot;TLS is disabled.&quot; http2=false
level=info ts=2022-01-26T08:04:14.560Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641478251052 maxt=1641513600000 ulid=01FRSEHC4YHV3N26JY5AMNZFRW
level=info ts=2022-01-26T08:04:14.593Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641513600037 maxt=1641578400000 ulid=01FRVCAP2VJGDF0Z9CS24EXAJJ
level=info ts=2022-01-26T08:04:14.624Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641578400038 maxt=1641643200000 ulid=01FRXA4AQHMHAEYWRKQFGP075M
level=info ts=2022-01-26T08:04:14.651Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641643200422 maxt=1641708000000 ulid=01FRZ7XQQ4RA96DCPPBP22D71N
level=info ts=2022-01-26T08:04:14.679Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641708000020 maxt=1641772800000 ulid=01FS15QDG6BS7H6M6Y09HG3E12
level=info ts=2022-01-26T08:04:14.707Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641772800011 maxt=1641837600000 ulid=01FS33GT38PRSB9VP56YFXT2M0
level=info ts=2022-01-26T08:04:14.736Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641963555381 maxt=1641967200000 ulid=01FS6MRNZEWT1Z6P697K09KHD7
level=info ts=2022-01-26T08:04:14.763Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641837600100 maxt=1641902400000 ulid=01FS6R88C70TCD8CYC4XJ95X23
level=info ts=2022-01-26T08:04:14.810Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641967200019 maxt=1642032000000 ulid=01FS8WXQP3YJ7EXBVNYBQG4DVY
level=info ts=2022-01-26T08:04:14.836Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642032000072 maxt=1642096800000 ulid=01FSATQBR4XBQRDM72ATFS9PQ2
level=info ts=2022-01-26T08:04:14.863Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642096800059 maxt=1642161600000 ulid=01FSCRHE2YBDX7GPRPSH6BNGRX
level=info ts=2022-01-26T08:04:14.895Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642161600091 maxt=1642226400000 ulid=01FSEPB1GPGAANVCQ2VKW9BQ4G
level=info ts=2022-01-26T08:04:14.948Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642226400026 maxt=1642291200000 ulid=01FSGM4J0G1D0A6H1GD3N9C372
level=info ts=2022-01-26T08:04:14.973Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642291200005 maxt=1642356000000 ulid=01FSJHY6W0FRYDHCXBVB5XPFYG
level=info ts=2022-01-26T08:04:15.002Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642356000027 maxt=1642420800000 ulid=01FSMFR96DASV6YPN66W7C86H9
level=info ts=2022-01-26T08:04:15.077Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642420800042 maxt=1642485600000 ulid=01FSPDHGWRT65D8CKWQ2JPRHW3
level=info ts=2022-01-26T08:04:15.105Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642485600006 maxt=1642550400000 ulid=01FSRBAVP2MW71H08F32D6HGB4
level=info ts=2022-01-26T08:04:15.130Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642550400028 maxt=1642615200000 ulid=01FST9482FD0Z3PHXHNW2W616E
level=info ts=2022-01-26T08:04:15.157Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642680000018 maxt=1642687200000 ulid=01FSW00TJKJ7CGCQ7JJS3XQK8G
level=info ts=2022-01-26T08:04:15.187Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642687200018 maxt=1642694400000 ulid=01FSW6WHTSEAXHWV5J7PQP94X7
level=info ts=2022-01-26T08:04:15.213Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642615200021 maxt=1642680000000 ulid=01FSW6XYH2Y429PG5YRM0K45XS
level=info ts=2022-01-26T08:04:15.275Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642694400018 maxt=1642701600000 ulid=01FSWDR92Y7H302NDZRX1V2PX9
level=info ts=2022-01-26T08:04:21.840Z caller=head.go:696 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:710 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=782.403397ms
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:716 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
level=info ts=2022-01-26T08:04:34.169Z caller=head.go:742 component=tsdb msg=&quot;WAL checkpoint loaded&quot;
level=info ts=2022-01-26T08:04:38.895Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=299 maxSegment=7511
level=warn ts=2022-01-26T08:04:46.423Z caller=main.go:645 msg=&quot;Received SIGTERM, exiting gracefully...&quot;
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:668 msg=&quot;Stopping scrape discovery manager...&quot;
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:682 msg=&quot;Stopping notify discovery manager...&quot;
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:704 msg=&quot;Stopping scrape manager...&quot;
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:678 msg=&quot;Notify discovery manager stopped&quot;
level=info ts=2022-01-26T08:04:46.425Z caller=main.go:698 msg=&quot;Scrape manager stopped&quot;
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:934 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:944 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
level=info ts=2022-01-26T08:04:46.426Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:872 msg=&quot;Notifier manager stopped&quot;
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:664 msg=&quot;Scrape discovery manager stopped&quot;
level=info ts=2022-01-26T08:04:46.792Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=300 maxSegment=7511
level=info ts=2022-01-26T08:04:46.870Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=301 maxSegment=7511
level=info ts=2022-01-26T08:04:46.901Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=302 maxSegment=7511
level=info ts=2022-01-26T08:04:46.946Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=303 maxSegment=7511
level=info ts=2022-01-26T08:04:46.974Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=304 maxSegment=7511
level=info ts=2022-01-26T08:04:47.008Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=305 maxSegment=7511
level=info ts=2022-01-26T08:04:47.034Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=306 maxSegment=7511
level=info ts=2022-01-26T08:04:47.067Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=307 maxSegment=7511
level=info ts=2022-01-26T08:04:47.098Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=308 maxSegment=7511
level=info ts=2022-01-26T08:04:47.124Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=309 maxSegment=7511
level=info ts=2022-01-26T08:04:47.158Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=310 maxSegment=7511
level=info ts=2022-01-26T08:04:47.203Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=311 maxSegment=7511
level=info ts=2022-01-26T08:04:47.254Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=312 maxSegment=7511
level=info ts=2022-01-26T08:04:47.486Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=313 maxSegment=7511
level=info ts=2022-01-26T08:04:47.511Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=314 maxSegment=7511
level=info ts=2022-01-26T08:04:47.539Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=315 maxSegment=7511
level=info ts=2022-01-26T08:04:47.564Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=316 maxSegment=7511

.
.
.
.
.
.
.
.
.

level=info ts=2022-01-26T08:05:15.161Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1401 maxSegment=7511
level=info ts=2022-01-26T08:05:15.182Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1402 maxSegment=7511
level=info ts=2022-01-26T08:05:15.205Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1403 maxSegment=7511
level=info ts=2022-01-26T08:05:15.229Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1404 maxSegment=7511
level=info ts=2022-01-26T08:05:15.251Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1405 maxSegment=7511
level=info ts=2022-01-26T08:05:15.274Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1406 maxSegment=7511
level=info ts=2022-01-26T08:05:15.297Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1407 maxSegment=7511
level=info ts=2022-01-26T08:05:15.323Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1408 maxSegment=7511
level=info ts=2022-01-26T08:05:15.349Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1409 maxSegment=7511
level=info ts=2022-01-26T08:05:15.372Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1410 maxSegment=7511
level=info ts=2022-01-26T08:05:15.426Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1411 maxSegment=7511
level=info ts=2022-01-26T08:05:15.452Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1412 maxSegment=7511
level=info ts=2022-01-26T08:05:15.475Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1413 maxSegment=7511
level=info ts=2022-01-26T08:05:15.498Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1414 maxSegment=7511
rpc error: code = NotFound desc = an error occurred when try to find container &quot;ae14079418f59b04bb80d8413e8fdc34f167bfe762317ef674e05466d34c9e1f&quot;: not found
</code></pre>
<p>So I deleted the deployment and redeployed to the the same storage account, the I got new error</p>
<pre><code>level=info ts=2022-01-26T11:10:11.530Z caller=main.go:418 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)&quot;
level=info ts=2022-01-26T11:10:11.534Z caller=main.go:423 build_context=&quot;(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)&quot;
level=info ts=2022-01-26T11:10:11.535Z caller=main.go:424 host_details=&quot;(Linux 5.4.0-1064-azure #67~18.04.1-Ubuntu SMP Wed Nov 10 11:38:21 UTC 2021 x86_64 prometheus-6b9d9d54f4-wnmzh (none))&quot;
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:425 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:426 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=info ts=2022-01-26T11:10:14.168Z caller=web.go:540 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
level=info ts=2022-01-26T11:10:15.385Z caller=main.go:795 msg=&quot;Starting TSDB ...&quot;
level=info ts=2022-01-26T11:10:16.022Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641837600024 maxt=1641902400000 ulid=01FS51ANKBFTVNRPZ68FGQQ5GA
level=info ts=2022-01-26T11:10:16.309Z caller=tls_config.go:191 component=web msg=&quot;TLS is disabled.&quot; http2=false
level=info ts=2022-01-26T11:10:16.494Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641902400005 maxt=1641967200000 ulid=01FS6Z46FGXN932K7D39D9166D
level=info ts=2022-01-26T11:10:16.806Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641967200106 maxt=1642032000000 ulid=01FS8WXRJ7Q80FKD4C8EJNR0AD
level=info ts=2022-01-26T11:10:17.011Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642032000003 maxt=1642096800000 ulid=01FSATQE1VMNR101KRW1X10Q75
level=info ts=2022-01-26T11:10:17.305Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642096800206 maxt=1642161600000 ulid=01FSCRGVT1E7562SF7EQN12JBM
level=info ts=2022-01-26T11:10:18.240Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642161600059 maxt=1642226400000 ulid=01FSEPAFP2CX03ANRB7Q1AG514
level=info ts=2022-01-26T11:10:21.046Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642226400051 maxt=1642291200000 ulid=01FSGM3WT0TKR0XW9BD4QSKPQE
level=info ts=2022-01-26T11:10:21.422Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642291200113 maxt=1642356000000 ulid=01FSJHXKHMANW0E6FXDXVM265G
level=info ts=2022-01-26T11:10:22.822Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642356000032 maxt=1642420800000 ulid=01FSMFQ6XJ97VJFKNCYQBVB4DZ
level=info ts=2022-01-26T11:10:23.536Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642420800021 maxt=1642485600000 ulid=01FSPDGM95FDDV2CDWX93BTDCS
level=info ts=2022-01-26T11:10:23.880Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642485600072 maxt=1642550400000 ulid=01FSRBA555RWY4QNP4HD9YKRBM
level=info ts=2022-01-26T11:10:25.021Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642550400031 maxt=1642615200000 ulid=01FST93N3C82K9VS20MKTMGGYC
level=info ts=2022-01-26T11:10:25.713Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642615200014 maxt=1642680000000 ulid=01FSW6X95FRNSN1XJZ2YK0MXW7
level=info ts=2022-01-26T11:10:26.634Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642680000012 maxt=1642744800000 ulid=01FSY4PXA7V1XQHHA3MC35JSWQ
level=info ts=2022-01-26T11:10:27.776Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642744800174 maxt=1642809600000 ulid=01FT02G9XGHPV8GME53ZPMYXE6
level=info ts=2022-01-26T11:10:28.760Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642809600070 maxt=1642874400000 ulid=01FT209WP8AXXVZB1NCSC55ACE
level=info ts=2022-01-26T11:10:29.618Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642874400253 maxt=1642939200000 ulid=01FT3Y3A4H72FFW318RKHEXXGA
level=info ts=2022-01-26T11:10:30.313Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642939200047 maxt=1643004000000 ulid=01FT5VX3YC838QN5VQFAERV1QX
level=info ts=2022-01-26T11:10:30.483Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643004000040 maxt=1643068800000 ulid=01FT7SPHC5EV0SS1R0WT04H9FR
level=info ts=2022-01-26T11:10:30.696Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643068800035 maxt=1643133600000 ulid=01FT9QFZXBZ7EYY2CTE8WXZTB9
level=info ts=2022-01-26T11:10:31.838Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643133600000 maxt=1643155200000 ulid=01FTA574G4M45WX97Z470DQF73
level=info ts=2022-01-26T11:10:33.686Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643176800008 maxt=1643184000000 ulid=01FTASSZCG8V5N2VGAGFBYJBSR
level=info ts=2022-01-26T11:10:36.078Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643184000000 maxt=1643191200000 ulid=01FTB0NP47JW5JCF808QZZ8WZQ
level=info ts=2022-01-26T11:10:36.442Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643155200065 maxt=1643176800000 ulid=01FTB0P9H3H09B2ADD5X1RXFW6
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:668 msg=&quot;Stopping scrape discovery manager...&quot;
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:682 msg=&quot;Stopping notify discovery manager...&quot;
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:704 msg=&quot;Stopping scrape manager...&quot;
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:678 msg=&quot;Notify discovery manager stopped&quot;
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:664 msg=&quot;Scrape discovery manager stopped&quot;
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:698 msg=&quot;Scrape manager stopped&quot;
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:934 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:944 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
level=info ts=2022-01-26T11:10:40.080Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
level=info ts=2022-01-26T11:10:40.080Z caller=main.go:872 msg=&quot;Notifier manager stopped&quot;
level=error ts=2022-01-26T11:10:40.080Z caller=main.go:881 err=&quot;opening storage failed: lock DB directory: resource temporarily unavailable&quot;
</code></pre>
<p>The yaml is provided by the <strong>Istio</strong> .Below is deployment yaml file.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: &quot;server&quot;
    app: prometheus
    release: prometheus
    chart: prometheus-14.6.1
    heritage: Helm
  name: prometheus
  namespace: istio-system
spec:
  selector:
    matchLabels:
      component: &quot;server&quot;
      app: prometheus
      release: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        component: &quot;server&quot;
        app: prometheus
        release: prometheus
        chart: prometheus-14.6.1
        heritage: Helm
        
        sidecar.istio.io/inject: &quot;false&quot;
    spec:
      enableServiceLinks: true
      serviceAccountName: prometheus
      containers:
        - name: prometheus-server-configmap-reload
          image: &quot;jimmidyson/configmap-reload:v0.5.0&quot;
          imagePullPolicy: &quot;IfNotPresent&quot;
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://127.0.0.1:9090/-/reload
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
              readOnly: true

        - name: prometheus-server
          image: &quot;prom/prometheus:v2.26.0&quot;
          imagePullPolicy: &quot;IfNotPresent&quot;
          args:
            - --storage.tsdb.retention.time=15d
            - --config.file=/etc/config/prometheus.yml
            - --storage.tsdb.path=/data
            - --web.console.libraries=/etc/prometheus/console_libraries
            - --web.console.templates=/etc/prometheus/consoles
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 4
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: azurefileshare
              mountPath: /data
              subPath: &quot;&quot;
      hostNetwork: false
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      terminationGracePeriodSeconds: 300
      volumes:
        - name: config-volume
          configMap:
            name: prometheus
        - name: azurefileshare
          azureFile:
            secretName: log-storage-secret
            shareName: prometheusfileshare
            readOnly: false
</code></pre>
<p><strong>Expected Behavior</strong>
When I mount the data to new container, It should load the data.</p>
<p><strong>Actual Behavior</strong>
Not able to load the data or not able bind the data with newly created pod when old pod dies</p>
<p>Help me out, to resolve the issue.</p>

<p><em><strong>Thank You <a href="https://stackoverflow.com/users/3781502/ywh">YwH</a> for your suggestion, Posting this an answer so it can help other community member if they encounter the same issue in future.</strong></em></p>
<p>As stated in this <a href="https://istio.io/latest/docs/ops/integrations/prometheus/" rel="nofollow noreferrer"><em><strong>document</strong></em></a> Istio provides a basic sample installation to quickly get Prometheus up and running:
This is intended for demonstration only, and is not tuned for performance or security.</p>
<blockquote>
<p>Note : Isio configuration is well-suited for small clusters and
monitoring for short time horizons, it is not suitable for large-scale
meshes or monitoring over a period of days or weeks</p>
</blockquote>
<p><em><strong>Solution : Prometheus is a stateful application, better deployed with a StatefulSet, not Deployment.</strong></em></p>
<p>StatefulSets are valuable for applications that require one or more of the following.</p>
<p><em><code>Stable, persistent storage. Ordered, graceful deployment and scaling.</code></em></p>
<p><em><strong>You can use this <a href="https://github.com/GoogleCloudPlatform/click-to-deploy/blob/master/k8s/prometheus/manifest/prometheus-statefulset.yaml" rel="nofollow noreferrer"><em><strong>Stateful</strong></em></a> code for deployment of prometheus container.</strong></em></p>
