# Question
Title: Monitoring checklist - What things should I be monitoring?
We are building a (Zabbix-based) monitoring system for our applications; hovewer, I'm having difficulties in defining what to monitor?

I have so far come up with the following general categories:

* **hardware data**: cpu, ram, swap, etc.
* **middleware data**: perfomance/health for MySQL instantces, Tomcat instances, JVMs, etc.
* **logical** or **application** data: the current status/health of the system, e.g. number of active users, page request, etc.
* **kpi data**: data for business, e.g. user registration over time.
* **dashboard**: quick overview of the system (e.g. microservices are running or not).

Are there any other fundamental categories for to monitor? Or is there another category system to use?

**UPDATE**: the purpose of the monitoring is

* the see if the system functions correctly (at high-level, e.g. no services are down, etc. - much like a smoke-test)
* see, if there are any indicators, that the system is likely to crash (e.g. historical data predicts that we will run out of disk space)
* if any of these occur, send a warning to the appropriate staff (e.g. via e-mail)

**UPDATE**: the complexity of our system does not demand an extra application for reporting (e.g. monitoring KPIs); also, we are running in local/local cloud infrastructure, so the **cost** of the application is not (that)relevant - but it might be someday :-)

# Answer
I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie

One of the key ideas (for me at least) is to realize the difference between *host* monitoring and *application* monitoring. Basically host monitoring tells you that something is fatally wrong *now*, but application monitoring should be able to *predict* problems by detecting higher error rate or that requests are taking longer time so you can fix problems *before* your users notice them.

(I'm not affiliated with weaveworks or the goto conference in any way, I just like the content and think there are some interesting ideas. Use the downvote button to let me know that this answer is not good :) )
> 5 votes

# Answer
I'm surprised nobody mentioned the four golden signals explicitly as an answer, so I'll add it. Lifted directly from Google's SRE Book chapter on monitoring distributed systems, it is suggested to **at least** collect metrics on the **"Four Golden Signals"**:

> 1. **Latency**:
> The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors.
> 2. **Traffic**:
> A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per second.
> 3. **Errors:**
> The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content.
> 4. **Saturation**:
> How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours."
>
> If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring.
> 3 votes

# Answer
It depends a lot on what your infrastructure situation is. If you're doing auto-scaling, the health of individual instances is mostly irrelevant. The important metrics are total cost, and cost per unit of work (e.g. per request). Personally I don't like to monitor individual instance state if I can possibly avoid it - I try to focus more on broader service-level and application-level metrics:

* Overall per-service uptime (% of the time that at least one instance is healthy and able to respond to new requests immediately)
* Overall end-user uptime (% of the time that the user perceives the product as being available - if some core service goes down, this might be user-facing downtime, but a lesser-used service or background worker might not be)
* 95th percentile response time of each service
* Message queue length and net queue growth per queue
* Consumer count per queue
* Average time to completion per queue (not just time in queue, but time from being queued to completed work for the message)
* Error rate for each service
* Cost per unit of work for each service
* For each server role, what is the ratio of the average CPU utilization to average memory utilization? This is useful for determining if we're scaling poorly - if CPU use is 70% but memory is 20%, we're giving instances too much memory, and vice-versa.
* Same for peak CPU/memory utilization
* Time to delivery for deployments

Some of your listed metrics like user registration over time, to me, don't belong in an infrastructure monitoring system like Zabbix. What is anyone watching Zabbing going to do about a 10% drop in registrations? Nothing. This is business reporting data that should be exposed to whoever wants it via a reporting DB, possibly rendered in a nice dashboard because pointy-haired bosses love dashboards.
> 2 votes

# Answer
For any given node, there are 4 basic resources with the following items to monitor:

* CPU
    + Total
    + Per-core
* Storage
    + Free space
    + Free inodes
    + Throughput
    + Backups
* Memory
    + Free
    + Swap
    + Buffers
    + Cache
* Network
    + Latency
    + Throughput
    + Jitter

These four basic resources will power your application or service at every layer of your application. This will be architected into up to 5 layers depending on your environment and you will want to monitor them at every layer. This might look like:

This can change depending on how your environment is architected. For example, if you run all of your data off of NFS mounts, the Storage layer would sit beside compute instead of behind it. If you have a C++ based application that workstations connect to, you might not have a front-end layer. If your application uses flat files, you might not have a database layer. If you do not use virtual machines or containers and use local storage, you might not have a compute and storage layer. You will want to monitor the above 4 basic resources at every layer.

This represents the basic monitoring that can be given by your vendors, software and hardware. Covering these four basic resources at every layer should be your first goal. Once 100% coverage is achieved, You can additionally build hooks into your application to report additional health statuses, but by their nature, these will typically be built as a reaction to outages and you would have to work with your internal developers to build these kinds of hooks.

Monitoring these 4 basic resources should catch probably 80% of issues that cause outages however, and then you can start working on the remaining 20%.
> 2 votes

---
Tags: monitoring
---


# Question
Title: Helm install or upgrade release failed on Kubernetes cluster: the server could not find the requested resource or UPGRADE FAILED: no deployed releases
Using helm for deploying chart on my Kubernetes cluster, since one day, I can't deploy a new one or upgrading one existed.

Indeed, each time I am using helm I have an error message telling me that it is not possible to install or upgrade ressources.

If I run `helm install --name foo . -f values.yaml --namespace foo-namespace` I have this output:

> Error: release foo failed: the server could not find the requested resource

If I run `helm upgrade --install foo . -f values.yaml --namespace foo-namespace` or `helm upgrade foo . -f values.yaml --namespace foo-namespace` I have this error:

> Error: UPGRADE FAILED: "foo" has no deployed releases

I don't really understand why.

This is my helm version:

```
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

```

On my kubernetes cluster I have tiller deployed with the same version, when I run `kubectl describe pods tiller-deploy-84b... -n kube-system`:

```
Name:               tiller-deploy-84b8...
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               k8s-worker-1/167.114.249.216
Start Time:         Tue, 26 Feb 2019 10:50:21 +0100
Labels:             app=helm
                    name=tiller
                    pod-template-hash=84b...
Annotations:        <none>
Status:             Running
IP:                 <IP_NUMBER>
Controlled By:      ReplicaSet/tiller-deploy-84b8...
Containers:
  tiller:
    Container ID:   docker://0302f9957d5d83db22...
    Image:          gcr.io/kubernetes-helm/tiller:v2.12.3
    Image ID:       docker-pullable://gcr.io/kubernetes-helm/tiller@sha256:cab750b402d24d...
    Ports:          44134/TCP, 44135/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 26 Feb 2019 10:50:28 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:44135/liveness delay=1s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:44135/readiness delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment:
      TILLER_NAMESPACE:    kube-system
      TILLER_HISTORY_MAX:  0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from helm-token-... (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  helm-token-...:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  helm-token-...
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Scheduled  26m   default-scheduler      Successfully assigned kube-system/tiller-deploy-84b86cbc59-kxjqv to worker-1
  Normal  Pulling    26m   kubelet, k8s-worker-1  pulling image "gcr.io/kubernetes-helm/tiller:v2.12.3"
  Normal  Pulled     26m   kubelet, k8s-worker-1  Successfully pulled image "gcr.io/kubernetes-helm/tiller:v2.12.3"
  Normal  Created    26m   kubelet, k8s-worker-1  Created container
  Normal  Started    26m   kubelet, k8s-worker-1  Started container

```

---

UPDATE

Following answer suggestion, I have run these two commands `helm upgrade foo . -f values.yaml --namespace foo-namespace --force` | `helm upgrade --install foo . -f values.yaml --namespace foo-namespace --force` and I still get an error:

```
UPGRADE FAILED
ROLLING BACK
Error: failed to create resource: the server could not find the requested resource
Error: UPGRADE FAILED: failed to create resource: the server could not find the requested resource

```

Notice that foo-namespace already exist. So the error don't come from the namespace name. Indeed, if I run `helm list`, I can see that the **foo** chart is in a `FAILED` status.

Is someone have faced the same issue ?

# Answer
> 3 votes
Yes this happens frequently when debugging helm releases. The problem happens when a previously failed release is preventing you from updating it.

If you run `helm ls` you should see a release in state `FAILED`. You might have deleted it in which case it might show up with `helm ls -a`. Such a release cannot be upgraded using the normal approach of having helm compare the new yaml to the old yaml to detect what objects to change as it is a failed release.

As this normally happens when trying to get something new running I typically `helm delete —purge` the failed release. That is slightly drastic though so you may want to try to do the upgrade adding `—force`

https://github.com/helm/helm/pull/2280

# Answer
> 1 votes
This can happen with releases in `FAILED` state as mentioned by simbo1905 in her/his answer.

Another case is if there's a previously deleted but not purged release with the same name.

Doing another delete with the purge option on the release will free the name for reuse.

```
helm ls -a
helm ls -a | grep -e NAME -e name_of_release
helm delete --purge name_of_release

```

Note: This issue https://github.com/helm/helm/issues/972 suggests that there is a parameter to force reusing the same name, however it also suggests to not use in production (i.e. do not automate with `replace` unless you know what you're doing).

```
> helm install --help
[...]
    --replace    re-use the given name, even if that name is already used. This is unsafe in production
[...]

```

---
Tags: docker, kubernetes, deployment, continuous-deployment, helm
---


# Question
Title: How to SSH from Jenkins node to Jenkins node safely?
We have a Jenkins Job that builds something on node A and then needs to SCP it to node B. You can just put the SSH keys for node B on Node A, and SCP with an `execute shell` build step, however I suspect that keeping SSH keys on a Jenkins node is a bad practice, and would like to avoid it. What is the best practice regarding this use case? Is there a plugin or a feature for this? Should we just install Hashicorp Vault on the node and configure the keys there separately?

# Answer
Yes, storing your ssh keys directly on a build node is a bad practice. Nodes can be replicated, deleted, or given access to from other systems, and you don't want to lose track of what systems have access to your secrets.

You should also not pass them directly into the build job, either as a parameter or as an environment variable. This can cause a huge headache with logging and tracking to make sure your keys aren't inadvertently output to places they shouldn't be. Instead, you should:

1. Use a centralized secret store (such as Hashicorp Vault) to retrieve the secrets at build run-time.
2. Use a plugin (such as the Credentials or SSH Credentials Plugin) to reference the keys during the build.
> 5 votes

# Answer
I would suggest using the Jenkins Credentials Plugin.

Another method which is more useful in case the Jenkins nodes are not able to connect via SSH is by pushing it to some remote bucket from NodeA and using triggers or a webhook to pull it to NodeB. This might however require access to the internet.

To apply this use https://wiki.jenkins.io/display/JENKINS/SSH+Credentials+Plugin in conjunction with https://jenkins.io/doc/book/using/using-credentials.
> 0 votes

---
Tags: jenkins, jenkins-plugins, key-management, ssh
---


# Question
Title: Kubernetes - How to show all service accounts
How do I show all service accounts In Kubernetes?

I have tried `kubectl get --all-namespaces all`. It does not show service accounts.

How can I use `kubectl` to list all service accounts?

# Answer
The command you listed will show you your resources.

Instead try: `kubectl get serviceAccounts`
> 15 votes

# Answer
kubectl get sa --all-namespaces

This will only provide the service accounts.

In general, you can have a comma separated list of resources to display.

Example:

kubectl get pods,svc,sa,deployments \[-FLAGS\]

The FLAGS would apply to all the resources.
> 9 votes

---
Tags: kubernetes, kubectl, gcp
---


# Question
Title: How to manage services inside the container?
Systemctl/service commands are not working on a container which is built by using centos:centos7 image.

Can anybody help me here?

# Answer
> 2 votes
Using **systemd in Docker** is not really that straightforward.

However, according to the doc on the docker hub, **systemd** is now included in both the `centos:7` and `centos:latest` base containers.
However, you will need to include text similar to the example Dockerfile :

```
FROM centos:7
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

```

**I strongly recommend to read the full description tab documentation on the hub.** This will help you make it run correctly.

---
Tags: docker, centos
---


# Question
Title: Why does docker not restart on a stack trace?
I'm running a container with the following: `docker run bot --restart-policy always`

However, the container is never actually restarted if it exits due to a stack trace, e.g.,:

```
Traceback (most recent call last):
  File "/jirabot/source/jbot.py", line 32, in <module>
    rtm = slack_client.rtm_read()
  File "/usr/local/lib/python2.7/site-packages/slackclient/client.py", line 235, in rtm_read
    json_data = self.server.websocket_safe_read()
  File "/usr/local/lib/python2.7/site-packages/slackclient/server.py", line 283, in websocket_safe_read
    data += "{0}\n".format(self.websocket.recv())
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 310, in recv
    opcode, data = self.recv_data()
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 327, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 358, in recv_data_frame
    self.pong(frame.data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 301, in pong
    self.send(payload, ABNF.OPCODE_PONG)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 250, in send
    return self.send_frame(frame)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 275, in send_frame
    l = self._send(data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 445, in _send
    return send(self.sock, data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_socket.py", line 117, in send
    return sock.send(data)
  File "/usr/local/lib/python2.7/ssl.py", line 707, in send
    v = self._sslobj.write(data)
socket.error: [Errno 32] Broken pipe

```

What am I missing about the purpose of the restart policy `always`?

# Answer
The command line is order sensitive. This command:

```
docker run bot --restart-policy always

```

Runs the `bot` image as a container with the value of `CMD` set to `--restart-policy always`. To modify the restart policy of the container being run, you need to pass the option after `run` and before your image name:

```
docker run --restart-policy always bot

```
> 4 votes

---
Tags: docker
---


# Question
Title: Is it safe to run "Repair - Reconcile component database from blob store job" in Nexus 3
We have a nexus 3.12.1 and its daily storing hundreds of artifacts which are build from Jenkins. The problem is there are too many files present in blob store are not showing in Nexus Console. Actually we don't want these files but that is filling up too much disk space. We ran all cleanup tasks on all of the repositories, but it is not cleaning the artifacts present in blob-store.

So our approach is to first run the task "Repair - Reconcile component database from blob store job" and then "Remove snapshots from Maven repository"

We have **dry run** the task "Repair - Reconcile component database from blob store job". The output of the log files is of 105079 lines, took ~15 min and we got only 2 warnings (no errors).

```
cat blobstore.rebuildComponentDB-20190823042827074.log | grep -v INFO
2019-08-23 04:33:04,902+0000 WARN  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.maven.internal.MavenRestoreBlobStrategy - Skipping as no maven coordinates found and is not maven metadata
2019-08-23 04:33:39,633+0000 WARN  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.maven.internal.MavenRestoreBlobStrategy - Skipping as no maven coordinates found and is not maven metadata

```

And summery logs are as below:

```
.......<truncated text>........
2019-08-23 04:37:43,653+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.maven.internal.MavenRestoreBlobStrategy - ::DRY RUN:: Restored asset, blob store: default, repository: snapshots, path: com/CMP/MODULE/FUNCTION/ui/MODULE-FUNCTION-ui/2.14.2-SNAPSHOT/MODULE-FUNCTION-ui-2.14.2-20190502.152829-43.pom.md5, blob name: com/CMP/MODULE/FUNCTION/ui/MODULE-FUNCTION-ui/2.14.2-SNAPSHOT/MODULE-FUNCTION-ui-2.14.2-20190502.152829-43.pom.md5, blob id: bf5327dd-e386-4fc4-a625-9da968e99f64
2019-08-23 04:37:43,666+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - ::DRY RUN:: Elapsed time: 9.276 min, processed: 376782, un-deleted: 0
2019-08-23 04:37:43,675+0000 INFO  [quartz-5-thread-20]  *SYSTEM com.sonatype.nexus.blobstore.restore.internal.NugetIntegrityCheckStrategy - Checking integrity of assets in repository 'r1' with blob store 'special'
2019-08-23 04:37:43,676+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r2-repo' with blob store 'special'
2019-08-23 04:37:43,681+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 4.655 ms, processed: 8, failed integrity check: 0
2019-08-23 04:37:43,681+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r3-release' with blob store 'special'
2019-08-23 04:37:43,726+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 44.85 ms, processed: 49, failed integrity check: 0
2019-08-23 04:37:43,726+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'thirdparty-software' with blob store 'special'
2019-08-23 04:37:43,754+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 28.02 ms, processed: 45, failed integrity check: 0
2019-08-23 04:37:43,754+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r4-repo' with blob store 'special'
2019-08-23 04:37:43,755+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r5-snapshots' with blob store 'special'
2019-08-23 04:37:43,756+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'java.net' with blob store 'special'
2019-08-23 04:37:43,758+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 2.017 ms, processed: 2, failed integrity check: 0
2019-08-23 04:37:43,759+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r5-central' with blob store 'special'
2019-08-23 04:37:43,759+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'central' with blob store 'special'
2019-08-23 04:38:25,568+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 41.81 s, processed: 63965, failed integrity check: 0
2019-08-23 04:38:25,568+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r6-snapshots' with blob store 'special'
2019-08-23 04:38:25,728+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 159.6 ms, processed: 226, failed integrity check: 0
2019-08-23 04:38:25,728+0000 INFO  [quartz-5-thread-20]  *SYSTEM com.sonatype.nexus.blobstore.restore.internal.NugetIntegrityCheckStrategy - Checking integrity of assets in repository 'r7-hosted' with blob store 'special'
2019-08-23 04:38:25,730+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r4-release' with blob store 'special'
2019-08-23 04:38:25,731+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r8am' with blob store 'special'
2019-08-23 04:38:25,735+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 4.032 ms, processed: 2, failed integrity check: 0
2019-08-23 04:38:25,735+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r9-tech' with blob store 'special'
2019-08-23 04:38:26,495+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 760.0 ms, processed: 1112, failed integrity check: 0
2019-08-23 04:38:26,495+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r5-releases' with blob store 'special'
2019-08-23 04:38:26,496+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r10-snapshot-repository' with blob store 'special'
2019-08-23 04:38:26,583+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 86.39 ms, processed: 100, failed integrity check: 0
2019-08-23 04:38:26,583+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r11-json' with blob store 'special'
2019-08-23 04:38:26,587+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'thirdparty' with blob store 'special'
2019-08-23 04:38:29,632+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 3.045 s, processed: 2379, failed integrity check: 0
2019-08-23 04:38:29,632+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'releases' with blob store 'special'
2019-08-23 04:39:29,639+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 1.000 min, processed: 83253, failed integrity check: 0
2019-08-23 04:39:37,551+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 1.132 min, processed: 94146, failed integrity check: 0
2019-08-23 04:39:37,552+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'download.java.net' with blob store 'special'
2019-08-23 04:39:37,553+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'snapshots' with blob store 'special'
2019-08-23 04:40:37,554+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 1.000 min, processed: 83879, failed integrity check: 0
2019-08-23 04:40:42,787+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 1.087 min, processed: 90913, failed integrity check: 0
2019-08-23 04:40:42,787+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r12-repository' with blob store 'special'
2019-08-23 04:40:42,935+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 148.3 ms, processed: 180, failed integrity check: 0
2019-08-23 04:40:42,936+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r13-snapshots' with blob store 'special'
2019-08-23 04:40:42,944+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 8.752 ms, processed: 1, failed integrity check: 0
2019-08-23 04:40:42,945+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'r3-milestone' with blob store 'special'
2019-08-23 04:40:43,136+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Elapsed time: 190.9 ms, processed: 234, failed integrity check: 0
2019-08-23 04:40:43,136+0000 INFO  [quartz-5-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - Task complete

```

Actually I am new to nexus and I don't understand what it says.
eg. *Elapsed time: 1.132 min, processed: 94146, failed integrity check: 0* : Will it restores 94146 files?
And looking into the logs above, is it safe to run this task? And will we need any downtime for this?

# Answer
> And looking into the logs above, is it safe to run this task? And will we need any downtime for this?

I would suggest to backup all artifacts and upload them to a new Nexus (I created for this) as I have seen multiple errors in Nexus that did not give me enough confidence to just run some tasks.

Once we deployed a new Nexus and included it in our build environment for two weeks we knew that everything was still working well. Then we applied some jobs to the old decommissioned Nexus and it turned out that it was fine.

In summary, ensure that you have a another Nexus that is working well before executing tasks that could go wrong.
> 2 votes

---
Tags: nexus, n3dr
---


# Question
Title: How to manage the versioning of Azure SQL Databases?
I am looking for a tool that will manage the versioning of Azure SQL Databases, and check for missing schema or migration scripts and apply those changes based on what is missing.

Any other solutions or tips would be great. I just want to make sure the database schema is flat across the board regardless, and apply changes based on version number.

Any suggestions?

# Answer
> 1 votes
I assume from the `ansible` tag in your question that you need something that can run from a command line, so that rules out plugins to SSMS like dbForge.

Try Flyway (has a free version) or Red Gate (commercial).

# Answer
> 0 votes
Did you try DbUp? DbUp homepage This utility can apply patches to database and maintain its version.

It writes execution log to databases and is able to run scripts that were not applied.

I'm using that project over 3 years now and it does it job well.

---
Tags: ansible, chef, azure
---


# Question
Title: Jenkins multi branch pipeline configuration for bitbucket
How to configure multi branch pipeline job for bitbucket ....i have to clone from different branches and build the jobs.... I found multi branch is the option... How to configure for bit bucket

# Answer
I recommend using the BitBucket Branch Source Plugin when using BitBucket. Documentation can be found here
> 1 votes

# Answer
You could watch this video about multi branch: https://www.youtube.com/watch?v=gXXwCki94ys

Basically it is about:

* create new item
* choose multibranch
* copy and paste the git url of the project
* choose a ssh key that has been added to the credentials settings
* choose a polling interval of e.g. 1 min in the "scan multibranch pipeline triggers" section
> 0 votes

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: How to share variables across instances?
I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the variables. Every time `molecule verify`, each test will run on each instance, so variables are independent on single instance.

How can I share variables across instances?

# Answer
Q: ***"Is it possible to share variables across instances?"***

A: Yes. The dictionary *hostvars* keeps the variables. *hostvars* lets you access variables for another host, including facts that have been gathered about that host. You can access host variables at any point in a playbook. For example,

```
{{ hostvars['test.example.com']['ansible_facts']['distribution'] }}

```

See:

---

With “Fact Caching” disabled, to share information among Ansible playbooks, it's possible to store all *hostvars* in a file. For example, given the inventory

```
shell> cat hosts
[test]
test_01
test_02
test_03

```

this template

```
shell> cat my_hostvars.json.j2
my_hostvars_all:
{% for my_host in ansible_play_hosts_all %}
  {{ my_host }}:
    {{ hostvars[my_host]|to_nice_json }}
{% endfor %}

```

and the playbook below

```
shell> cat pb1.yml
- hosts: test

  tasks:

    - set_fact:
        test_var: "test_var_in_{{ inventory_hostname }}"

    - template:
        src: my_hostvars.json.j2
        dest: "{{ playbook_dir }}/my_hostvars.json"
      delegate_to: localhost
      run_once: true

```

store *hostvars* of all hosts into the dictionary *my\_hostvars\_all* and put it into the file

```
{{ playbook_dir }}/my_hostvars.json

```

at localhost(master). The dictionary can be included in the next playbook. For example, the playbook

```
shell> cat pb2.yml
- hosts: test

  tasks:

    - include_vars: my_hostvars.json

    - debug:
        msg: "{{ my_hostvars_all[inventory_hostname]['test_var'] }}"

```

gives (abridged)

```
ok: [test_01] =>
  msg: test_var_in_test_01
ok: [test_02] =>
  msg: test_var_in_test_02
ok: [test_03] =>
  msg: test_var_in_test_03

```
> 3 votes

---
Tags: ansible, testing, python, molecule
---


# Question
Title: Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?
# Initial Problem

We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps. These are metrics that would be problematic if they happened during the day when customers are using the service, but at night there are absolutely no customers so there's no motivation by devs to make the nightly jobs less error-prone. So now I want to make it so it doesn't alert every night. Between New Relic and PagerDuty I would think I could figure that out, but so far I've come up with nothing practical.

# New Relic

I haven't spent tons of time with New Relic so I was learning as I tried adjusting things, but based on this thread on New Relic's site it seems clear that time-boxing alerts is not a feature they current support or will in the forseeable future. I've dealt with using history as a predictor for future results using a few technologies and it is a mess that would not be a reasonable replacement for the simple alerts that are in place currently. Going down that path would create more noisy alerts without detecting any more problems than we already find.

\[ It doesn't seem very DevOps-friendly to me to ignore feature requests for simple features that would solve lots of people's problems and only offer complicated quagmires as an alternative. I'm sure they're all jazzed about 24-7 cloud stuff, but ignoring the market of people who have prehistoric needs is sad. My audience is literally captive and they are going to stay that way until we stop putting people in prison. This use case will not go away regardless of how much cloud pixie dust you sprinkle on it. \]

# PagerDuty

I've used PagerDuty merrily for years and I was sure there would be some way to handle this after the alert left New Relic, but:

* Schedules apply to people and come after an alert has been through a Service. If you could add a Service to a Schedule we could hack this, but no, that's not there.
* Event Rule-based Suppression doesn't include time-of-day conditions.
* Maintenance Windows might work, but I was really hoping this wouldn't turn into another cron job I have to keep an eye on.

# The Question

Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?

# Answer
We ended up creating a new PagerDuty service with a schedule attached to it. That schedule only had anybody assigned for half of the day. This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.
> 2 votes

---
Tags: monitoring, pagerduty
---


title: using copy module with wildcards

text:
# Question
Title: using copy module with wildcards
I need to copy a database backup file from a primary server to a standby server. Since I can't use `synchronize`, I was following this method:

1. copy a file from primary to the local ansible server,
2. copy the file from local ansible server to standby.

The name of backup files are different on every machine. For example:

```
TEST1.0.db2inst1.DBPART000.20190729162630.001
TEST3.0.db2inst1.DBPART000.20180729172631.002

```

The playbook I was using looked like this:

```
 - hosts: primary
   remote_user: root
   tasks:
   - name: copy backup to local file
     fetch: src={{item}} dest=/data1/backup
     with_fileglob: /data1/{{dbname}}.0.db2inst1.*
  tags: transfer

 - hosts: standby
   remote_user: root
   tasks:
   - name: copy backup to standby
     copy: src={{item}} dest=/data1 mode=0755 owner=db2inst1
     with_fileglob: /data1/backup/{{dbname}}.0.db2inst1.*
   tags: transfer

```

(The playbook is run with the command: `ansible-playbook -i hostfile transfer.yml --extra-vars "dbname=TEST1"`)

However, this doesn't work; I get this output:

```
primary  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0
standby  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0

```

I think the name of files are problem. What should I do?

# Answer
> 3 votes
`with_fileglog` creates a loop with the `fileglob` lookup. As all other lookups it runs on the controller machine, not on the remote server.

Since I guess you don't already have those backup files on the controller in the path you referenced, your loop is empty.

As you have guessed, this approach will not work for your scenario. I suggest you have a look at the `find` module to create the list of files to fetch from your primary server.

# Answer
> 2 votes
any `with_*` is actully a local lookup.

So as Zeitounator mentioned, you're going to need to use something like the `find` module to fetch the files you want.

so I imagine that something like this would work:

```
- hosts: primary
  remote_user: root
  tasks:
    - name: gather list of files to fetch
      find:
        paths: "/data1/"
        recurse: no
        patterns: "{{ dbname }}.0.db2inst1.*"
        use_regex: no
      regisrer: db_backup_src
    - name: fetch backups to local disk
       fetch:
         src: "{{ item['path'] }}"
         dest: /data1/backup/
         flat: yes
      loop: "{{ db_backuo_src['files'] }}"

- hosts: standby
  remote_user: root
  tasks:
    - name: copy backup to standby
      copy:
        src: "{{ item }}"
        dest: /data1
        mode: 0755
        owner: db2inst1
      with_fileglob:
        - "/data1/backup/{{dbname}}.0.db2inst1.*"

```

---
Tags: ansible, linux, centos
---


# Question
Title: Is it possible to have pair of IPs in Ansible?
I have so many HA machines that one of them is primary and the other is standby. I have this playbook that copies a config file from primary and then from standby on local machine then use those files as input for a python script:

```
- hosts: primary
  remote_user: root
  tasks:
  - name: copy the config of primary to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/primary
      flat: yes
- hosts: standby
  remote_user: root
  tasks:
  - name: copy the config of standby to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/standby
      flat: yes
- hosts: 127.0.0.1
  connection: local
  remote_user: root
  tasks:
  - name: run python script
    shell: /AnsibleDir/driver.py
    register: myshell_output
  - name: copy the output to a local file
    copy:
      content: "{{ myshell_output.stdout }}"
      dest: "/AnsibleDir/output.{{ host }}"

```

and I use this command:
`ansible-playbook -i /Ansible/hostfile /Ansible/python.yml --extra-vars "host=test1"`

in `hostfile` I have this configuration:

```
[primary]
192.168.107.69
[standby]
192.168.107.70

```

I think that I should edit this file every time I want to run my playbook. so is it possible to have so many pairs of IPs?
for example something like this:

```
primary:1.1.1.1,standby:1.1.1.2
primary:1.1.1.3,standby:1.1.1.4
primary:1.1.1.5,standby:1.1.1.6

```

or can I assign each pair the variable I want instead of writing it in my command every time? like:

```
primary:1.1.1.1,standby:1.1.1.2,host:test1
primary:1.1.1.3,standby:1.1.1.4,host:test2

```

I searched a lot but I could't find anything and I'm new to ansible. Any suggestions?

# Answer
> Q: *Is it possible to have a pair of IPs in Ansible?*

A: Yes. It is possible. For example, it might be useful to create groups in the inventory with all pieces of information.

```
[primary]
1.1.1.1
1.1.1.3
1.1.1.5

[standby]
1.1.1.2
1.1.1.4
1.1.1.6

[ha]
1.1.1.1 stdby=1.1.1.2
1.1.1.3 stdby=1.1.1.4
1.1.1.5 stdby=1.1.1.6

```

It's possible to simplify the fetching of the configuration.

```
- hosts: primary, standby
  remote_user: root
  tasks:
  - name: copy the config to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/conf

```

Without `flat: yes` the configuration files will be stored in

```
/AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl

```

Then it's possible to reference the configuration files by the inventory hostname. For example `groups['primary']` (`secondary` the same way). Take a look at `myshell_output` and select the data for `content` and `dest` from the `items`

```
- hosts: 127.0.0.1
  connection: local
  remote_user: root
  tasks:
  - name: loop primary config
    command: "/AnsibleDir/driver.py
              /AnsibleDir/conf/{{ item }}/data1/cronjobs/gen_ddl"
    register: myshell_output
    loop: "{{ groups['primary'] }}"
  - name: copy the output to a local file
    copy:
      content: "{{ <SELECT-FROM-ITEM> }}"
      dest: "/AnsibleDir/output.{{ <SELECT-FROM-ITEM> }}"
    loop: "{{ myshell_output.results }}"

```

To reference primary/standby pairs use group `ha`. For example the play below changes the configuration both at primary and standby for all hosts in `ha`.

```
- hosts: ha
  tasks:
    - name: Change configuration at primary
      lineinfile:
        path: /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>
    - name: Change configuration at standby
      lineinfile:
        path: /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>

```

> Q: *I need to find the differences of configuration files in 2 machines. they need to be the same. In python script, it will find the differences and fix them. So I need to have both configurations on my local then run the python script for them.*

A: Fix the configuration files in 2 machines

```
- hosts: ha
  tasks:
    - name: Fix primary and standby config
      command: "/AnsibleDir/driver.py
                /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl
                /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_ddl"

```
> 2 votes

---
Tags: ansible
---


# Question
Title: Terraform | Error "transport is closing" when "terraform apply" after upgrading to 0.12
```
Terraform v0.12.7
+ provider.aws v2.25.0
+ provider.null v2.1.2
+ provider.template v2.1.2

```

Hello, when I try run `terraform apply` after upgrading to 0.12 I get error:

```
module.elasticsearch.module.ec2.aws_instance.this[0]: Modifying... [id=i-075090c94e5117ce3]
module.elasticsearch.module.ec2.aws_instance.this[1]: Modifying... [id=i-01fddbae2487e2573]

Error: rpc error: code = Unavailable desc = transport is closing

Error: rpc error: code = Unavailable desc = transport is closing

Releasing state lock. This may take a few moments...

```

Also I found next errors in logs:

```
[DEBUG] plugin: plugin process exited: path=/home/user/env-beta/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.25.0_x4 pid=25405 error="exit status 2"
2019/08/29 14:47:18 [DEBUG] module.elasticsearch.module.ec2.aws_instance.this[1]: apply errored, but we're indicating that via the Error
pointer rather than returning it: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [DEBUG] module.elasticsearch.module.ec2.aws_instance.this[0]: apply errored, but we're indicating that via the Error
pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalApplyPost, err: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing

```

And I notice strange thing, terraform wants add to this ec2 `credit_specification {}` but instance type is i3.large...

Maybe someone can help me. Thanks in advance.

# Answer
It's a bug in terraform aws ec2 module, last working version is 2.5.0
> 0 votes

---
Tags: terraform
---


# Question
Title: Docker bridge network Configuration for containers
I have an Angular application deployed in the container . If I log in to the container I can curl to the port 4200.

I have mapped to port 9001 to connect from my host .

But I can't curl from my host machine. What could be the issue? I think I should configure with docker bridge how to do that??

My Docker file:

```
FROM alpine:latest

#installing required pakages
RUN apk update
RUN apk fix
RUN apk add nodejs nodejs-npm apache2 curl yarn
RUN npm install -g @angular/cli
RUN npm install -g typescript
RUN mkdir -p /var/opt/angular

WORKDIR /opt

COPY . .

WORKDIR /opt/frontend

RUN npm install
RUN ng build
COPY start.sh /opt/frontend/start.sh
RUN chmod +x start.sh

EXPOSE 4200
CMD ["sh", "start.sh"]

```

Start-up script(start.sh):

```
#!/bin/bash

ng serve --host 0.0.0.0

```

While running the docker file I have specified. -p 9001:4200

# Answer
Since you are using the tag "Docker-Machine" I'm assuming that you are using Docker Toolbox to manage Docker. The problem is most likely an issue with Virtual Box configuration than your Docker configuration.

When you try pinging your container on port 9001, use the ip `0.0.0.0` instead of `127.0.0.1` or `localhost`. Another way to test would be to get the IP address of the docker-machine, `docker-machine ip`, and use this IP to ping your container (e.g. use `192.168.99.100:9001`.

If you are trying to access the container from another computer you may have to look into a solution like port forwarding as described here.
> 1 votes

---
Tags: docker, networking, ports, docker-machine
---


# Question
Title: Consul cluster does not work as expected when one of server is down
I guess, the main purpose of a cluster is failure tolerance. However, when I start the following consul cluster, in docker, it is not the case and I don't understand why.

```
services:

  # docker network create --driver=bridge discovery-network

  # SERVICE DISCOVERY
  consul-server-0:
    image: consul:1.6.0
    container_name: consul-server-0
    command: "agent -server -bootstrap-expect 2 -client 0.0.0.0 -datacenter datacenter-1 -node consul-server-0"
    networks:
      - discovery-network

  consul-server-1:
    image: consul:1.6.0
    container_name: consul-server-1
    command: "agent -server -retry-join consul-server-0 -client 0.0.0.0 -datacenter datacenter-1  -node consul-server-1"
    networks:
      - discovery-network
    depends_on:
      - consul-server-0

  consul-client-1:
    image: consul:1.6.0
    container_name: consul-client-1
    command: "agent -retry-join consul-server-0 -ui -client 0.0.0.0  -datacenter datacenter-1  -node consul-client-1"
    ports:
      - "8500:8500" # GUI
    networks:
      - discovery-network
    depends_on:
      - consul-server-0

networks:
  discovery-network:
    external:true

```

When I stop one of servers, the cluster does not work anymore. I am unable to register anymore service (through consul-client).

In the remaining server's logs, I can see the message `Failed to make RequestVote RPC` In the client's logs, I can see the message `No cluster leader`

What is wrong with my configuration?

# Answer
Based on my understanding of the topology you have described you essential have:

3 Nodes - 2 Servers and 1 Client

This layout is immediately a problem because the clustering protocol that consul uses relys on being able to easily elect a leader, which requires an odd number of servers.

If you consider the failure scenario of a 2 node consul cluster - as soon as a single node goes down your one remaining node cannot achieve consensus by itself because there's no way to get a majority of the servers voting on something.

In the consul docs the consensus protocol is reasonably well described Consensus docs

They even include a summary table which explicitly states how many server members could be lost for varying cluster sizes.

Following their advice, your 2 server cluster cannot withstand any fault because it would then become impossible to achieve quorum.

Really a minimum number of servers for a functional consul cluster is 3.
> 2 votes

---
Tags: docker, docker-compose, consul
---


# Question
Title: Is Cypress ok for acceptance tests on APIs in a CI/CD pipeline?
I have a small team but a gigantic application. We don't have PHP unit testing, so we will be using Cypress as a baseline for quality. Everything will go through a CI/CD pipeline and use regression end to end to make sure we haven't broken any modules or projects. One example is a FedEx API for shipping and we have multiple addresses, using GUI testing will be too slow, so I wonder if using Cypress is okay for that task as opposed to put another system (Postman with CLI) in the pipeline running testing with dynamic data (multiple Postal Codes and addresses? )

# Answer
> 2 votes
Cypress is designed for front-end end-to-end testing. It is designed for testing a users interaction with your application and verifying that it works correctly (e.g. logging in with a valid username and password load the landing page). Tools like this are designed to allow running in a headless mode which will let you run the tests in CI tool without loading a GUI.

Since it appears that your goal is to test the API interaction you are probably better off using a tool designed for that purpose. Most API test tools allow you to send a request to your API and verify certain things about the response (i.e., response code, response headers, response bodies).

While using an end-to-end tool (like Cypress) will ultimately test your API, it will take longer to setup, run, and often be more brittle. To use your above example, a tool like Cypress will require a script/test steps that log into your system, selecting multiple addresses for shipping, and verifying that everything displays properly. A tool like Postman would just test that when you send a request to your FedEx API with a specific body and headers, you get the expected response, body, and headers back. Setting up something like Postman will require a new tool, but will ultimately save time

Many of the API tools also provide a command line test runner and integrate with various CI tools (non-exhaustive list):

1. Postman \- Free, relatively easy to setup, and includes an automated testing tool (newman).
2. SmartBear \- Commercial product that integrates with most CI tooling.
3. Rest-Assured \- Open source option for REST API testing.

---
Tags: continuous-integration, testing, regression
---


# Question
Title: How to set 'image tag mutability' using CloudFormation?
When one would like to create an Elastic Container Registry then it is possible to set 'image tag mutability':

> Image tag mutability
>
> The image tag mutability setting for the repository. Select "immutable" to prevent image tags from being overwritten by subsequent image pushes using the same tag. Mutable Immutable

According to this user guide it does not seem to be possible to configure this using AWS CloudFormation.

# Answer
The `ImageTagMutability` property in ECR is not currently supported in CloudFormation, as verified by experimentation (nor is it part of the LifecyclePolicy property). There is sometimes lag between feature availability in CloudFormation, and the mutability flag was only introduced in late July.

This comes up elsewhere in the corners of AWS, including:

* AWS Global Accelerator - this entire service is not provisionable using CloudFormation.
* AWS WAF geo match rules cannot be created in CFN.
> 3 votes

---
Tags: aws-cloudformation, aws-ecr
---


title: How to handle a new Dev Ops position?

text:
# Question
Title: How to handle a new Dev Ops position?
At my current job, I started as a Software Developer, and I've worked myself into a DEVOPS position. I am worried about getting fired, not because I'm doing something wrong, but because I'm used to proactively doing things(writing code), whereas most of my time now is sitting around waiting for things to happen....

My daily duties(and I asked my manager in a detailed email listing these same ones) are,

> Monitor 5 different servers' performance.
>
> Monitor 2 of those servers Active Batch execution agents
>
> Handle requests as they are created.
>
> Handle project tasks as they are created.
>
> Monitor Email/Skype/Phone
>
> Build whatever is requested that isn't directly assigned thru the ticketing system.

It seems like a lot at first glance, but what is actually involved is little more than watching 1 monitor for new errors, and another monitor for messages. I can usually complete tickets in minutes-1 hour. Is this normal for Dev Ops-just sitting around waiting? If so, what can I do to be more useful? If not, am I in danger of getting fired?

# Answer
I agree with those who have commented that the role you describe doesn't really match a typical DevOps role. However, there are certain things you won't be able to change in your organization. Instead, focus on the things you can do.

## Understand the Objectives

The best place to start is with a clear understanding of Service Level Objectives. What kinds of things do external customers actually care about? If they don't care about uptime, don't make uptime measurements a high priority. Perhaps latency, software updates, content accuracy, or some other measure is more important. Also be aware of your internal customers' needs (your boss, coworkers, etc). What things do they care about?

Once you understand what is most important to your customers, **measure, measure, measure!** Get as much data as you can about those items that matter. How long does it take to deploy code? When you close a ticket, what percentage of the time does it come back unresolved? And so on. Use those measurements to improve over time.

Here are some areas you can focus your efforts (varies, depending on the company you work for):

### Configuration

If someone logged into a server and broke the configuration, how hard would it be to correct the error?

* You should be able to consistently build and replace resources with minimal effort.
* Configuration management should be complete so that all meaningful changes are tracked.
* Ideally, build up processes or tooling as needed so that developers can modify configuration themselves in a controlled and collaborative way (:D DevOps!).

### CI/CD

When new code is released, how does it get delivered?

* When code is pushed to source control, it should trigger automated testing and deployment. If possible, this should happen in a pipeline that deploys multiple environments in sequence as tests pass (dev -\> qa -\> uat -\> prod).
* Improve visibility so that you can easily see where different versions of the code are running, what errors are occurring, and how performance is changing.
* Enable your software developers to deliver code themselves (:D DevOps!).
* Collaborate with developers to improve this process (:D DevOps!)

### Logging/Metrics

When something goes wrong, how quickly can you find out what the problem was? If something went wrong 2 months ago, can you still find out what the problem was?

* Collect logs in a central place with tools that help you search and visualize.
* Aggregate metrics with tools that help visualize.
* Reduce noise! If it's not meaningful, don't keep it (but be a little conservative, sometimes things turn out to be meaningful later).
* Work with developers to understand how logging and metrics would help them deliver better software (:D DevOps!).

### Alerting/Incident Response

Ideally, you should not ever need to look at a graph, dashboard, or log message, unless there's an actual problem to solve. Sitting and watching a dashboard is not a good task for humans.

* Set up alerting so that you get notified when problems arise.
* Alerts should go to **ONE** person, who is the RIGHT person.
* Alerts should be *urgent* (needs to be addressed immediately), and *actionable* (the person getting the alert *can* fix it).
* In the spirit of paging the right person, you may be able to work with developers on getting them into some kind of on-call rotation (even just M-F 8-5 for a subset of code issues). (DevOps?)

### Disaster Recovery

If your servers were destroyed today, how long would it take to be up and running again?

* Take backups of data and set up backup infrastructure.
* Test backup recovery regularly.
* Test failover to secondary infrastructure regularly.
* Work with developers to better understand how code will operate in a failover scenario and improve as needed (:D DevOps!).

### Documentation

Heaven forbid something happen to you, but if it does, how well can someone else replace you. Document what you've built and improved so clearly that another person *can* replace you. Don't worry, this makes you *less* likely to be fired.

### Education

Always, educate yourself. Take online courses, read books, work on certifications. I don't know what the path looks like for you. I believe that spending a few hours every week specifically dedicated to improving your own skills is a valuable use of your time (even for your employer!).

A book like Site Reliability Engineering (Beyer, Jones, Petoff & Murphy) or The Phoenix Project (Kim, Behr & Spafford) might be good to read on this topic.

---

Totally possible I've entirely missed the point of your question. If so, I apologize. However, I think this ought to give you some ideas of how you can spend your time in ways that is worthwhile to both your employer and yourself.
> 1 votes

# Answer
Below comments will work for you:

> Monitor 5 different servers' performance.
>
> Monitor 2 of those servers Active Batch execution agents

* understand and priorities tasks/faults as per solution and business impact
* build jobs to alert you & your team (may be scripts already there)
* build rollbacks/optimizer according alerts (some rollback scripts may be already there)

> Handle requests as they are created.
> Handle project tasks as they are created.

* make sync closely with developer/testing teams and follow up / understand there approach, scripts and solutions what they have developed

Monitor Email/Skype/Phone

> Build whatever is requested that isn't directly assigned thru the ticketing system.

follow up manually tasks/ solution designs that to be build and understand the requirement. To build jobs, you should educate your self at least with below technologies:

* data format json, xml, yml
* python , python/ linux scripting
* Python framework like Robot, Ansible
* Rest API
* CI/CD Jankins, Git
* CI/CD suite xtesting
* Linux
* solution specific dependencies
> 0 votes

---
Tags: operations
---


# Question
Title: How to make a goreleaser script to build a deb file locally?
I am an absolute novice in Golang but I want to modify a Go build script to build a file locally only instead of publishing it to GitHub.

https://github.com/dahendel/docker-machine-driver-cloudstack/blob/master/.goreleaser.yml

How to proceed?

I've created a fork with a Dockerfile to capture the release environment I can so far "reverse-engineer" but as I run it I get the following error and don't know how what and how to modify in the .goreleaser.yml.

```
   • releasing using goreleaser 0.117.1...
   • loading config file       file=.goreleaser.yml
   • RUNNING BEFORE HOOKS
   • LOADING ENVIRONMENT VARIABLES
      • pipe skipped              error=publishing is disabled
   • GETTING AND VALIDATING GIT STATE
      • releasing v1.0.5, commit d47a87ff8c671ec70b99a125fd6aadc45949905d
      • pipe skipped              error=disabled during snapshot mode
   • PARSING TAG
   • SETTING DEFAULTS
      • LOADING ENVIRONMENT VARIABLES
      • SNAPSHOTING
      • GITHUB/GITLAB/GITEA RELEASES
      • PROJECT NAME
      • BUILDING BINARIES
      • ARCHIVES
            • DEPRECATED: `archive` should not be used anymore, check https://goreleaser.com/deprecations#archive for more info.
      • LINUX PACKAGES WITH NFPM
            • DEPRECATED: `nfpm` should not be used anymore, check https://goreleaser.com/deprecations#nfpm for more info.
      • SNAPCRAFT PACKAGES
      • CALCULATING CHECKSUMS
      • SIGNING ARTIFACTS
      • DOCKER IMAGES
      • ARTIFACTORY
      • S3
      • BLOB
      • HOMEBREW TAP FORMULA
         • optimistically guessing `brew[0].installs`, double check
      • SCOOP MANIFEST
   • SNAPSHOTING
   • CHECKING ./DIST
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • pipe skipped              error=not available for snapshots
   • BUILDING BINARIES
      • running hook              hook=dep ensure
   ⨯ release failed after 0.01s error=pre hook failed:

```

# Answer
I have cloned your repo and try that in my local machine. Here is the steps :

1. Git clone
2. executing Dry run (testing everything before doing a release "for real" :

$ goreleaser release --skip-publish
3. show there is no error

```
 SIGNING ARTIFACTS
  • pipe skipped              error=artifact signing is disabled
• DOCKER IMAGES
  • pipe skipped              error=docker section is not configured
• PUBLISHING
  • pipe skipped              error=publishing is disabled
• release succeeded after 20.75s

```
4. execute goreleaser for release

$ goreleaser release
5. goreleaser will created **dist** folder inside project and this folder will consist of distribution packages (deb, rpm).

I have encounter some issues and here is what I do :

* error=missing GITHUB\_TOKEN, GITLAB\_TOKEN and GITEA\_TOKEN

create github or gitlab token ( https://github.com/settings/tokens) and put it as environment variabel

```
export GITHUB_TOKEN=xxxxyyyyyzzzzz

```

resolve the issue.

* pre hook failed: xxxx is not within a known GOPATH/src

as I see in your goreleaser.yaml

```
hooks:
pre: dep ensure

```

you're using dep ensure, checking $GOPATH and make sure $GOPATH pointing to right path of your Go project.

* error=dist is not empty, remove it before running goreleaser or use the --rm-dist flag

dist folder has been created before, you can either manually delete the folder or add flags --rm-dist when executing goreleaser command

```
$ goreleaser release --skip-publish --rm-dist

```
* error=nfpm failed: rpmbuild not present in $PATH

this error occured as I was running on mac machine so there is no rpmbuild installed, installing rpm, rpmbuild solve the issue

```
$brew install rpm

```
* error=git is currently in a dirty state, please check in your pipeline what can be changing the following files: M Gopkg.lock

Goreleaser seems to check file diff, so as because running hook (dep ensure) updating the Gopkg.lock and this changes/updates are not pushed to git. The solution is always pushing the changes to git.

* error=git tag v1.0.5 was not made against commit 3ae83eebd904d33cc549117254e857ebea04df90

reading from GoReleaser documentation which is "GoReleaser enforces semantic versioning and will error on non-compliant tags. Your tag should be a valid semantic version. If it is not, GoReleaser will error."

after pushing to git, make sure you have to update the tags, in this case I updates the tags to v1.0.6 (previously v1.0.5).

make sure release text is there.
> 1 votes

---
Tags: build-pipeline
---


# Question
Title: Is it possible to install RPM build on Debian?
I have a Docker-based Debian Sid build environment for Go. The build script of an is configured to build both .deb and .rpm.

The rpm build process of a component I wish to build requires though the tool `rpmbuild`.

Is it possible to install rpmbuild on Debian or should I better try another Linux distro?

# Answer
You can indeed build RPMs on Debian. You need the `rpm` and `rpmbuild8` packages:

I tried it with a docker container:

```
docker run --rm -it debian bash -c 'apt-get update && \
apt-get -y install rpm librpmbuild8 && which rpmbuild'

```

---

More info on packages:

```
The following additional packages will be installed:
  dbus debugedit libapparmor1 libarchive13 libdbus-1-3 libdw1 libexpat1 libgdbm-compat4 libgdbm6 libicu63 liblua5.2-0 libmagic-mgc libmagic1 libnspr4
  libnss3 libperl5.28 libpopt0 librpm8 librpmio8 librpmsign8 libsqlite3-0 libxml2 netbase perl perl-modules-5.28 rpm-common rpm2cpio
Suggested packages:
  default-dbus-session-bus | dbus-session-bus rpm-i18n lrzip gdbm-l10n file sensible-utils perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make libb-debug-perl liblocale-codes-perl alien python elfutils rpmlint rpm2html

```
> 2 votes

---
Tags: docker, build-pipeline
---


# Question
Title: How to trigger jenkins pipeline only on merge to develop branch from github webhook trigger?
I am having a problem to trigger Jenkins pipeline job based on merge any branches to develop branch only. I also did not find any webhook for merge as like bitbucket. I also tried using pull request but that triggered the job on any changes to the PR branch which is not my requirement.

I have also shared my code below and my Jenkins Configuration. Jenkins Configuration also please note that it is develop branch but in the image I've shown `*/feature` I have changed it to develop but still the issue is the same.

```
def CONTAINER_NAME=""
def CONTAINER_TAG="${env.BUILD_NUMBER}"
def CONTAINER_PORT=""
def ECR_REPO=""
def HOST_PORT=""
def URL=""
def INSTANCE_IP=""

pipeline {
  agent any
  stages {
    stage('Image Build Develop') {
      when {
        expression {
         return env.BRANCH_NAME != 'feature_cicd'
         echo env.BRANCH_NAME
        }
      }
      steps {
        git branch: "develop", url: 'https://github.com/jenkinsci/git-parameter-plugin.git'
        imageBuildDevelop(CONTAINER_NAME, CONTAINER_TAG)
        }
    }

    stage('Image Tag') {
      steps {
        imageTag(CONTAINER_NAME, CONTAINER_TAG, ECR_REPO)
      }
    }

    stage('Image Push') {
      steps {
        imagePush(CONTAINER_NAME, CONTAINER_TAG, ECR_REPO)
      }
    }

    stage('Image Deploy') {
      steps {
        imageDeploy(HOST_PORT, env.BUILD_NUMBER, INSTANCE_IP)
      }
    }
  }
}

def imageBuildDevelop(containerName, tag) {
  echo "#------------------- Checkout Develop Branch -------------------#"
  git branch: "feature_cicd", url: 'https://github.com/jenkinsci/git-parameter-plugin.git'
  echo "#------------------- Login into ECR Repository -------------------#"
  sh "`/var/jenkins_home/.local/bin/aws ecr get-login --no-include-email`"
  echo "#------------------- Build Docker Image for Media-API -------------------#"
  sh "docker build -t $containerName:$tag -t $containerName ."
  echo "#------------------- Image Build Complete -------------------#"
}

def imageTag(containerName, tag, repo) {
  echo "#------------------- Tag media-api image -------------------#"
  sh "docker tag $containerName:$tag $repo:$containerName-$tag"
  echo "#------------------- Image Tag Complete ------------------#"
}

def imagePush(containerName, tag, repo) {
  echo "#------------------- Login into ECR Repository -------------------#"
  sh "`/var/jenkins_home/.local/bin/aws ecr get-login --no-include-email`"
  echo "#------------------- Push Image into ECR Repository -------------------#"
  sh "docker push $repo:$containerName-$tag"
  echo "#------------------- Image Push to ECR Repository Complete -------------------#"
}

def imageDeploy(hostPort, tag, ip) {
  echo "#------------------- SSH into media-api instance -------------------#"
  sh "ssh ubuntu@$ip \"BUILD_NUMBER=$tag\" \
  ' cd reactApi_test/ &&\
    echo \"TAG=${BUILD_NUMBER}\" > .env &&\
    `aws ecr get-login --no-include-email` &&\
    docker-compose up -d --build &&\
    ./test.sh '"
  echo "#------------------- Media-API container started on Port '${hostPort} (http)' -------------------#"
}

```

# Answer
You'll want to use the CHANGE\_TARGET environment variable. If that variable exists then you are dealing with a merge request.

You can use the following code snipped to then determine the branch name and alter the job's logic depending on which branch the code is being merged into.

`branchName = env.CHANGE_TARGET ? env.CHANGE_TARGET : env.BRANCH_NAME`

In your code example you would replace

```
        expression {
         return env.BRANCH_NAME != 'feature_cicd'
         echo env.BRANCH_NAME
        }

```

with

```
        expression {
         branchName = env.CHANGE_TARGET ? env.CHANGE_TARGET : env.BRANCH_NAME
         return branchName != 'feature_cicd'
         echo branchName
        }

```
> 2 votes

---
Tags: jenkins-pipeline
---


# Question
Title: How to clean up old images in docker registry with jenkins
I have a jenkins multi branch project, that builds on every commit and pushes the image to our private registry:

```
docker-reg:5000/<project>/<branch>:latest
docker-reg:5000/<project>/<branch>:<build-server>-<build-number>

```

How can I delete the labels in the docker registry in order to let the garbage collector kick in upon:

* build gets orphaned after some time
    + remove `<project>/<branch>:<build-server>-<build-number`
* branch gets merged and jenkins sub project gets automatically deleted
    + remove `<project>/<branch>:latest`

# Answer
We have a script which we run periodicaly which scans our docker registry for orphaned items, and then deletes them.

Unfortunately, it requires multiple API calls to get all the data required, and then to delete each image. But basically, if an image hash is lacking a 'latest' tag or a tag that starts with 'v' it gets culled.

You could probably also try deleting based on last modified date.

We run this as a cron job in kubernetes, but you could just as easily run a script like this as a periodic jenkins job.
> 1 votes

# Answer
This is a very complicated issue for a few reasons:

1. The biggest problem is Jenkins does not have any way of notifying you when a branch is deleted or a OR is merged. There are a lot of issues in Jenkin's jira, but from what I know - this one is the monitored one.
2. You would somehow need to keep a record of all of the published tags.

Overall I don't see how you'd achieve this as it's simply not possible from Jenkin's side. It would be cool if somebody else has found a solution to this. I have looked for one a few times already without any luck.
> 0 votes

---
Tags: docker, jenkins, docker-registry
---


title: Get vars in Jenkins Pipeline

text:
# Question
Title: Get vars in Jenkins Pipeline
In FreeStyle Project, I have lots of env vars, which are available from any part of Jenkins' job. I also used ghprb plugin to trigger that job against GH PRs, which added some extra env vars.

Later, that job became more complex, so I've decided to put it on Jenkins Pipeline. But now, the issue is, I can't retrieve those env vars in a pipeline, which were available before, in the Freestyle Project.

I'm a newbie in writing Jenkins Pipeline, that's why I decided to use declarative approach for that purpose (but I don't think it's root cause). So, my question is, how to retrieve env vars in Jenkins Pipeline, such as: `${BUILD_CAUSE}`, `${ghprbTargetBranch}`, `${ghprbActualCommitAuthorEmail}`?

# Answer
> 2 votes
If you want to use environment variables with the `${VAR}` syntax in the declarative pipeline, then you need to define those variables in the `environment { }` block.

for example:

```
environment {
        DISABLE_AUTH = 'true'
        DB_ENGINE    = 'sqlite'
    }

    stages {
        stage('Build') {
            steps {
                echo "Database engine is ${DB_ENGINE}"
                echo "DISABLE_AUTH is ${DISABLE_AUTH}"
                sh 'printenv'
            }
        }
    }

```

If you want to use environment variables that come from other plugins etc, then you need to access them with the `env.VAR` syntax. For example:

`echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"`

# Answer
> 0 votes
Not sure if you did, but put the vars into double quotes:

```
... "${BUILD_CAUSE}"

```

---
Tags: jenkins-pipeline
---


# Question
Title: What's the way to deploy a resource once on a helm chart that is deployed multiple times?
in my use case i've got a helm chart for my application.

One of the resources is actually a **Grafana Dashboard** for the application.

This dashboard actually monitors all instances of the application, and i want it to be deployed only once.

Considering i'm performing 2 **helm install** commands for the same apps, which will cause the dashboard to be deployed **twice**, what's the best practices to have it deployed only once?

* Should it be a totally different "release"? (and be deployed individualy?)
* Should I specify in the 2nd installation not to deploy the dashboard? (seems bad)

What's the best practice?

# Answer
> 1 votes
I've eventually created an umbrella chart with alias:

```
dependencies:
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child1
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child2

```

And in values.yaml i've changed the needed values, then added the grafana dashboard in the bundled yaml file.

---
Tags: kubernetes, helm
---


# Question
Title: SqlPackage.exe: Error SQL0: Required contributor with id 'id' could not be loaded
while trying to build dacpac using sqlpackage.exe, we are getting the following error

```
Microsoft.SqlServer.Dac.DacServicesException: An error occurred during deployment plan generation. Deployment cannot continue.
Error SQL0: Required contributor with id 'BcDevKit.DeploymentContributors.DacMacroSqlDeploymentContributor' could not be loaded.

```

The above error can be fixed if I copy the contributors from `C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150\Extensions`

to `C:\Program Files (x86)\Microsoft SQL Server\150\DAC\bin\Extensions`

I want sqlpackage to search for contributors only in Visual Studio directory. Can anyone help?

# Answer
I have figured the answer on my own and wanted to help someone like me.

So, the problem is I was referencing sqlpackage.exe from `C:\Program Files (x86)\Microsoft SQL Server\150\DAC\bin` directory, that is why it was looking for the contributor in it's sub-directory `\Extensions`.

I was looking around in Visual Studio installation directory and found the sqlpackage.exe in `C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150` directory, updated the path in environment variables and it worked.
> 2 votes

---
Tags: sql
---


# Question
Title: How to mitigate "[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release" in quay.io?
\[DEPRECATION NOTICE\] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the quay.io registry NOW to avoid future disruption

This issue occurs if a `docker pull` is issued or the API is consulted.

# Answer
> 6 votes
You don't need to do anything; this warning is a mistake. See https://github.com/moby/moby/pull/39736

---
Tags: quay.io
---


# Question
Title: How to execute shell command or call URL on new gitlab issue?
I want to execute a shell command or call a URL when a user creates a new issue. The reason behind this is to synchronize our Gitlab issues with our CalDav calendar infrastructure. We are using Gitlab community.

# Answer
> 1 votes
That's mostly what gitlab webhooks are for I think.

For your case that would be an issue event and you'll have to create something to parse the posted json and do whatever is needed on your CalDav infra to match your needs.

---
Tags: git, gitlab
---


# Question
Title: What strategies exist to transition DB Migrations with Stored Procedures to Continuous Delivery?
Our company has broken up most of our large Monolith application into smaller Microservices and we have a clear plan on how to get new services on Kubrenetes into a CI/CD pipeline.

However, we still have some parts of our applications which have not been broken up because of their database dependencies and the fact that they require downtime to deploy because of DB Migration issues. Part of the problem is that we have a large number of stored procedures which contain business logic in them.

I've seen plenty of information about how to refactor code and APIs so that we can have Blue/Green/Canary deployments. But how do we this for DB changes or migrations without downtime?

# Answer
> 1 votes
It is quite difficult to migrate the databases without downtime, however, using some approaches you can do it with very minimum impact, nearly a zero downtime.

The main idea in this is to create new databases with new schema. Then make changes in your code. First migrate the data from the old database to the new database. Then make changes in your code such that it starts writing to both the databases, old and new. You need to decide a time (time when there will be least writes to the database). Then stop all the writes. Remove the old database and make sure all the writes should now happen to new database from now on. Then start the writes back.

In this way, you can achieve DB migration with minimal downtime.

You can check this for a detailed information on how this can be done.

---
Tags: databases, migration
---


# Question
Title: How do I automate the process of pulling the latest code and restarting a service on git push?
I have an application (Logstash) running on a Linux VM and the settings for it are on a GitLab repo. What I want is for the VM to pull the latest version of the settings file and restart the service when I push an update to the repo.

What is the easiest way of achieving this?

# Answer
There's no "easiest" way per se, there's the better way for you. I'd recommend using a configuration management tool like Chef or Puppet (for a pull model) for which there's already resources to configure and handle logstash restarts.

That said logstash is able to watch its config file and reload it itself, quoting elastic.co documentation:

> Starting with Logstash 2.3, you can set Logstash to detect and reload configuration changes automatically.
>
> To enable automatic config reloading, start Logstash with the `--config.reload.automatic` (or `-r`) command-line option specified. For example:
>
> ```
> bin/logstash –f apache.config --config.reload.automatic
>
> ```

So you may just set a crontab to `git pull` your repository periodically and have logstash reload itself.
> 1 votes

# Answer
I would suggest `jenkins` for this as it can easily integrate wit `github`, `gitlab` or any other SCM. Jenkins jobs can be setup such that the build runs whenever you push a commit to the `gitlab` repo. In the job, you can mention to update the logstash config on the linux VM and then reload/restart your service.

Here are some articles which can help you in proceeding further.

http://fourkitchens.com/blog/article/trigger-jenkins-builds-pushing-github/

https://oncomputingwell.princeton.edu/2018/01/triggering-a-jenkins-build-every-time-changes-are-pushed-to-a-git-branch-on-github/

NOTE: If you want to setup this just for a single node, You can also create a very simple script which should pull from github, say every 5 mins, and if the changes are there, it can update logstash config and simply restart the logstash.
> 0 votes

---
Tags: git, gitlab, cicd
---


# Question
Title: AWS Production Configuration ASP.NET Core & Docker
So we have an asp.net core application we are hoping to start using with docker and host this all on aws.

We have multiple clients that we set up this project for, each with their own configuration settings, such as connection strings, application name, token secret, mailing server information, etc...

The problem we are going to have/are having, is how to make production configuration not built into the source code/docker whilst being accessible for multiple instances/containers that will need to use this on ECS/EC2.

So the idea we currently have, which i was hoping someone could check or suggest a better solution if there is one, is that we will build the container free of the specific configuration, then access the appSettings.production.json which we would mount from outside of the container. Because this would be on amazon it would be stored in EFS if capable or S3.

Is this possible and an appropriate solution?

Thanks

# Answer
> 1 votes
If you are going full on AWS, I would suggest to have a look at AWS SSM parameters. When EC2 or ECS container are starting, you have a script that is going to fetch the parameters. Then you can do whatever you need to do: replace variables in config files with the parameters fetched previously for example. You could also use files stored in S3. You would be able to copy it and place it in the right application folder when your EC2 or ECS container are starting.

---
Tags: docker, amazon-web-services, amazon-ecs
---


# Question
Title: gitlab -ci script not being updated on runner
I have a problem I can't get my head around in gitlab-ci: I have a monorepo where I store some infra code (IaC), in this repo I have a script which allow me to lint the saltstack states code, as :

```
#!/bin/bash

# Std var
_hostname=$(hostname -f)

for file in $( find . -iname "*.sls" ! \( -path '*/files/orchestration_states/*' -or -path '*/files/reactor_states/*' -or -name "top.sls" \) ); do
    state_path_name=$( echo ${file%.*}| tr -d "."| tr "/" ".")
    if ! sudo salt-call --file-root=. --pillar-root=/home/gitlab-runner/salt_pillars/testBed/ --local --retcode-passthrough state.show_sls ${state_path_name}  pillar='{"tgt_client_name": "dummy-dummy.domain}' ; then
        echo -e "> Error found while validating state file: please fix it!\
        \n
        \nDetails:\
        \n - File(s) name(s): ${state_path_name}\
        \n - Pipeline ID: ${CI_JOB_ID}\
        \n
        \n - Job link: ${CI_JOB_URL}"\
        |mail -s "${CI_PROJECT_NAME} | Pipeline #${CI_JOB_ID} has failed" -aFrom:${_hostname}@spie.com "mail@domain.com"
        exit 1
    fi
done

```

In this script I need to update the `--pillar-root=/home/gitlab-runner/salt_pillars/testBed/` to replace the `testBed` string by something lese, let's say `toto`. But when I update the string inside the repo the runner keeps using the old version of this script (which failed because the path is not correct).

I can see the correct version of the script inside the repo, but the runner does not update this script when fetching the branch to lint the states.

Any ideas about why the runner does not update the script ?

Thanks

# Answer
> 0 votes
OK, as usual the error was between the keyboard and the screen... I did use the "try again" button on the jobs list, which, of course was making use of the same commit, and therefore the changed made to the script were not taken into account ...

---
Tags: salt, gitlab-ci-runner, linting
---


# Question
Title: Looping through hosts from an inventory group in a group_vars file
Is it possible to do something like this:

```
# inventories/inventory1/group_vars/group1

filewatches:
    {% for host in groups['group2'] %}
    - type: "copy"
      directory_name: "{{ host }}"
      shell_script: "{{ script_dir }}/ascript.sh"
     {endfor %}
    - type: "anotherone"
      ...

```

```
# inventories/inventory1/hosts

[group1]
somemachine01

[group2]
machine1
machine2
machine3
...

```

Basically, I want to generate a bunch configuration that is similar except for just one variable (in this case the directory\_name variable). I would rather do this than making 20+ different entries which all differ by JUST the directory\_name variable.

I did not include the playbook or roles information for the sake of clarity/terseness. I hope I was clear enough and gave enough context, and I appreciate any help.

Thanks!

# Answer
> 2 votes
Yes. It is possible to create a YAML data structure with Jinja and read the variable in with the filter from\_yaml.

The play

```
- hosts: group1
  vars:
    script_dir: /tmp
  tasks:
    - debug:
        var: filewatches

```

with the group\_vars

```
$ cat group_vars/group1.yml
filewatches: "{{ filewatches_var|from_yaml }}"
filewatches_var: |
    {% for host in groups['worker'] %}
    - type: "copy"
      directory_name: "{{ host }}"
      shell_script: "{{ script_dir }}/ascript.sh"
    {% endfor %}

```

and with the inventory

```
$ cat hosts
...
group1:
  hosts:
    test_01

group2:
  hosts:
    test_02
    test_03

```

gives

```
ok: [test_01] => {
    "filewatches": [
        {
            "directory_name": "test_02",
            "shell_script": "/tmp/ascript.sh",
            "type": "copy"
        },
        {
            "directory_name": "test_03",
            "shell_script": "/tmp/ascript.sh",
            "type": "copy"
        }
    ]
}

```

---
Tags: ansible
---


# Question
Title: docker build throws UnknownHostException with jenkins dind slave + kubernetes plugin
So Jenkins is installed inside the cluster with this official helm chart. And this is my installed plugins as per helm release values:

```
  installPlugins:
  - kubernetes:1.18.1
  - workflow-job:2.33
  - workflow-aggregator:2.6
  - credentials-binding:1.19
  - git:3.11.0
  - blueocean:1.19.0

```

my Jenkinsfile relies on the following pod template to spin up slaves:

```
kind: Pod
spec:
  # dnsConfig:
  #   options:
  #     - name: ndots
  #       value: "1"
  containers:
  - name: dind
    image: docker:19-dind
    command:
    - cat
    tty: true
    volumeMounts:
    - name: dockersock
      readOnly: true
      mountPath: /var/run/docker.sock
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock

```

Slaves (pod /dind container) starts nicely as expected whenever there is new Build.

However, it broke at the step of "docker build" in ( Jenkinsfile pipeline `docker build -t ...` ) and it breaks there :

```
Step 16/24 : RUN      ../gradlew clean bootJar

 ---> Running in f14b6418b3dd

Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip

Exception in thread "main" java.net.UnknownHostException: services.gradle.org

    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)

    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)

    at java.base/java.net.Socket.connect(Socket.java:591)

    at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)

    at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)

    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)

    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)

    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)

    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)

    at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)

    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)

    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)

    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)

    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)

    at org.gradle.wrapper.Download.downloadInternal(Download.java:67)

    at org.gradle.wrapper.Download.download(Download.java:52)

    at org.gradle.wrapper.Install$1.call(Install.java:62)

    at org.gradle.wrapper.Install$1.call(Install.java:48)

    at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)

    at org.gradle.wrapper.Install.createDist(Install.java:48)

    at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)

    at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)

The command '/bin/sh -c ../gradlew clean bootJar' returned a non-zero code:

```

At the first galance, I thought it's DNS resolution issue with the Slave container (`docker:19-dind`) since it is alpine. That's why I debug its `/etc/resolv.conf` by adding `sh "cat /etc/resolv.conf"` in the Jenkinsfile.

I got :

```
nameserver 172.20.0.10
search cicd.svc.cluster.local svc.cluster.local cluster.local ap-southeast-1.compute.internal
options ndots:5

```

I removed the last line `options ndots:5` as per recommendation of many thread on the internet.

But it does not fix the issue.

I thought again and again and I realized that the container responsible for this error is not the Slave (docker:19-dind), instead, it is the intermediate containers that are opened to satisfy `docker build`.

As consequence, I added `RUN cat /etc/resolv.conf` as another layer in the Dockerfile (which starts by `FROM gradle:5.5-jdk11`).

Now, the `resolv.conf` is different :

```
Step 15/24 : RUN cat /etc/resolv.conf

 ---> Running in 91377c9dd519

; generated by /usr/sbin/dhclient-script

search ap-southeast-1.compute.internal

options timeout:2 attempts:5

nameserver 10.0.0.2

Removing intermediate container 91377c9dd519

 ---> abf33839df9a

Step 16/24 : RUN      ../gradlew clean bootJar

 ---> Running in f14b6418b3dd

Downloading https://services.gradle.org/distributions/gradle-5.5-all.zip

Exception in thread "main" java.net.UnknownHostException: services.gradle.org

```

Basically, it is a different nameserver `10.0.0.2` than the nameserver of the slave container `172.20.0.10`. There is NO `ndots:5` in resolv.conf this intermediate container.

I was confused after all these debugging steps and a lot of attempts.

## Architecture

```
Jenkins Server (Container )
     ||
(spin up slaves)
     ||__ SlaveA (Container, image: docker:19-dind)
             ||
       ( run "docker build" )
             ||
             ||_ intermediate (container, image: gradle:5.5-jdk11 )

```

# Answer
Found the answer here

Just add `--network=host` to `docker build` or `docker run`.

```
 docker build --network=host foo/bar:latest .

```
> 2 votes

# Answer
Have you considered not using DinD? I would look to tools like Kaniko, Buildah, Img, Umochi, FTL, or Bazel.

Here's a good example from the Jenkins Kubernetes plugin that builds containers without using Docker or DinD in Kubernetes.

https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/kaniko.groovy
> 0 votes

---
Tags: jenkins, amazon-web-services, kubernetes, dns, docker-in-docker
---


# Question
Title: How to get other than "<no value>" when interpolating {{ .Environment.SOME_VAR }} (with SOME_VAR set) into cache key?
Using this configuration:

```
version: 2
jobs:
  test:
    docker:
    - image: nixorg/nix:circleci
    environment:
      NIXPKGS_REV: 3c83ad6ac13b67101cc3e2e07781963a010c1624
    steps:
    - restore_cache:
        name: Restore Nix Store Paths
        keys:
        - paymentserver-nix-store-v1-{{ .Environment.NIXPKGS_REV }}
        - paymentserver-nix-store-v1-

```

The job tries to restore the cache using the key `paymentserver-nix-store-v1-<no value>`.

https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-job says this is the correct way to set an environment variable. https://circleci.com/docs/2.0/caching/#using-keys-and-templates says interpolating environment variables this way is supported for cache keys.

What's going on? How do I use the value of an environment variable in a cache key?

# Answer
> 1 votes
A closer reading of the documentation about using environment variables in cache keys reveals the issue (emphasis mine):

> The environment variable variableName (supports any environment variable exported by CircleCI or added to a specific Context—**not any arbitrary environment variable**).

"exported by CircleCI" seems to mean the list of mostly-`CIRCLE_...`-prefixed variables (documented as "Built-In Environment Variables") while (unsupported) "any arbitrary environment variable" includes variables set in an `environment` section in `config.yml`.

I ended up working around this limitation something like this:

```
jobs:
  build:
    docker:
      - image: "nixorg/nix:circleci"

    environment:
      NIXPKGS_REV: "3c83ad6ac13b67101cc3e2e07781963a010c1624"

    steps:
      - "run":
          name: "Prepare For Cache Key"
          command: |
            echo "${NIXPKGS_REV}" > nixpkgs.rev

      - restore_cache:
          name: "Restore Nix Store Paths"
          keys:
            - paymentserver-nix-store-v1-{{ checksum "nixpkgs.rev" }}
            - paymentserver-nix-store-v1-

```

That is, instead of trying to use the value of `NIXPKGS_REV` from the environment in the cache key, an earlier step writes the value of that environment variable to a file and then the cache key is constructed using the checksum of that file. Computing file checksums for use in cache keys is well supported so this produces a cache key which is derived from the value of the environment variable. The key doesn't *contain* the value of the environment variable but since the key's only purpose is to be unique for certain unique inputs, using the checksum works just as well.

---
Tags: circleci-2.0
---