# Question
Title: why can't map local path to docker's inside?
I want to build private docker hub. and I did it with:

```
 docker run -it --restart=always -p 7901:5000   -v /data/registry:/tmp/registry registry /bash/bin

```

And my disk informations as below:

```
[root@10-19-47-24 docker]# df -lh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  4.4G   16G  22% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  401M  3.5G  11% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vdb         60G  572M   60G   1% /data
tmpfs           783M     0  783M   0% /run/user/0
tmpfs           783M     0  783M   0% /run/user/1001
tmpfs           783M     0  783M   0% /run/user/1000
overlay          20G  4.4G   16G  22% /var/lib/docker/overlay2/9923f757c58c0792526f4cb32750401847f127b9e33c0dc78a29d7933bcde5bf/merged
shm              64M     0   64M   0% /var/lib/docker/containers/51959fbdc1663f968eff770d54c56bc04ddaba2c2d3623058c90a00f582fa40c/mounts/shm

```

And my version of docker is:

```
[root@10-19-47-24 docker]# docker --version
Docker version 18.06.0-ce, build 0ffa825

```

And I can push `hello-world` to the hub, but I can't see anything inside the `/data/registry`. My question is where is my registry? Is there any mistake of my command?

# Answer
Oh! I have search for a while, it's should be like:

```
docker run -d --name=ak_registry  --restart=always -p 7900:5000   -v /data/.docker:/var/lib/registry    registry

```

Used `/var/lib/registry` instead of `/tmp/registry`
> 1 votes

---
Tags: docker, centos
---


# Question
Title: Determine the URL for a SCM trigger from inside the build
My Jenkins is affected by a known Jenkins bug where it will trigger jobs that have an SCM poll set if a change to the shared library happens. I can't have this, so I thought I would get the trigger info from inside the job and if the URL was from the library repo I would abort the build. This has proven to be extremely difficult to do.

This doesn't work...

```
println ("Cause: " + currentBuild.causes[0])

```

or

```
print currentBuild.rawBuild.getCause(hudson.triggers.SCMTrigger$SCMTriggerCause)

```

or

```
def causes = currentBuild.rawBuild.getCauses()
for (cause in causes) {
    print cause
}

```

The first example barfs

```
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified field org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper causes
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.unclassifiedField(SandboxInterceptor.java:397)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:381)

```

but doesn't allow me to whitelist it on the script security page.

The last 2 print thing `hudson.triggers.SCMTrigger$SCMTriggerCause@3` but it seems the only thing I can do is call `getShortDescription()` and it prints `Started by an SCM change`.

If you know how to print more information about an SCM trigger such as the URL please share =).

# Answer
Some bad news and some good news :)

## Bad news:

not a bug. this is a feature (default behaviour)

I don't understand why this bug is still open, either regression, or miscommunication.

## Good news:

During setup of Global Shared Libraries jenkins allows to mark whether you should trigger user jobs upon changes in the library. And, you can also mark this during library usage in the job's groovy.

Please refer to:
> 1 votes

# Answer
Recently, I used this code snippet (that's part of a regular groovy script located inside the 'vars/' directory) to get the project URL as set in the pipeline project config. Hope this helps.

```
scm.getUserRemoteConfigs()[0].getUrl()

```

FYI, it does not expect to have any SCM operation to have taken place.

For more info, refer to this SO thread.

Hopefully, am not violating the devops stackexchange ground rules with this reference to stack overflow.
> 1 votes

# Answer
Unfortunately I had to disable the Groovy sandbox because I ran into so many situations where methods I wanted to use in my jobs did not appear available for whitelist on the script security page. Instead of adding a bunch of individual method calls to a whitelist or disabling the sandbox, you can also use a global shared library, since global shared libraries are automatically whitelisted in the sandbox. (The shared library approach worked well for me at first, but eventually I ran into situations where I did not feel that the code I was writing was appropriate for a shared library, so I just disabled the sandbox as it had never provided any benefit to me anyway. Just as a warning, disabling the sandbox is usually fine in single-tenancy situations, but not in multi-tenancy Jenkins instances.)

As for your code, unfortunately `SCMTriggerCause` doesn't appear to have any properties pointing to the actual URL of the SCM source (I can't confirm this as I don't use SCM polling on my Jenkins instance). Instead you could try something like this:

```
currentBuild.rawBuild.changeSets.collect { it.browser.repoUrl }

```
> 0 votes

---
Tags: jenkins-pipeline
---


# Question
Title: How to optimize blue/green AWS deployments?
I got the following setup with AWS code deploy:

However it takes about 7 minutes from the moment I push a git branch remotely to the moment the deploy is completed. The time is broken down like so:

* bitbucket deploy: 45 seconds

* provisioning a new EC2 instance: 3:40 minutes

* installing application on replacement instance: 40 seconds

* 2 minutes rerouting traffic from old instance to new one \*this one is extremely long, if it were done manually it would literally take 10 seconds)
* 17 seconds terminating second instance

All together it's around 7 minutes.

How can I optimize this? I'm looking to shrink it to below 1 minute.

# Answer
got this answer from aws support

> Hello,
>
> Andrew from AWS Support here.
>
> I understand that you would like to know if there is a way to speed up Code Deploy as currently it takes 7 minutes to complete. Please correct me if needed.
>
> After examining the link and the Code Deploy environment I can see that the step which involves rerouting traffic from the old instance to the new instance can potentially be tuned to try and decrease the overall deployment time. In addition, using a different deployment configuration may also help to decrease the deployment time.
>
> To try and reduce the time taken to perform a blue/green deployment edit the health check on the Elastic Load Balancer (ELB). Reduce the health check interval from 30 seconds to 5 seconds and reduce the response timeout from 20 seconds to the default of 5 seconds \[1\]. This should decrease the time it takes the ELB to check whether an instance is healthy and ready to receive traffic.
>
> Another setting that may also help decrease the time it takes to deploy is the deployment configuration, set the deployment configuration from the currently used "one at a time" to "all at once". CodeDeploy "all at once" will at once attempt to deploy to as many instances as possible, the application status will be shown as succeeded if the application is deployed successfully to one or more of the instances. This configuration will also reroute all the traffic in the placement group to all instances at once and will succeed if the traffic is successfully routed to at least once instance \[2\].
>
> To reduce this deployment from 7 to 1 minute will be high unlikely as EC2 launches are managed by CodeDeploy, there may be ways to improve to EC2 launch times manually but this is not the case when using CodeDeploy.
>
> I hope you find this information useful, if you have any other questions or need clarification on anything above please let me know, I'd be happy to help
>
> Take care,
>
> References: \[1\] Configure Health Checks for Your Classic Load Balancer - Health Check Configuration - https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html#health-check-configuration
>
> \[2\] Working with Deployment Configurations in AWS CodeDeploy - Predefined Deployment Configurations for an EC2/On-Premises Compute Platform - https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-configurations.html#deployment-configurations-predefined
>
> Best regards,
>
> Andrew v. Amazon Web Services
> 2 votes

# Answer
> bitbucket deploy: 45 seconds

This seems correct. Not sure of your repo size, but if it is hosted outside of your region, that's not out of the ordinary.

> bitbucket deploy: 45 seconds

Again, seems correct. Not much room for optimization here.

> provisioning a new ec2 instance: 3:40 minutes

What size of instance are you provisioning? Are you provisioning one or more instances? The obvious solution would be to use in place deployment vs blue-green, but I think the extra time is well worth it. If this is for production/staging, I think you are fine. If this is in a testing environment, and you are aiming for quick feed back, you could consider provisioning new instances after a successful blue-green so that, if there is no AMI change, they are ready to go for installation immediately.

> installing application on replacement instance: 40 seconds

Nothing to optimize here

> 2 minutes rerouting traffic from old instance to new one \*this one is extremely long, if it were done manually it would literally take 10 seconds)

My guess is that step is performing extra steps that you do not perform when you manually reroute traffic. It's possible that CodeDeploy is performing extra checks to make sure that connections are draining, new connections are established before old ones are closed, and other test/checks. This is probably your best area to optimize, but not at the sacrifice of checks and balances.

> 17 seconds terminating second instance

Nothing to optimize here.

**Summary:** I don't think it is feasible or worthwhile to attempt to optimize your deployment further. The amount of effort it would take and sacrifices you would need to make in your test and checks far out ways the potential minute or two you would save in my opinion. I also don't think it is possible to achieve a \<1 minute deployment. That would average ~15-20 seconds a step, which would require a lot of sacrifices in your deployment strategy.

If your goal is for quick iterations for QA, code testing, integration testing, etc., and you absolutely want sub 1 minute deployments, I recommend creating a Cloud Formation stack with your exact deployment except skipping the blue/green deployment model for an in place deployment model. This way, you can quickly see the results of your code. You should then later run your full suite blue/green deployment after your testing in either another testing environment or your staging environment.
> 2 votes

# Answer
Depending on the value of a faster deployment in your situation, some options are:

* Maintain spare capacity in your ECS cluster (either larger instance types, or more smaller ones) to allow for side-by-side deployment of new versions of the service along with the old, so that new *tasks* can be started without having to provision new *instances*. This obviously has a dollar cost, but again, depending on the value of rapid deployments to your organization, and the amount of deployments you expect to do per day, it might be worthwhile.
* Use multiple sets of target groups for optimal Blue-Green deployments. This still requires extra cluster capacity, and extra work in your pipeline, but you can ensure that once tested, the replacement set of tasks all go live at once (and are fully tested).
> 0 votes

---
Tags: aws-codedeploy
---


# Question
Title: Is there an established scheme for creating diagrams about Containers, Services etc.?
We have UML to describe classes, components, workflows. We have BPMN to describe business processes. We have more old-fashioned forms used in times past for structural diagrams.

Is there anything established or semi-established in the DevOps world? The information contained in a simple diagram should be something like:

* Which containers are there in a containerized app, including their associations (i.e. which statically depends on which for building) and the (micro)services they provide?
* How do they interact dynamically (networking etc.)?
* What responsibility does each of the containers fulfill?
* What requirements does each container have on the host, if relevant?
* ... and so on.

I'm neither asking for a list of any private schemes somebody may fancy, nor about opinions on which is the most beautiful.

I'm strictly asking about whether there *exist* any that either by purpose are industry standards or have turned out to just be used in many places by chance/convention.

The purpose of using such a scheme would be to have something that would instantly be recognized by fellow DevOps people; or which could easily be shown in contexts where it is not appropriate to look into Dockerfiles or other Infrastructure-as-code sources (e.g., when scribbling on a board, or holding a more abstract presentation).

# Answer
> 2 votes
The established method is infrastructure as code.

Want to find out what processes run in a container? Check the Dockerfile. Infrastructure dependencies? Check the Cloudformation template/Terraform blueprint. Interactions between containers? Check the docker-compose.yml or Kubernetes yaml file.

# Answer
> 0 votes
If you are in AWS, there is a very large native set of icons. Confluence has an imported drawing system (draw.io) that includes the AWS icons, it works but its a bit buggy. You can download the AWS icons. In Kubernetes there might be some new mapping tools. You can draw anything with FOSS Inkscape using imported icons. There are likely many other tools. At my shop developers just hack away at their own drawings and copy each other's style.

However I think you were asking for a UML-type of graphical language for cloud representation. Wouldn't it be great to have a 3D system that shows network flows and security connections? A startup for Kubernetes would be a great idea. I think New Relic and/or Datadog have some ability to map network flows, too.

---
Tags: culture, documentation
---


# Question
Title: Looking for a Desired State and Configuration as Code central configuration management framework
I am looking for a central configuration management framework that supports the following scenarios:

1. Desired State Configuration. My expectation is to express the desired configuration in code and have the framework ensure it constantly to prevent the configuration drift. And, of course, it should support the pull mode.
2. All the configuration is managed in the Source Control (git, for example). I.e. no ad hoc scripts.

I come from the software development environment where having code outside of the Version Control is unthinkable. I want to treat my DSC specs the same - everything is in the version control, so when the configuration is changed by anyone it is easy to know exactly when, why, how and who changed it.

I started to look at the Azure Automation DSC, not sure if it fits the bill though - see Is it possible to keep all the DSC configurations in Version Control and make Azure Automation fetch it from there automatically?

Our servers are Windows Servers. Mostly 2012R2 with 2016 being slowly introduced.

# Answer
> 3 votes
Ansible is one example of a tool that that allows you to define all your configuration as scripts stored in your SCM and can deal with configuration drift. Ansible has modules to work with Windows in multiple ways including DSC through run books. Full information is available at: https://docs.ansible.com/ansible/latest/user\_guide/windows.html.

For desired state configuration specifically - https://docs.ansible.com/ansible/latest/user\_guide/windows\_dsc.html#what-is-desired-state-configuration

There is also additional information in writing scripts to insure the system stays in the desired state. It is important to insure the way your writing the script does not in itself make differences in the system. - https://shadow-soft.com/ansible-idempotency-configuration-drift/

I know of many environments where the full system is defined and deployed with Ansible and it is used to remove any drift.

There are other ways to also help with the situation including using dynamic provisioning to shorten the life of systems. Only keeping them up for the length of a test or other short time frame.

---
Tags: configuration-management, casc
---


title: Can jenkins control baremetal with ipmi

text:
# Question
Title: Can jenkins control baremetal with ipmi
I have a setup with a bunch of blades that pxe boot into coreos using terraform/matchbox setup. I currently control them all manually using ipmitool from my desktop.

Is there a way to let jenkins control this?

an example use case would be if a bunch of different builds are being run and a new one is kicked off jenkins will know to use one of the slaves not running a build so it will power reset the machine and wait for it to come up and then run my docker stuff and the build.

# Answer
> 3 votes
Generic answer yes, it is possible with a lot of work...

Is it something wise to do? In my opinion no, you should convert your bare metal nodes to hypervisors with low maintenance and use common tooling to start slaves Virtual Machines on those hypervisors.

# Answer
> 0 votes
Jenkins is but a scheduler. It is oriented to respond to code changes, and do whatever you want.

About IPMI, if you cannot follow @Tensibai's advice, I would suggest to wrap the operations at IPMI with something like ansible (Depending on what you actually need to do there).

Usually ipmi environment is used to TEST baremetal operations, in which case you have no other choice but use things like `ipmi_boot` or `ipmi_power` ansible modules, or actually write shell based ipmitool invocations to match incompatible hardware special saltos you need to do. Or even to use other "older generatino" things (chef, puppet, even cfengine etc.)

You still may find some consolation in Foreman/Satellite to control BMs via ipmi using PXE boot, and all the other "oldies" technologies.

All that of course, if you cannot actually wrap your BMs somehow to make them either k8s cluster with flexible slave-pods OR VMs, or something ... more "up-to-date" (Sometimes you can't I know).

I believe if you don't have network/io constraints you should go with Kubernetes (a.k.a. k8s), and not with Docker.

With both you can work with locally created images, but Kuberenets will allow you much more convenient housekeeping of containers, groups of containers, and even better - a "service", "operator" and other cool entities.

Besides you can version control your deployments with Helm charts and many many many other nice toys.

Good luck!

---
Tags: jenkins, jenkins-plugins, terraform
---


# Question
Title: Tool to collect developer environment parameters
I would like to start collecting parameters from local developer environments on my team, such as to see how they configure their properties files (not in source control), what directory they use as their application root, where they have this and that software installed etc.

So I was envisioning having a script that each would run daily and which would post that data to some web service within some application that does this kind of stuff within the DevOps realm -- team management etc. Is there such an app for local developer management, just like Jenkins does builds or Chef does deployments?

Ideally, I would go into this console and get an idea of how each developer on my team has their environment configured without that developer letting me see their laptop, provided that, of course, my data collection script includes all the relevant parameters.

# Answer
Instead of trying to watch, how about working together to define a scripted way of defining environments that is scm managed. Having the configuration for deployment in the SCM insures they are always consistent and allows the collaboration and communication between team members about the ways they want to work.

There are many tools that can be used to script the configuration of the system level as well as the application level definitions. Using this kind of definition based environment configuration not only helps with the development level, but then also helps with the deployment to later stages capabilities.

This would also facilitate the communication between team members a fundamental principle of DevOps.
> 4 votes

# Answer
I agree with @Tensibai and rather using a tool I would suggest to talk with your team members about the tools they use rather than sniffing their system.

If one would use a tool to spy other systems then that is not aligned with DevOps. DevOps is about getting rid of silos. Using a tool for "secretly" monitoring colleague system is a contradiction. Imagine that the developers would know that such a tool is used, then silos will be created again and that is not DevOps. DevOps is about building bridges between team members and departments, e.g. Ops, Dev and QA.
> 2 votes

# Answer
Optionally, you could consider vortual workbench like Che (in case it is mature enough for your purposes, can't judge on this.) And, such actions must be definitely in sync with privacy protection law and colleagues informed. I see a use case where it is not spying but rather establishing standards (a good workshop topic).
> 0 votes

---
Tags: configuration-management
---


# Question
Title: Modify sudoers file with ansible playbook template
I am trying to create a sudoers file with ansible template. The sudoers file should look like below:

```
Cmnd_Alias LS = /bin/ls
Cmnd_Alias LESS = /usr/bin/less
Cmnd_Alias DU = /usr/bin/du

%support1 ALL=(ALL) NOPASSWD: LS, LESS, DU

```

What I have managed so far is below:

```
Cmnd_Alias LS = ls
Cmnd_Alias LESS = less
Cmnd_Alias DU = du

%support1 ALL=(ALL) NOPASSWD: LS, LESS, DU

```

The template looks like below:

```
{% for item in commands %}
Cmnd_Alias {{ item|upper }} = {{ item }}
{% endfor %}

%{{ group }} ALL=(ALL) NOPASSWD: {% for item in commands %}
{{ item|upper}}{% if not loop.last %}, {% endif %}
{% endfor %}

```

vars

```
commands:
  - ls
  - less
  - du

```

As far as I know, ansible template module does not have anything which will execute the command in the remote server and print the output otherwise I was thinking of changing template file to look like below:

```
{% for item in commands %}
Cmnd_Alias {{ item|upper }} = `which {{ item }}`
{% endfor %}

%{{ group }} ALL=(ALL) NOPASSWD: {% for item in commands %}
{{ item|upper}}{% if not loop.last %}, {% endif %}
{% endfor %}

```

and the output will be like what I wanted.

Is there any other method which can make it simple?

BTW I have already checked this post

# Answer
**TL;DR:** KISS. Don't use less.

People often make a mistake with ansible by trying to make variable things that don't need to be. Unless there are multiple places where you define the list of commands that support can access, it is perfectly acceptable to just put them in the template create file:

**templates/etc/sudoers.d/support1**

```
Cmnd_Alias LS = /bin/ls
Cmnd_Alias LESS = /bin/cat
Cmnd_Alias DU = /usr/bin/du

%support1 ALL=(ALL) NOPASSWD: LS, LESS, DU

```

or even explicitly as you don't reuse the Cmnd\_Alias anywhere

```
%support1 ALL=(ALL) NOPASSWD: /bin/ls
%support1 ALL=(ALL) NOPASSWD: /bin/cat
%support1 ALL=(ALL) NOPASSWD: /usr/bin/du

```

And add some task like:

```
- name: add templates
  template:
    src: {{ item }}
    dest: /{{ item }}
    owner: root
    group: root
    mode: 0640
  with_items:
    - etc/sudoers.d/support1

```

You would only use templates instead of files because late on there might be some variable to add to all those or the group name might come from variable if you get another role that creates the groups.

If you need to use variable, the thing you can do is to use a list of hashes like this:

```
sudoers.support1.commands:
- { alias: "LS", path: "/bin/ls" }
- { alias: "DU", path: "/usr/bin/du" }

```

Then in the template:

```
{% for item in sudoers.{{ group }}.commands %}
Cmnd_Alias {{ item.alias }} = {{ item.path }}
{% endfor %}
%{{ group }} ALL=(ALL) NOPASSWD: {{ sudoers.{{ group }}.commands | map(attribute='alias') | join(', ') }}

```

**It is not safe to use /usr/bin/less**

In all this you did not notice much important thing and that is the use of less as viewer. Sadly that is a security hole. You can type '!bash' for invoking bash. And by pressing 'v' you get into editor based on VISUAL, EDITOR or LESSEDIT variables. So you can give them '/bin/cat' and they can always pipe the content into less themselves. Note this is still a security hole as some files in unix are very intentionally restricted for example:

```
/etc/shadow
/etc/sudoers
/etc/ssh/ssh_host_rsa_key
$HOME/.ssh/id_rsa

```
> 6 votes

# Answer
### Firstly

you should refer to Jinja handling of spaces/newlines. your current template would create new lines, And I think this will confuse `sudo` and fail syntax validation (IIRC, the right syntax is to add `-` like: `-%}` at the for loop, but you should "play" and see what happens). To render a template you can do it on your workstation, without running it on actual target machine.

Also, I think creating the template with 1 command at instruction line is more readable:

```
{% for command in commands %}
%{{group}} ALL=(ALL) NOPASSWD: {{command}}
{% endfor %}

```

### Secondly

I do not recommend to edit existing global files with ansible. Instead create your custom template under `/etc/sudoers.d/` (like you've mentioned you saw).

This is the right way to do it, because:

* the system will have all defaults as is.
* your template will be far shorter.
* if you do mistakes, you would be sure where to look - in your template.

### Thirdly

I think executing `which` inside `sudoers` is an original idea, but should not work.
> 1 votes

---
Tags: ansible
---


# Question
Title: Docker swarm join linux container Error - remote CA does not match fingerprint
**Start docker swarm :** docker swarm init --advertise-addr MANAGER-IP

**Join docker swarm:** docker swarm join --token MANAGER-IP:2377

I am using Windows 10, it is working fine with Windows container mode, but gives below error in Linux container mode.

**Error:**

Error response from daemon: remote CA does not match fingerprint. Expected: 91030413f17ec7c023a2a796ee05a024915080ca8dfd646a597c7e966f667df6

**Docker swarm manager command: docker node ls**

```
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
2zf1l2o7sl2a1qka55s2vi77x *   moby                Ready               Active              Leader

```

Host name is **moby**, when running in Windows container mode it gives machine host correctly.

# Answer
> 3 votes
If you configured and joined correctly you should see two hosts: one for Linux and one for Windows.

*Windows hosts cannot natively run inside the Linux hosts and vice-versa*. This means you have to have different kernels for Linux hosts and Windows hosts in the same swarm. This way they can communicate across the same Docker network.

Refer to Docker Engine on Windows for more info.

# Answer
> 1 votes
your token is wrong.

you can get worker token in the manager node:

```
docker swarm join-token -q worker

```

it works for me.

https://docs.docker.com/engine/reference/commandline/swarm\_join/

---
Tags: docker, containers, docker-swarm
---


title: What is a glob in salt-stack?

text:
# Question
Title: What is a glob in salt-stack?
I was going through targeting minions in salt. There I came across a term called `glob`. Can someone explain with an example what is a **`glob`** in Salt-Stack?

# Answer
Globbing is a term used for matching file name patterns with wild cards. You can see a more detailed definition here https://searchsecurity.techtarget.com/definition/globbing

As it relates to Salt Stack, it's finding the minion id using wild cards. Additional details here: https://docs.saltstack.com/en/latest/topics/targeting/globbing.html
> 4 votes

---
Tags: deployment, terminology, salt
---


# Question
Title: Logstash multiple pipelines going into same index
I'm trying to set up a centralized syslog for multiple log sources.

So I have a logstash that has two separate inputs and two separate outputs

However for some reason the data from one of the inputs ends up in both indexes

What am i doing wrong?

Below are both pipelines' configs

```
input {
  tcp {
    port => 5052
    codec => "json_lines"
  }
}
output {
  elasticsearch {
    hosts => "10.50.6.116"
    index => "remote"
  }
  file {
    path => "/var/log/logstash/remote-tcp.log"
  }
  stdout { codec => rubydebug }
}

input {
  file {
    path => "/data/vmlist/*.csv"
    start_position => "beginning"
    sincedb_path => "/tmp/sincedb"
  }
}
filter {
  csv {
    separator => ","
    columns => ["VM Name","Creation Date","Owner","Type","Message"]
  }
}
output {
  elasticsearch {
    hosts => "http://10.50.6.116:9200"
    index => "vms"
    document_type => "csv"
  }
  stdout{ codec=> rubydebug}
}

```

# Answer
> 2 votes
There are two ways to accomplish this, though one of them was only available recently.

The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...

```
input {
  tcp {
    port => 1525
    codec => json_lines
    tags => [ 'tcp' ]
  }
}

input {
  file {
    path => '/var/log/app.log'
    codec => 'json'
    tags => [ 'file' ]
  }
}

output {
  if 'file' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'files'
    }
  }
  if 'tcp' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'tcp'
    }
  }
}

```

This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with a way to do multiple pipelines in separate files.

```
- pipeline.id: tcp-inputs
  path.config: '/etc/logstash/pipelines/tcp.cfg'
  pipeline.workers: 3
- pipeline.id: file-inputs
  path.config: '/etc/logstash/pipelines/files.cfg'
  pipeline.workers: 2

```

This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.

# Answer
> 0 votes
Once parsed your config create one and only one pipeline, with various inputs, various filters and various outputs.

You have to use some conditional constructs to apply filter and output only to specific messages, usually using a special tag or field set on input.

Some exemple are available in logstash documentation:

> ```
> output {
>   if [type] == "apache" {
>     if [status] =~ /^5\d\d/ {
>       nagios { ...  }
>     } else if [status] =~ /^4\d\d/ {
>       elasticsearch { ... }
>     }
>     statsd { increment => "apache.%{status}" }
>   }
> }
>
> ```

---
Tags: elastic, logstash
---


title: Configure AWS cli inside chef cookbook

text:
# Question
Title: Configure AWS cli inside chef cookbook
I am working on setting up something with S3 and Chef, I have AWS CLI installed and working but I need to figure out how to specify the `access ID` and the `secret access key`. I know there are cookbooks out there for this but I have tried a few with no luck so I am not using anyone else's cookbooks for this project anymore.

Suggestions on how I can run `aws configure` and then output the correct value for each of the 4 things it asks?

Code in my script to set env variables:

```
export AWS_ACCESS_KEY_ID=<key>

export AWS_SECRET_ACCESS_KEY=<secret_key>

export AWS_DEFAULT_REGION=<region>

```

# Answer
> 2 votes
Have you tried setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables? By setting these variables **before** you run your cli commands, you should not have to run the `aws configure` command.

> **Environment Variables**
>
> Environment variables override configuration and credential files and can be useful for scripting or temporarily setting a named profile as the default. The AWS CLI supports the following environment variables.
>
> `AWS_ACCESS_KEY_ID` – AWS access key.
>
> `AWS_SECRET_ACCESS_KEY` – AWS secret key. Access and secret key variables override credentials stored in credential and config files.
>
> `AWS_SESSION_TOKEN` – Specify a session token if you are using temporary security credentials.
>
> `AWS_DEFAULT_REGION` – AWS region. This variable overrides the default region of the in-use profile, if set.
>
> ....

---
Tags: amazon-web-services, chef, aws-cli
---


# Question
Title: Jenkins Pipeline "Build when a change is pushed to GitLab"
I have a Jenkins declarative pipeline that is ran via a webhook from GitLab whenever a change is pushed to my repo. In Jenkins, I have the following trigger option selected:

Is there a way for me to add this trigger to my Jenkins declarative pipeline, if so how?

# Answer
Looks like you are using the Gitlab plugin for jenkins? if so this link should be of help https://github.com/jenkinsci/gitlab-plugin#declarative-pipeline-jobs
> 6 votes

---
Tags: jenkins, jenkins-pipeline, gitlab
---


# Question
Title: Bitbucket Webhook plugin for Jenkins - differentiate between branches
We use Jenkins and have the Bitbucket plugin installed. We've got multiple repos and use webhooks for lots of jobs coming from our `foo` repo. No problems there, the jobs go off without a hitch. Now however we want to start running a couple jobs via webhook from our `bar` repo.

### The Problem

The default behavior here is that as soon as we attach a webhook to `bar`, commits to either repo will trigger any job with polling. I know we can hardcode in logic in each job to either only build when commits match a certain pattern, or ignore any commits matching a pattern.

Given the number of jobs we have that currently use polling this would be insanely unideal to have to code in to make it backwards compatible, not to mention potentially doing this again if we start running unit tests on even more repos in the future.

### The Ideal Solution

I'm hoping I might be missing some way to differentiate between commits made from `foo` vs `bar` on the Bitbucket side, rather than needing to change anything in each Jenkins job. Ideally, there's some way to change the webhook so each repo gets its own url. Does this, or some other blanket solution, exist? (I will accept "no" as an answer if that's the case).

# Answer
I believe it is now possible to configure the webhook on per-repo basis:

https://mohamicorp.atlassian.net/wiki/spaces/DOC/pages/381419546/Configuring+Webhook+To+Jenkins+for+Bitbucket+Bitbucket+Branch+Source+Plugin
> 5 votes

# Answer
You could:

* prune your branches. (guessing you'd have done this by now if feasible.)
* implement Jenkinsfile, a code-driven pipeline, and simply remove the jenkinsfile from the branches you don't want built or triggered by SCM-polling.
> 2 votes

# Answer
One could use SCM polling and specify different branches that need to be polled.
> 1 votes

# Answer
So... I had this issue and solved it by making sure I passed the pull request ID to the Jenkins job and the Branch ID from Pull Request. I've attached a screen shot.

And this as well:

you can *absolutely* control what Jenkins pulls from your SCM w/the proper configurations - these should get you there.

Also: I have polling disabled - you really don't want to use polling as it is expensive - configure a BitBucket PullRequest plugin to call your Jenkins job upon creation as you are doing and you shouldn't *need* to poll, although I don't know that particular plugin... this one is great:

https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket
> 0 votes

---
Tags: jenkins, jenkins-plugins, bitbucket, bcbsn
---


title: When to get rid of '-SNAPSHOT'

text:
# Question
Title: When to get rid of '-SNAPSHOT'
I'm working on a Git workflow (not git-flow) for a Java project that uses Maven and Artifactory for artifact storage. On develop and feature/\* branches the version number contains '-SNAPSHOT'. It is not desirable to have release/\* or master branches with '-SNAPSHOT' so I'm trying to figure out when (and how) to get rid of it.

I came up with a partial workflow, as below, but it doesn't handle removing 'SNAPSHOT':

```
                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v
develop         *            * v1.0-snap
                v1.0-snap    |
                             |
                             v
release/1.1        v1.0-snap * -----> *
                                    v1.1-snap

```

I came up with 2 options, but both might be wrong.

**Option 1**

```
                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v             v1.1-snap
develop         *            * v1.0-snap  *
                v1.0-snap    |           /
                             |          /
                             v         /
release/1.1        v1.0-snap * -----> * v1.1-snap
                                       \
                                        \
                                         \
master                                    * --------> *
                                          v1.1-snap   v1.1

```

**Option 2**

```
                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v             v1.1-snap
develop         *            * v1.0-snap  *
                v1.0-snap    |           /
                             |          /
                             v         /
release/1.1        v1.0-snap * -----> * ----- >* v1.1
                                   v1.1-snap    \
                                                 \
                                                  \
master                                             *
                                                  v1.1

```

Both options seems weird to me, though.

Is there any best practice for this kind of flow?

# Answer
I have a similar setup.

For `feature/*` and `develop`, I use something like `18.08-SNAPSHOT` in my pom. When we're ready to start release hardening, I'll cut a release branch (`release/18.08`) and change the version in the pom/s to `18.08`.

I'll also bump the version on develop to `18.09-SNAPSHOT`, for example.

From there, bugfix and finalise the release, merge to `master` when appropriate.

We're not properly CI/CD yet, but the issue I bumped in to with release builds was that I still needed unique builds in Artifactory. So, at build time (for release branches) Jenkins will change the version in the pom to something like `18.08-326` (326 being build number). It won't commit this change, but the artefact that gets built and pushed to Artifactory will be unique (`foobar-18.08-326.war`)
> 2 votes

---
Tags: continuous-integration, git, maven
---


# Question
Title: System monitoring and reporting with Jenkins
Are there any plugins that would enable me to distribute system monitoring and reporting via Jenkins, if I provided it with data in a required format. E.g. I would like to write a simple script that would display the status of every server and maybe some metrics. If that runs, say every hour, and dumps data in an XML or CSV or whatever conventional format of the plugin, can I distribute reports with that data using Jenkins using a special plugin?

I understand there are separate apps that do system monitoring but the fact of the matter I am trying to make Jenkins my one-stop-shop for what my less technical usership needs so, because I enabled them to run builds and deployments using Jenkins as they wish, I would prefer not to give them a different interface with a different login and other overhead.

# Answer
> 1 votes
having "ab"used Jenkins to be many things, the most sensible (IMHO) you can do is expose monitoring dashboards via build report page. I think you should not even put Jenkins and monitoring under the same url, due to the "who will guard the guard" problem.

Also, nowadays monitoring tools are too versatile and have too many "faces" (a face for each characteristic user, and some specific users prefer to have their own personal "face" to monitoring) About "reporting" I don't know what do you mean. if its reporting of builds statuses - READ ON. Also there are other things to let user go through (logs, alerts, test reports)

So what I would suggest to address monitoring visibility is:

1. monitor your Jenkins in your monitoring :) (builds, statuses, etc.)
2. define monitoring url inside Jenkins (e.g. as Global Env Variable)
3. let's assume you have a job that deploys service `$svc` on environment `$env`. using Groovy you can generate:
    1. generate a url to point at that monitoring service "dashboard" for that service in that env. let's call it `serviceMonitorUrl`
    2. `environmentMonitorUrl`, and others
    3. using `addSummary` and related stuff you can add to the build report a list of links to monitoring
    4. you can decorate your build history column with "labels" pointing at branch, env, hash, whatever
    5. your userdom will get build page with those links, and will be able to "click" them.

Having said that, there are Jenkins pipelines visualizations you can also use to visualize Jenkins jobs statuses, etc. e.g. this one.

Generally: Jenkins is good for CI related scheduling. But the truth is - CI should not be the focal point of users. CI should be "transparent" until something is broken. Thus making users "aware" of Jenkins (CI), and doing everything through it, unless business requirement is met by it (like company's product is being controlled/operated though Jenkins) - you should not do this. This kind of "knowledge" is not really serving the users. They need to be able to see where to look for information easily, not be specific application's advanced users.

Think what happens when you decide to migrate from Jenkins to something else. you are hooked now.

(it isn't necessarily bad, just be aware of it)

# Answer
> 0 votes
If I understand the question correctly, you want a way to distribute status information of other servers, not specifically the Jenkins server, to less-technical users. Is that correct? I guess the follow-up question would be, do the servers in question already have some sort of API or other interface to query the status?

If so, then your script could do something as simple as using sh to call `curl` or `wget` to query the information from those servers. Since you seem to be ok with using whatever output format that a plugin would need, perhaps you can save the output as HTML and then use the Jenkins HTML Publisher plugin to generate reports. These reports would be viewable on the Jenkins build page.

Alternatively, you could output the status information as text and include it in the body of an e-mail sent to the less technical users via the Jenkins Email-ext plugin

# Answer
> 0 votes
# SSO

Centralizing user logins is certainly a reasonable goal. There are various "single sign-on" solutions out there. But even something hacky like copying a `htpasswd` file around can work. But squeezing everything into the jenkins UI just because that is the only login people have seems like asking for trouble. I'd let everyone see my dashboards without logins before I tried to make it all work through jenkins. You'll want to ease logins to other systems like email or PagerDuty eventually so finding a nice general solution to this will be handy for a long time.

# Dashboards

There are lots of nice ways to do dashboards. If you want to do more yourself and not pay monthly charges I'd look at Prometheus. If you want to do less yourself and monthly charges are survivable I'd look at datadog. There are are other choices available of course, but these are some of the better result-for-the-effort-put-in choices out there right now. Either way it will be a much more natural process than reinventing the wheel to make it work through jenkins.

Any of these solutions will take data from a variety of inputs and let you have a coherent dashboard. Most of them will also let you interact with the graphs to zoom in on certain ranges of time or subsets of servers. You can usually let your users create their own dashboards as well which will reduce the number of innane tweaks that you're asked to make over time.

---
Tags: jenkins, monitoring
---


# Question
Title: when does currentBuild.changeSets in Jenkins gets filled?
Some background: The project I'm working on is in SVN. Very very large. A lot of tests are running on Jenkins. I want to implement a new tests which should be executed only if files in certain directory have been changed.

The idea that I had was:

* to trigger the job using a pollSCM every ten minutes
* check the files in currentBuild.changeSets. If a file contains the directory, continue.
* Only then I do a full checkout of the repository, compile, and run the tests.

But before I write the code, I was wondering when does currentBuild.changeSets? Will it have the required information when triggered by a pollSCM? I could not find this in the documentation

Yes, this could be done in a better way using git and hooks and other tools but that's what I have right now.

# Answer
It will be set after any `checkout` steps (or derivatives of `checkout` such as `git` or `svn`).

So for instance:

```
println(currentBuild.changeSets) // should print an empty set

checkout(scm)

println(currentBuild.changeSets) // should print out any changes in the current build

```
> 5 votes

---
Tags: jenkins
---


# Question
Title: Is it ok to logout after calling docker stack deploy?
If I immediately terminate the process that called `docker stack deploy` will the docker swarm continue?

My use case is this:

1. In my CI job I ssh into my swarm cluster node.
2. I copy the new stack.yml file and issue
3. `docker stack deploy -c stack.yml stack`
4. Since this immediately returns, my CI job disconnects from ssh.

My question is whether this can lead to some unforeseen side-effects? Is my deploy being terminated midway? Or will docker swarm be smart enough to keep going on its own.

Thanks

# Answer
> 1 votes
It *is* safe. As soon as docker returns it is safe to logout/leave the shell.

The issue was that Docker 18.06.0-ce **broke** `docker stack deploy`. Reverting to 18.03.1 (specifically `18.03.1~ce-0~ubuntu`) made it work.

---
Tags: docker, docker-swarm, cicd
---


# Question
Title: Alarms management with Elasticsearch and Kibana
I'm working on my first implementation of the Elasticsearch-Logstash-Kibana stack. I have everything working, but now I need to:

* Set alarms on certain thresholds/events
* Send notifications on alarms
* (desirable) Manage alarms from within Kibana (i.e. alert/acknowledge)

I've read some partial solutions using another tools beside ELK, as Nagios, but I'm wondering what is the industry's "best" approach to this need.

**Note**: I've already taken a look at X-Pack and seems to be one of the acceptable solutions, but the licencing details and costs are still vague - not publicly available. I've contacted Elastic but need to answer a few questions with data I still don't have.

# Answer
> 7 votes
I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part)

But besides that, it sounds like your interested in monitoring. In my opinion ELK is not a classic monitoring tool (more: data collection, data processing and data visualization). You should take a look at Icinga2 (https://www.icinga.com/products/icinga-2/) which can can be configured highly dynamical and is compatible with nagios plugins

# Answer
> 0 votes
You can use emergent AI based alerts by totalcloud. You can customize the thresholds and see a dry run of the alarms too.

# Answer
> 0 votes
> Manage alarms through Kibana

That specifically is one of the licensing points for X-Pack, so there won't be much non-X-pack stuff out there that does this.

I've been doing alarms through Logstash config, but it isn't managed through X-Pack. We've had better luck managing logstash `output {}` blocks through GitHub-style pull-request workflows. It will require knowing how to write logstash code, and some sense of the object model of your events. But if people are fluent in writing Kibana dashboards, they already have that fluency; most of your training will be in scoping the logstash code.

---
Tags: monitoring, notifications
---


title: Jenkins UUIDs ... oh the humanity

text:
# Question
Title: Jenkins UUIDs ... oh the humanity
I have an environment block

```
environment {
  SUITE_RUN_ID = UUID.randomUUID().toString()
  SMOKE_CMD = "runtests.sh SUITE_RUN_ID=${SUITE_RUN_ID}"
  FAILED_TESTS = "output_rerun_info.rb SUITE_RUN_ID=${SUITE_RUN_ID}"
}

```

When I use SMOKE\_CMD and FAILED\_TESTS in their respective stages I get different UUIDs... why?

# Answer
`def suiteRunId = UUID.randomUUID().toString()` worked at the top of the `Jenkinsfile`.

Thanks all for your answers.
> 6 votes

# Answer
The block needs to be resolved for each stage, otherwise you could not use stage dependent methods in the block, so there would need to be some check on what methods to use and what to not use, it would lead to even more problems. Further each stage can be executed on different node, so the code needs to re-resolve it. Otherwise it would just behave differently with single node or in parallel. Just deal with it.

Instead of setting the variables in the environment block, simply set them in the first stage. And in the agent set the `reuseNode true`.
> 4 votes

# Answer
If only you would add the missing quote for SMOKE\_CMD (credits to ilhicas who first mentioned something about that in a comment below your question), you might get UUIDs that are no longer different ... Similar to what you seem to have done in revision 2 of your question already.
> 0 votes

---
Tags: jenkins-pipeline, groovy
---


# Question
Title: How to prevent group from deploy to environment but allow same to redeploy?
In our pipeline, the dev team will do multiple releases to their own test environment, when they are satisfied, they will queue a release for test. This signifies to our QA team that the devs are ready for QA to begin their testing. QA approves the queued build, kicking off the automation process of deploying the artifacts into their environment.

Next, QA does their testing and puts in work items for bug that arise. The dev team then makes code changes and repeat the cycle until QA approves a release. The release will then go to 3rd party compliance testing before being packaged up to go onto remote devices.

We would like to prevent the QA team from deploying to test when a build hasn't yet been queued by the dev team. However, we want our QA to be able to re-deploy any build that the dev team had previously queued, without requiring the dev team to queue the build.

This is all in the VSTS release environment.

# Answer
You can add a post deployment aprroval to the dev environment that the dev team has to approve. And then you can add a pre deployment approval to the QA environment that the QA team has to approve.

Then a release will be automatically triggered for the QA environment after the dev team has approved the post deployment approval on the dev environment. But it will not deploy to the QA environment until the QA team has approved the pre-deployment approval.

Additionally, the QA team can re-release any previous release to the QA environment at will by going back to the release and clicking redeploy.
> 1 votes

---
Tags: continuous-deployment, azure-devops
---


# Question
Title: Persist Changes to New Docker Image - neo4j
I am trying to create a new image based on the `neo4j:latest` image. What is the proper way to persist to a new image and run a container based on the image?

I have these goals:

* Save the new password I assigned to the database.
* Keep data stored in the database.

I am using Windows 10 Docker Community Edition Version 18.06.0-ce-win72 (19098)

Here is what I have tried:

1. Start a container from neo4j:latest.

I used this docker-compose.yml:

```
version: "3"
services:
  neo4j:
    image: neo4j:latest
    ports:
      - "7474:7474"
      - "7687:7687"
networks:
  webnet:

```
2. Connect to neo4j using a browser. I set a new password, as is required.

`http://localhost:7474/browser/`

3. Load data into the database using a script run on my host. I verified that the database now contains nodes and relationships.
4. Commit the container to an image.

`docker commit --author="Jacob Quisenberry" c5c3f2998895 jquisenberry/graphdb:latest`

5. Push the image to my repository.

`docker push jquisenberry/graphdb:latest`

6. Run the image

`docker run --expose=7474 --expose=7687 -p 7474:7474 -p 7687:7687 jquisenberry/graphdb:latest`

7. Connect using a browser.

I expected not to have to enter a new password and for my data to be intact.Instead, I am required to enter a new password. My database now lacks nodes and relationships.

# Answer
Regarding to Neo4j with Docker and Insallation: Docker

> /data to allow the database to be persisted outside its container.
>
> /logs to allow access to Neo4j log files.

I understand that we should mount the `/data` to the host machine so that the new password is persisted as the following example.

```
docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

```

Note, the `/path/to/neo4j` is a physical directory on the host machine.

### Edit1

To set password, the Insallation: Docker mentions as

> By default Neo4j requires authentication and requires you to login with `neo4j/neo4j` at the first connection and set a new password.
>
> You can set the password for the Docker container directly by specifying `--env NEO4J_AUTH=neo4j/<password>` in your run directive. Alternatively, you can disable authentication by specifying `--env NEO4J_AUTH=none` instead.

Then the docker run should be

```
docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --env NEO4J_AUTH=neo4j/<password> \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

```

Anyhow for the `/data`, I only achieve with mounting it to the host machine.
> 1 votes

---
Tags: docker, docker-compose
---


title: Docker ADD always invalidates cache

text:
# Question
Title: Docker ADD always invalidates cache
I've got problem with my Dockerfile. After building an image it is pushed to repository as `cache-image`, so on next builds I can pull it and use with `--cache-from`.

Problem is that build stops using cache after first `ADD` command.

```
Step 20/30 : WORKDIR $PROJECT_PATH
---> Using cache
---> d79e2ba28d96
Step 21/30 : ADD requirements $PROJECT_PATH/requirements
---> 6261bb0bd3e6
Removing intermediate container 26707f4107c9

```

I ran both images (cache and built one) and checked content of requirements directory, they're both identical. Whole build is done on Bamboo worker.

# Answer
Check the image history of your two images with `docker image history`. Pay careful attention to the sha256 checksum of the files being added on the `ADD` line. Docker performs a checksum on the files, including some of the metadata on these files, so a single byte change or a permission update will result in a cache miss.

Once you've checked this, if it is not your issue, please update the question with the pull command, the build command and full output, and the history command on both images.

Side note: you likely want to use `COPY` instead of `ADD` unless you are pulling remote files or extracting a tar file into a directory.
> 4 votes

---
Tags: docker, dockerfile, cache
---


# Question
Title: Different download speeds from AWS S3 CLI vs Console/Websites
I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.

Is there a known reason for this?

Is there something that can be done about it?

# Answer
**TL;DR** Try reducing the number of concurrent connections used by awscli to 1 using this command: `aws configure set default.s3.max_concurrent_requests 1`

---

You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the `aws s3` commands (not `aws s3api`).

In the cli user guide under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.

I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of `lsof` I could see my browser was only opening a single connection for a ~500MB file download, whereas using `aws cp` 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.

Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times: `s3.max_concurrent_requests=10 8m22.965s
s3.max_concurrent_requests=1 6m47.772s`

So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.

You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html.
> 1 votes

---
Tags: amazon-web-services, amazon-s3, aws-cli
---


# Question
Title: Does anyone have resources, tools, ideas, for automating database create, update, modify, delete
I've been tasked to find a way to automate promotion of database creation, changes and other modification related to application changes in a devops world.
My databases include MS SQL, Oracle, MySQL and could include NoSQL databases. I've been doing what I can with web searches and all I seem to be able to find are powershell maintenance scripts (backups, rebuild indexes, etc). Nothing really about taking DDL out of a VCS like subversion, using jenkins or other tool to then apply those changes to either an existing database or create a new database. Our databases are a mixture of on-prem and AWS, with future Azure possible.
I did find a TOAD tool that will be available later this year that seems to do this, however I'm trying to see if there is a less expensive way.
I've been developing applications, databases and supporting the same for 40 years. I can figure it out - however I'd like to not have to recreate the wheel if one exists. Or if there is a best practices, or if there is someone else that has tools they use.
Thanks in advance for any assistance.

The suggested question from over a year ago has some ideas, however it is not close to what I'm looking for. All the answers are over a year old, and as we know software changes at almost a weekly rate. The answers in that question help if I have to "roll my own" solution.

# Answer
> 1 votes
Database *migrations* are a pretty standard thing for a long time, I believe - not related that much to DevOps, though obviously needed.

There are ready-made tools which are agnostic of your programming environment, i.e., not directly tied to whatever ORM the devs are using.

Strategies of how to actually decide which DDLs/DMLs to feed to the DB might vary; i.e., in the simplest case you just keep track of one global "DB version"; or you have a table which lists individual migrations that have been applied already; or you define a "wanted" state of the DB, and the tool then creates the need DDLs itself to change whatever state the DB is in right now to the new one.

One example of such a standalone tool would be Liquibase (open source), which has a plethora of ways to be configured. I mention that because you say you don't have much time, and it is rather quick to get going - it has one mode where the "configuration" is just a simple file of SQL intermixed with flat (and small) meta information as SQL comments.

The biggest challenge, in my experience is how to handle *changing* of data structures; one way would be to make all migrations such that they never lose any data, and never break existing code (or data). I.e., you would almost never include a `DROP TABLE` or `ALTER TABLE DROP COLUMN` in just any migration, but would let stuff drop out of use, and when a table, or column (or whatever other DB object) finally is not in use anymore, and is inconceivable to be used again, *then* you drop it. To change, for example, the type of a column, you would create a new one with the correct type, and phase the data over; and so on. This places the work straight on the shoulders of the devs, and not on the migration tool. Which is fine, as this stuff must be in tight correspondence with the DB-centric code of your application, anyways (even if hidden behind an ORM layer).

This is pretty much the same approach as for the software development itself; i.e., creating new features but with a switch (deploying the software with the switch "off", and then being able to toggle quickly), and rolling out parallel implementations of the same feature; only removing the old code when the new one is absolutely proven.

---
Tags: jenkins, deployment, databases, subversion
---


# Question
Title: Executing a command in running docker swarm service
I am trying to run a Django project in docker swarm mode. The problem is I want to index the Elastic search container which needs to be run only after postgres completes the SQL dumping process (only in the first run). How can I do this? Any solutions.

# Answer
I suggest making a `cronjob` container that does the SQL dump and elastic index using `docker exec` commands toward the corresponding containers.
> 1 votes

---
Tags: docker, docker-swarm, elastic
---


# Question
Title: Ansible YAML inventory - Groups that are allways members of other groups
Situation:

I have a group "webservers" where I put all webservers in. As well as a group "mysqlservers" for the database servers respectively.

Now I do also manage some LAMP servers and I want them to be always part of the webservers and mysqlservers group, as well as manage them in a group "lamp"

This is my approach:

```
---
all:
  hosts:
    host1:
    host2:
  children:
    webservers:
      hosts:
        web1:
        web2:
        web3
      children:
        lamp:
    mysqlservers:
      hosts:
        db1:
        db2:
        db3:
      children:
        lamp:
    lamp:
      hosts:
        lamp1:
        lamp2:
        lamp3:
...

```

Is that the correct approach? If I followed the ansible docs examples, I would have defined the "lamp" hosts as children of webservers and mysqlservers, but I really want to reduce redundancy. So is it possible to create the group as a child of "all" and just reference it as children of the specified groups without repeating the hosts?

# Answer
> 4 votes
According to this best practices post, your file will look like this

file: `test`

```
[webservers]
web1
web2
web3

[mysqlservers]
db1
db2
db3

[lamp]
lamp1
lamp2
lamp3

[misc]
host1
host2

[all:children]
misc
webservers
mysqlservers
lamp

[webservers:children]
lamp

[mysqlservers:children]
lamp

```

You can analyse your file by using `ansible-inventory`:

```
ansible-inventory --inventory-file=./test --list

```

**output:**

```
{
    "_meta": {
        "hostvars": {
            "db1": {},
            "db2": {},
            "db3": {},
            "host1": {},
            "host2": {},
            "lamp1": {},
            "lamp2": {},
            "lamp3": {},
            "web1": {},
            "web2": {},
            "web3": {}
        }
    },
    "all": {
        "children": [
            "lamp",
            "misc",
            "mysqlservers",
            "ungrouped",
            "webservers"
        ]
    },
    "lamp": {
        "hosts": [
            "lamp1",
            "lamp2",
            "lamp3"
        ]
    },
    "misc": {
        "hosts": [
            "host1",
            "host2"
        ]
    },
    "mysqlservers": {
        "children": [
            "lamp"
        ],
        "hosts": [
            "db1",
            "db2",
            "db3"
        ]
    },
    "ungrouped": {},
    "webservers": {
        "children": [
            "lamp"
        ],
        "hosts": [
            "web1",
            "web2",
            "web3"
        ]
    }
}

```

---
Tags: ansible, ansible-inventory
---


# Question
Title: Docker & Django - what directory does localhost point to?
I'm completely new to Docker and Django and I'm having trouble getting my Docker app to serve inside Django. Currently, I'm running my docker container with this command:

```
docker run -it --name web -p 8000:8000 --link mysql:db -v ~/directory:/app django

```

where `mysql` and `django` refer to containers defined earlier.

Next, I'm using `manage.py` to make migrations and migrate, then serving with the following command:

```
mod_wsgi-express start-server --reload-on-changes --working-directory /app/voting_project /app/voting_project/voting_project/wsgi.py &

```

When I visit `localhost:8000` in my browser, it successfully renders the HTML template, but the CSS and images which I see locally are missing.

The following errors are logged in the console:

I've played with the structure of my directories a bit, but I can't seem to get the browser to find anything in `localhost:8000/static` or `localhost:8000/images`.

I'm assuming 8000 is mapping to somewhere weird in my local directories, but I realize this could also be caused by a number of other things. Please let me know if I'm making any obvious mistakes.

Thanks so much!

# Answer
This is most likely a problem with the django configuration better fitting to stackoverflow, and not so much a devops problem.

Check the `settings.py` for the following entries:

```
INSTALLED_APPS = [
    (...)
    'django.contrib.staticfiles',
]

```

At the end of this file

```
STATIC_ROOT='staticfiles'
STATIC_URL = '/static/'
STATICFILES_DIRS = [
     os.path.join(BASE_DIR, 'static'),
]

```

source **Django manual:** How to static-files
> 1 votes

---
Tags: docker
---


# Question
Title: Windows Container (Nano Server) with ASP.NET Core
I'm trying to make an Windows Nano Server container that runs an ASP.NET Core app.
I'm building it on Windows Server 2016. I can make it work, but there is one odd problem.

The only way I can make it work is to build the image, run the container, *then* start an interactive Powershell session and use `Invoke-WebRequest http://localhost:5000`. Once I do that, the app is visible from other servers (I can't browse the container locally from the Win2016 Server due to a known NAT bug.)

Here's my Dockerfile:

```
FROM microsoft/dotnet:sdk-nanoserver
ARG source
WORKDIR /app
COPY /ContainerPOC/ .
RUN dotnet restore --runtime win10-x64 .

RUN dotnet build --framework netcoreapp1.1 --runtime win10-x64 .

EXPOSE 5000
ENV ASPNETCORE_URLS http://0.0.0.0:5000
CMD dotnet run --framework netcoreapp1.1

```

Any idea why this won't "just work?"

# Answer
> 1 votes
As it is unclear what command and subcommands you used to run the docker container it could be possible that you run the container without using the `-p` subcommand. In this case `docker run -p 5000:5000 <image_name>` could solve the issue.

# Answer
> -1 votes
You need run entrypoint like as:

```
FROM microsoft/aspnetcore:1.1.1
...

ENTRYPOINT ["dotnet", "netcoreApp.dll"]

```

---
Tags: docker, windows, containers
---


title: How to monitor ELK Stack Load?

text:
# Question
Title: How to monitor ELK Stack Load?
We have tried Prometheus and Grafana. Currently it is not set up on our cluster and we are tight with resources.

* What are the best ways to monitor load on an ELK stack?
* Is there anything built in?

# Answer
> 1 votes
Load can mean a lot with an ELK stack.

For **ingestion** load, the **logstash parsing** side, you're looking at a combination of one or two things:

* CPU load on the ingestion boxes.
* If used, back-pressure in your ingestion-queue (such as Redis, AMQP, or others)
    + This can be simple stuff like queue-size, or a fancy number like age of the oldest member of the queue.

If you're in a cloud-provider that gives you a way to auto-scale your ingestion nodes, keying on CPU functions pretty well in recent Logstash versions.

For **ingestion and searching**, the **ElasticSearch** side, it can depend on a couple of things due to your architecture. If you're using physical servers, it is very easy to have far more CPU capacity than I/O capacity to store the processed results. If you're using physical servers, keep an eye on **I/O queue depth** in addition to CPU loading. Also beware of the Compressed Object Pointer barrier, around 32GB of JVM HEAP. Stay under 32GB. Just do it.

If you're on cloudy instances, its harder to out-strip your I/O due to the reduced core-counts. Even so, queue-depth and cpu-loading are good things to keep an eye on. If your JVM monitoring tools are up to it, tracking Garbage Collection frequency is worth it until you have a feel for what you need for RAM.

For all of this, keeping an eye on consumed disk-space will tell you when you need to add more ES data-nodes or adjust your index retention settings.

For **display and searching**, such as with Kibana or Grafana, I've found this to be the part of the stack I never bother looking at. At least in any way other than up/down tracking. We track CPU, but we don't alarm anything since in our experience even a 2 core box with 7GB of RAM is enough to keep up without breaking a sweat. Both of these work OK with horizontal scaling if you do run into problems.

---

ELK stacks come in many sizes, from all in one boxes, all the way up to systems with routing logic to bring events to parsing and ES clusters owned by completely different parts of the organization. The monitoring needs of a small architecture (all in one) are rather different than medium architectures (discrete parsing tier, usually queue-mediated), and are different again from large architectures (multiple parsing-tiers with multiple event-stores).

# Answer
> 0 votes
I'm using Datadog which has a really great EKS plugin that works really well with the other AWS integrations.

Downside is cost, which is $15 per node on the basic plan.

---
Tags: monitoring, elastic
---


title: AWS: Fargate charging model

text:
# Question
Title: AWS: Fargate charging model
According to Amazon's official documentation,

> For Amazon ECS, AWS Fargate pricing is calculated based on the vCPU and memory resources used from the time you start to download your container image (docker pull) until the Amazon ECS Task\* terminates, rounded up to the nearest second. A minimum charge of 1 minute applies.

Does this mean that

a) It **knows** when my containers consume cpu/ram and applies charges **only** for this time frames?

b) It aplies charges based on pre-allocations I do when creating the tasks?

# Answer
## Orchestration Timers

Regardless of the internal implementation, there is most likely an orchestrator that starts and stops a timer based on particular signals or events within the Fargate service. Basically, the charges are from when the pull command are executed until an orchestration process issues or receives the termination command/signal for your process. Conceptually, think of it as similar to wrapping a shell script in `time { :; }`.

Again conceptually, you might think of the way it charges as the following pseudocode:

```
cpu_charges = $num_cpus * cpu_rate_per_second
mem_charges = $mem_size * mem_rate_per_second
charge_rate = cpu_charges + mem_charges

elapsed_time = time () {
    docker pull foo_image
    docker run --cpus $num_cpus -m $mem_size -it --rm foo_image
}

bill_for elapsed_time * charge_rate

```

I am not currently aware of any public documentation that exposes the *real* mechanics of how this is internally implemented by AWS, but this is likely accurate enough for most practical purposes.
> 1 votes

---
Tags: amazon-web-services, amazon-ecs
---


# Question
Title: How to not "fire and forget" when doing a remote trigger of a Jenkins build?
I am attempting to trigger a parameterized Jenkins build from Bamboo, doing the below:

> POST - `http://jenkins-url.com/job/jobname/buildWithParameters?ENVIRONMENT=dev&APPLICATION=hello-world`

But I will get a 201 instantly which tells me that the build was created. How do I make this request wait and return the success status of the build instead of the fire and forget?

It is apparently possible as per Parameterized-Remote-Trigger-Plugin:

Edit: created this in the end if you need it. https://github.com/owenmorgan/jenkins-remote-builder

# Answer
> 6 votes
I've created the jenkins-remote-builder script which will follow your remote build to completion.

Some more details about it (from its README.md):

> ## Example
>
> ```
> jenkins=https://user:pass@jenkins.mydomain.com:8080
> jenkins_job=MyApp-Deploy
>
> environment=dev
> application=myapp
> revision=9fd71f63b351b8208264daf86d292ced580a2f60
>
> ./jenkins_remote_trigger.sh \
>             -h ${jenkins} \
>             -j ${jenkins_job} \
>             -p "ENVIRONMENT=${environment}&APPLICATION=${application}&REVISION=${revision}"
>
> ```
>
> ## Usage:
>
> ```
> -h HOST     | --host=HOST                       Jenkins host
> -j JOBNAME  | --jobname=test-build-job          The name of the jenkins job to trigger
> -p JOBPARAM | --jobparam=environment=uat&test=1 Jenkins job paramiters
> -q          | --quiet                           Don't output any status messages
>
> ```

# Answer
> 5 votes
When setting up the remote trigger, you have to click "Block until the remote triggered projects finish their builds."

# Answer
> 1 votes
If you need to trigger a job from CLI and wait for its completion you can use "Jenkins CLI" (see here).

However jenkins CLI does not support promotions so for them I came up with the following script:

```
#!/bin/bash
# Trigger a promotion and wait for its completion
#
# For triggering jobs jenkins cli is sufficient: https://support.cloudbees.com/hc/en-us/articles/228392127-How-to-wait-for-build-to-finish-when-triggering-from-CLI-
#
# The script is dependent on the current jenkins implementation of:
# - the promotion web page (links for triggering/re-executing the promotions)
# - the behaviour of the XML of the promotion status
#
# The behaviour of the job run status XML is:
#   - if the the promotion is not yet been triggered than the response is 404 not found
#   - is the the promotion has been triggered
#     - ... but it's still waiting for an executor:  the response is 404 not found or <duration> is empty
#     - ... and has started:                         <duration> is empty or 0
#     - ... and it's finished:                       <duration> is a non-zero number
#
#
#  run syntax:
#    ./trigger_promotion_and_wait_for_completion.sh \
#       <job_name> \
#       <job_run_number_to_promote> \
#       <promotion_name> \
#       <jenkins_user> \
#       <jenkins_pwd> \
#       <jenkins_url> \
#       <script_workspace_folder> \
#       '{"name": "prom_param_1", "value": "stringvalue" } , {"name": "prom_param_2", "value": true }'
#
#   example:
#    ./trigger_promotion_and_wait_for_completion.sh \
#       job1 \
#       22 \
#       promotion1 \
#       admin \
#       password \
#       http://localhost:8080/jenkins/ \
#       . \
#       '{"name": "prom_param_1", "value": "stringvalue" } , {"name": "prom_param_2", "value": true }'

set -uexo pipefail

#other debug options:
#PS4='+\t '
#set -v

JOB_NAME="${1}"
JOB_RUN_NUMBER="${2}"
DEPLOY_PROMOTION_NAME="${3}"
BUILDER_USER="${4}"
BUILDER_PASSWORD="${5}"
JENKINS_URL="${6}"
WORKSPACE_FOLDER="${7}"
PROMOTION_ARGUMENTS="${8}"

TIMEOUT=900

echo "retrieving the promotion nextBuildNumber (so we can poll the promotion status and check if it's finished)..."
PROMOTION_RUN_NUMBER="$( curl -s  -u${BUILDER_USER}:${BUILDER_PASSWORD} "${JENKINS_URL}job/${JOB_NAME}/promotion/process/${DEPLOY_PROMOTION_NAME}/api/xml"  | grep -P  '<nextBuildNumber>(.*)</nextBuildNumber'   | sed -re 's/.*<nextBuildNumber>(.*)<\/nextBuildNumber.*/\1/g' )"

echo "running the promotion..."
echo 'only the first promotion can be triggered with "Approve", while subsequent promotions are triggered with "Re-execute promotion"'
echo 'so we check in the web page if the approve link is present'

#the link to search in the web page
PROMOTION_APPROVE_STRING="promotionProcess/${DEPLOY_PROMOTION_NAME}/promotionCondition/hudson.plugins.promoted_builds.conditions.ManualCondition/approve"

PROM_STATUS_URL="${JENKINS_URL}job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/"

if curl -s -vvv -u${BUILDER_USER}:${BUILDER_PASSWORD}  "${PROM_STATUS_URL}" | grep "${PROMOTION_APPROVE_STRING}" ; then
    echo "The job has not yet been promoted, triggering it with 'Approve'"
    WEB_PATH="job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/promotionProcess/${DEPLOY_PROMOTION_NAME}/promotionCondition/hudson.plugins.promoted_builds.conditions.ManualCondition/approve"
    SUBMIT="Approve"
else
    echo "The job has already been promoted, triggering it with 'Re-execute promotion'"
    WEB_PATH="job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/${DEPLOY_PROMOTION_NAME}/build"
    SUBMIT="Re-execute+promotion"
fi

#note for the troubleshooting: in case the following curl fails then the error cause can be found near the string "stack trace"
CURL_OUTPUT="$(  curl -s -vvv -XPOST -u${BUILDER_USER}:${BUILDER_PASSWORD} "${JENKINS_URL}${WEB_PATH}"  \
                       --data 'json={
                                 "parameter": [
                                                '"${PROMOTION_ARGUMENTS}"'
                                              ]
                                    }&Submit='"${SUBMIT}" 2>&1 )"

if ( echo "${CURL_OUTPUT}" |  grep -P "< HTTP/1.1 5\d\d" ) || ( echo "${CURL_OUTPUT}" |  grep -P "< HTTP/1.1 4\d\d" ) ; then
  echo  'error in triggering the job/promotion! exiting...'
  exit 1
else
  echo  'curl good'
fi

echo "checking promotion status until promotion is finished"
FINISHED=no

INITIAL_TIME="$(date +%s)"
while [ "${FINISHED}" != "ok" ]
do
    sleep 2

    #checking if promotion is finished (we check the value of <duration> XML element in the job run status)
    ERROR="" ; DURATION="$(curl -s  -u${BUILDER_USER}:${BUILDER_PASSWORD} "${JENKINS_URL}job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/${DEPLOY_PROMOTION_NAME}/promotionBuild/${PROMOTION_RUN_NUMBER}/api/xml"    | grep -Po  '<duration>.*</duration>'   | sed -re  's/<duration>(.*)<\/duration>/\1/g' )"   || ERROR="yes"
    if [[ $ERROR == "yes" ]] ; then
      echo " the promotion has been queued but not yet started, waiting for it to start..."
      curl -s  -u${BUILDER_USER}:${BUILDER_PASSWORD} "${JENKINS_URL}job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/${DEPLOY_PROMOTION_NAME}/promotionBuild/${PROMOTION_RUN_NUMBER}/api/xml"
      ERROR=""
      continue
    fi  ;   ERROR=""

    #we interrupt the polling of the job/promotion status if the promotion
    #  - is terminated
    #  - is taking too long (there is some problem)
    #(in the XML of the job run status the <duration> XML element value is initially empty, than it is 0, and eventually is the number of seconds of the run duration )

    POLLING_TIME="$(date +%s)"
    let "ELAPSED_TIME=POLLING_TIME-INITIAL_TIME"
    echo "ELAPSED_TIME=${ELAPSED_TIME}"

    if  (( ${ELAPSED_TIME} \> $TIMEOUT ))  ; then
      echo "error: the promotion has taken too long... exiting"
      exit 1
    fi

    if  [[ "${DURATION}" != "" ]] ; then
      re='^[0-9]+$'
      if [[ $DURATION =~ $re ]] ; then
        if (( "${DURATION}" \> "0" )) ; then
          FINISHED=ok
        else
          : #do nothing (the value of <duration> is 0 , that is the job/promotion has been started in a slave and is still running)
        fi
      else
        echo "error: the promotion duration is not a number. exiting..."
        exit 1
      fi
    else
      :  # the job/promotion has not yet started
    fi

    echo "waiting for the promotion to finish..."
done

echo "Promotion finished"

echo "Promotion output:"
curl -s  -u${BUILDER_USER}:${BUILDER_PASSWORD} "${JENKINS_URL}job/${JOB_NAME}/${JOB_RUN_NUMBER}/promotion/${DEPLOY_PROMOTION_NAME}/promotionBuild/${PROMOTION_RUN_NUMBER}/consoleText" > ${WORKSPACE_FOLDER}/promotionOutput

cat ${WORKSPACE_FOLDER}/promotionOutput

if [[ ! "$(tail -n1 ${WORKSPACE_FOLDER}/promotionOutput)" =~ "SUCCESS"  ]] ; then
      echo "Promotion did not successfully terminate"
      exit 1
else
      echo "Promotion successfully terminated"
fi

```

---
Tags: jenkins, jenkins-plugins, bamboo
---


title: Problem with Anaconda installation

text:
# Question
Title: Problem with Anaconda installation
This is what comes up any time I start up Anaconda Prompt -

```
\Xtralien was unexpected at this time.
C:\Users\User Name>            @SET "PATH=C:\Program Files (x86)\QuantumWise\VNL-ATK-2016.3\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Intel\IntelSGXPSW\bin\x64\Release\;C:\Program Files\Intel\IntelSGXPSW\bin\win32\Release\;C:\Program Files (x86)\MiKTeX 2.9\miktex\bin\;C:\Program Files (x86)\LyX 2.1\Perl\bin;C:\Program Files\MATLAB\R2017a\runtime\win64;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files (x86)\Calibre2\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files (x86)\QuantumWise\VNL-ATK-2016.3\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Intel\IntelSGXPSW\bin\x64\Release\;C:\Program Files\Intel\IntelSGXPSW\bin\win32\Release\;C:\Program Files (x86)\MiKTeX 2.9\miktex\bin\;C:\Program Files (x86)\LyX 2.1\Perl\bin;C:\Program Files\MATLAB\R2017a\runtime\win64;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files (x86)\Calibre2\;C:\Program Files (x86)\Skype\Phone\;"C:\Program Files (x86)\Xtralien Python Distribution\Scripts";"C:\Program Files (x86)\Xtralien Python Distribution";c:\FullProf_Suite"
C:\Users\User Name>

```

I am guessing it's because of a conflict with an old Xtralien python installation that I had and forgot about. I uninstalled that and reinstalled Anaconda but it hasn't resolved the problem.

# Answer
> 2 votes
@PrestonM indicated that:

> Whenever I see an "unexpected at this time" error it's in my batch scripts, and it's either because I didn't use the correct escape character, or an absolute path is incorrect. I don't see Anaconda anywhere in your PATH, but I do see Xtralien. Is it possible that you need to add Anaconda?

OP feedback was:

> Not changing the default directory and adding it my path variable during the installation seems to have done the trick. Thanks a bunch for the advice!

# Answer
> 2 votes
We recommend not adding Anaconda to the PATH environment variable, since this can interfere with other software read more ...

Anaconda 4.4 added a feature to the Windows installer so that the “verbose” install information is printed out to a special debug streame read more ...

---
Tags: windows, python, installation
---


# Question
Title: CI/CD free for hobby side projects. Jenkins on Heroku as alternative, with ephemeral dyno
I have four side projects which are my hobby (NOT A JOB! NOT A WORK!).

I work on that at night, when I have time and when I want to.

All projects are javascript backend and frontend. Just experiments.

What I need is a Jenkins instance (or something like that for CI (continuous integration) and maybe also CD (continuous delivery)) but **free**. Maybe like free heroku dyno which is amazing: I don't work everyday on it, I work sometimes, when I have time and, like my free dyno on Heroku it's amazing the auto-sleep and go on just when I need it.

Is there something for Jenkins out there but free, with auto-sleep?

**Jenkins on Heroku** is still an option given the ephemeral state of dyno on Heroku? Is Jenkins still saving config on filesystem?

I can't afford to pay something for that because it's just an experiment.

Any idea?

# Answer
> 1 votes
You should try CI / CD like:

Openshift, as @JamesKnott mention, but this one is a full PaaS, so it is meant to host your app too.

Hope it helps

---
Tags: jenkins, continuous-integration, continuous-deployment, continuous-delivery, travis-ci
---