# Question
Title: How can I persuade developers on my team to embrace "You build it, you run it"?
How can I persuade developers on my team to embrace "You build it, you run it"? By that, I have this quote from Werner Vogels in mind:

> Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.

I'm specifically thinking of a set of developers that:

* Were hired into a developer role, with little/no mention of ops-related tasks.
* Traditionally have "thrown code over the wall" to an ops team.
* Traditionally have a 9-5 work schedule, and are actively hostile to the idea of "pager duty", participating in disaster recovery, writing post-mortems, etc, *especially* outside of normal business hours. (Note: I only have very infrequent outages in mind for this; I am not proposing that we add after-hours customer support to this team's workload.)
* Are not currently responsible for writing/supporting monitoring or alerting on their applications.

Suppose there is a team that is rapidly developing new cloud micro-services with a profile that is getting to be such that handing these services off to an ops team is sub-optimal because they can't keep up in regards to gaining deep knowledge of the services that is required to effectively manage and monitor them. "You build it, you run it" would work better for this team because tasks could delegated to each responsible team member. So this team would begin taking part in designing infrastructure, monitoring/alerting tools for the services, and (very infrequently) responding to outage events.

I am specifically interested in methodologies, backed up by real world examples. How this been successfully implemented in other workplaces, and if there any canonical steps to follow while implementing this? Any links to write-ups that can support answers would be very helpful.

# Answer
I think the easiest way is to change their performance goals so they are based off reliability as well as delivering code. Sell it as the company cannot succeed without both so why should the developers only be measured on one? The best way for them then to meet their performance goals is to be involved in operations.

Ultimately you need to convince them this is the best way for the company to succeed and therefore for them to succeed. That's hard and you can't expect them to be onboard from the start. They need to be sold on the value too.
> 22 votes

# Answer
When it comes to affecting business culture, the best way is probably via the well-known "boil the frog" method. You have to introduce these tasks to developers slowly, because I know I myself (as a dev) would balk at having all this new responsibility at once.

First, start out by introducing one or two new tasks only to be performed during normal business hours. They need to learn how to do devops which could be quite the learning process for a (to this point) code-only dev and might require some supervision. They will also likely be hostile to the idea of changing their work-life balance since you mention they're used to 9-5. At this point, record data on the new processes for use later (have them write this code, data is always useful).

Later as you're running out of new devops-y tasks to introduce (so the first, second, and fourth bullet points are almost complete), bring the first tasks you've introduced up as candidates to be performed outside of standard work hours. You may see some backlash at this and you might even see some attrition depending on how strongly you push this and how heavily the work-ends-at-five culture is ingrained. To defend against this, hopefully your data supports the idea that work beyond standard hours will be rare, only occur in extreme situations, and will greatly benefit both the business and the customer. If your data doesn't support this, then you'd better be ready to deal with the consequences of this choice.

Even with data, it may still be easier to have the developers write the monitoring/alerting code (so they become *dev*ops but still mainly dev) and keep the alternate ops team as front-line support (as a few others have suggested). Like I said, small changes are important to avoid backlash. Integrating work for devs beyond standard hours will be challenging, as they may know they can look elsewhere for employment if they don't like it since the market for devs is strong right now, especially if they already had devops skills. Caveat emptor!
> 12 votes

# Answer
IMHO `You build it, you run it` should not be taken literally. To begin - it almost sounds like a punishment ;)

No single person or even small developer team can *reasonably* support a tool or a toolset used in the software development process (i.e. in production!) for long periods of time. Been there, done that :)

Support duties tend to expand as the tool(set) customer base grows. If assuming those duties exclusively, the development team could end up doing mostly support, with little/no time left for development down the road. Effectively a dead-end - how many developers would want to stick around in such environment?

Having a professional 1st line of support team is crucial to prevent frustration, stress and other effects of long-term exposure to support duties over your developer team members.

The 1st line support team would, of course, fallback to the developer team (again, team, not single-person!) for issues they cannot cover directly. Yes, it may be difficult at first, but thing will get better. It should be a collaboration - that's part of what DevOps is about.
> 10 votes

# Answer
Looking outside of DevOps specifically, if you are talking about large (ish) enterprise environments then the SAFe methodology has a fairly good way of encouraging this kind of behavior.

Essentially it boils down to a few key stages:

* projects start as release trains. The intent (and the expectation of whoever is funding it) is that it will be long running. I'm talking years long, none of this "stand up a team for 3 months then stand them down" business.
* over the course of a project the release train will inevitably amass more people as more of both new project requirements based on newly realised business opportunities as well as the ongoing tax of maintenance style work.
* I almost always see the trains run their first increment as 100% project/change teams, 2nd increment they allocate a percentage of time to Run work. 3rd increment management realise they are about to have a problem and starting adding teams to try and handle the Run overflow to give their now seasoned developers and engineers time to keep working on new requirements.
* if the right balance is struck of project teams being able to keep delivering project outcomes without being bogged down to much in maintenance work and new teams who join the train aren't immediately expected to just be 100% support teams, but instead take on a fair portion of the change work that was going to be handled then you end up with delivery teams who are owning the products they are building by default, it doesn't need to come at them all at once that now they are a Run team, instead it just slowly integrates into their day to day activities.

Where this model obviously has some weaknesses is in the pricing for a business. Generally I'd expect to be paying a lot more for a change resource than a run resource, usually run contracts with major vendors are fixed price and the intent is that they make their money on continuous improvement and therefore cost savings.

That being said, it isn't all that difficult to consider the change teams stood up as part of a release train to simply be delivering continuous improvements. If there is something they can build or do which will improve their ability to focus on new project deliveries and be less concerned with "business as usual" work then that should be backlogged and prioritised based on the perceived benefits by whoever holds the money for the release train.
> 10 votes

# Answer
This ultimately depends on the size and structure of the company. There are really three stages that your company can be in:

1. The startup stage (under 150 engineers). Of course, the developers need to run their own software. Everyone does so in a startup. You might not even have operations team to begin with, but even if you do, it is small and the speed of progress is so fast that there is no way to pass the knowledge required to run it successfully on another team quickly enough for them to be successful.
2. The medium sized business (over 150 engineers, but a single operations team). At this stage the churn in the company starts to be too high, the engineers who build software do not necessarily stick around also to run it. You don't know everyone anymore and it is hard to communicate effectively for everyone to be in operations. It would start to turn into chaos. At this stage you want to turn to the Google model, where every team has to operationalize their software, but not necessarily operate it. They will operate it at the start, but a big part of building software is to reduce the cost of operating it to a point, where the load is small enough that the operations team can sign on running it. Only then it is considered done.
3. Large enterprise with multiple business units, (where each has its own operations team): At this stage you can turn back to the Amazon model, where every team essential develops and operates their own services. Each of the business units has to be small enough for everyone to know each others, so under about 150 engineers and you operate each essentially as a startup. Amazon has each AWS service operating more or less separately and it works for them.

You have to figure out which stage your company is in and/or moving into and act accordingly. There is no "one size fits all" solution.
> 7 votes

# Answer
My take on this (if I was faced with such commandement, or whatever you'd call it), would be something like "***`What else would you expect?!?!`***". Because, accidently:

* I wouldn't even be able to live without it, and
* I love to eat my own (dog)food.

Let me explain a bit further ...

My business / hobby / passion is , more specifically in environments. And wherever I go (to finetune stuff to fit the customers needs), the very first requirement the ***I impose*** (in my contract), is that any tuning done to the system we implement, is via that very same system. And by doing so (true, that takes like a few hours, say half a day at max), we get these benefits from it (incomplete list):

* I hardly document anything I did to get the system going. If any questions are asked, I just query the system (and teach the customer how to query the system without my help).
* If I get called in in X months/years (to upgrade to a new release?), I want to know (remember) what "I" have been doing before (and the only thing I trust is what the system tells me, aka reminds me about).
* I only need to ask the customer once about how to do specific things in their site (naming conventions are hard to remember), or to convince them to grant required permissions to "the system" (not to me ...).
* You're ***`continuously`*** QA-testing your own system ... in production. Chances are that you'll experience bugs, logical errors or missing features (and what not) yourself. And those are extremely motivating to get them addressed ASAP ... eg to become more productive.
* ... and if you'd want I can add another dozen of reasons ...

However, before you try this at home yourself, be aware of some pitfalls to avoid:

* you want *everybody* to join the party (use the system), because only 1 "exception" (aka the manager/owner of the company?) may ruin the party (you'd think you can trust your system, but then somebody did something without asking/using the system). Those case may cost you days to discover ...
* new users might complain that by using this (new) system, it takes them way longer to get their job done (as compared to whatever they used before). And wherever it makes sense, they'll use that as an excuse why they are late to complete their job. At that point, you better have upper management covering you, because otherwise your project/system may get the blame.
* make sure you do know your own system, and how to configure it to fit your needs. As a sample (in my case): you want to configure the system so that you use the operational environment to deliver to your experimental environment, and not the other way around ... I've seen that happening in the past ... using (abusing) the test system to deliver to highly secured environments.
> 4 votes

# Answer
What you propose is that every developer in your company should be a Jack of all trades, master of none. You should keep in mind that the Amazon model (constant startup mode) causes people to burn out from constant context switching, pager duty, and not focusing on what they do best. If you build an airplane or a formula 1 car, that does not mean that you should fly that airplane or drive that car because some people will do it better than you, while they would not be good at building.

When you run software like Jenkins, MySQL, Cassandra, Tomcat, do you expect the developers who developed that software to support your installation and every other installation in the world? Of course not, but you do expect to have tools and means to monitor performance, failures, memory consumption and other issues that might occur.

As Jiri Klouda explained in his answer Google uses that philosophy where it expects teams to operationalize their software, but not necessarily operate it. They also employ SREs while Amazon does not so developers have to pay the toll and do something they do not excel in.

> Traditionally have a 9-5 work schedule and are actively hostile to the idea of "pager duty", participating in disaster recovery, writing post-mortems, etc, especially outside of normal business hours. (Note: I only have very infrequent outages in mind for this; I am not proposing that we add after-hours customer support to this team's workload.)

If you are certain that outages will be infrequent, are you ready to pay for the extra time when an outage occurs?

Other problems frequently happen in teams where developers are not just developers, but admins, customer support, pr etc. Because they do ops and dev, they do not make good documentation and there is a lot of tribal knowledge. They do not make it easy for some other team to run their software so when they leave the company it is hard to support their service. You should persuade your developers to make their software operationalized so anyone can run it with little or no help.
> 1 votes

# Answer
Different organisations have different organisational structures, needs and cultures. Some developers think that operational issues are not their issues. However, a good development culture is one where developers care about the smooth operations of their systems, they also care about operation colleagues who need to support their system. Us developers used to carry pagers in case there were issues that operators cannot solve. Whilst there are clear team responsibilities, there is also a fuzzy line due to the complexity of developing and operating a system. If some staff do not appreciate that, then it can be difficult in some situations. I am not suggesting that development staff should be supporting ops all the time. If that happens too often, then there are issues somewhere that need fixing. Allow flexibility and try to look after each other.
> 1 votes

---
Tags: culture
---


# Question
Title: Ansible cli_command loop register issues
I can't seem to get the register to works with cli\_command and looping of multiples commands. Any suggestions?

-

```
name:  Run show commands
  hosts:
    - all
  connection: network_cli
  gather_facts: no

  vars:
     shcmd:
       - show bgp sum | match 3356
       - show config protocols bgp group Level3

  tasks:

    - name: create report directory if needed
      set_fact:
        report_dir: "{{ user_data_path }}/reports"

    - name: confirm report directory
      file:
        path: "{{ report_dir }}"
        state: directory

    - name: Run commands
      cli_command:
          command: "{{item}}"
      register: "showcmds"
      with_items: "{{shcmd}}"

    - debug:
        msg: "{{ showcmds.stdout_lines }}"
        with_items: "{{shcmd}}"

```

# Answer
> 2 votes
There are a couple of things here:

1. The indent of "with\_items" on the debug module is wrong, it should the same level as the "debug" word.
2. More importantly, in this case the "with\_items" variable on the debug is wrong. When you added "register" to the cli\_command module, you actually said "collect the output from every iteration and return the whole lot". Therefore, you need to iterate over the variable you created with "register", not the variable you passed into the command itself. Something like below (I'm not at a computer atm so this will need to be checked):

```
    - debug:
        msg: "{{ item.stdout_lines }}"
      with_items: "{{ showcmds }}"

```

edit: OK, sounds like this specific module doesn't return stdout\_lines. Try running the debug with just "{{ showcmds }}" and without the "with\_items" to get the actual structure of the variable.

---
Tags: ansible, networking
---


# Question
Title: Ansible: Does the `hosts: all` pattern in a play only match hosts specified in inventory, or does it also match playbook `add_host` hosts?
I know "all" doesn't match the implicit localhost; the title succinctly expresses my question.

Motivation: I have a playbook that needs to be run on cloud hosts generated dynamically and added to groups with `add_host` during a playbook, but I also need to run the whole playbook on a vagrant vm, as well as in CI (molecule), so I want to use "all" for the latter two cases, so that when I specify inventory, certain plays will target the inventoried hosts. These plays will also have the names of the add\_host groups, so that I don't have to duplicate plays. I'm just wondering if I can write

`hosts: all` \# instead of

`hosts: all:group1:group1...`

for these plays, and have the `add_host` hosts included, without specifying their names. I will only specify inventory for the vagrant and molecule scenarios, and not for the live cloud deployment scenario, so that I don't target my entire cloud infrastructure.

# Answer
> 2 votes
Q: *"**Does the `hosts: all` pattern only match hosts specified in inventory, or does it also match `add_host` hosts?**"*

A: Simple test shows `hosts: all` includes also added hosts by `add_host`. In the same playbook, of course. The dynamic groups live inside a playbook only.

Given the inventory

```
shell> cat hosts
host1
host2

```

The playbook

```
---
- name: Only 2 hosts in the inventory
  hosts: all
  gather_facts: false
  tasks:
    - debug:
        var: ansible_play_hosts_all
      run_once: true

- name: Add host3 to the group1
  hosts: all
  gather_facts: false
  tasks:
    - add_host:
        name: host3
        groups: group1
      run_once: true

- name: All 3 hosts are available to this play
  hosts: all
  gather_facts: false
  tasks:
    - debug:
        var: ansible_play_hosts_all
      run_once: true

```

gives

```
PLAY [Only 2 hosts in the inventory] **************************************

TASK [debug] **************************************************************
ok: [host1] =>
  ansible_play_hosts_all:
  - host1
  - host2

PLAY [Add host3 to the group3] ********************************************

TASK [add_host] ***********************************************************
changed: [host1]

PLAY [All 3 hosts are available to this play] *****************************

TASK [debug] **************************************************************
ok: [host3] =>
  ansible_play_hosts_all:
  - host3
  - host1
  - host2

```

---
Tags: ansible, ansible-inventory
---


title: Access vault secret from bash script

text:
# Question
Title: Access vault secret from bash script
I am looking for help on a direction on where to go from here. I have a Hasicorp Vault server running and accessible over DNS (local). I need to have a script get a secret from it then use that secret. I am not sure where to go from there. I know I want to use a bash script for this but past that I don't know what to do.

Until now I have been using curl to test it.

Suggestions?

# Answer
> 3 votes
The vault docs mention a `-field` parameter for the `read` subcommand. So you should be able to put this into a shell script:

```
SECRET=$(vault read -field foo secret/mysecret)

```

Other vault docs use the `vault kv get` in the same way so you might try:

```
SECRET=$(vault kv get -field foo secret/mysecret)

```

# Answer
> 3 votes
This mighty be a lengthy procedure but worth implementing, **creating child tokens to fetch information from Vault Server**.

Execute following procedure at Vault Server.

> 1. Create policy to restrict the access for clients.

```
cat auth-policy.hcl
path "secret/*" {
  capabilities = ["read"]
}

```

> 2. Write the policy

```
vault policy write client-access auth-policy.hcl
Success! Uploaded policy: client-access

```

> 3. Create token

```
vault token create -policy=client-access -period=768h

Key                Value
---                -----
token              *********************
token_accessor     *********************
token_duration     768h
token_renewable    true
token_policies     [client-access default]
-----------------------------------------------
vault policy write client-access auth-policy.hcl
Success! Uploaded policy: client-access

```

Configure `-period` in hours. Token should be renewed after configured period of time. Configure expiry period accordingly by considering `max_lease_ttl` and `default_lease_ttl` in `config.hcl`.

Configure these parameters in `config.hcl` during Vault startup.

```
max_lease_ttl = "1000h"
default_lease_ttl = "1000h"

```

Now using this child token, from your laptop/machine, you will be able to fetch the information from Vault server, using APIs.

```
$ curl -sH "X-Vault-Token: CHILD-TOKEN" -X GET VAULT-URL/v1/secret/SECRET | jq
{
  "request_id": "e53887a2-fe5a-2f27-2121-c716a697f0e8",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 25920000,
  "data": {
    **************
    **************
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

```

With this child token, **users can only read the data but neither write or delete data as configured in `auth-policy`**.

If you still want to secure child token you can save it in Jenkins credentials and will be fetched into bash script when build triggered.

# Answer
> 1 votes
I've ended up at exactly the same spot.

We have a Vault server holding a good deal of secrets for different environments and various endpoints for each environment, so we set up kv engines for every environment and added a descriptive path for every secret. For instance, we have called our engines constant-text/environment/engine-name and all Database users are somewhere in db/schema-name/username, db/schema-name/password, db/schema-name/tns-string (we use Oracle).

I will not cover SSL/TLS setup, as you said you have TLS installed.

Sources: in Vault CLI there is a section about environment variables.

This is what I did:

```
!#/bin/bash
export VAULT_ADDR=https://address.of.vault.server/
export VAULT_TOKEN=value-of-my-root-token-or-an-authorized-token
vault kv get const/environment/engine-name db/schema/tns-string | command

```

where 'command' will read the secret from stdin. It will be required to parse the vault output format (example given in subsection "Reading Data" in ) but a manual test shows more than 1000 words. As already mentionned by Tensibai above, there are ways to parse JSON, which I haven't implemented yet. I'm afraid it can't get much simpler. It can get more complex and more secure if you pass the address and the token to the script and then use

```
export VAULT_ADDR=$1
export VAULT_TOKEN=$2

```

to pass them on to the environment, but don't forget adding

```
set VAULT_ADDR=
set VAULT_TOKEN=

```

at the end to clear the variables again.

# Answer
> 0 votes
If you want pure-play bash, then you're already doing the right thing, `curl`, and although you don't mention it, it's a good idea to use `jq` when using REST APIs for parsing results, just remember to request json.

If you can install extra tools, then you can install the vault cli itself. Personally, I never install anything except docker, so I'd run that in a container:

```
$ alias vault="docker run -it --rm --cap-add IPC_LOCK vault"
$ vault --version
Vault v1.5.3 (9fcd81405feb320390b9d71e15a691c3bc1daeef)

```

As I'm sure you know, the vault cli, just like `curl`, can return json from most of its commands, which you can pipe to `jq` (again) for parsing (which is a lot easier than using go-templates IMO).

However, having said that, I have a suspicion that maybe you're asking the wrong question. Getting a secret on your laptop isn't really what vault is for and maybe you're just writing the code on your laptop ready to deploy to a server somewhere in order to provide a secret to that server. If that's the case then doing this in bash may be the wrong approach due to the issue of initial trust.

If you need to get a secret from vault then you have to authenticate first and get a token that has the correct authority (policies) to access the secret you're interested in. Authentication requires some sort of credentials (jwt/username-password/etc.). How do you get those creds to your bash script? You can't commit them to git.

Perhaps you're provisioning an ec2 instance with user-data that contains the creds the script needs (or any other cloud or on-prem equiv. of this process). If that's the case, then you can work out how to provision the machine with the secret it needs instead of the creds to get the secret. This is of course much easier with containers and kubernetes, but you can do the same thing with VMs of various types. In AWS you can do it with Lambda. In Azure with Azure Functions. On-prem you can do it using VMWare. With this approach, you'd be pushing the secrets management function into the platform (shift-left if you like that vocab.), rather than making it a responsibility of your application layer.

Maybe this is going too far and you're just playing around and learning about vault, but you did ask for help with direction. But if it's more than that, you should look at things like SPIFFE and check out the patterns hashicorp has implemented with its K8S sidecar.

# Answer
> -1 votes
Vault has a REST API available for reading.

https://www.vaultproject.io/api/index.html

In your script, just curl the API.

---
Tags: hashicorp-vault, bash, curl
---


title: Root ssh key for ansible control node

text:
# Question
Title: Root ssh key for ansible control node
**Scenario:** Deploy 30 VMs using terraform and then run ansible - how many ssh keys do you generate? Virtual machine definition in terraform requires administrator ssh key or password provided.

Do you provide only the public key of ansible control node? Or do you create ssh keys for each and every VM? If the latter is true, how do you automate that and configure ansible inventory?

# Answer
There are pros and cons for both options:

A single master key for your Ansible account would reduce complexity but you would need to ensure the private key is **very** secure, even if the key were password protected you would want to ensure the key is encrypted at rest with a tool like BlackBox and backed up somewhere.

System unique SSH keys would help with security but it's preferable to have a tool like HashiCorp Vault to help manage access to those keys. This shifts the security burden onto your secrets server but the added benefits could offset the extra work involved with setting it up and locking down access.

Another way you can go about this is to have a master Ansible ssh key but prevent root level access on the account and use something like PowerBroker to control and log privilege escalation from Ansible.
> 1 votes

---
Tags: ansible, terraform, cloud, ssh
---


# Question
Title: Autoscaling Postgres Database using Kubernetes
Is it possible to autoscale a Postgres Database using Kubernetes ? I usually run a Docker container with very specific settings to install a postgres database for my application. Does using Kubernetes to provision this db container help with scaling, load balancing if all the replicas would point to the same volume on the host server ?

# Answer
> 2 votes
Yes, you can create a Postgres cluster using k8s StatefulSets. take a look at this example.

---
Tags: kubernetes, postgresql
---


# Question
Title: Is it possible or common to use a Proxy server to refer to a database
From my experience I have used proxy servers to refer to different servers (nginx).

I have previously worked on building a mvc framework. Basically there were 8 databases with 10 accounts each. The main database had the server addresses and db credentials for each of the other databases. Whenever a user logged in, the main db will disconnect and connect to the appropriate user database.

I was wondering if there in a more common way to do that other than hard-coding this to the database class(s).

I have been thinking about proxy servers that basically refer a user to servers. So maybe it's possible to do the same but for db servers? (Just a thought..)

# Answer
The most usual strategy is a direct connection from application (java, php, c#, python, ruby, etc) to the database (mysql, oracle, sql server, postgres, mongo, etc)

In your development stage, your app and database are in your laptop. In this scenario is not common to apply security regulations.

In the next stages like testing, staging, uat, production, etc the security should be applied:

* Developers should not know the database credentials of another environments, especially **production**
* Database credentials should not be in the source code. The application should use configurations files or environment variables.
* Database must be under ssl

With these security regulations or standards, you will not need crazy and unmaintainable database switch to **protect** your final database.
> 2 votes

---
Tags: databases
---


# Question
Title: Should I shut down the staging environment when the deployment test ends?
I have a small application deployed on an EC2 instance.

I need a staging environment so I've thought of two choices:

* Deploy the same app again on the same instance, and that would be a staging environment. But I would really like to separate things and NOT put the two environments into the same instance.
* Or use another instance especially for staging, but it will be expensive. So, is it good to **start the staging instance right before deployment and shut it down when the tests end**?

Thank you.

# Answer
You can approach this case with multiple ways.

* For example you can set an auto-shutdown for 20:00 pm and auto-start for 8:00 am.
* Use **Terraform** to create your environment and, after deployment/testings, you can easily destroy it, saving your costs.
* And many other ways.

I commented on paths that you can follow, so you should choose one that brings you greater benefit and agility both in creating your environments and in testing them.

Regards
> 2 votes

# Answer
How big is the app? How much usage will it get in staging?

If you spin it up on a t2.micro, the extra cost will be negligible, especially if you use auto-shutdown or Terraform to only have the box running when you need it.

It's critical to have separate environments for staging and production.

Finally, consider dockerizing your app if you can. Much more flexibility, and you can run it in Fargate.
> 2 votes

# Answer
It's better to create a new instance for staging, deploy the app, test it, stop the instance when you finish testing.
> 1 votes

---
Tags: amazon-ec2, deployment, testing, environments
---


# Question
Title: choice parameter and extended choice parameters not working in Blue Oean interface in Jenkins
choice parameter and extended choice parameters not working in Blue Ocean interface in Jenkins.

I have a huge list in the drop down menu. I wanted to have a drop down with filterable option or make it as a extended choice for the users to select the option easily in the Jenkins. I have tried to use Choice and extended choice parameter in the input. choice parameter and extended choice parameters are not supported in Blue Ocean interface in Jenkins.

Please let me know how to proceed?

# Answer
> 0 votes
BlueOcean only supports these types:

```
export const supportedInputTypesMapping = {
    BooleanParameterDefinition: Boolean,
    ChoiceParameterDefinition: Choice,
    TextParameterDefinition: Text,
    StringParameterDefinition: String,
    PasswordParameterDefinition: Password,
};

```

https://github.com/jenkinsci/blueocean-plugin/blob/master/blueocean-core-js/src/js/parameter/index.js#L20

So either raise a Jenkins ticket or use Classic view or translate your job to use those types only.

---
Tags: jenkins
---


title: Bootstrapping Galera Cluster in Docker

text:
# Question
Title: Bootstrapping Galera Cluster in Docker
I'm trying to bootstrap a MariaDB Galera cluster in Docker containers. The following configuration works when I set `network_mode: host`, but then I can't directly access the MariaDB from inside other containers, and I'd rather use ports published by Docker instead of directly binding to the host interface. Every container is running on a separate host. This is not a Docker Swarm or Kubernetes setup, just three separate hosts with Docker.

Basically I've been converting the commands from the Bitnami Galera image documentation into an Ansible playbook.

```
tasks:
  - name: Deploy mysql bootstrap container
    docker_container:
      name: mariadb
      image: "bitnami/mariadb-galera:10.5"
      pull: yes
      restart_policy: "unless-stopped"
      # network_mode: host
      ports:
        - "0.0.0.0:3306:3306"
        - "0.0.0.0:4567:4567"
        - "0.0.0.0:4567:4567/udp"
        - "0.0.0.0:4568:4568"
        - "0.0.0.0:4444:4444"
      volumes:
        - "{{ mariadb_datadir }}:/bitnami/mariadb"
      env:
        MARIADB_DATABASE: "{{ nextcloud_db_name }}"
        MARIADB_USER: "{{ nextcloud_db_username }}"
        MARIADB_PASSWORD: "{{ nextcloud_db_password }}"
        MARIADB_ROOT_PASSWORD: "{{ mariadb_root_password }}"
        MARIADB_GALERA_CLUSTER_NAME: "{{ mariadb_galera_cluster_name }}"
        MARIADB_GALERA_MARIABACKUP_PASSWORD: "{{ mariadb_galera_mariabackup_password }}"
        MARIADB_GALERA_CLUSTER_BOOTSTRAP: "{{ mariadb_galera_cluster_bootstrap }}"
        MARIADB_REPLICATION_PASSWORD: "{{ mariadb_replication_password }}"
        MARIADB_GALERA_CLUSTER_ADDRESS: "gcomm://"
    when: inventory_hostname == groups[mariadb_galera_cluster_name][0]
  - name: Wait for Galera Cluster status
    shell:
      cmd: "docker exec -i mariadb mysql -u root -p{{ mariadb_root_password }} -s -e \"SELECT variable_value FROM information_schema.global_status WHERE variable_name='WSREP_LOCAL_STATE_COMMENT'\""
    register: galera_status
    retries: 60
    delay: 10
    until: galera_status.stdout == 'Synced'
    when: inventory_hostname == groups[mariadb_galera_cluster_name][0]
  - name: Deploy mysql cluster containers
    docker_container:
      name: mariadb
      image: "bitnami/mariadb-galera:10.5"
      pull: yes
      restart_policy: "unless-stopped"
      # network_mode: host
      ports:
        - "0.0.0.0:3306:3306"
        - "0.0.0.0:4567:4567"
        - "0.0.0.0:4567:4567/udp"
        - "0.0.0.0:4568:4568"
        - "0.0.0.0:4444:4444"
      volumes:
        - "{{ mariadb_datadir }}:/bitnami/mariadb"
      env:
        MARIADB_ROOT_PASSWORD: "{{ mariadb_root_password }}"
        MARIADB_GALERA_CLUSTER_NAME: "{{ mariadb_galera_cluster_name }}"
        MARIADB_GALERA_MARIABACKUP_PASSWORD: "{{ mariadb_galera_mariabackup_password }}"
        MARIADB_REPLICATION_PASSWORD: "{{ mariadb_replication_password }}"
        MARIADB_GALERA_CLUSTER_ADDRESS: "gcomm://{{ groups[mariadb_galera_cluster_name] | map('extract', hostvars, ['ansible_default_ipv4', 'address']) | join(':4567,') }}:4567"
    when: inventory_hostname != groups[mariadb_galera_cluster_name][0]

```

When I comment out the `network_mode` and publish the ports directly the containers start up, but the nodes can't join the cluster, I get a lot of these messages:

> 2021-07-28 9:31:51 0 \[Warning\] WSREP: Member 0.0 (f38e8cff30b6) requested state transfer from '\*any\*', but it is impossible to select State Transfer donor: Resource temporarily unavailable

which ends with:

> WSREP\_SST: \[ERROR\] Possible timeout in receiving first data from donor in gtid stage: exit codes: 124 0 (20210728 09:31:22.716)

Full log of one of the nodes:

```
mariadb 09:26:21.27
mariadb 09:26:21.28 Welcome to the Bitnami mariadb-galera container
mariadb 09:26:21.28 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
mariadb 09:26:21.29 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
mariadb 09:26:21.29
mariadb 09:26:21.29 INFO  ==> ** Starting MariaDB setup **
mariadb 09:26:21.31 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 09:26:21.34 INFO  ==> Initializing mariadb database
mariadb 09:26:21.36 INFO  ==> Updating 'my.cnf' with custom configuration
mariadb 09:26:21.37 INFO  ==> Setting wsrep_node_name option
mariadb 09:26:21.38 INFO  ==> Setting wsrep_node_address option
mariadb 09:26:21.39 INFO  ==> Setting wsrep_cluster_name option
mariadb 09:26:21.42 INFO  ==> Setting wsrep_cluster_address option
mariadb 09:26:21.43 INFO  ==> Setting wsrep_sst_auth option
mariadb 09:26:21.45 INFO  ==> ** MariaDB setup finished! **

mariadb 09:26:21.49 INFO  ==> ** Starting MariaDB **
mariadb 09:26:21.49 INFO  ==> Setting previous boot
2021-07-28  9:26:21 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.5.11-MariaDB-log) starting as process 1 ...
2021-07-28  9:26:21 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2021-07-28  9:26:21 0 [Note] WSREP: wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
2021-07-28  9:26:21 0 [Note] WSREP: wsrep_load(): Galera 4.8(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
2021-07-28  9:26:21 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-07-28  9:26:21 0 [Warning] WSREP: Could not open state file for reading: '/bitnami/mariadb/data//grastate.dat'
2021-07-28  9:26:21 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2021-07-28  9:26:21 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 0
UUID: 00000000-0000-0000-0000-000000000000
Seqno: -1 - -1
Offset: -1
Synced: 0
2021-07-28  9:26:21 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
2021-07-28  9:26:21 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 172.17.0.2; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc
2021-07-28  9:26:21 0 [Note] WSREP: Start replication
2021-07-28  9:26:21 0 [Note] WSREP: Connecting with bootstrap option: 0
2021-07-28  9:26:21 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2021-07-28  9:26:21 0 [Note] WSREP: protonet asio version 0
2021-07-28  9:26:21 0 [Note] WSREP: Using CRC-32C for message checksums.
2021-07-28  9:26:21 0 [Note] WSREP: backend: asio
2021-07-28  9:26:21 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2021-07-28  9:26:21 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
2021-07-28  9:26:21 0 [Note] WSREP: restore pc from disk failed
2021-07-28  9:26:21 0 [Note] WSREP: GMCast version 0
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2021-07-28  9:26:21 0 [Note] WSREP: EVS version 1
2021-07-28  9:26:21 0 [Note] WSREP: gcomm: connecting to group 'nextcloud_int', peer '10.0.28.26:4567,10.0.28.31:4567,10.0.28.32:4567'
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.0.28.31:4567
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') connection established to d9e457ac-8a0d tcp://10.0.28.26:4567
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2021-07-28  9:26:21 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') connection established to df17c022-aae4 tcp://10.0.28.32:4567
2021-07-28  9:26:22 0 [Note] WSREP: EVS version upgrade 0 -> 1
2021-07-28  9:26:22 0 [Note] WSREP: declaring d9e457ac-8a0d at tcp://10.0.28.26:4567 stable
2021-07-28  9:26:22 0 [Note] WSREP: declaring df17c022-aae4 at tcp://10.0.28.32:4567 stable
2021-07-28  9:26:22 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2021-07-28  9:26:22 0 [Note] WSREP: Node d9e457ac-8a0d state prim
2021-07-28  9:26:22 0 [Note] WSREP: view(view_id(PRIM,d9e457ac-8a0d,2) memb {
        d9e457ac-8a0d,0
        def73391-b951,0
        df17c022-aae4,0
} joined {
} left {
} partitioned {
})
2021-07-28  9:26:22 0 [Note] WSREP: save pc into disk
2021-07-28  9:26:22 0 [Note] WSREP: gcomm: connected
2021-07-28  9:26:22 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2021-07-28  9:26:22 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2021-07-28  9:26:22 0 [Note] WSREP: Opened channel 'nextcloud_int'
2021-07-28  9:26:22 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
2021-07-28  9:26:22 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2021-07-28  9:26:22 1 [Note] WSREP: Starting rollbacker thread 1
2021-07-28  9:26:22 0 [Note] WSREP: STATE EXCHANGE: sent state msg: df444d26-ef85-11eb-8936-be401295de77
2021-07-28  9:26:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: df444d26-ef85-11eb-8936-be401295de77 from 0 (e4a7a8af0b78)
2021-07-28  9:26:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: df444d26-ef85-11eb-8936-be401295de77 from 2 (cd8aaaad506b)
2021-07-28  9:26:22 2 [Note] WSREP: Starting applier thread 2
2021-07-28  9:26:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: df444d26-ef85-11eb-8936-be401295de77 from 1 (f38e8cff30b6)
2021-07-28  9:26:22 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 1,
        members    = 1/3 (joined/total),
        act_id     = 15,
        last_appl. = 14,
        protocols  = 2/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = d6aa7099-ef85-11eb-9fbe-dbb821267db3
2021-07-28  9:26:22 0 [Note] WSREP: Flow-control interval: [28, 28]
2021-07-28  9:26:22 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 16)
2021-07-28  9:26:22 2 [Note] WSREP: ####### processing CC 16, local, ordered
2021-07-28  9:26:22 2 [Note] WSREP: Process first view: d6aa7099-ef85-11eb-9fbe-dbb821267db3 my uuid: def73391-ef85-11eb-b951-cb71f524bed3
2021-07-28  9:26:22 2 [Note] WSREP: Server f38e8cff30b6 connected to cluster at position d6aa7099-ef85-11eb-9fbe-dbb821267db3:16 with ID def73391-ef85-11eb-b951-cb71f524bed3
2021-07-28  9:26:22 2 [Note] WSREP: Server status change disconnected -> connected
2021-07-28  9:26:22 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-07-28  9:26:22 2 [Note] WSREP: ####### My UUID: def73391-ef85-11eb-b951-cb71f524bed3
2021-07-28  9:26:22 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2021-07-28  9:26:22 0 [Note] WSREP: Service thread queue flushed.
2021-07-28  9:26:22 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2021-07-28  9:26:22 2 [Note] WSREP: State transfer required:
        Group state: d6aa7099-ef85-11eb-9fbe-dbb821267db3:16
        Local state: 00000000-0000-0000-0000-000000000000:-1
2021-07-28  9:26:22 2 [Note] WSREP: Server status change connected -> joiner
2021-07-28  9:26:22 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-07-28  9:26:22 0 [Note] WSREP: Joiner monitor thread started to monitor
2021-07-28  9:26:22 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '172.17.0.2' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid'
WSREP_SST: [INFO] SSL configuration: CA='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20210728 09:26:22.594)
WSREP_SST: [INFO] Streaming with mbstream (20210728 09:26:22.684)
WSREP_SST: [INFO] Using socat as streamer (20210728 09:26:22.688)
WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/opt/bitnami/mariadb//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20210728 09:26:22.709)
2021-07-28  9:26:23 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 16, STRv: 3
2021-07-28  9:26:23 2 [Note] WSREP: IST receiver addr using tcp://172.17.0.2:4568
2021-07-28  9:26:23 2 [Note] WSREP: Prepared IST receiver for 0-16, listening at: tcp://172.17.0.2:4568
2021-07-28  9:26:23 0 [Note] WSREP: Member 1.0 (f38e8cff30b6) requested state transfer from '*any*'. Selected 0.0 (e4a7a8af0b78)(SYNCED) as donor.
2021-07-28  9:26:23 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 16)
2021-07-28  9:26:23 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2021-07-28  9:26:23 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
2021-07-28  9:26:23 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> d6aa7099-ef85-11eb-9fbe-dbb821267db3:16
2021-07-28  9:26:23 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:24 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:25 0 [Note] WSREP: (def73391-b951, 'tcp://0.0.0.0:4567') turning message relay requesting off
2021-07-28  9:26:25 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:26 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:27 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:28 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:26:29 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable

[... message repeats every second ...]

2021-07-28  9:31:20 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:31:21 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
2021-07-28  9:31:22 0 [Warning] WSREP: Member 2.0 (cd8aaaad506b) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
WSREP_SST: [ERROR] Possible timeout in receiving first data from  donor in gtid stage: exit codes: 124 0 (20210728 09:31:22.716)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20210728 09:31:22.718)
WSREP_SST: [INFO] Removing the sst_in_progress file (20210728 09:31:22.721)
WSREP_SST: [INFO] Cleaning up temporary directories (20210728 09:31:22.724)
2021-07-28  9:31:22 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '172.17.0.2' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid: 32 (Broken pipe)
2021-07-28  9:31:22 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2021-07-28  9:31:22 3 [Note] WSREP: SST received
2021-07-28  9:31:22 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
2021-07-28  9:31:22 3 [Note] WSREP: SST succeeded for position 00000000-0000-0000-0000-000000000000:-1
2021-07-28  9:31:22 2 [ERROR] WSREP: Application received wrong state:
        Received: 00000000-0000-0000-0000-000000000000
        Required: d6aa7099-ef85-11eb-9fbe-dbb821267db3
2021-07-28  9:31:22 0 [Note] WSREP: Joiner monitor thread ended with total time 300 sec
2021-07-28  9:31:22 2 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
2021-07-28  9:31:22 2 [Note] WSREP: ReplicatorSMM::abort()
2021-07-28  9:31:22 2 [Note] WSREP: Closing send monitor...
2021-07-28  9:31:22 2 [Note] WSREP: Closed send monitor.
2021-07-28  9:31:22 2 [Note] WSREP: gcomm: terminating thread
2021-07-28  9:31:22 2 [Note] WSREP: gcomm: joining thread
2021-07-28  9:31:22 2 [Note] WSREP: gcomm: closing backend
2021-07-28  9:31:22 2 [Note] WSREP: view(view_id(NON_PRIM,d9e457ac-8a0d,2) memb {
        def73391-b951,0
} joined {
} left {
} partitioned {
        d9e457ac-8a0d,0
        df17c022-aae4,0
})
2021-07-28  9:31:22 2 [Note] WSREP: PC protocol downgrade 1 -> 0
2021-07-28  9:31:22 2 [Note] WSREP: view((empty))
2021-07-28  9:31:22 2 [Note] WSREP: gcomm: closed
2021-07-28  9:31:22 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2021-07-28  9:31:22 0 [Note] WSREP: Flow-control interval: [16, 16]
2021-07-28  9:31:22 0 [Note] WSREP: Received NON-PRIMARY.
2021-07-28  9:31:22 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 16)
2021-07-28  9:31:22 0 [Note] WSREP: New SELF-LEAVE.
2021-07-28  9:31:22 0 [Note] WSREP: Flow-control interval: [0, 0]
2021-07-28  9:31:22 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2021-07-28  9:31:22 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 16)
2021-07-28  9:31:22 0 [Note] WSREP: RECV thread exiting 0: Success
2021-07-28  9:31:22 2 [Note] WSREP: recv_thread() joined.
2021-07-28  9:31:22 2 [Note] WSREP: Closing replication queue.
2021-07-28  9:31:22 2 [Note] WSREP: Closing slave action queue.
2021-07-28  9:31:22 2 [Note] WSREP: /opt/bitnami/mariadb/sbin/mysqld: Terminated.
210728  9:31:22 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.11-MariaDB-log
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336801 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc554000c18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc5780e2e18 thread_stack 0x49000
/opt/bitnami/mariadb/sbin/mysqld(my_print_stacktrace+0x2e)[0x555bfab25a3e]
Printing to addr2line failed
/opt/bitnami/mariadb/sbin/mysqld(handle_fatal_signal+0x485)[0x555bfa5e2ad5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fc57ab51730]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x1fd)[0x7fc57a678611]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x1cf6ac)[0x7fc57a44f6ac]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x5d863)[0x7fc57a2dd863]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x79fe5)[0x7fc57a2f9fe5]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x681da)[0x7fc57a2e81da]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x689a3)[0x7fc57a2e89a3]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x68eeb)[0x7fc57a2e8eeb]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x918f0)[0x7fc57a3118f0]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x91b61)[0x7fc57a311b61]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x676e0)[0x7fc57a2e76e0]
/opt/bitnami/mariadb/lib/libgalera_smm.so(+0x440a8)[0x7fc57a2c40a8]
/opt/bitnami/mariadb/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x555bfabb35ce]
/opt/bitnami/mariadb/sbin/mysqld(+0xbd84f3)[0x555bfa8854f3]
/opt/bitnami/mariadb/sbin/mysqld(_Z15start_wsrep_THDPv+0x2cf)[0x555bfa87603f]
/opt/bitnami/mariadb/sbin/mysqld(+0xb67ffb)[0x555bfa814ffb]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fc57ab46fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc57a74f4cf]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 2
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway.
Query:

Writing a core file...
Working directory at /bitnami/mariadb/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    0                    bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            1048576              1048576              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       63703                63703                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E

```

# Answer
> 1 votes
For now I gave up on this approach after stumbling over this paragraphs in the Redis cluster documentation:

> Currently Redis Cluster does not support NATted environments and in general environments where IP addresses or TCP ports are remapped.
>
> Docker uses a technique called *port mapping*: programs running inside Docker containers may be exposed with a different port compared to the one the program believes to be using. This is useful in order to run multiple containers using the same ports, at the same time, in the same server.
>
> In order to make Docker compatible with Redis Cluster you need to use the **host networking mode** of Docker. Please check the `--net=host` option in the Docker documentation for more information.

I can only assume that Galera has the same problem as Redis. If I'm wrong, please correct me and post an answer, I'll leave the question open for that.

---
Tags: docker, mysql
---


# Question
Title: Deploy an application using dll files to IIS on a VM
I hope I am posting this in the correct forum, but to me at least, it seemed that the focus was more on deployment.

I am currently stuck after having successfully refactored an application by extracting certain responsibilities into class libraries (language: C#, .NET 4.8).

The application is hooked up to Azure CI builds using Azure DevOps, deploying the application to an IIS on a VM. Pipelines are triggered upon commit.

The pipeline looks roughly like this:

* `NuGet restore`
* `Build Solution`
* `Test Assemblies (Visual Studio Tests)`
* `Publish symbols path`
* `Publish Artifact`

The `.dll` files are located in a dedicated folder within the application. I am referencing the `.dll` files in the application and have already changed their property `Copy to Output Directory` to `Copy if newer`. When building the application on my local machine, the required custom `.dll` files are being copied to the `bin` folder as expected. However, when analysing the deployment logs, I noticed that the custom `.dll` files are not being updated / copied.

How can I make sure that the custom `.dll` files are deployed properly (same as the standard `.ddl` files)?

# Answer
> 1 votes
It seems that there is no other way than creating NuGet packages from those `.dll` files required for the build, as this post on SO points out.

---
Tags: continuous-integration, azure-devops, deployment, azure-pipelines
---


# Question
Title: Minikube will not start again after a stop
Running gentoo linux... I am new to kubernetes and learning it on my dev cloud. I am able to start minikube with no issues (other than the second node is missing...not sure why). Then I stop minikube. When I start it again it will not start with `GUEST_PROVISION_CONTAINER_EXITED`. I have to reboot my server to resolve it.

Anyone know how to fix this?

```

one@work ~
% minikube start --nodes=2

ð  minikube v1.22.0 on Gentoo 2.7 (amd64)
â¨  Using the docker driver based on existing profile
ð  Starting control plane node minikube in cluster minikube
ð  Pulling base image ...
ð¤·  docker "minikube" container is missing, will recreate.
ð¥  Creating docker container (CPUs=2, Memory=7300MB) ...
ð³  Preparing Kubernetes v1.21.2 on Docker 20.10.7 ...
    âª Generating certificates and keys ...
    âª Booting up control plane ...
    âª Configuring RBAC rules ...
ð  Verifying Kubernetes components...
    âª Using image gcr.io/k8s-minikube/storage-provisioner:v5
ð  Enabled addons: storage-provisioner, default-storageclass
â  The cluster minikube already exists which means the --nodes parameter will be ignored. Use "minikube node add" to add nodes to an existing cluster.
ð  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
one@work ~
% minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

one@work ~
% minikube stop
â  Stopping node "minikube"  ...
ð  Powering off "minikube" via SSH ...
ð  1 nodes stopped.
one@work ~
% minikube start --nodes=2
ð  minikube v1.22.0 on Gentoo 2.7 (amd64)
â¨  Using the docker driver based on existing profile
ð  Starting control plane node minikube in cluster minikube
ð  Pulling base image ...
ð  Restarting existing docker container for "minikube" ...
ð¤¦  StartHost failed, but will try again: provision: get ssh host-port: unable to inspect a not running container to get SSH port
ð  Restarting existing docker container for "minikube" ...
ð¿  Failed to start docker container. Running "minikube delete" may fix it: provision: get ssh host-port: unable to inspect a not running container to get SSH port

â  Exiting due to GUEST_PROVISION_CONTAINER_EXITED: Docker container exited prematurely after it was created, consider investigating Docker's performance/health.

```

# Answer
> 1 votes
This is generic docker debugging help. I've installed minikube, but haven't ran into this particular inconvenience.

When it says "consider investigating Docker's performance/health" it probably means to check the docker logs

`docker ps --all`

should give you a list of images, if this is your only docker project, then it oughta be easier to find. The 3rd column in that list is the container\_name parameter you want to feed to `docker logs [container_name]`

Whatever caused your container to stop prematurely ought to be in there.

Assuming you're running a production build of minikube, the chances are good it's something with your setup, probably filesystem related, but who can tell until you debug it.

---
Tags: kubernetes, linux, minikube
---


# Question
Title: Configure WebSocket on AWS EC2 Load Balancer throws Connection to 'wss://chat.domain.com:8080/' failed
I have built a Websocket on Ubuntu 20.04 (LAMP) stack using Ratchet.

I followed this article https://www.twilio.com/blog/create-php-websocket-server-build-real-time-even-driven-application to build the websocket.

I Followed this article https://stackoverflow.com/questions/39336033/does-an-application-load-balancer-support-websockets to configure my webserver.

1. Configured Security Group - Inbound rules TCP 8080
2. Configured Load balancer
3. Created Target group (TCP) for port 8080

4. Enabled Stickyness (1 hour)
5. SSL is configured and created in AWS Certificate Manager

5. Apache configuration

```
<VirtualHost *:80>
    ServerName chat.domain.com
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/websites/chat.domain.com/public
       <Directory /var/www/websites/chat.domain.com/public/>
           Options Indexes FollowSymLinks MultiViews
           AllowOverride All
           Require all granted
       </Directory>

    ErrorLog ${APACHE_LOG_DIR}/chat.domain.com/error.log
    CustomLog ${APACHE_LOG_DIR}/chat.domain.com/access.log combined
</VirtualHost>

<VirtualHost *:443>
    ServerName chat.domain.com
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/websites/chat.domain.com/public
       <Directory /var/www/websites/chat.domain.com/public/>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Require all granted
        </Directory>

    ProxyPass / https://chat.domain.com:8080/
    ProxyPassReverse / https://chat.domain.com:8080/
    <Location "/">
        ProxyPass "wss://chat.domain.com:8080/"
    </Location>

    ErrorLog ${APACHE_LOG_DIR}/chat.domain.com/error.log
    CustomLog ${APACHE_LOG_DIR}/chat.domain.com/access.log combined
</VirtualHost>

```

Every thing is in place and running but I get

> WebSocket connection to 'wss://chat.domain.com:8080/' failed:

Here is my chrome inspect

I have even tried to Open traffic to ALL ports (Inbound) just to check the security group but still getting the same error.

I doubt the problem is from configuring Load Balancer, Security Group and Target group?

Any help or suggestion?

# Answer
> 1 votes
Finally I found the solution.

* The port 8080 should have been added to firewall
* create a new target group for port 8080 which
* create a load balancer with port http:8080 and point to the target group

---
Tags: amazon-web-services
---


# Question
Title: Jenkins JobDSL organizationFolder and ghprb - even possible?
I'm working on getting JobDSL working with organizationFolder (the seed job builds just fine), and ghprb. 'githubPullRequest' doesn't exist under JobDSL's organizationFolder/triggers section, but 'ghprbTrigger' does.

When I try a minimal config, I'm told:

```
the following options are required and must be specified: adminlist, whitelist,
orgslist, cron, triggerPhrase, onlyTriggerPhrase, permitAll,
autoCloseFailedPullRequests, displayBuildErrorsOnDownstreamBuilds,
commentFilePath, skipBuildPhrase, blackListCommitAuthor,
allowMembersOfWhitelistedOrgsAsAdmin, msgSuccess, msgFailure,
commitStatusContext, gitHubAuthId, buildDescTemplate, blackListLabels,
whiteListLabels, includedRegions, excludedRegions

```

I put in some dummy values to alleviate their string & boolean hunger, and am finally rewarded with:

```
java.lang.ClassCastException: jenkins.branch.OrganizationFolder  cannot be cast
to hudson.model.Job

```

and a giant groovy/java error dump.

I don't know what that last error means or how to track it down. This is pretty frustrating. Any help would be appreciated. Thanks!

# Answer
The answer, as it turns out, is not to use ghprb at all. Simply aim your entire github org at your Jenkins web-hook and you're done.

Per the Cloudbees support docs, install the GitHub plugin and GitHub Branch Source plugin, then setup the org folder job as normal. Then, at your Github org, go to Settings, Webhooks, Add Webhook, and add your webhook. Give it Pull Request, Push, and Repository events.
> 1 votes

---
Tags: jenkins
---


# Question
Title: ERROR This job is stuck because the project doesn't have any runners online assigned to it. even though i have added runner and it is alive
Despite of the fact runner is alive. Why this error is coming

# Answer
> 3 votes
You need to either configure the runner to run untagged jobs, or tag your job to use the runner.

To configure the runner to run untagged jobs:

> 1. Go to the project’s Settings \> CI/CD and expand the Runners section.
> 2. Find the runner you want to pick untagged jobs and make sure it’s enabled.
> 3. Click the pencil button.
> 4. Check the Run untagged jobs option.
> 5. Click the Save changes button for the changes to take effect.

Ref: https://docs.gitlab.com/13.12/ee/ci/runners/#use-tags-to-limit-the-number-of-jobs-using-the-runner

To add a tag to your job:

```
job:
  tags:
    - ci

```

Ref: https://docs.gitlab.com/ee/ci/yaml/#tags

---
Tags: gitlab, cicd
---


# Question
Title: How to add a string to a line when it doesn't exist?
I'm trying to solve a problem where I want to add a setting 'pci=noaer' to all kernel definitions in /boot/grub/grub.conf, but only if the setting doesn't exist. I've managed to get the basic functionality to work with the replace module, in that the module WILL add the setting to the relevant lines, but the problem is in NOT adding the setting to lines where it already exists. In other words, if I run the playbook once, it'll add the setting as needed, but every other run of the playbook after that adds another instance of the setting to each line. The following is an example of the line I want to update and the regexp and replace lines I currently have configured to update it:

Line to change:
`kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3`

After the playbook runs the above line should look like:
`kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3 pci=noaer`

Relevant replace module parameters:

```
regexp: '(^\s+kernel /vmlinuz.+(?!pci=noaer).*$)'
replace: '\1 pci=noaer'

```

I've tried using the \b sequences to clearly define the boundaries of pci=noaer (e.g. (?!\bpci=noaer\b)), but with no luck. I've tested this a bit on regex101 but haven't really been able to definitively understand why the negative lookahead isn't doing what it's supposed to be doing (looking for cases where the pci=noaer string isn't there) - or how I'm using it within the replace module. I've had similar issues with negative lookaheads/lookbehinds in lineinfile, and I'm perfectly willing to admit this is total user error, given that I'm fairly new to ansible and python, and my regex skills are fairly limited thus far. But I've searched and searched and searched and I can't seem to find any good indication of why the negative lookaround sequences don't work the way I expect them to within these modules. Could this be a case of not properly managing the greediness of my regex configuration?

For what it's worth, I have seen that lookahead/lookbehind might have issues in some of the 2.x versions of Ansible, so perhaps this is a version problem? I'm running Ansible 2.6.20 with an underlying Python version of 2.6.6. If this is indeed an issue with my version, is there a way around it without having to update to something more recent?

A bit of additional information: This is being done across multiple CentOS 6-based systems where I don't feel like I can necessarily expect the kernel lines in the grub.conf file to be the same from one system to the next (perhaps they should be, but that's a topic for a different discussion). That's why the fairly non-specific regex. Also, since it's CentOS 6, there is no equivalent of update-grub or grub2-mkconfig for that OS. It's just a matter of editing the /etc/grub.conf or /boot/grub/grub.conf file and rebooting.

# Answer
> 1 votes
Your *regexp* can't work because greedy `.+` in front of the lookahead assertion will consume the line till the very end. The next problem is that you require at least one space at the beginning of the line `\s+`.

Try this

```
        regexp: '^(?!.* pci=noaer.*)(\s*kernel /vmlinuz.*)$'
        replace: '\1 pci=noaer'

```

It's important to understand that *regexp* in Ansible module *replace* uses MULTILINE and does not use DOTALL. Quoting:

> Uses MULTILINE mode, which means ^ and $ match the beginning and end of the file, as well as the beginning and end respectively of each line of the file. Does not use DOTALL, which means the . special character matches any character except newlines.

---

<sup>

If it does not work below is a brute force solution. Given the testing file

```
shell> cat grub.cfg
kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3

```

get the lines

```
    - command: cat grub.cfg
      register: grub
      check_mode: false

```

For simplicity, I used *command* but you might want to use *fetch*, *slurp*, or any other suitable module instead.

Test it. In debug, iterate the matched lines, split the properties of the line, and find the missing properties

```
    - debug:
        msg:
          - "{{ item }}"
          - "{{ missing }}"
      loop: "{{ grub.stdout_lines|select('match', match) }}"
      loop_control:
        label: "{{ line.1 }}"
      when: missing|length > 0
      vars:
        match: '^\s*kernel /vmlinuz.+$'
        line: "{{ item.split() }}"
        missing: "{{ props|difference(line) }}"
        props:
          - pci=noaer

```

gives

```
ok: [localhost] => (item=/vmlinuz-2.6.32-279.22.1.el6.x86_64) =>
  msg:
  - kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
  - - pci=noaer
ok: [localhost] => (item=/vmlinuz-2.6.32-279.22.1.el6.x86_64) =>
  msg:
  - kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
  - - pci=noaer
ok: [localhost] => (item=/vmlinuz-2.6.32-279.22.2.el6.x86_64) =>
  msg:
  - kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
  - - pci=noaer
ok: [localhost] => (item=/vmlinuz-2.6.32-279.22.2.el6.x86_64) =>
  msg:
  - kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3
  - - pci=noaer

```

If the configuration seems to be working use the module *replace* in check mode

```
    - replace:
        path: grub.cfg
        regexp: "{{ item }}"
        replace: "{{ (line + missing)|join(' ') }}"
      loop: "{{ grub.stdout_lines|select('match', match) }}"
      loop_control:
        label: "{{ line.1 }}"
      when: missing|length > 0
      vars:
        match: '^\s*kernel /vmlinuz.+$'
        line: "{{ item.split() }}"
        missing: "{{ props|difference(line) }}"
        props:
          - pci=noaer

```

gives

```
--- before: grub.cfg
+++ after: grub.cfg
@@ -1,4 +1,4 @@
-kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
+kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3 pci=noaer
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3

changed: [localhost] => (item=/vmlinuz-2.6.32-279.22.1.el6.x86_64)
--- before: grub.cfg
+++ after: grub.cfg
@@ -1,4 +1,4 @@
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
-kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
+kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3 pci=noaer
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3

changed: [localhost] => (item=/vmlinuz-2.6.32-279.22.1.el6.x86_64)
--- before: grub.cfg
+++ after: grub.cfg
@@ -1,4 +1,4 @@
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
-kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
+kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3 pci=noaer
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3

changed: [localhost] => (item=/vmlinuz-2.6.32-279.22.2.el6.x86_64)
--- before: grub.cfg
+++ after: grub.cfg
@@ -1,4 +1,4 @@
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro recovery root=/dev/sda3
 kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro root=/dev/sda3
-kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3
+kernel /vmlinuz-2.6.32-279.22.2.el6.x86_64 ro recovery root=/dev/sda3 pci=noaer

changed: [localhost] => (item=/vmlinuz-2.6.32-279.22.2.el6.x86_64)

```

Run the play if this is what you want.

---

In case of grub2 you should modify `GRUB_CMDLINE_LINUX` or `GRUB_CMDLINE_LINUX_DEFAULT` in /etc/default/grub and notify handler `update-grub`. For example

```
tasks:
  - name: Configure /etc/default/grub
    ansible.builtin.lineinfile:
      dest: /etc/default/grub
      regexp: '^\s*{{ item.var }}\s*=(.*)$'
      line: "{{ item.var }}={{ item.value }}"
      backup: true
    loop:
      - var: GRUB_CMDLINE_LINUX_DEFAULT
        value: "quiet splash pci=noaer"
    notify: update grub

handlers:
  - name: update grub
    ansible.builtin.command:
      cmd: >
        /usr/sbin/update-grub

```

Fit the configuration data to your needs.

</sup>

---
Tags: ansible, linux, automation, python
---


title: Best Practice for writing DockerFile

text:
# Question
Title: Best Practice for writing DockerFile
I have mostly seen people using following type of pipeline steps(it varies, but in general this is the flow of steps I have observed)

> Process: clone repo -\> build -\> test -\> create docker image -\> push binary to artifactory -\> push image to docker registry

Recently I came across a docker file (multi-stage docker file) which was executing gradle build and test while building image.

I wanted to reach out to this community and check what are the advantages/dis-advantages of having a docker file which does gradle/maven build and test as a part of `docker build`?

Please Guide

# Answer
I don't know if this is a good practice or not, but I usually create 2 docker files. One just runs the application and the other runs a multi-stage build (including test), the latter is only used by the developer, the former is the one used on CI.

The CI steps depend mostly on what language is the code, what types of tests you want to run, and what process (or development flow) the company use.

For a compiled language (like java or golang):

tests that does not need to build the code (unit tests, SAST..) --\> build (the code and the container image if needed) --\> tests that need a runnable version of the code (end-to-end) --\> artifact repository

For a non compiled language (like php or python): tests --\> tag
> 2 votes

# Answer
It would be consistent with most other languages' setups to run `gradle build` in a Dockerfile. You may want to set up a multi-stage build so that the first stage has a complete JDK, `COPY`s in the `build.gradle` and source code, and builds it; and then the second stage only contains the JRE and the jar file `COPY --from=` the first stage. (Java is unusual in that it's *possible* to build a portable jar file outside Docker and `COPY` it directly into an image; most other languages either interpret the source directly, or build a native artifact that's not as portable.)

The advantages of doing this are IMHO somewhat minimal. If you have a multi-language environment, being able to `git clone; docker build` every service the same way is useful. You will use the JVM specified by the Dockerfile's `FROM` line, instead of what's installed on the host, but you will probably need a JVM anyways for day-to-day development and IME the precise patch level of the JRE isn't that important in practice.

---

I would not run tests in a Dockerfile.

At a philosophical level, they don't really belong there. The Dockerfile has one task: it takes source files or a jar file in, and produces a Docker image out. The Dockerfile system is designed for it to be hard for there to be side effects, so it's very difficult to start a support process or access a database even if you otherwise would want to. Running tests are in some way a "side effect" like this.

Mechanically, it makes sense to run tests both before and after the image build, but not necessarily during. You can break your tests into *unit tests* and *integration tests*. The unit tests should only depend on your application code, with mocks for any external dependencies; since they don't depend on anything but the code and the JRE, you should get consistent results whether you run them on the host or in a container, and you can first run unit tests and then only build the image if they pass. The integration tests depend on the entire system, including things you can't connect to from the Dockerfile, and so a good practice here is to run integration tests against the built image as you're about to ship it.

```
   Source code      (checked in to a repository)
        |
        v
   Unit tests       (on the host, no external dependencies)
        |
        v
Docker image build  (push to a repository now if needed)
        |           (only this step is the Dockerfile)
        v
Integration tests   (on the built container, with dependencies
        |            running in other containers, either launched
        v            from the host or a dedicated test-runner container)
     Deploy

```

I would avoid complex Dockerfiles that have multiple optional stages that sometimes include test frameworks and sometimes don't. Build one image and run it in all environments, and keep the Dockerfile simple and readable.
> 0 votes

---
Tags: docker, dockerfile
---


title: Installing NVIDIA driver on EKS nodes

text:
# Question
Title: Installing NVIDIA driver on EKS nodes
I have a EKS cluster that has two node pools. One CPU general compute, the other one has GPU support (g4dn). I want to install NVIDIA driver on to the GPU node group. The best I can do right now is to manually ssh onto each node and install. Is there a better way to achieve this?

# Answer
> 1 votes
GPU Operator helps manage NVIDIA drivers in Kubernetes clusters, you can deploy it using Helm.

---
Tags: kubernetes, amazon-web-services, eks
---


# Question
Title: Can not install mysql client and server using a playbook Ansible
Do you know how I can fix this error?

I'm getting **404** response from my node machibe when I'm trying to install **default-mysql-client and server** using Ansible-playbook apt module or shel.

**Facts**

I do can install another software like `htop`, If I use the `shel:` option I'm getting the same error, but when I install the packages trough `ssh` works like a charm. The node with the problem has installed ansible, I `purge` it to check if that could be the problem, did not help me.

I have 2 nodes and the second do not experience the same issue, They have the same `/etc/apt/source.list`.

**Edit:** I paid attention in the error message that it is trying to install  `mariadb-client-core-10.3_10.3.27-0+deb10u1_amd64.deb` instead of `name: ['default-mysql-client','default-mysql-server']` .

**Source.list node**

```
#

# deb cdrom:[Debian GNU/Linux 10.8.0 _Buster_ - Official amd64 NETINST 20210206-10:34]/ buster main

# deb cdrom:[Debian GNU/Linux 10.8.0 _Buster_ - Official amd64 NETINST 20210206-10:34]/ buster main

deb http://deb.debian.org/debian/ buster main
deb-src http://deb.debian.org/debian/ buster main

deb http://security.debian.org/debian-security buster/updates main
deb-src http://security.debian.org/debian-security buster/updates main

# buster-updates, previously known as 'volatile'
deb http://deb.debian.org/debian/ buster-updates main
deb-src http://deb.debian.org/debian/ buster-updates main

# Added by aby for ansible course
#deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.

```

**Host**

```
ansible --version

ansible [core 2.11.3]
  python version = 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
  jinja version = 2.11.2
  libyaml = True

```

```
lsb_release -a
Debian GNU/Linux 10 (buster)

```

**Nodes**

```
lsb_release -a
Debian GNU/Linux 10 (buster)

```

**Playbook**

```
- name: Deploy a web app
  hosts: db_and_web_server1 , db_and_web_server2
  become: yes
  tasks:
    - name: Update and Upgrade system
      apt:
        upgrade: yes
        update_cache: yes
        cache_valid_time: 0

    - name: Install all required dependencies
      apt:
        name: ['python', 'python-setuptools', 'python-dev', 'build-essential', 'python-pip','htop']
        update_cache: yes
        state: latest

    - name: Install Mysql and htop
      apt:
        update_cache: yes
        cache_valid_time: 0
        name: ['default-mysql-client','default-mysql-server']
        #state: latest
        #shell: apt install -y default-mysql-client
        #shell: "apt install -y --fix-missing default-mysql-client"
      #ignore_errors: true
      #shell: apt install -f -y
    - name: Start MySql Service
      service:
        name: mysql
        state: started
        enabled: yes

```

**Error**

```
fatal: [db_and_web_server1]: FAILED! =>
{
    "changed": true,
    "cmd": "apt install -y default-mysql-client",
    "delta": "0:00:00.940517",
    "end": "2021-08-02 11:09:07.829191",
    "msg": "non-zero return code",
    "rc": 100,
    "start": "2021-08-02 11:09:06.888674",
    "stderr": "\nWARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n\nE: Failed to fetch http://deb.debian.org/debian/pool/main/m/mariadb-10.3/mariadb-client-core-10.3_10.3.27-0+deb10u1_amd64.deb  404  Not Found [IP: 2a04:4e42:62::644 80]\nE: Failed to fetch http://deb.debian.org/debian/pool/main/m/mariadb-10.3/mariadb-client-10.3_10.3.27-0+deb10u1_amd64.deb  404  Not Found [IP: 2a04:4e42:62::644 80]\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?",
    "stderr_lines": ["",
        "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.",
        "",
        "E: Failed to fetch http://deb.debian.org/debian/pool/main/m/mariadb-10.3/mariadb-client-core-10.3_10.3.27-0+deb10u1_amd64.deb  404  Not Found [IP: 2a04:4e42:62::644 80]",
        "E: Failed to fetch http://deb.debian.org/debian/pool/main/m/mariadb-10.3/mariadb-client-10.3_10.3.27-0+deb10u1_amd64.deb  404  Not Found [IP: 2a04:4e42:62::644 80]",
        "E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?"
    ],
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n  libaio1 libconfig-inifiles-perl libdbd-mysql-perl libdbi-perl libreadline5\n  mariadb-client-10.3 mariadb-client-core-10.3\nSuggested packages:\n  libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl\nThe following NEW packages will be installed:\n  default-mysql-client libaio1 libconfig-inifiles-perl libdbd-mysql-perl\n  libdbi-perl libreadline5 mariadb-client-10.3 mariadb-client-core-10.3\n0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 6,904 kB/7,986 kB of archives.\nAfter this operation, 54.5 MB of additional disk space will be used.\nErr:1 http://deb.debian.org/debian buster/main amd64 mariadb-client-core-10.3 amd64 1:10.3.27-0+deb10u1\n  404  Not Found [IP: 2a04:4e42:62::644 80]\nErr:2 http://deb.debian.org/debian buster/main amd64 mariadb-client-10.3 amd64 1:10.3.27-0+deb10u1\n  404  Not Found [IP: 2a04:4e42:62::644 80]",
    "stdout_lines": [
        "Reading package lists...",
        "Building dependency tree...",
        "Reading state information...",
        "The following additional packages will be installed:",
        "  libaio1 libconfig-inifiles-perl libdbd-mysql-perl libdbi-perl libreadline5",
        "  mariadb-client-10.3 mariadb-client-core-10.3",
        "Suggested packages:",
        "  libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl",
        "The following NEW packages will be installed:",
        "  default-mysql-client libaio1 libconfig-inifiles-perl libdbd-mysql-perl",
        "  libdbi-perl libreadline5 mariadb-client-10.3 mariadb-client-core-10.3",
        "0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.",
        "Need to get 6,904 kB/7,986 kB of archives.",
        "After this operation, 54.5 MB of additional disk space will be used.",
        "Err:1 http://deb.debian.org/debian buster/main amd64 mariadb-client-core-10.3 amd64 1:10.3.27-0+deb10u1",
        "  404  Not Found [IP: 2a04:4e42:62::644 80]",
        "Err:2 http://deb.debian.org/debian buster/main amd64 mariadb-client-10.3 amd64 1:10.3.27-0+deb10u1",
        "  404  Not Found [IP: 2a04:4e42:62::644 80]"
    ]
}

```

# Answer
> 1 votes
Maybe specify other package names that one can see at http://deb.debian.org/debian/pool/main/m/mariadb-10.3/

like:

```
- name: Install Mysql and htop
  apt:
    update_cache: yes
    cache_valid_time: 0
    state: present
    name: "{{packages}}"
  vars:
    packages:
    - mariadb-server
    - mariadb-client
    - htop

```

---
Tags: ansible
---


# Question
Title: How would I change the docker's base directory
All my docker related folders are saved in /var/lib/docker. But my / space is very limited. Hence, I would like all of the content present inside this location to go to say /home/User/my\_space. How would I do that? I could use symbolic link. But, that will only move the current contents of the location. I would like to move the future changes of the docker also. Please help me :)

# Answer
> 1 votes
Symlink is an universal solution which works, I've successfully used it plenty of times. I do not recommend using the path you suggested, though, since the files should be owned by either the `root` or `docker` users, depending on your distro/OS. Therefore, if you want to use the `/home` partition, then:

* shut down docker
* `mv /var/lib/docker /home/`
* `ln -s /home/docker /var/lib/docker`
* start docker

This will usually persist through upgrades unless you uninstall and reinstall Docker.

Now, more recent versions of Docker support an option in `/etc/docker/daemon.json`, namely the `data-root` option. So just edit the file and add

```
{
  "data-root": "/home/docker"
}

```

You can find the whole schema for the file in the docs.

---
Tags: docker, volume
---


# Question
Title: How to do docker build image from dockerfile on Kubernetes 1.20
Kubernetes 1.20 moved from docker to CRI-O as container runtime. I have a scenario where my Jenkins run on kubernetes and does docker build my dockerfile .But the build breaks as docker daemon is not available now. I tried using Kaniko to build image but the downfall was once image is built i have no chance to scan the image before pushing to container registry. Once the image is built i have to push it immediately. How can i scan it before pushing to registry.

Current process:

1. Jenkins runs on Kubernetes.
2. Pulls code and does the docker build and tag it.
3. Once docker build is successful will a scan the image for vulnerabilities.
4. Once scan passes we push to the container registry.

With Kaniko i can do till step 2 later image goes away I am not able to find the image locally but i can push directly. And pull it and do scan. Any suggestions or alternatives so i can use my old dockerfiles build,scan and push without major changes

# Answer
You can take a look at podman, is like run docker but without the need of a daemon.

You can also check buildah
> 1 votes

---
Tags: docker, jenkins, kubernetes
---


# Question
Title: Ansible with OpenSUSE transactional server
When using an OpenSUSE transactional server, transactional-update must be used instead of `Zypper`. Is there any way of doing this with Ansible?

# Answer
> 1 votes
I think currently the only way is to use the command (or shell) module, as there's no native support for transactional-updates in Ansible yet. However, there's a fresh feature request issue: https://github.com/ansible-collections/community.general/issues/3159

For example:

```
- name: install a package with transactional-update
  ansible.builtin.command:  transactional-update --continue pkg install man

```

Multiple package names can be added. The parameter `--continue` is important, as otherwise only the last call to `transactional-update` will get effective.

Another downside is, that a reboot is required, which can be triggered from Ansible, but that kills the Ansible run....

---
Tags: ansible, linux
---


# Question
Title: mounting overlayfs within kubernetes POD - "cannot mount overlay read-only"
I try to "manually" mount an overlayfs within a Kubernetes POD, to create a copy-on-write layer on top of a shared persistent volume.

(For running a read-only postgres - postgres cannot run on a read-only filesystem because of locking.)

So I created a volume

```
spec:
  volumes:
    - name: tempfs-volume
      emptyDir:
        medium: Memory
        sizeLimit: "1Gi"

```

and mount it under /opt/overlay:

spec:
  containers:
    - volumeMounts:
        - mountPath: /opt/pgdata_readonly
          name: my-postgres-volume
        - mountPath: /opt/overlay
          name: tempfs-volume```

When I create the directories mentioned in the command below and do

</code></pre>

mount -t overlay -o lowerdir=/opt/pgdata\_readonly,upperdir=/opt/overlay/upper,workdir=/opt/overlay/work overlay /opt/overlay/target

```

I get the error `mount: /opt/overlay/target: cannot mount overlay read-only.`

I googled for quite a while and could not find a hint what could possibly go wrong here.
This is in the Google cloud.

```

# Answer
After a while I found out that I need to declare the container to have a privileged security context. Suddenly it works!

```
spec:
  containers:
    - securityContext:
        privileged: true

```

There is also an article showing how to have a two-layered setup where the container that hosts the actual application does not need to be privileged: https://itnext.io/using-overlay-mounts-with-kubernetes-960375c05959
> 1 votes

---
Tags: kubernetes, postgresql, google-kubernetes-engine
---


title: What does "build step" mean?

text:
# Question
Title: What does "build step" mean?
The term ***build step*** is used everywhere, yet I searched both google.com and qwant.com to no avail. That baffles my mind.

I also searched for a developer terminology index. Anyone has a good explanation of this term?

# Answer
In a CI/CD pipeline a build step is whatever you want it to be. It is the equivalent of a function or procedure from a traditional programming language. Examples of build steps from a CI/CD environment:

* setup a clean build or test environment by spinning up instances, starting databases, loading data, etc.
* get code from github/gitlab/etc.
* compile a component of the application
* send artifact from previous build step to a repo

Notice that in the CI/CD world a build step is not necessarily building anything, but it might be preparing something to be built or doing something with an artifact of a build.
> 3 votes

# Answer
In my experience "build step" is synonymous with "compilation", e.g. it's the step that builds a set of output files (binary, etc.).
> 2 votes

# Answer
The precise answer will depend on the context in which build step is used. However, on a high level as per me, it is always relative to the deploy process. Build step is just before the deploy step.

Basically build involves making the application completely ready (compiling, linking, uploading artifacts etc or whatever is involved) so that we are good to go ahead and deploy it to webservers/containers that will eventually serve the requests
> 1 votes

# Answer
This gives a good overview, in my experience, build steps seem synonymous with deployment steps not just compiling code.

https://docs.platform.sh/overview/build-deploy.html
> 0 votes

# Answer
With all respect, but the previous 2 answers appear to me like misleading ...

In the big iron world, aka mainframe, the build step is often, though not always, a process that consists of at least 2 steps:

* step 1 uses a program source, coded by a developer in whatever programming language (cobol, c, fortran, pl1, assembler, etc) as input, and then transforms that input in an object deck (which is not yet an executable ...). That transformation is typically done by a program called a "compiler" (each program language can have a zillion versions of such compiler, for cobol there is cobol vs, cobol le, cobol 2, and what not).
* step 2 then uses the object deck (output from step1), to transform it into an executable program, e.g by staticically linking (called) sub-programs into it also. That transformation is typically done by a program called a "linker".

The complete set of compile-steps and link-steps is often bundled together in some sort of script (procedure), which in mainframe is often done via JCL (if compile/link is done in batch), or (eg) REXX (if compile/link is done online). And this script is actually what is called a "***build procedure***".

With the above in mind, go checkout any other OS where you also have compiles, links, etc. Sooner or later, you'll recognize similar compilers and linkers.

Attention: the result of such build procedure (an executable) is typically stored in some sort of work area (like a temp folder). And from there it still needs to be shipped (copied, FTPed, whatever) to somewhere else (e.g. an authorized folder or library), before it can actually be used to execute the executable (e.g. after appropriate approvals for an emergency fix have been gathered). That shipment process is what deployment of an executable is all about (as compared to building the executable).

And better not use that deployment target directly as output of your build procedure (if you do that, you basically open up your life environments to ... pretty much any developer, which is not what you want if you are a bank, or if your fixing some piece of software in a plane ... flying at 30.000 feet).
> 0 votes

# Answer
E.g. Under the Docker context...

If you take it literally, you can define 'Build' step to wrap up all the stuff build the actual image. But stage names don't quite follow any name conventions. So build can be literally whatever you want to.
> 0 votes

---
Tags: terminology, builds, software-engineering
---