# Question
Title: Trying to understand sample windows 10 packer template at https://github.com/StefanScherer/packer-windows
I am a beginner to Packer. I am using a template found at https://github.com/StefanScherer/packer-windows for Windows 10, and have been able to create a virtual machine in Hyper-V using the template and my own .ISO.

Now I want to take the next step and add some of my own PowerShell scripts to the process, but I notice this template has multiple references to 'floppy\_files' at various parts of the template, yet they are the same script files repeated? Can someone explain why? Do I need to put my script in each floppy\_files collection? I'd rather understand why before I do that even if it does happen to work. Template shown below for clarity.

```
{
  "builders": [
    {
      "accelerator": "kvm",
      "boot_wait": "6m",
      "communicator": "winrm",
      "cpus": "2",
      "disk_size": "{{user `disk_size`}}",
      "floppy_files": [
        "{{user `autounattend`}}",
        "./floppy/WindowsPowershell.lnk",
        "./floppy/PinTo10.exe",
        "./scripts/fixnetwork.ps1",
        "./scripts/disable-screensaver.ps1",
        "./scripts/disable-winrm.ps1",
        "./scripts/enable-winrm.ps1",
        "./scripts/microsoft-updates.bat",
        "./scripts/win-updates.ps1"
      ],
      "headless": true,
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_url": "{{user `iso_url`}}",
      "memory": "{{user `memory`}}",
      "output_directory": "windows_10-qemu",
      "qemuargs": [
        [
          "-drive",
          "file=windows_10-qemu/{{ .Name }},if=virtio,cache=writeback,discard=ignore,format=qcow2,index=1"
        ],
        [
          "-drive",
          "file={{ user `iso_url` }},media=cdrom,index=2"
        ],
        [
          "-drive",
          "file={{ user `virtio_win_iso` }},media=cdrom,index=3"
        ]
      ],
      "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"",
      "type": "qemu",
      "vm_name": "{{user `vm_name`}}",
      "winrm_password": "vagrant",
      "winrm_timeout": "{{user `winrm_timeout`}}",
      "winrm_username": "vagrant"
    },
    {
      "boot_wait": "6m",
      "communicator": "winrm",
      "configuration_version": "8.0",
      "cpus": "2",
      "disk_size": "{{user `disk_size`}}",
      "floppy_files": [
        "{{user `autounattend`}}",
        "./floppy/WindowsPowershell.lnk",
        "./floppy/PinTo10.exe",
        "./scripts/fixnetwork.ps1",
        "./scripts/disable-screensaver.ps1",
        "./scripts/disable-winrm.ps1",
        "./scripts/enable-winrm.ps1",
        "./scripts/microsoft-updates.bat",
        "./scripts/win-updates.ps1"
      ],
      "guest_additions_mode": "none",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_url": "{{user `iso_url`}}",
      "memory": "{{user `memory`}}",
      "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"",
      "switch_name": "{{user `switch_name`}}",
      "type": "hyperv-iso",
      "vm_name": "{{user `vm_name`}}",
      "winrm_password": "vagrant",
      "winrm_timeout": "{{user `winrm_timeout`}}",
      "winrm_username": "vagrant"
    },
    {
      "boot_command": "",
      "boot_wait": "6m",
      "communicator": "winrm",
      "cpus": 2,
      "disk_adapter_type": "lsisas1068",
      "disk_size": "{{user `disk_size`}}",
      "disk_type_id": "{{user `disk_type_id`}}",
      "floppy_files": [
        "{{user `autounattend`}}",
        "./floppy/WindowsPowershell.lnk",
        "./floppy/PinTo10.exe",
        "./scripts/fixnetwork.ps1",
        "./scripts/disable-screensaver.ps1",
        "./scripts/disable-winrm.ps1",
        "./scripts/enable-winrm.ps1",
        "./scripts/microsoft-updates.bat",
        "./scripts/win-updates.ps1"
      ],
      "guest_os_type": "windows9-64",
      "headless": "{{user `headless`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_url": "{{user `iso_url`}}",
      "memory": "{{user `memory`}}",
      "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"",
      "type": "vmware-iso",
      "version": "{{user `vmx_version`}}",
      "vm_name": "{{user `vm_name`}}",
      "vmx_data": {
        "RemoteDisplay.vnc.enabled": "false",
        "RemoteDisplay.vnc.port": "5900"
      },
      "vmx_remove_ethernet_interfaces": true,
      "vnc_port_max": 5980,
      "vnc_port_min": 5900,
      "winrm_password": "vagrant",
      "winrm_timeout": "{{user `winrm_timeout`}}",
      "winrm_username": "vagrant"
    },
    {
      "boot_command": "",
      "boot_wait": "6m",
      "communicator": "winrm",
      "cpus": 2,
      "disk_size": "{{user `disk_size`}}",
      "floppy_files": [
        "{{user `autounattend`}}",
        "./floppy/WindowsPowershell.lnk",
        "./floppy/PinTo10.exe",
        "./scripts/fixnetwork.ps1",
        "./scripts/disable-screensaver.ps1",
        "./scripts/disable-winrm.ps1",
        "./scripts/enable-winrm.ps1",
        "./scripts/microsoft-updates.bat",
        "./scripts/win-updates.ps1"
      ],
      "guest_additions_mode": "disable",
      "guest_os_type": "Windows10_64",
      "headless": "{{user `headless`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_url": "{{user `iso_url`}}",
      "memory": "{{user `memory`}}",
      "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"",
      "type": "virtualbox-iso",
      "vm_name": "{{user `vm_name`}}",
      "winrm_password": "vagrant",
      "winrm_timeout": "{{user `winrm_timeout`}}",
      "winrm_username": "vagrant"
    },
    {
      "boot_command": "",
      "boot_wait": "6m",
      "communicator": "winrm",
      "cpus": 2,
      "disk_size": "{{user `disk_size`}}",
      "floppy_files": [
        "{{user `autounattend`}}",
        "./floppy/WindowsPowershell.lnk",
        "./floppy/PinTo10.exe",
        "./scripts/fixnetwork.ps1",
        "./scripts/disable-screensaver.ps1",
        "./scripts/disable-winrm.ps1",
        "./scripts/enable-winrm.ps1",
        "./scripts/microsoft-updates.bat",
        "./scripts/win-updates.ps1"
      ],
      "guest_os_type": "win-10",
      "iso_checksum": "{{user `iso_checksum`}}",
      "iso_url": "{{user `iso_url`}}",
      "memory": "{{user `memory`}}",
      "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"",
      "type": "parallels-iso",
      "parallels_tools_flavor": "win",
      "parallels_tools_mode": "disable",
      "prlctl": [
        [
          "set",
          "{{.Name}}",
          "--adaptive-hypervisor",
          "on"
        ],
        [
          "set",
          "{{.Name}}",
          "--efi-boot",
          "off"
        ]
      ],
      "vm_name": "{{user `vm_name`}}",
      "winrm_password": "vagrant",
      "winrm_timeout": "{{user `winrm_timeout`}}",
      "winrm_username": "vagrant"
    }
  ],
  "post-processors": [
    {
      "keep_input_artifact": false,
      "output": "windows_10_{{.Provider}}.box",
      "type": "vagrant",
      "vagrantfile_template": "vagrantfile-windows_10.template"
    }
  ],
  "provisioners": [
    {
      "execute_command": "{{ .Vars }} cmd /c \"{{ .Path }}\"",
      "remote_path": "/tmp/script.bat",
      "scripts": [
        "./scripts/enable-rdp.bat"
      ],
      "type": "windows-shell"
    },
    {
      "scripts": [
        "./scripts/vm-guest-tools.ps1",
        "./scripts/debloat-windows.ps1"
      ],
      "type": "powershell"
    },
    {
      "restart_timeout": "{{user `restart_timeout`}}",
      "type": "windows-restart"
    },
    {
      "scripts": [
        "./scripts/set-powerplan.ps1",
        "./scripts/docker/disable-windows-defender.ps1"
      ],
      "type": "powershell"
    },
    {
      "execute_command": "{{ .Vars }} cmd /c \"{{ .Path }}\"",
      "remote_path": "/tmp/script.bat",
      "scripts": [
        "./scripts/pin-powershell.bat",
        "./scripts/compile-dotnet-assemblies.bat",
        "./scripts/set-winrm-automatic.bat",
        "./scripts/uac-enable.bat",
        "./scripts/dis-updates.bat",
        "./scripts/compact.bat"
      ],
      "type": "windows-shell"
    }
  ],
  "variables": {
    "autounattend": "./answer_files/10/Autounattend.xml",
    "disk_size": "61440",
    "disk_type_id": "1",
    "memory": "2048",
    "headless": "false",
    "iso_checksum": "sha256:668fe1af70c2f7416328aee3a0bb066b12dc6bbd2576f40f812b95741e18bc3a",
    "iso_url": "https://software-download.microsoft.com/download/sg/17763.107.101029-1455.rs5_release_svc_refresh_CLIENT_LTSC_EVAL_x64FRE_en-us.iso",
    "restart_timeout": "5m",
    "vhv_enable": "false",
    "virtio_win_iso": "~/virtio-win.iso",
    "vm_name": "windows_10_pos",
    "winrm_timeout": "6h",
    "vmx_version": "14"
  }
}

```

# Answer
> 1 votes
You're looking at a list of different *builders*. there can be multiple builder entries, and they're indicated by a certain 'type' - see the field named *type*.

In the example from Stefan there are *qemu*, *hyperv-iso*, *vmware-iso*, *virtualbox-iso* and finally *parallels-iso*.

Each of them contains

* a full set of parameters for building a VM "artifact"
* the right parameters for that respective hypervisor.
* the scripts to install the specific guest tools
* the scripts for shared tasks they all need, i.e. the automatic update scripts.

An artifact here is an installed VM in the format of this specific hypervisor / an iso of that / something.

And so that's why you can have some duplication, and as such, that's why *you* probably only need to copy one builder section .

---
Tags: windows, packer, vm
---


# Question
Title: Checking Jenkins build status from only one pipeline and avoid from other pipeline pointing to same gitlab repo in PR build
Hi I am having two jenkins pipeline pointing to same git repo one pipeline is named as **mainpipeline** and other is **sit** pipeline. Before **sit pipeline** our git **PR** build is checking the build status from mainpipeline. But after introducing **sit pipeline** which is pointing to same git repo as mainpipeline, our git build status is start checking from sit pipeline . How should i avoid checking the git build status from the sit pipeline and check only from mainpipeline.

I have suppress the build check status in sit pipeline but sometimes its working and sometimes not below is the screen shot of configuration

# Answer
As far as I can tell from reading the source code, the build status notifications are basically hard-coded into the GitLab Branch Source plugin and cannot be disabled. Therefore, without writing your own plugin or something like that, the only way I can think of to disable the build status notifications would be to use a different method of checking out the repository that does not use the GitLab plugin, e.g. using the `checkout` step with a direct url.
> 1 votes

---
Tags: jenkins-pipeline, gitlab
---


# Question
Title: Should I struggle for minimalistic context in GitLab CI jobs?
I'm designing a build & deploy pipeline for our project. We have two jobs:

* `build` -- builds a docker image with our app and pushes it to a container registry.
* `deploy` -- communicates with an orchestrator, which deploys the new version.

There's one detail that I'm not sure about. Which choice is better:

1. Get the full repository in the `deploy` job, while I need only one static config (deployment spec.).
2. Use artifacts to pass the required config file from `build` job to `deploy` job, and set `GIT_STRATEGY` to `none`.

These two options differ in the `deploy` job environment:

1. The `deploy` has access to the same context as `build` (full repository):

**Pros**:

* The default behavior (requires no extra code and actions for this option to work).

**Cons**:

* The environment in `deploy` job is dirty.
    * The true context of the `deploy` job is implicit.
    * Someone could misuse it in the future.
2. The `build` job publishes an artifact with the required config for the `deploy` job:

**Pros**:

* The `deploy` job explicitly has no access to anything unnecessary.

**Cons**:

* Need to configure the `deploy` job not to fetch the repository.
    * Need to configure the `build` job to publish the artifact.
    * Need to configure the `deploy` job to consume the artifact.
    * The artifact always contains an unchanged file from the repository (I'm afraid that I misuse artifacts in that case).

Please, help me choose.

# Answer
> 2 votes
Your use case is very simple, so relying on `GIT_STRATEGY=fetch` is probably sufficient and as you said the default behavior.

You should be cautious using artifacts unnecessarily as they are uploaded to the gitlab server.

The answer to your question generally depends on a few things, like the size of your repository, network connection, runner executor type, and so on. One option is to configure a cache (in this case at the pipeline level and for the deploy just set the `GIT_STRATEGY=none`:

```
default:
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - deployment.spec.config

build:
  stage: build
  script:
    - build something

deploy:
  stage: deploy
  variables:
    GIT_STRATEGY: none
  script:
    - push-deployment ./deployment.spec.config

```

An added benefit here, depending on your build process, is the possibility of caching your dependencies when running multiple pipelines against the same branch. Note that you can only cache files relative to the project build directory.

---
Tags: continuous-integration, gitlab, deployment, continuous-deployment, build-pipeline
---


# Question
Title: Unexpected type error in ansible community.aws.cloudfront_distribution: "origins" parameter: "'list' object has no attribute 'get'"
I get an unexpected type error, for what I assume to be the "origins" argument to the community.aws.cloudfront\_distribution module: **"Error validating distribution origins: 'list' object has no attribute 'get'"**

I filed a bug report: https://github.com/ansible-collections/community.aws/issues/504

The documentation says that **origins** should be a list with dictionary elements, which is what I have.

Is this a bug, or a syntax error on my part? I've checked my indentation and yaml thoroughly.

The error message:

```
    "msg": "Error validating distribution origins: 'list' object has no attribute 'get'"

```

Ansible version:

```
ansible 2.10.7
  config file = /home/bryant/src/api_guys/ansible-deploy-vst/ansible.cfg
  configured module search path = ['/home/bryant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/bryant/.pyenv/versions/vst-ansible-deployment/lib/python3.9/site-packages/ansible
  executable location = /home/bryant/.pyenv/versions/vst-ansible-deployment/bin/ansible
  python version = 3.9.1 (default, Feb 16 2021, 14:25:16) [GCC 9.3.0]

```

My code, using `community.aws.cloudfront_distribution`: (it's long, but the error is in the **origins:** parameter, I believe)

```
- name: Create Cloudfront distribution for the interface and api reverse proxy
  community.aws.cloudfront_distribution:
    profile: "{{ vst_nongov_boto_profile }}"
    state: present
    enabled: yes
    caller_reference: "{{ cloudfront_domain_alias }}"
    alias: "{{ cloudfront_domain_alias }}"
    aliases:
      - "{{ cloudfront_domain_alias }}"
    origins:
      # reverse proxy for api
      - id: "{{ base_domain }}"
        domain_name: "{{ base_domain }}"
        custom_origin_config:
          http_port: 80
          https_port: 443
          origin_keepalive_timeout: 5
          origin_protocol_policy: "https-only"
          origin_read_timeout: 10
          origin_ssl_protocols:
            - "TLSv1.2"
      # s3 bucket for interface
      - id: "{{ s3_bucket_www_domain_name }}"
        domain_name: "{{ s3_bucket_domain_name }}"
        custom_origin_config:
          http_port: 80
          https_port: 443
          origin_keepalive_timeout: 5
          origin_protocol_policy: "http-only"
          origin_read_timeout: 10
          origin_ssl_protocols:
            - "TLSv1.2"
    default_cache_behavior:
      #   this contains as many settings as possible that are able to be
      # replicated from the UI option caching policy
      # "Managed-CachingOptimized"; some are guesses, which may need to be
      # changed if we notice performance issues
      target_origin_id: "{{ s3_bucket_domain_name }}"
      forwarded_values:
        allowed_methods:
          - GET
          - HEAD
        items:
          - GET
          - HEAD
        compress: yes
        cookies:
          forward: none
        default_ttl: 86400
        max_ttl: 31536000
        min_ttl: 1
        query_string: no
        smooth_streaming: no
        trusted_signers:
          enabled: no
        viewer_protocol_policy: "redirect-to-https"
    cache_behaviors:
      - path_pattern: "api/*"
        target_origin_id: "{{ base_domain }}"
        forwarded_values:
          allowed_methods:
            - GET
            - HEAD
          items:
            - GET
            - HEAD
            - OPTIONS
            - PUT
            - POST
            - PATCH
            - DELETE
          compress: no
          cookies:
            forward: all
          default_ttl: 0
          headers:
            - Authorization
            - Referer
            - User-Agent
          max_ttl: 0
          min_ttl: 0
          query_string: yes
          query_string_cache_keys:
            - '*'
          smooth_streaming: no
          trusted_signers:
            enabled: no
          viewer_protocol_policy: "redirect-to-https"
      - path_pattern: "testapi*"
        target_origin_id: "{{ base_domain }}"
        forwarded_values:
          allowed_methods:
            - GET
            - HEAD
          items:
            - GET
            - HEAD
          compress: no
          cookies:
            forward: all
          default_ttl: 0
          headers:
            - Authorization
            - User-Agent
          max_ttl: 0
          min_ttl: 0
          query_string: yes
          query_string_cache_keys:
            - '*'
          smooth_streaming: no
          trusted_signers:
            enabled: no
          viewer_protocol_policy: "redirect-to-https"
    custom_error_responses:
      - error_caching_min_ttl: 10
        error_code: 403
        response_code: 200
        response_page_path: "/index.html"
      - error_caching_min_ttl: 10
        error_code: 404
        response_code: 200
        response_page_path: "/index.html"
    viewer_certificate:
      acm_certificate_arn: "{{ nongov_acm_cert_upload_result.certificate.arn }}"
      cloudfront_default_certificate: no
      minimum_protocol_version: "TLSv1.2_2019"
      ssl_support_method: "sni-only"
    http_version: http2
    ipv6_enabled: no
    tags:
      Name: "{{ generic_name_tag }}"
    wait: yes
    default_root_object: "index.html"
    price_class: "{{ cloudfront_price_class }}"
    comment: "Cloudfront distribution for {{ base_interface_domain }} interface and API reverse proxy"
  notify: primary_cloudfront_invalidation
  tags: cloudfront

```

The full output, including the error:

```
TASK [aws_full_deployment : Create Cloudfront distribution for the interface and api reverse proxy] ***
task path: /home/bryant/src/api_guys/ansible-deploy-vst/roles/aws_full_deployment/tasks/main.yml:501
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: bryant
<127.0.0.1> EXEC /bin/sh -c 'echo ~bryant && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/bryant/.ansible/tmp `"&& mkdir "` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" && echo ansible-tmp-1616796020.2343066-177081-217806658917877="` echo /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877 `" ) && sleep 0'
Using module file /home/bryant/.ansible/collections/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py
<127.0.0.1> PUT /home/bryant/.ansible/tmp/ansible-local-175967cbvvbt8u/tmpamrug82t TO /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/bryant/.pyenv/versions/vst-ansible-deployment/bin/python /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/AnsiballZ_cloudfront_distribution.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/bryant/.ansible/tmp/ansible-tmp-1616796020.2343066-177081-217806658917877/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1665, in validate_origins
  File "/tmp/ansible_community.aws.cloudfront_distribution_payload_vqd_hi9m/ansible_community.aws.cloudfront_distribution_payload.zip/ansible_collections/community/aws/plugins/modules/cloudfront_distribution.py", line 1729, in validate_origin
AttributeError: 'list' object has no attribute 'get'
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.16.25",
    "botocore_version": "1.19.25",
    "changed": false,
    "invocation": {
        "module_args": {
            "alias": "v6-full-deployment.icloudcms.com",
            "aliases": [
                "v6-full-deployment.icloudcms.com"
            ],
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "cache_behaviors": [
                {
                    "forwarded_values": {
                        "allowed_methods": [
                            "GET",
                            "HEAD"
                        ],
                        "compress": false,
                        "cookies": {
                            "forward": "all"
                        },
                        "default_ttl": 0,
                        "headers": [
                            "Authorization",
                            "Referer",
                            "User-Agent"
                        ],
                        "items": [
                            "GET",
                            "HEAD",
                            "OPTIONS",
                            "PUT",
                            "POST",
                            "PATCH",
                            "DELETE"
                        ],
                        "max_ttl": 0,
                        "min_ttl": 0,
                        "query_string": true,
                        "query_string_cache_keys": [
                            "*"
                        ],
                        "smooth_streaming": false,
                        "trusted_signers": {
                            "enabled": false
                        },
                        "viewer_protocol_policy": "redirect-to-https"
                    },
                    "path_pattern": "api/*",
                    "target_origin_id": "api.v6-full-deployment.icloudcms.com"
                },
                {
                    "forwarded_values": {
                        "allowed_methods": [
                            "GET",
                            "HEAD"
                        ],
                        "compress": false,
                        "cookies": {
                            "forward": "all"
                        },
                        "default_ttl": 0,
                        "headers": [
                            "Authorization",
                            "User-Agent"
                        ],
                        "items": [
                            "GET",
                            "HEAD"
                        ],
                        "max_ttl": 0,
                        "min_ttl": 0,
                        "query_string": true,
                        "query_string_cache_keys": [
                            "*"
                        ],
                        "smooth_streaming": false,
                        "trusted_signers": {
                            "enabled": false
                        },
                        "viewer_protocol_policy": "redirect-to-https"
                    },
                    "path_pattern": "testapi*",
                    "target_origin_id": "api.v6-full-deployment.icloudcms.com"
                }
            ],
            "caller_reference": "v6-full-deployment.icloudcms.com",
            "comment": "Cloudfront distribution for v6-full-deployment.icloudcms.com interface and API reverse proxy",
            "custom_error_responses": [
                {
                    "error_caching_min_ttl": 10,
                    "error_code": 403,
                    "response_code": 200,
                    "response_page_path": "/index.html"
                },
                {
                    "error_caching_min_ttl": 10,
                    "error_code": 404,
                    "response_code": 200,
                    "response_page_path": "/index.html"
                }
            ],
            "debug_botocore_endpoint_logs": false,
            "default_cache_behavior": {
                "forwarded_values": {
                    "allowed_methods": [
                        "GET",
                        "HEAD"
                    ],
                    "compress": true,
                    "cookies": {
                        "forward": "none"
                    },
                    "default_ttl": 86400,
                    "items": [
                        "GET",
                        "HEAD"
                    ],
                    "max_ttl": 31536000,
                    "min_ttl": 1,
                    "query_string": false,
                    "smooth_streaming": false,
                    "trusted_signers": {
                        "enabled": false
                    },
                    "viewer_protocol_policy": "redirect-to-https"
                },
                "target_origin_id": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com"
            },
            "default_origin_domain_name": null,
            "default_origin_path": null,
            "default_root_object": "index.html",
            "distribution_id": null,
            "e_tag": null,
            "ec2_url": null,
            "enabled": true,
            "http_version": "http2",
            "ipv6_enabled": false,
            "logging": null,
            "origins": [
                {
                    "custom_headers": {
                        "quantity": 0
                    },
                    "custom_origin_config": {
                        "h_t_t_p_port": 80,
                        "h_t_t_p_s_port": 443,
                        "origin_keepalive_timeout": 5,
                        "origin_protocol_policy": "https-only",
                        "origin_read_timeout": 10,
                        "origin_ssl_protocols": [
                            "TLSv1.2"
                        ]
                    },
                    "domain_name": "api.v6-full-deployment.icloudcms.com",
                    "id": "api.v6-full-deployment.icloudcms.com",
                    "origin_path": ""
                },
                {
                    "custom_origin_config": {
                        "http_port": 80,
                        "https_port": 443,
                        "origin_keepalive_timeout": 5,
                        "origin_protocol_policy": "http-only",
                        "origin_read_timeout": 10,
                        "origin_ssl_protocols": [
                            "TLSv1.2"
                        ]
                    },
                    "domain_name": "v6-full-deployment-s3-bucket.s3-website-us-gov-west-1.amazonaws.com",
                    "id": "v6-full-deployment-s3-bucket-www.s3-website-us-gov-west-1.amazonaws.com"
                }
            ],
            "price_class": "PriceClass_100",
            "profile": "vst_nongov",
            "purge_aliases": false,
            "purge_cache_behaviors": false,
            "purge_custom_error_responses": false,
            "purge_origins": false,
            "purge_tags": false,
            "region": null,
            "restrictions": null,
            "security_token": null,
            "state": "present",
            "tags": {
                "Name": "v6-full-deployment"
            },
            "validate_certs": true,
            "viewer_certificate": {
                "acm_certificate_arn": "xxxxxxxxxxOBFUSCATED",
                "cloudfront_default_certificate": false,
                "minimum_protocol_version": "TLSv1.2_2019",
                "ssl_support_method": "sni-only"
            },
            "wait": true,
            "wait_timeout": 1800,
            "web_acl_id": null
        }
    },
    "msg": "Error validating distribution origins: 'list' object has no attribute 'get'"
}

```

# Answer
It's a bug; `origin_ssl_protocols` is defined as a list here, then the code tries to treat it as a dict here (which is the line referenced in your stack trace.)

As a workaround, it looks like the code might accept

```
          origin_ssl_protocols:
            items:
             - "TLSv1.2"

```
> 2 votes

---
Tags: amazon-web-services, ansible, amazon-cloudfront
---


# Question
Title: One azure-pipeline.yaml for staging and production
Currently I have two separate Azure DevOps Pipeline config files:

* `azure-pipelines-staging.yaml`
* `azure-pipelines-production.yaml`

Really the only differences in them are the:

```
trigger:
  branches:
    include:
    - master # or staging

```

And the directory where my k8s manifests are:

```
manifests: |
  $(Pipeline.Workspace)/manifests/azure/prod/api.yaml

# or

manifests: |
  $(Pipeline.Workspace)/manifests/azure/staging/api.yaml

```

So ideally it would be something like:

```
trigger:
  branches:
    include:
    - master
    - staging
...
manifests: |
  $(Pipeline.Workspace)/manifests/azure/$(branch)/api.yaml

```

Not quite seeing how to get `$(branch)` name in there. Suggestions?

# Answer
> 1 votes
Ok, found it here:

```
Build.SourceBranch

The branch of the triggering repo the build was queued for. Some examples:

 - Git repo branch: refs/heads/master
 - Git repo pull request: refs/pull/1/merge
 - TFVC repo branch: $/teamproject/main
 - TFVC repo gated check-in: Gated_2016-06-06_05.20.51.4369;username@live.com
 - TFVC repo shelveset build: myshelveset;username@live.com

When your pipeline is triggered by a tag: refs/tags/your-tag-name
When you use this variable in your build number format, the forward slash characters (/) are replaced with underscore characters _).

Note: In TFVC, if you are running a gated check-in build or manually building a shelveset, you cannot use this variable in your build number format.

```

# Answer
> 0 votes
If your after the branch name, and your branches don't include "directories" (example: releases/my\_release), then `Build.SourceBranchName` will you give you "The last path segment in the ref". So, if `Build.SourceBranch` is `refs/heads/master`, then Build.SourceBranch will be `master`.

Your YAML:

```
trigger:
  branches:
    include:
    - master
    - staging
...
manifests: |
  $(Pipeline.Workspace)/manifests/azure/${{ variables.Build.SourceBranchName }}/api.yaml

```

---
Tags: azure-devops, azure, azure-pipelines, branch
---


# Question
Title: Azure Pipeline PowerShell@2 Task can't run 'invoke-build' command as inline script (build PowerShell Module)
I'm trying to build a PowerShell module in an Azure pipeline (dev.azure.com).

I've setup an initial pipeline and then building it up incrementally in small steps, because inexplicable errors are encountered far too easily on Azure.

I've added a 'Prepare' stage:

```
name: $(Build.DefinitionName)_$(Date:yyyyMMdd))

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  major: 0
  minor: 0
  patch: $(Build.BuildID)
  buildVersion: $(major).$(minor).$(patch)

stages:
- stage: Prepare
  jobs:
    - job: Prepare
      steps:
      - powershell: .\bootstrap.ps1
        displayName: 'Install pre-requisites'

```

This is based on the original yaml that you get by default when you start a new Pipeline, so the part invoking the bootstrap file is what I got generated for free (This part works ok).

The bootstrap uses a dependencies file into which you can declare your dependencies and these are subsequently imported. One of those dependencies is InvokeBuild which contains the Invoke-Build command. So I would expect to be able to use the Invoke-Build command in subsequent stages.

The next part of the yaml pipeline is as follows:

```
- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'invoke-build build'

```

The *invoke-build build*, in itself is ok and valid and works on a local host.

The *Build* job fails with the following error:

```
/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/edb3758b-8e16-448d-bea2-9ba18acb9693.ps1'
Invoke-Build: /home/vsts/work/1/s/do-build.ps1:3
Line |
   3 |  Invoke-Build build
     |  ~~~~~~~~~~~~
     | The term 'Invoke-Build' is not recognized as the name of a
     | cmdlet, function, script file, or operable program. Check the
     | spelling of the name, or if a path was included, verify that
     | the path is correct and try again.

```

I discovered there is another way to invoke the build with the PowerShell@2 task using the *filePath* input:

```
- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/do-build.ps1'

```

the do-build.ps1 script is defined as:

```
Set-Location .\Elizium.FakeBuddy
Invoke-Build build

```

(Elizium.FakeBuddy is the name of my PowerShell test module being built and that directory contains the *InvokeBuild* build script)

but this fails with the exact same error.

So why is the *invoke-build* command not available in the PowerShell session at the build stage, even though the Prepare stage has already imported the module?

The only thing I can think of is that different stages somehow don't share the same PowerShell session, but I don't know if this is correct, this seems unlikely, it doesn't make sense that different stages don't share the same session.

EDIT: I tried an alternative version where the Build job is declared inside the same stage as Prepare, but this makes no difference, still a problem with Invoke-Build command.

# Answer
> 1 votes
I'm not familiar with Invoke-Build, but appearantly it's a part of the InvokeBuild PowerShell module? In that case, you'll have to install the module as part of the script where you're using it. Your YAML should be something like this:

```
- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: |
              Install-Module InvokeBuild
              Invoke-Build build

```

As @Michael Erpenbeck said "the hosted agent starts off as a clean slate". That is to say that it starts off as a clean slate for each job in your pipeline. I'm not sure what bootstrap.ps1 is supposed to do so I can't really comment on whether that script and your `Invoke-Build` should be run as steps in the same job. But the main point to remember is that any requirements beyond what the standard Azure Pipelines agents offer should be taken care off within each job.

---
Tags: azure-devops, azure, azure-pipelines, powershell
---


# Question
Title: Multi-Stage Azure Pipeline with a path trigger - first stage always runs
I have a multi-stage pipeline for my terraform code. This is my first pipeline so I am sure I'm missing something but there appears to be some inconsistent behaviour that is rather frustrating.

The pipeline trigger is defined as follows:

```
trigger:
  branches:
    include:
      - master
  paths:
    include:
      - terraform/*

```

My repo has the following folder structure:

```
IaC
 |
 +-- terraform
 |    |
 |    +-- main.tf
 |
 |
 +-- azure-pipeline.yml
 +-- readme.md

```

My two stages are defined as follows (I cut out the job steps, its just tarraform commands):

```
stages:
  - stage: plan
    displayName: 'terraform plan'
    condition: eq(variables['Build.Reason'], 'PullRequest')
    ...
  - stage: apply
    displayName: 'terraform apply'
    condition: ne(variables['Build.Reason'], 'PullRequest')
    ...

```

Whenever I update code in the terraform directory and create a PR, the pipeline works exactly as expected. It does the terraform plan for the reviewer to see and once the PR is approved, the second stage fires and it does a terraform apply.

The issue I am having is whenever I update a file outside the terraform directory and create a PR, the first stage still runs! Oddly, once the PR is approved, the second stage does not.

Are my stage conditions to simple? Does a multi-stage pipeline always have to do something?

# Answer
Your pipeline runs for pull requests not because of the trigger, but because of a build validation policy you've set for one or more target branches ("whenever I create a PR to this branch, start this pipeline"). It's the build validation, as part of a branch's policies: https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops#build-validation

You probably didn't specify a path filter in the build validation settings. Here's an example of three different build validations in a single branch policy, all with different path filters:

In short: The path trigger in your YAML has no effect on how a pipeline is triggered for a PR. The trigger you've specified is one that triggers on commits to the master branch, if the commit includes files in the terraform directory.
> 2 votes

---
Tags: azure-pipelines
---


# Question
Title: Easiest way to manage an elasticsearch cluster in GCP?
I'm trying to create my own Elasticsearch cluster to use for security analytics. I'm familiar with the function and setup of a single instance deployment, but I'm having trouble figuring out the simplest strategy to set up and manage the cluster of any sizable scale in GCP. The two ways I've played around with are:

1. Building the ES cluster using Compute Engine VM's in GCP.
2. Managing the cluster using the Kubernetes Engine and the official Elastic docker image.

Since I don't have much experience with containerization, I'm trying to figure out if learning and managing a kubernetes deployment would save me some headache in the long run.

# Answer
We have been using elastic/helm-charts for more than a year for our product, and it is easy to manage and scale on-demand. Let me know if you need any help in deploying the helm chart.
> 1 votes

# Answer
One could use helm to install elastic on managed k8s in GCP.
> 0 votes

---
Tags: kubernetes, deployment, google-cloud-platform, elastic
---


# Question
Title: Connect to RDS Private Instance from Docker Container
I am running a docker swarm cluster and I was trying to connect to a private RDS instance within the same VPC from a docker container which is running in an EC2 within the same VPC but I get an error like **"unknown host"**. I have enabled the database port number(3306) in RDS security group for my EC2 instances.

However, I am able to connect to the database from the host level but not from the container level.

# Answer
As per the error, it looks like the resolution is not happening for host `xxxxx-xxxxx-xxxx.xxxxxxxxx.us-east-2.rds.amazonaws.com'`. I mean the hostname is not getting resolved to the IP address from inside the container.

You may need to check if the DNS resolver is working properly from inside the container or not. For testing, you can try to use the IP instead of the RDS endpoint hostname and see if it works.
> 1 votes

# Answer
I had to enable both DNS Hostname and DNS resolution under the AWS VPC settings.

Sorry for answering late.
> 0 votes

---
Tags: docker, amazon-rds
---


# Question
Title: Azure deploy pipeline with/without approval on separated enviroments
I have two separate environments under my repository:

* development (Approvals are not configured but the environment exists)
* production (Approvals rules are configured via dev.azure.com UI)

The problem is, even we try to deploy to the development environment, the pipeline asking for approval. How we can fix the issue? Below, you can see simple two deploy jobs from YAML:

```
jobs:
    - deployment: Deploy_Dev
      displayName: 'Deploy Develop to Dev Machine'
      pool:
        name: Development
      condition: contains(variables['Build.SourceBranch'], 'refs/heads/develop')
      environment: 'development'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                    echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false

    - deployment: Deploy_Master_to_Hotfix
      displayName: 'Deploy Master to Hotfix Machine'
      pool:
        name: Hotfix
      condition: contains(variables['Build.SourceBranch'], 'refs/heads/master')
      environment: 'production'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                      echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false

```

# Answer
> 1 votes
Environment approvals work per-stage. Split your jobs into separate stages:

```
stages:
  - stage: development
    condition: contains(variables['Build.SourceBranch'], 'refs/heads/develop')
    jobs:
    - deployment: Deploy_Dev
      displayName: 'Deploy Develop to Dev Machine'
      pool:
        name: Development
      environment: 'development'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                    echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false

  - stage: production
    condition: contains(variables['Build.SourceBranch'], 'refs/heads/master')
    jobs:
    - deployment: Deploy_Master_to_Hotfix
      displayName: 'Deploy Master to Hotfix Machine'
      pool:
        name: Hotfix
      environment: 'production'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                      echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false

```

---
Tags: git, azure-devops, deployment, azure, azure-pipelines
---


# Question
Title: How to build, package and deploy a react-native-web app in a Docker container?
Given there is a react-native-web solution, what needs to be done to build it and run inside a Docker container?

* What is the build environment/toolchain?
* Which environment is required to run?

From what I can find so far, it's not clear how this should work.

UPDATE: Project examples.

# Answer
It's a pretty general question, I will try to answer the best I can: If you want to just build and run an image: Build the project: npx expo-cli init react\_native\_app --npm

1. You need to create a Dockerfile file in your repository
2. Docker build -t {tag\_name} . (You can replace '.' eith the Dockerfile path)
3. Docker run...
4. If you want you can create docker compose file to better control your app. Reference: https://www.rockyourcode.com/how-to-run-react-native-expo-web-in-a-docker-container/
> 1 votes

---
Tags: docker, containers, cicd, software-engineering
---


# Question
Title: Enable TLSv1 TLSv1.1 for specific Ingress with Nginx Ingress
I try to enable TLSv1 and TLSv1.1 for one specific ingress on a AKS cluster.

By default, only TLSv1.2 is enabled.

I don't want to apply this configuration on my all cluster, just on a specific webserver service.

Regarding the documentation, I need to add this annotation if I don't want to change nginx config by ConfigMap :

`nginx.ingress.kubernetes.io/proxy-ssl-protocols: TLSv1 TLSv1.1 TLSv1.2`

but this has no effect.

Is it really possible to do this by annotation ?

Did I do something wrong ?

Thank you !

# Answer
> 2 votes
The ConfigMap and Annotation configuration have different purposes.

As per nginx-configuration/configmap/#ssl-protocols:

> Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3
>
> ngx\_http\_ssl\_module.html#ssl\_protocols
> Context: http, server

i.e., enable the specified protocols **to your Nginx Controller**.

`client --ssl-protocols-> controller --> backend`

---

On nginx-configuration/annotations/#backend-certificate-authentication, we can read:

> Enables the specified protocols for requests to a proxied HTTPS server.
>
> ngx\_http\_proxy\_module.html#proxy\_ssl\_protocols http, server, location

i.e., enables the specified protocols for requests **to a proxied backend using HTTPS**.

`client --> controller --proxy-ssl-protocols-> backend`

---

As you can see, to set the protocols served by your Nginx controller, you only have `http, server` contexts available (i.e., you can't set this to specific ingress objects).

---
Tags: kubernetes, nginx, ingress, azure-kubernetes-service
---


# Question
Title: Testing docker image builds when low level libraries change
**Background**

In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:

* `myorg/python-lib1`: produces package `myorg.lib1`
* `myorg/python-lib2`: produces package `myorg.lib2`
* `myorg/python-app`: produces a package `myorg.app`, depends on `myorg.lib1` and `myorg.lib2`

In our CI for `myorg/python-lib1` and `myorg/python-lib1` we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.

In our CI for `myorg/python-app1` we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:

```
# Builds image reistry.myorg.com/python-app
FROM reistry.myorg.com/python-base:latest

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

```

***Note**: We also have some projects using build-packs to achieve a similar goal.*

---

**Issue**

Because we only run the integration tests for `myorg/python-app1`, we can't get as much confidence when a change to `myorg/python-lib1` and `myorg/python-lib2` happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.

I'm looking to restructure the way we build images in `myorg/python-app1` (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.

I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.

# Answer
> 1 votes
So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.

Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.

Steps to make this work:

**1. Base image from ARG**

I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:

```
# Builds image reistry.myorg.com/python-app
ARG BASE=reistry.myorg.com/python-base:latest
FROM $BASE

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

```

**2. Upload test versions of library**

For each python change:

* Clone the repo
* `python -m build` the repo
* `devpi upload` the wheels

**3. Create a layer with devpi pip conf**

Make a pip conf that can access your devpi instance and configure to your index name:

```
[global]
index-url = http://<some ip>:<some port>/root/dev/+simple/
trusted-host = <some ip>
disable-pip-version-check = true

[search]
index = http://<some ip>:<some port>/root/dev/

```

And add that to your base images pip conf:

```
FROM reistry.myorg.com/python-base:latest

COPY pip.conf /etc/pip.conf

```

**4. Chain the images together**

```
docker build -t devpi-base -f devpi.Dockerfile .
docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
docker tag my-app reistry.myorg.com/my-app:test1
docker push reistry.myorg.com/my-app:test1

```

---

Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!

---
Tags: docker, continuous-integration, containers, python, testing
---


title: Octopus server installation on Linux

text:
# Question
Title: Octopus server installation on Linux
I learnt that,

Octopus server can be installed on Windows 2008, 2012, 2016 server editions

I see msi packages on Download page

---

Can Octopus server be installed on Ubuntu?

Can tentacle software be installed on Linux box?

# Answer
As of 2023-04-16 this answer is out of date, Diego Marinelli's answer should be accepted as correct.

## Original Answer

~~Historically Octopus Deploy was a solution to deploy software on Windows only, therefore the server component only supports installation on Windows.~~

~~This is for a couple of technical reasons:<br>1. Octopus Server takes a hard dependency on Internet Information Services (IIS) which itself only runs on Windows.
2. Octopus Server is written in the .NET Framework doesn't run on Linux.~~

~~Once you have the Octopus Deploy Server software installed, you can manage Linux hosts via SSH, you don't need tentacles to manage Linux hosts.~~
> 2 votes

# Answer
The document, Octopus Server Linux Container provides detailed instructions on how to operate Octopus Server on Linux using an image provided by Octopus Deploy on Docker Hub:

> Octopus Server Linux Containers launched as part of 2020.6 and it's our recommended way to use containers with Octopus Deploy.
> 4 votes

---
Tags: deployment, continuous-deployment, octopus-deploy
---


# Question
Title: kubernetes nodeport service load balancing
Let's assume I have three node named `n1`, `n2`, `n3`. created four pods named `p1`, `p2`, `p3`, `p4`, `k8s` scheduler placed pods in nodes as follows

n1 -\> p1

n2 -\> p2

n3 -\> p3, p4

Now, to access these pods I have created a `nodeport` service. `Nodeport` service will open same numbered ports in each nodes. Nodes will have `iptables` configured to forward traffics from service to pods.

Let's also assume that I will have a NLB configured top of it. how traffic will be distributed? is it like below?

p1 -\> 33.3330%

p2 -\> 33.3330%

p3 -\> 16.6665%

p4 -\> 16.6665%

# Answer
> 1 votes
`service` is a cluster-wide object. In each node, `kube-proxy` will create `iptable` load balancing entry with all available pod's IP based on selector for a `service`. So NLB will distributed traffic evenly among all nodes. Then each will loadbalance the traffic to all available pods using round robin algorithm.

We will have those traffic distributed among all pods like below.

p1 -\> 33.3330%

p2 -\> 33.3330%

p3 -\> 16.6665%

p4 -\> 16.6665%

This will deviate a little bit.

---
Tags: kubernetes, microservices
---


# Question
Title: Sprints section shows tasks as unparented
I am not sure if this has been answered before, I have looked all over Google and can't seem to find any real answer. Apparently Microsoft allows you to view parent items to tasks in the Sprints section, and I've seen this done at my workplace when we view the Taskboard.

In my own personal DevOps configuration, I've made a custom work item type called "Goal", and assigned "Task" items as children to these "Goals". However, when viewing the Taskboard, all of my tasks show as "Unparented". My goals and tasks are on the same area path and iteration path.

In my Backlog levels settings, I've created a top level portfolio backlog named "Goals" that only uses the "Goals" work item type. When I change the Iteration backlog to include "Goal" work items, it puts the work item type on the same iteration as "Task" work items, and the Taskboard does not properly populate.

Is there something I'm missing? I've tried to read multiple documentation pages on Microsoft's website to no avail.

https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/resolve-backlog-reorder-issues?view=azure-devops

https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/set-up-your-backlog?view=azure-devops#taskboard-types

https://docs.microsoft.com/en-us/azure/devops/organizations/settings/select-backlog-navigation-levels?view=azure-devops

https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/organize-backlog?view=azure-devops

https://docs.microsoft.com/en-us/azure/devops/organizations/settings/work/customize-process-backlogs-boards?view=azure-devops&viewFallbackFrom=vsts#add-or-edit-portfolio-backlogs

# Answer
I was able to contact DevOps Support and get an answer. The issue was that I could only create Portfolio Backlogs, not Requirement Backlogs. By changing the default Requirement Backlog to include my custom work item, those showed in the Sprints section.
> 2 votes

---
Tags: azure-devops
---


# Question
Title: How to get information about resources in an Azure Pipelines run?
I work in a platform team that builds and supports Azure Pipelines YAML templates and various custom scripting for these pipelines. People in my organisation use these YAML templates like:

```
resources:
  repositories:
    - repository: templates
      type: git
      name: PROJECT/cicd_stuff
      ref: refs/heads/releases/v0.21
extends:
  template: our_awesome_template.yml@templates
  parameters:
    ...

```

In order to reliably match the version of our scripts to the version of the YAML templates, we now include an extra build stage in our pipeline templates that checks out the repo `templates` repo and puts all our scripts in an artifact to be used in that run. Using "release branches" allows us to safely put out and test new features and fixes to our pipelines: Teams can upgrade their pipeline on their own pace, with older version remaining supported for a while.

I'd like to start using Azure Artifacts for our script artifacts, but now I'm wondering "how can I determine which version of my scripts I should be downloading from Azure Artifacts?" The information included in the `resources` block would work for me, but I can't seem to access that with an expression or with a predefined variable. The only solution I can currently think of is to use the az pipelines cli. Any thoughts?

# Answer
You can do:

```
variables:
  templates.ref: $[ resources.repositories['templates'].ref ]

```

See Repository Details docs.
> 4 votes

---
Tags: azure-devops, azure-pipelines, yaml
---


# Question
Title: Watch folder, when direct changes are made push to repo
So I have an azure devops repo and pipeline that copys the repo to my server when changes are pushed. However there are times when some of our developers may make a direct edit to the production server. Is there a way for me to have the azure agent that is installed on the server watch the files for any additions/changes and if/when this happens it pushes the files to the repo?

I have tried to find a solution for this but have been unsuccessful.

# Answer
First, I'd highly recommend not doing this - a better approach would be to have a single source of truth (that is your repo). Force developers to commit & push changes and let the pipeline do the rest. If it's not in repo, it didn't happen.

If you absolutely have to, then I'd go with "brute-force" way - have a pipeline that triggers periodically (every couple of hours or so) and does a diff of server files against repo files. If something changed, send a threat email to the developer who did this sync back to repo.
> 1 votes

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Why do companies use JFrog Artifactory or other repositories instead of Maven central repository?
I have usually seen that when someone uses Maven on a company network to download dependencies, then user is redirected to the JFrog Artifactory or any other, instead of downloading dependencies from official Maven Central Repository.

What's the benefits to companies/organisations using other Repositories like JFrog?

# Answer
> 5 votes
1. Saved bandwith and faster downloads: Artifactory stores the artifacts that are downloaded from maven central. So if another developer needs the same dependencies they don't need to be downloaded again from maven central but instead they can be delivered from the local artifactory instance. This makes downloading faster because company networks are usualy faster then the internet connection and it also saves bandwith.
As example imagine that a developer adds a new dependency to a project. This dependency then needs to be downloaded into the local maven cache of every other developer that is working on the same project. On large projects with a lot of developers this means that the dependency might be downloaded 30 times.
2. Stable builds: Dependencys might be removed from maven central. If you have them in your local artifactory instance the project who depends on it can still be build.
3. Easier maintenance: Another benefit that has nothing to do with maven: Artifactory provides several types of repositories like for Docker or Vagrant. So instead of setting up a maven artifacts repository, a docker registry and some kind of vagrant store you could create three repositories in artifactory. Thus you only need to maintain one application (artifactory) instead of three separate applications.

Some or all of these benefits might also apply to other binary repository managers but as I only know artifactory I can't say something about that.

---
Tags: build-pipeline, builds, maven, artifactory, dependency-management
---


# Question
Title: How do I make NFS volumes reliable in Docker?
I host various docker containers on my Ubuntu 18 machine. A few of them require storing their data on my Synology NAS. At first, I was using the host machine's `/etc/fstab` to control NFS mounts, which I then mounted in the containers (as a mount, not a volume).

However, I figured it would be a better idea to have the containers map NFS to mount points in the containers. The host really has no use for the mounts, so it didn't make sense to maintain them there.

At the moment, I am configuring my NFS volumes like this (using Docker Compose v3 format):

```
volumes:
  data:
    driver_opts:
      type: nfs
      o: addr=192.168.1.51,nolock,soft,rw
      device: :/volume2/nextcloud

```

This works great when the NAS is booted and working normally. However, I had a power outage and noticed all sorts of problems. Also, the timing of which machines boot first (NAS vs Ubuntu box) affects reliability of my docker container volumes. In my last situation, the NAS was not powered on. So when the container was started, it failed:

> ERROR: for app Cannot start service app: error while mounting volume '/var/lib/docker/volumes/nextcloud\_data/\_data': error while mounting volume with options: type='nfs' device=':/volume2/nextcloud' o='addr=192.168.1.51,nolock,soft,rw': no route to host

What would be nice is if docker would keep retrying to mount the volume until the NAS was powered on again. That would make this hands-free and prevent any timing issues (on which devices boot first across the network) from causing permanent failures like this.

I'm also not sure what happens if a volume is created, and the NAS is powered off at any point. Does the volume stay available? Does docker keep trying to reconnect the NFS mount? I feel like there is very little control here.

Note I just use Docker Compose. I don't use Swarm for technical reasons I won't go into here. Can someone recommend a way to resolve these reliability issues? Are NFS volumes in Docker the way to go? Should I go back to mounting on the host? Is there an amazing 3rd option out there somewhere?

# Answer
> 2 votes
There are options in docker that will retry the startup of a container, restart in version 2 and restart-policy in version 3 (you'd need compatibility mode enabled for the version 3 syntax to work). However, I believe they only work when the issue is from the application inside the container fails, not when there is an issue creating the container like you see with a volume mount failing (or would also happen if the image couldn't be retrieved from a registry).

To handle the failing volume mount, I believe swarm mode is your best option despite your objections. You can create a single node cluster with `docker swarm init` and deploy your compose file with `docker stack deploy -c docker-compose.yml stack_name`, making an easy transition from docker-compose. Swarm mode looks at the overall state of the service and continuously tries to make the current state match your target state (as defined in the compose file), which will handle a failing volume mount that eventually corrects itself. I don't have an NFS server to test on right now, but here's a scenario with a bind mount to a missing folder:

```
$ cat docker-compose.vol-bind.yml
version: '3'
volumes:
  bind-test:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /home/bmitch/data/docker/test/missing
services:
  bind-test:
    image: busybox
    command: tail -f /dev/null
    volumes:
      - bind-test:/bind-test

$ docker stack deploy -c docker-compose.vol-bind.yml voltest
Creating network voltest_default
Creating service voltest_bind-test

$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                             PORTS
omzaeo7mrour        voltest_bind-test      replicated          0/1                 busybox:latest

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
kpz0l79eucaw        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Ready               Ready 2 seconds ago
j6fylzhvcv60         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 5 seconds ago    "starting container failed: er…"
61o6raohp0xl         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 12 seconds ago   "starting container failed: er…"

$ docker inspect kpz0l79eucaw
[
    {
        "ID": "kpz0l79eucaw1856obmwvcak1",
        "Version": {
            "Index": 445
        },
        "CreatedAt": "2019-04-29T12:57:25.925788528Z",
        "UpdatedAt": "2019-04-29T12:57:34.3467203Z",
        "Labels": {},
        "Spec": {
            "ContainerSpec": {
                "Image": "busybox:latest",
                "Labels": {
                    "com.docker.stack.namespace": "voltest"
                },
...
        "Status": {
            "Timestamp": "2019-04-29T12:57:33.936048295Z",
            "State": "failed",
            "Message": "starting",
            "Err": "starting container failed: error while mounting volume '/home/var-docker/volumes/voltest_bind-test/_data': failed to mount local volume: mount /home
/bmitch/data/docker/test/missing:/home/var-docker/volumes/voltest_bind-test/_data, flags: 0x1000: no such file or directory",
            "ContainerStatus": {
                "ContainerID": "4c75c851bd43b5ef57b4785a4611f78501279933822eab90c693e1108f53ee82",
                "PID": 0,
                "ExitCode": 128
            },
            "PortStatus": {}
        },
...

$ mkdir missing

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
okylli7smu1c        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Running             Running 4 seconds ago
9tpm188ysu2k         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 14 seconds ago   "starting container failed: er…"
kpz0l79eucaw         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 21 seconds ago   "starting container failed: er…"

```

From the above, you can see as soon as the missing directory was created, the bind mount succeeded, and the container that failed to be created as retried and started successfully.

# Answer
> 0 votes
I know the question is old, but as I assume others also will run into this requirement...

As OP is bound to this exactly host anyway (reason to use docker-copose over swarm), I suggest another route.

NFS on Synology is a PITA. One has to manage UIDs to be the same on server and client. It's impossible to change UIDs on Synology, hence only client side UIDs can be adopted to fit server UIDs, which can cause ID clashes and is cumbersome. For this reasons I went the SMB/CIFS route.

To be able to mounted a Synology share to a host directory vie the `/etc/fstab` file. Authentication is done by `credentials=/.smbcredentials_3` option. This subdirectory of this then is mounted to the container. This is an easy setup in the compose.yml.

Regarding reboots of the NAS. I run this now since two years without having any issues. I can't remember how it's done, but AFAIR the remount is done autamaically by the host.

My containers are short lived and created by a cron job. In case of a failure due to the NAS shutdown / reboot, the next run will find the mounted NAS share again.

---
Tags: docker, docker-compose
---


# Question
Title: How to only include directory contents in Azure Publish Artifact without full filepath?
I've been using Azure Pipelines and Releases to build and publish artifacts.

In SolutionX, there are projects ProjectY and ProjectYTests.

I build everything in the solution (to run the tests) and then I want to publish just ProjectY's built contents, in an artifact called drop.

I deploy it to a folder called FolderZ.

When I download the published artifact, the filepath is like: `FolderZ/drop/ProjectY/bin/Release/*`. I'd much rather it was just `FolderZ/*`.

How can I get rid of (at least) the bin/Release pipeline bit?

# Answer
In the copy files task, there is an optional parameter flattenFiles (bool). This does the job.
> 2 votes

---
Tags: azure, cicd
---


# Question
Title: How can I experiment with Cloud (Azure, AWS, Google, etc) without going broke?
Cloud technologies are very hot right now, but they can be expensive. What are the best strategies for learning/trying cloud services without racking up a huge bill?

# Answer
The three major cloud platforms you mentioned have either free trials or a free tier; these all include spending and resource caps which prevent you from spending more than your free trial allows:

## Azure

Azure offer £150/$200/€170 for the first one month of usage, in addition many of Azure's services have a free or low cost tier:

* App Service Free Tier
* SQL Azure Basic Instance @ £0.005/hour
* Azure Machine Learning Free Tier
* Azure Service Bus @ £0.04/million operations
* Azure Functions @ £0.15 per million executions

If you have a MSDN or Visual Stuido Online account you can also get £100/$150/€130 per month of credit just because you are a subscriber to one of these services.

If you are a startup you may qualify for $120,000 of Azure Credits to host your application through Microsoft's BizSpark programme.

## Amazon Web Services

Amazon have a Free Tier which includes:

* 750 Hours of EC2 which equates to about a month of a t2.micro instance
* 750 Amazon RDS
* 5GB S3
* 250,000 AWS IoT Messages
* **... and more**

In addition there perpetually free usage allowances on most services:

* AWS Lambda: 1 million free requests a month
* DynamoDB: 25GB Storage
* Amazon SNS: 1 million publishes
* **... and more**

## Google Compute Platform

Google offer a $300 free trial for 12 months in a similar deal to Amazon. In addition they have free usage tiers for many of their core offerings:

* Google App Engine: 28 instance hours / day
* Google Cloud Database: 1GB Storage
* Google Compute Engine: 1 f1-micro instance / month
* Google Pub/Sub: 10GB messages / month
* Google Cloud Functions: 10 million executions / month
* **... and more**

If all of that wasn't enough for you most of the cloud providers will compensate you in credits if you blog about their services in detail, you do have to put in a substantial effort up front and "appear" on their radar by writing blog posts, presenting at meetup and conferences but once they see you they will gladly put $100 a month towards your further learning.
> 20 votes

# Answer
# Emulate the cloud on your own hardware

Doing anything substantial with the free tier is challenging. But you probably have some hardware lying around that you can build your own minicloud on. If you need more memory you can dig through your junk drawer. If you build something serious there's some chance you'll decide to buy more hardware, but that is less risky than needing to pay $$$ monthly to AWS to keep your doors open.

## LocalStack

LocalStack spins up local emulators of a slew of AWS-style services:

> * API Gateway at http://localhost:4567
> * Kinesis at http://localhost:4568
> * DynamoDB at http://localhost:4569
> * DynamoDB Streams at http://localhost:4570
> * Elasticsearch at http://localhost:4571
> * S3 at http://localhost:4572
> * Firehose at http://localhost:4573
> * Lambda at http://localhost:4574
> * SNS at http://localhost:4575
> * SQS at http://localhost:4576
> * Redshift at http://localhost:4577
> * ES (Elasticsearch Service) at http://localhost:4578
> * SES at http://localhost:4579
> * Route53 at http://localhost:4580
> * CloudFormation at http://localhost:4581

It is completely open-source and it seems like Atlassian is putting a lot of engineering effort into keeping it current.

## AppScale

<sub>(source: jujucharms.com)</sub>

AppScale can help you emulate the Google cloud:

> AppScale is an open-source cloud computing platform that automatically deploys and scales unmodified Google App Engine applications over public and private cloud systems and on-premise clusters. AppScale is modeled on the App Engine APIs and has support for Python, Go, PHP and Java applications.

> AppScale is supported and maintained by AppScale Systems, in conjunction with Google.

## Eucalyptus

Eucalyptus has been around for a bit longer and it has free and paid features. It supports emulating EC2, S3, and IAM.
> 14 votes

# Answer
There are quite a few services that offer 'free tiers' of some description to help new developers get started with a cloud platform without the cost you've mentioned.

AWS have quite a generous free tier:

> ## Always Free
>
> * 1 million free Lambda requests/month
> * 25 GB of DynamoDB storage
> * 20,000 free requests to the AWS Key Management Service
> * 1 million requests to Amazon SQS
> * various other services
>
> ## 12 Months Free
>
> * 750 hours of EC2 computing time (t2.micro)
> * 5 GB of Amazon S3 storage
> * 250,000 messages on AWS IoT
> * various others

Azure offer US$200 (or £150) of free credit to use *in any way you like* within one month (but they expire after a month):

> You receive £150 of Azure credits with the Free Trial. The choice is now yours as to how you use your Azure credits. Use them on any Azure service based on your needs, including, Virtual Machines, Websites, Cloud Services, Mobile Services, Storage, SQL Database, Content Delivery Network, HDInsight, Media Services and many more.
>
> Use the pricing calculator to estimate how much you could use with £150 of Azure credits. Here are a few usage scenarios that consume no more than £150 per month:
>
> * Running 2 small Virtual Machine instances for the entire month, or
> * Storing 800 GB of data in Storage, or
> * Developing and testing a web application using Cloud Services, with 3 web roles and 2 worker roles on medium instances, for 10 hours a day, 5 days a week, or
> * Running two S2 SQL databases for the entire month

Google Cloud give 12 months with $300 of credit, and have some services which are 'always free', broadly similar to Amazon's offering.

If you're careful about your timing, you could also use a spot instance (but be careful that you're not paying more than the on-demand price!). Since spot instances can be terminated at any time if you're outbid, they may not be ideal for learning, as you might not be able to use the service exactly when you want it, but they are often *far* cheaper.

Students can also claim *"up to $110 in bonus AWS credits for a total of $75-$150"* with the GitHub Student Developer Pack. Perhaps worth checking if your favourite cloud service has similar offers if you're eligible!
> 6 votes

# Answer
Turn things off when you are not using them. Tools like www.parkmycloud.com are very helpful.
> 0 votes

# Answer
They are intentionally confusing. Scott Adams coined the term "confusopoly" to describe payment structures so complex that it's impossible to compare two things like for like. Phone tariffs are a great example.

Having recently rolled out some cloud-based infra at my org, the advice I'd give is:

1. Start tiny and scale up.
2. Monitor your costs at each step of the way.
> 0 votes

---
Tags: amazon-web-services, azure, cloud
---


title: Scheduling a build using Kubernetes

text:
# Question
Title: Scheduling a build using Kubernetes
The doc https://docs.openshift.com/container-platform/3.9/dev\_guide/cron\_jobs.html provides details of creating a cron job.

To start a scheduled task that executes a build every 10 mins I use the command:

```
oc run run-build 161/my-app --image=myimage --restart=OnFailure --schedule='*/10 * * * *'

```

Which returns:

```
cronjob.batch/run-build created

```

But the job fails to start:

The log of pod displays:

```
Error: unknown command "161/my-app" for "openshift-deploy"
Run 'openshift-deploy --help' for usage.

```

Have I configured the command ( `oc run run-build 161/my-app --image=myimage --restart=OnFailure --schedule='*/10 * * * *'` ) to start the cron job incorrectly ?

# Answer
According the documentation you're giving, you need to remove `run-build` from the command :

```
$ oc run NAME --image=<image> \
    [--generator=<resource>] \
    [--port=<port>] \
    [--replicas=<replicas>] \
    [--dry-run=<bool>] \
    [--overrides=<inline_json>] \
    [options]

```
> 1 votes

---
Tags: kubernetes, openshift, kubectl
---


# Question
Title: How I can retrieve aws lambda cloudwatch logs in txt format so I can search on them?
A lambda that I have deployed has gone wrong. Therefore, I want to look on logs and find out whats wrong about it. Hence I look upon cloudwatch logs for any error message.

But the cloudwatch environment allows no easy way to search for a string in logs. Therefore I need to download a portion of it in text format so I can `grep` string in it. Do you know how I can do that?

# Answer
> 1 votes
There are several ways for exporting logs from CloudWatch in searchable text format:

The easiest way of downloading logs in text format is to go to a log group -\> Select a log stream -\> go to **Actions** and select **Download search results as CSV**:

This will download the logs in a `.csv` format which may or may not be what we want, nevertheless I think it is good for searching a analysis.

A more involved way to export logs would be to select a log group and from the **Actions** and for that log group select **Export Data to S3**:

This may export every stream from the log group to an S3 bucket, from which we can download the logs in text format. In order to be able to write to an S3 bucket, we need a policy. There is a full documentation on the AWS docs about how to do this.

I need to mention the CloudWatch provides a way to analyze logs without being able to export them by using CloudWatch Logs Insights. CloudWatch Logs Insights has its own query language.

---
Tags: amazon-web-services, aws-lambda, amazon-cloudwatch
---


title: AWS ECS Update with new docker image

text:
# Question
Title: AWS ECS Update with new docker image
I've setup a cluster on AWS ECS, it's hosting nginx and a webapp.

I've used these commands to create my resources

```
aws ecr create-repository --repository-name <repo-name>

ecs-cli configure --cluster <cluster-name> --region eu-west-3 --config-name <conf-name> --cfn-stack-name <stack-name> --default-launch-type EC2

ecs-cli up --keypair <keypair> --capability-iam --size 1 --instance-type t2.medium --cluster-config <conf-name>

docker-compose build

docker-compose push

ecs-cli compose up --cluster-config <conf-name>

```

I've also setup a service in that cluster connected to the task created above.

My goal is to be able to push the docker images (That already works) and then use some command to get ECS to fetch the new images and load them up (this is where I'm stuck). I have to do this through the CLI since I want to automate this later on.

My question is: Should I just delete the running task so that the service will replace it? This works if I do it in the AWS Management console. But it causes downtime of my webservice.

Another options is to use the command below which I found here:

```
aws ecs update-service --cluster <cluster name> --service <service name> --force-new-deployment

```

However, when I run that command I get this error:

```
EVENTS  2021-04-21T05:32:45.842000-07:00        2c47e2c0-3a76-4dc1-bfef-21392b65871b    (service <service-name>) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance 25785de8f80041e0afd7b0adfda5f2fa) is already using a port required by your task. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.
EVENTS  2021-04-21T05:29:11.592000-07:00        4279fb7c-9c36-477b-acb3-173edb8e2c31    (service <service-name>) has reached a steady state.

```

I'm guessing it can't create another EC2 instance because I've set the number of instances to 1 on creation of my cluster. But I tried with 2 instances and it seems overkill to have two instances up and running all the time. How can I achieve this?

# Answer
> 2 votes
AWS container service team member here. IF you are still early on in your journey to automate this via a CLI I would suggest you consider moving past the `ecs-cli`. The follow-on version of this CLI is called AWS Copilot and has more sophistication to implement those workflows. Note that Copilot dropped support for `docker compose` because we have implemented a native integration with Docker and the new `docker compose cli`. You can read more here and here.

---
Tags: docker, amazon-web-services, amazon-ecs, aws-ecr
---


# Question
Title: How to build a complex parallel Jenkins pipeline?
I've been interested in converting our bespoke Jenkins integrations into a pipeline. However, I can't seem to figure out how to do it.

Can anyone help me with the Jenkins script that could do the following?

```
1---2---3-----------9---10
    |           |
    |---4-------|
    |           |
    |---5---6---|
        |       |
        |---7---|

1: Start pipeline
10: End pipeline
5: Build some files
   * needed by 6, 7,
   * needed as artifacts at the end
2, 3, 4, 6, 7: Have jUnit result files, should be available at end of
   test (somewhere), even if one failed

```

Is this even possible? Or should I just join after 3, 4, 5? Like this:

```
1---2---3-------6-------9---10
    |       |   |   |
    |---4---|   7---|
    |       |
    |---5---|

```

# Answer
> 11 votes
Based on the comments to my question, and some basic testing the following seems to work:

# Answer
> 5 votes
I had a similar situation in which I wanted to nest other parallel jobs threads inside another parallel one. This code worked for me:

```
def performDeploymentStages(String node, String app) {
    stage("build") {
        echo "Building the app [${app}] on node [${node}]"
    }
    stage("deploy") {
        echo "Deploying the app ${app}] on node [${node}]"
    }
    stage("test") {
        echo "Testing the app [${app}] on node [${node}]"
    }
}

pipeline {
    agent {
        label 'master'
    }
    parameters {
        string(name: 'NODES', defaultValue: '1,2,3', description: 'Nodes to build, deploy and test')
        choice(name: 'ENV', choices: 'qa', description: 'Environment')
        string(name: 'APPS', defaultValue: 'app01,app02', description: 'App names')
    }

    stages {
        stage('parallel stage') {
            steps {
                script {
                    def nodes = [:]
                    for (node in params.NODES.tokenize(',')) {
                        def apps = [:]
                        for (app in params.APPS.tokenize(',')) {
                            performDeploymentStages(node, app)
                        }
                        parallel apps
                    }
                    parallel nodes
                }
            }
        }
    }
}

```

To fully benefit from parallel run remember to assign enough executors.

# Answer
> 1 votes
Working off of the existing answers and the Jenkins example documentation I came up with the following nested parallel solution. The Stage View (2.19) plugin and Blue Ocean (1.24.5) do not display them as you would hope, so I used the Groovy Postbuild (2.5) plugin (whitelisted) to show a result summary on the build page of the Classic View (the Badge plugin may work as well).

Current environment is using Jenkins LTS 2.277.2, a main "trigger" job, and 4 downstream jobs that build the pieces of my application, all of which are Multibranch pipelines.

I added a method to trigger my downstream Multibranch Pipeline jobs, since they all do the same thing. This is at the end of my Jenkinsfile.

```
// In case you want to work on the results of the downstream jobs
buildResults = []

// Common code to trigger a downstream job
// Outputs a URL that you can follow in Blue Ocean and Classic View
def performTriggerJob(String jobID) {
    // We need to wrap what we return in a Groovy closure, or else it's invoked
    // when this method is called, not when we pass it to parallel.
    // To do this, you need to wrap the code below in { }, and either return
    // that explicitly, or use { -> } syntax.
    return {
        GIT_BRANCH_FIXUP = env.BRANCH_NAME.replace("/","%2F")
        // propagate: false so we can work after it returns. Try/catch will also work
        res = build( job: "../${jobID}/${GIT_BRANCH_FIXUP}", propagate: false)
        echo "${jobID}: ${res.getResult()} - ${res.buildVariables.get('RUN_DISPLAY_URL')}"
        // Add to buildResults
        buildResults << res

        text = "Downstream Run Result: <a href=\"${res.buildVariables.get('RUN_DISPLAY_URL')}\">${jobID} - ${res.getResult()}</a>"

        if (res.getResult().equals("SUCCESS")) {
            manager.createSummary("green.gif").appendText(text)
        } else {
            manager.createSummary("red.png").appendText(text)
            error 'JOB FAILED' // this fails the stage
        }
    }
}

```

Then in the main pipeline of the main Triggering job, I use `script` blocks to start the nested parallel jobs.

```
pipeline {
agent any
stages {
stage('Build Nested parallel') {
   steps {
      script {
         // Runs all these builds in parallel, the 2 nested Jobs
         // have a requirement on the cmake build, but can also run in
         // parallel after that finishes. Declarative cannot nest, but
         // Scripted can.

         // This is a map of 'name': action
         // performTriggerJob returns a Closure which is why this is just a method "call"
         // as Groovy will invoke .call() on parallel 'actions' automatically
         def builds = [
            "subJob1": performTriggerJob('subJob1'),
            "subJob2": performTriggerJob('subJob2'),
            "subJob3": { script {
               // Since we have more work to do than just trigger a job,
               // we have to .call() this Closure directly in the script block to get it to run
               performTriggerJob('subJob3').call()

               // You can also pass a map directly into the parallel
               parallel (
                  'nestedJob1': performTriggerJob('nestedJob1'),
                  'nestedJob2': performTriggerJob('nestedJob2')
               )
            } }
         ]
         parallel builds
      }
   }
}
// Other stages
}
}

```

Since `performTriggerJob` adds the results of the jobs to the variable `buildResults` we can do something with all the results once everything is done. I copy all artifacts from the last successful build and then zip them up so they're available on the trigger job.

`buildResults` is a list of (what I believe are) RunWrapper objects, so the methods in this javadoc are what you can call on this object.

```
// Other stages
stage("Post steps") {
steps {
script {
   for( finishedBuildObj in buildResults ) {
      // Obj to use. Could be this build or the last successful build
      buildObj       = finishedBuildObj
      projectName    = buildObj.getFullProjectName()
      if (buildObj.getResult() != "SUCCESS") {
         buildObj = buildObj.getPreviousSuccessfulBuild()
      }
      copyArtifacts(
         projectName: projectName,
         // Allow this build to continue even if no build is found matching the "Which build" condition,
         // the build's workspace does not exist or is inaccessible, or no artifacts are found matching the specified pattern.
         // By default this build step fails the build if no artifacts are copied.
         optional: true,
         // Ignore the directory structure of the artifacts in the source project and copy all matching artifacts directly into the specified target directory.
         // By default the artifacts are copied in the same directory structure as the source project.
         flatten: true,
         // Target base directory for copy, or leave blank to use the workspace.
         // Directory (and parent directories, if any) will be created if needed.
         //May contain references to build parameters like $PARAM.
         target: 'allArtifacts',
         fingerprintArtifacts: true,
         // How to select the build to copy artifacts from, such as latest successful or stable build, or latest "keep forever" build. Other plugins may provide additional selections.
         // The build number of the selected build will be recorded in the environment for later build steps to reference. For details, see the help of "Result variable suffix" in "Advanced" section.
         selector: specific(buildObj.getNumber().toString())
      )
   }
}}}

```

---
Tags: jenkins, jenkins-pipeline, build-pipeline, jenkinsfile
---


# Question
Title: How to create custom AWS CloudWatch metric for i-node
I have created custom CloudWatch alarm for Disk Utilization for root and mount volume to monitor the disk space for my Ubuntu (16.04) instances, but quite often it happens that disk is only 60% full (df -h) and still it's unable to create any more file as the i-node (df -i) already reached its limit. To keep an eye on this, I want to introduce a custom metric for i-node utilization and then want to configure an alarm on it.

Please provide a solution, considering I am relatively new in AWS.

# Answer
> 1 votes
AWS have published a plugin for `collectd` which allows to send metrics collected using `collectd` to CloudWatch.

https://github.com/awslabs/collectd-cloudwatch

The `collectd` monitoring tool runs on the server and can be configured to collect various metrics, among them metrics on disks and filesystems using their `DF` plugin. https://collectd.org/wiki/index.php/Plugin:DF

You can turn the `ReportInodes` option of the CollectD DF plugin and then have these metrics reported to CloudWatch as custom metrics using the AWS CollectD CloudWatch plugin.

# Answer
> 1 votes
AWS have a CloudWatch agent which will submit a number of useful (configurable) metrics, including `inodes_free`, `inodes_used`, and `inodes_total`. They'll appear, by default, under the `CWAgent` namespace in CloudWatch metrics from where you can view them, add them to dashboards, create alarms etc.

For the full details, these links will be useful:

Alternatively, the quickest way to get started is to:

1. Download the agent from https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
2. Unzip and run `install.sh`
3. Start the agent by running `/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start` (this will create a default config file)
4. Edit the default config file at `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json` and modify the `metrics_collected` to include the inode metrics under the `disk` category (see below for sample).
5. Stop (above command but with `-a stop` instead) and start the agent for the new config to take effect.

You should then see the new metrics appear in CloudWatch within a minute or so. The agent should also automatically start at boot.

Here's a sample of defining the inode metrics (for full details see the configuration docs):

```
{
  "metrics": {
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "inodes_free",
          "inodes_used",
          "inodes_total"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      }
    }
  }
}

```

# Answer
> 0 votes
SSH into the instance and add the script that will collect the inode usage and send metrics to the alarm

```
!/bin/bash
USAGE=80
INODE_USED=$(df -ih | sed -n '4p'| awk '{print $5}'| grep -v U | cut -d% -f1)
aws cloudwatch put-metric-data --metric-name inode-usage --dimensions Instance=<Instance id>  --namespace "Custom" --value $INODE_USED

```

---
Tags: amazon-web-services, amazon-cloudwatch
---


title: Continuous Deployment with jenkins

text:
# Question
Title: Continuous Deployment with jenkins
I am a beginner in DevOps. I created a continuous integration cycle using jenkins(docker image) and bitbucket. I used a simple spring boot hello-world project and after CI i got an executable Jar. I created a dockerfile and docker-compose for the deployment process. I tried this on my local server and it worked successfully(hello world successfully displayed on browser). Now i want to begin the continuous deployment process on remote server how can i do that ?

# Answer
> 1 votes
Your next step will be to identify a solution for *container orchestration*. For a quick dev test solution, you could host your own docker registry, push your built image to the registry, and add a deployment step that stops the existing container and spins up a new one from your registry.

However, in a production setting there are many other aspects to consider such as downtime, versioning, security, etc. There are several products and tools that are designed to handle containers such as AWS Elastic Container Solution (ECS), Kubernetes.

---
Tags: docker, jenkins, docker-compose, dockerfile, spring-boot
---


# Question
Title: How to automate initial server configuration?
I’m trying to add new nodes (Linux servers) to my cluster but I have to first configure my nodes properly, i.e. setting up the firewall, closing ports, disabling password login, setting up git, etc… This is all very time consuming and can go wrong easily due to human mistakes. How do I automate this process ideally? What tools are used for this? I read about Terraform, Ansible, but I feel kinda lost with all those different technologies. My desired result would be whenever I buy a new VPS, I run a script (or connect it to a master node) and then configuration happens based on a template or so

Thanks in advance

# Answer
> 6 votes
What you are trying to accomplish is referred to as *infrastructure as code*.

Which tools you select will depend on where you are operating, and what your business needs are. If you are operating in AWS exclusively, you can skip 3rd party solutions and focus on their CloudFormation service. Terraform (infrastructure) and Ansible (configuration) are popular cloud agnostic tools if you are operating On-Prem or across multiple cloud providers.

---
Tags: configuration, infrastructure-as-code
---


# Question
Title: Add an Actuator Button to Grafana Dashboard
Hi everyone and Thank You for reading. I have built a dashboard with Grafana. The sensor readings comes to me from a MQTT broker and are stored in Inflixdb. Now I want to add a button on the Dashboard in order to toggle an actuator (a water pump). Can you help me and suggest me how to do? I do not see any widget in Grafana to add an actuator. Thank you for your help.

# Answer
I think to have found a suitable grafana plugin that can activate/deacttivate an actuator by posting a value with a call api rest. The name of this plugin that can be easily donwloaded and istalled in the grafana is cloudspout button panel plugin. and can be found at grafana.com/grafana/plugins/cloudspout-button-panel
> 1 votes

---
Tags: grafana
---


# Question
Title: Limit throughput between Jenkins and Jenkins Agent
I am trying to find a way to throttle the thruput on the request of our Network folks. They want me to not exceed 5.5Mb/sec. Is there a way to do this within Jenkins. Looking at the config files under

Manage Jenkins \> Nodes \> "Node name"

I couldn't find anything that would do this.

Also, googling for a solution only resulted in simultaneous build info.

Thanks

# Answer
hmmm, tuff one. dont think it's possible do do this *directly*

counter question, why do you wish to limit traffic ? If you could do this, say through a network config of some kind, it owuld negatively impact build time.

From experience things that have an impact on traffic between agent and sever :

* Artifacts: if you are storing artifacts within your builds they will be stored on the server. So at these points in your build process you will upload them from the agent to the server and depending on artifact size it will generate traffic.
* Sync: Depending on what source control you use (if any) and how you use it it's possible that the sync to occur on the master and then transfered over to the agent. You can probably change this so the sync only occur on the agent which will pretty much eliminate this part of the traffic between them.
* Logs: all logs generated on the your jobs will be transfered back to the server. Though to max out a connection here you'dhave to generate a whole lot of it !

It's posible some plugins can generate their own traffic. Without knowing more on your tasks it's difficult to go further.

As to the actual question my first reflex here would be to look for ways to achieve this through a firewall or some other network system to throttle traffic. Some will allow to target a specific application in addition to source/target IPs in their rules.
> 1 votes

# Answer
It's a strange question coming from network as they control it but then again... network folks ;).
I'm assuming you have a Jenkins master node and a separate Jenkins slave node. For a quick blunt fix you could take a look at https://github.com/magnific0/wondershaper and run it on the remote slave node. It will slow things down considerably when there's stuff to be pushed/pulled over the network but at least you are complying to their request. Another option is true "Traffic Shaping" but it's a hassle to set up whereas wondershaper is a quick but less elegant fix. Perhaps you could enable wondershaper at the start of the Jenkins job and disable it afterwards or you could add another nic and create dedicated routes between master/slave so that network card is used for jenkins communication and run wondershaper on that nic alone leaving normal/default network traffic at max speed.

Last option is to stake your ground and respond to the network people "You guys run the network, can't you configure appropriate Traffic Shaping on the router/firewall?"
> 1 votes

---
Tags: jenkins, networking
---