Maintain an edit in a file in a zone delimited by two line markers

    .. versionadded:: 2014.1.0
    .. versionchanged:: 2017.7.5,2018.3.1
        ``append_newline`` argument added. Additionally, to improve
        idempotence, if the string represented by ``marker_end`` is found in
        the middle of the line, the content preceding the marker will be
        removed when the block is replaced. This allows one to remove
        ``append_newline: False`` from the SLS and have the block properly
        replaced if the end of the content block is immediately followed by the
        ``marker_end`` (i.e. no newline before the marker).

    A block of content delimited by comments can help you manage several lines
    entries without worrying about old entries removal. This can help you
    maintaining an un-managed file containing manual edits.

    .. note::
        This function will store two copies of the file in-memory (the original
        version and the edited version) in order to detect changes and only
        edit the targeted file if necessary.

        Additionally, you can use :py:func:`file.accumulated
        <salt.states.file.accumulated>` and target this state. All accumulated
        data dictionaries' content will be added in the content block.

    name
        Filesystem path to the file to be edited

    marker_start
        The line content identifying a line as the start of the content block.
        Note that the whole line containing this marker will be considered, so
        whitespace or extra content before or after the marker is included in
        final output

    marker_end
        The line content identifying the end of the content block. As of
        versions 2017.7.5 and 2018.3.1, everything up to the text matching the
        marker will be replaced, so it's important to ensure that your marker
        includes the beginning of the text you wish to replace.

    content
        The content to be used between the two lines identified by
        ``marker_start`` and ``marker_end``

    source
        The source file to download to the minion, this source file can be
        hosted on either the salt master server, or on an HTTP or FTP server.
        Both HTTPS and HTTP are supported as well as downloading directly
        from Amazon S3 compatible URLs with both pre-configured and automatic
        IAM credentials. (see s3.get state documentation)
        File retrieval from Openstack Swift object storage is supported via
        swift://container/object_path URLs, see swift.get documentation.
        For files hosted on the salt file server, if the file is located on
        the master in the directory named spam, and is called eggs, the source
        string is salt://spam/eggs. If source is left blank or None
        (use ~ in YAML), the file will be created as an empty file and
        the content will not be managed. This is also the case when a file
        already exists and the source is undefined; the contents of the file
        will not be changed or managed.

        If the file is hosted on a HTTP or FTP server then the source_hash
        argument is also required.

        A list of sources can also be passed in to provide a default source and
        a set of fallbacks. The first source in the list that is found to exist
        will be used and subsequent entries in the list will be ignored.

        .. code-block:: yaml

            file_override_example:
              file.blockreplace:
                - name: /etc/example.conf
                - source:
                  - salt://file_that_does_not_exist
                  - salt://file_that_exists

    source_hash
        This can be one of the following:
            1. a source hash string
            2. the URI of a file that contains source hash strings

        The function accepts the first encountered long unbroken alphanumeric
        string of correct length as a valid hash, in order from most secure to
        least secure:

        .. code-block:: text

            Type    Length
            ======  ======
            sha512     128
            sha384      96
            sha256      64
            sha224      56
            sha1        40
            md5         32

        See the ``source_hash`` parameter description for :mod:`file.managed
        <salt.states.file.managed>` function for more details and examples.

    template : jinja
        Templating engine to be used to render the downloaded file. The
        following engines are supported:

        - :mod:`cheetah <salt.renderers.cheetah>`
        - :mod:`genshi <salt.renderers.genshi>`
        - :mod:`jinja <salt.renderers.jinja>`
        - :mod:`mako <salt.renderers.mako>`
        - :mod:`py <salt.renderers.py>`
        - :mod:`wempy <salt.renderers.wempy>`

    context
        Overrides default context variables passed to the template

    defaults
        Default context passed to the template

    append_if_not_found : False
        If markers are not found and this option is set to ``True``, the
        content block will be appended to the file.

    prepend_if_not_found : False
        If markers are not found and this option is set to ``True``, the
        content block will be prepended to the file.

    insert_before_match
        If markers are not found, this parameter can be set to a regex which will
        insert the block before the first found occurrence in the file.

        .. versionadded:: Neon

    insert_after_match
        If markers are not found, this parameter can be set to a regex which will
        insert the block after the first found occurrence in the file.

        .. versionadded:: Neon

    backup
        The file extension to use for a backup of the file if any edit is made.
        Set this to ``False`` to skip making a backup.

    dry_run : False
        If ``True``, do not make any edits to the file and simply return the
        changes that *would* be made.

    show_changes : True
        Controls how changes are presented. If ``True``, the ``Changes``
        section of the state return will contain a unified diff of the changes
        made. If False, then it will contain a boolean (``True`` if any changes
        were made, otherwise ``False``).

    append_newline
        Controls whether or not a newline is appended to the content block. If
        the value of this argument is ``True`` then a newline will be added to
        the content block. If it is ``False``, then a newline will *not* be
        added to the content block. If it is unspecified, then a newline will
        only be added to the content block if it does not already end in a
        newline.

        .. versionadded:: 2017.7.5,2018.3.1

    Example of usage with an accumulator and with a variable:

    .. code-block:: jinja

        {% set myvar = 42 %}
        hosts-config-block-{{ myvar }}:
          file.blockreplace:
            - name: /etc/hosts
            - marker_start: "# START managed zone {{ myvar }} -DO-NOT-EDIT-"
            - marker_end: "# END managed zone {{ myvar }} --"
            - content: 'First line of content'
            - append_if_not_found: True
            - backup: '.bak'
            - show_changes: True

        hosts-config-block-{{ myvar }}-accumulated1:
          file.accumulated:
            - filename: /etc/hosts
            - name: my-accumulator-{{ myvar }}
            - text: "text 2"
            - require_in:
              - file: hosts-config-block-{{ myvar }}

        hosts-config-block-{{ myvar }}-accumulated2:
          file.accumulated:
            - filename: /etc/hosts
            - name: my-accumulator-{{ myvar }}
            - text: |
                 text 3
                 text 4
            - require_in:
              - file: hosts-config-block-{{ myvar }}

    will generate and maintain a block of content in ``/etc/hosts``:

    .. code-block:: text

        # START managed zone 42 -DO-NOT-EDIT-
        First line of content
        text 2
        text 3
        text 4
        # END managed zone 42 --

def blockreplace(
        name,
        marker_start='#-- start managed zone --',
        marker_end='#-- end managed zone --',
        source=None,
        source_hash=None,
        template='jinja',
        sources=None,
        source_hashes=None,
        defaults=None,
        context=None,
        content='',
        append_if_not_found=False,
        prepend_if_not_found=False,
        backup='.bak',
        show_changes=True,
        append_newline=None,
        insert_before_match=None,
        insert_after_match=None):
    '''
    Maintain an edit in a file in a zone delimited by two line markers

    .. versionadded:: 2014.1.0
    .. versionchanged:: 2017.7.5,2018.3.1
        ``append_newline`` argument added. Additionally, to improve
        idempotence, if the string represented by ``marker_end`` is found in
        the middle of the line, the content preceding the marker will be
        removed when the block is replaced. This allows one to remove
        ``append_newline: False`` from the SLS and have the block properly
        replaced if the end of the content block is immediately followed by the
        ``marker_end`` (i.e. no newline before the marker).

    A block of content delimited by comments can help you manage several lines
    entries without worrying about old entries removal. This can help you
    maintaining an un-managed file containing manual edits.

    .. note::
        This function will store two copies of the file in-memory (the original
        version and the edited version) in order to detect changes and only
        edit the targeted file if necessary.

        Additionally, you can use :py:func:`file.accumulated
        <salt.states.file.accumulated>` and target this state. All accumulated
        data dictionaries' content will be added in the content block.

    name
        Filesystem path to the file to be edited

    marker_start
        The line content identifying a line as the start of the content block.
        Note that the whole line containing this marker will be considered, so
        whitespace or extra content before or after the marker is included in
        final output

    marker_end
        The line content identifying the end of the content block. As of
        versions 2017.7.5 and 2018.3.1, everything up to the text matching the
        marker will be replaced, so it's important to ensure that your marker
        includes the beginning of the text you wish to replace.

    content
        The content to be used between the two lines identified by
        ``marker_start`` and ``marker_end``

    source
        The source file to download to the minion, this source file can be
        hosted on either the salt master server, or on an HTTP or FTP server.
        Both HTTPS and HTTP are supported as well as downloading directly
        from Amazon S3 compatible URLs with both pre-configured and automatic
        IAM credentials. (see s3.get state documentation)
        File retrieval from Openstack Swift object storage is supported via
        swift://container/object_path URLs, see swift.get documentation.
        For files hosted on the salt file server, if the file is located on
        the master in the directory named spam, and is called eggs, the source
        string is salt://spam/eggs. If source is left blank or None
        (use ~ in YAML), the file will be created as an empty file and
        the content will not be managed. This is also the case when a file
        already exists and the source is undefined; the contents of the file
        will not be changed or managed.

        If the file is hosted on a HTTP or FTP server then the source_hash
        argument is also required.

        A list of sources can also be passed in to provide a default source and
        a set of fallbacks. The first source in the list that is found to exist
        will be used and subsequent entries in the list will be ignored.

        .. code-block:: yaml

            file_override_example:
              file.blockreplace:
                - name: /etc/example.conf
                - source:
                  - salt://file_that_does_not_exist
                  - salt://file_that_exists

    source_hash
        This can be one of the following:
            1. a source hash string
            2. the URI of a file that contains source hash strings

        The function accepts the first encountered long unbroken alphanumeric
        string of correct length as a valid hash, in order from most secure to
        least secure:

        .. code-block:: text

            Type    Length
            ======  ======
            sha512     128
            sha384      96
            sha256      64
            sha224      56
            sha1        40
            md5         32

        See the ``source_hash`` parameter description for :mod:`file.managed
        <salt.states.file.managed>` function for more details and examples.

    template : jinja
        Templating engine to be used to render the downloaded file. The
        following engines are supported:

        - :mod:`cheetah <salt.renderers.cheetah>`
        - :mod:`genshi <salt.renderers.genshi>`
        - :mod:`jinja <salt.renderers.jinja>`
        - :mod:`mako <salt.renderers.mako>`
        - :mod:`py <salt.renderers.py>`
        - :mod:`wempy <salt.renderers.wempy>`

    context
        Overrides default context variables passed to the template

    defaults
        Default context passed to the template

    append_if_not_found : False
        If markers are not found and this option is set to ``True``, the
        content block will be appended to the file.

    prepend_if_not_found : False
        If markers are not found and this option is set to ``True``, the
        content block will be prepended to the file.

    insert_before_match
        If markers are not found, this parameter can be set to a regex which will
        insert the block before the first found occurrence in the file.

        .. versionadded:: Neon

    insert_after_match
        If markers are not found, this parameter can be set to a regex which will
        insert the block after the first found occurrence in the file.

        .. versionadded:: Neon

    backup
        The file extension to use for a backup of the file if any edit is made.
        Set this to ``False`` to skip making a backup.

    dry_run : False
        If ``True``, do not make any edits to the file and simply return the
        changes that *would* be made.

    show_changes : True
        Controls how changes are presented. If ``True``, the ``Changes``
        section of the state return will contain a unified diff of the changes
        made. If False, then it will contain a boolean (``True`` if any changes
        were made, otherwise ``False``).

    append_newline
        Controls whether or not a newline is appended to the content block. If
        the value of this argument is ``True`` then a newline will be added to
        the content block. If it is ``False``, then a newline will *not* be
        added to the content block. If it is unspecified, then a newline will
        only be added to the content block if it does not already end in a
        newline.

        .. versionadded:: 2017.7.5,2018.3.1

    Example of usage with an accumulator and with a variable:

    .. code-block:: jinja

        {% set myvar = 42 %}
        hosts-config-block-{{ myvar }}:
          file.blockreplace:
            - name: /etc/hosts
            - marker_start: "# START managed zone {{ myvar }} -DO-NOT-EDIT-"
            - marker_end: "# END managed zone {{ myvar }} --"
            - content: 'First line of content'
            - append_if_not_found: True
            - backup: '.bak'
            - show_changes: True

        hosts-config-block-{{ myvar }}-accumulated1:
          file.accumulated:
            - filename: /etc/hosts
            - name: my-accumulator-{{ myvar }}
            - text: "text 2"
            - require_in:
              - file: hosts-config-block-{{ myvar }}

        hosts-config-block-{{ myvar }}-accumulated2:
          file.accumulated:
            - filename: /etc/hosts
            - name: my-accumulator-{{ myvar }}
            - text: |
                 text 3
                 text 4
            - require_in:
              - file: hosts-config-block-{{ myvar }}

    will generate and maintain a block of content in ``/etc/hosts``:

    .. code-block:: text

        # START managed zone 42 -DO-NOT-EDIT-
        First line of content
        text 2
        text 3
        text 4
        # END managed zone 42 --
    '''
    name = os.path.expanduser(name)

    ret = {'name': name,
           'changes': {},
           'result': False,
           'comment': ''}
    if not name:
        return _error(ret, 'Must provide name to file.blockreplace')

    if sources is None:
        sources = []
    if source_hashes is None:
        source_hashes = []

    (ok_, err, sl_) = _unify_sources_and_hashes(source=source,
                                                source_hash=source_hash,
                                                sources=sources,
                                                source_hashes=source_hashes)
    if not ok_:
        return _error(ret, err)

    check_res, check_msg = _check_file(name)
    if not check_res:
        return _error(ret, check_msg)

    accum_data, accum_deps = _load_accumulators()
    if name in accum_data:
        accumulator = accum_data[name]
        # if we have multiple accumulators for a file, only apply the one
        # required at a time
        deps = accum_deps.get(name, [])
        filtered = [a for a in deps if
                    __low__['__id__'] in deps[a] and a in accumulator]
        if not filtered:
            filtered = [a for a in accumulator]
        for acc in filtered:
            acc_content = accumulator[acc]
            for line in acc_content:
                if content == '':
                    content = line
                else:
                    content += "\n" + line

    if sl_:
        tmpret = _get_template_texts(source_list=sl_,
                                     template=template,
                                     defaults=defaults,
                                     context=context)
        if not tmpret['result']:
            return tmpret
        text = tmpret['data']

        for index, item in enumerate(text):
            content += six.text_type(item)

    try:
        changes = __salt__['file.blockreplace'](
            name,
            marker_start,
            marker_end,
            content=content,
            append_if_not_found=append_if_not_found,
            prepend_if_not_found=prepend_if_not_found,
            insert_before_match=insert_before_match,
            insert_after_match=insert_after_match,
            backup=backup,
            dry_run=__opts__['test'],
            show_changes=show_changes,
            append_newline=append_newline)
    except Exception as exc:
        log.exception('Encountered error managing block')
        ret['comment'] = (
            'Encountered error managing block: {0}. '
            'See the log for details.'.format(exc)
        )
        return ret

    if changes:
        ret['changes']['diff'] = changes
        if __opts__['test']:
            ret['result'] = None
            ret['comment'] = 'Changes would be made'
        else:
            ret['result'] = True
            ret['comment'] = 'Changes were made'
    else:
        ret['result'] = True
        ret['comment'] = 'No changes needed to be made'

    return ret