title: creating data stream through logstash
body: <p>I have installed elasticsearch cluster v 7.14.</p>
  <p>I have created ILM policy and Index template. However data stream parameters mentioned under logstash pipeline file are giving error.</p>
  <p><code>ILM policy</code> -</p>
  <pre><code>{
    &quot;testpolicy&quot; : {
      &quot;version&quot; : 1,
      &quot;modified_date&quot; : &quot;2021-08-28T02:58:25.942Z&quot;,
      &quot;policy&quot; : {
        &quot;phases&quot; : {
          &quot;hot&quot; : {
            &quot;min_age&quot; : &quot;0ms&quot;,
            &quot;actions&quot; : {
              &quot;rollover&quot; : {
                &quot;max_primary_shard_size&quot; : &quot;900mb&quot;,
                &quot;max_age&quot; : &quot;2d&quot;
              },
              &quot;set_priority&quot; : {
                &quot;priority&quot; : 100
              }
            }
          },
          &quot;delete&quot; : {
            &quot;min_age&quot; : &quot;2d&quot;,
            &quot;actions&quot; : {
              &quot;delete&quot; : {
                &quot;delete_searchable_snapshot&quot; : true
              }
            }
          }
        }
      },
      &quot;in_use_by&quot; : {
        &quot;indices&quot; : [ ],
        &quot;data_streams&quot; : [ ],
        &quot;composable_templates&quot; : [ ]
      }
    }
  }
  </code></pre>
  <p><code>Index temaplate</code> -</p>
  <pre><code>{
    &quot;index_templates&quot; : [
      {
        &quot;name&quot; : &quot;access_template&quot;,
        &quot;index_template&quot; : {
          &quot;index_patterns&quot; : [
            &quot;test-data-stream*&quot;
          ],
          &quot;template&quot; : {
            &quot;settings&quot; : {
              &quot;index&quot; : {
                &quot;number_of_shards&quot; : &quot;1&quot;,
                &quot;number_of_replicas&quot; : &quot;0&quot;
              }
            },
            &quot;mappings&quot; : {
              &quot;_routing&quot; : {
                &quot;required&quot; : false
              },
              &quot;dynamic_date_formats&quot; : [
                &quot;strict_date_optional_time&quot;,
                &quot;yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z&quot;
              ],
              &quot;numeric_detection&quot; : true,
              &quot;_source&quot; : {
                &quot;excludes&quot; : [ ],
                &quot;includes&quot; : [ ],
                &quot;enabled&quot; : true
              },
              &quot;dynamic&quot; : true,
              &quot;dynamic_templates&quot; : [ ],
              &quot;date_detection&quot; : true
            }
          },
          &quot;composed_of&quot; : [ ],
          &quot;priority&quot; : 500,
          &quot;version&quot; : 1,
          &quot;data_stream&quot; : {
            &quot;hidden&quot; : false
          }
        }
      }
    ]
  }
  </code></pre>
  <p>logstash pipeline config file -</p>
  <pre><code>input {
    beats {
      port =&gt; 5044
    }
  }
  
  filter {
  
   if [log_type] == &quot;access_server&quot; and [app_id] == &quot;pa&quot;
    {
       grok {
      match =&gt; {
      &quot;message&quot; =&gt; &quot;%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}&quot;    
      }
      }
      mutate {
               replace =&gt; {
                 &quot;[type]&quot; =&gt; &quot;access_server&quot;
               }
             }
    }
  }
  
  output {
     if [log_type] == &quot;access_server&quot; {
    elasticsearch {
      hosts =&gt; ['http://10.10.10.76:9200']
          user =&gt; elastic
      password =&gt; xxx
       data_stream =&gt; &quot;true&quot;
       data_stream_type =&gt; &quot;logs&quot;
       data_stream_dataset =&gt; &quot;access&quot;
       data_stream_namespace =&gt; &quot;default&quot;
       ilm_rollover_alias =&gt; &quot;access&quot;
       ilm_pattern =&gt; &quot;000001&quot;
       ilm_policy =&gt; &quot;testpolicy&quot;
       template =&gt; &quot;/tmp/access_template&quot;
       template_name =&gt; &quot;access_template&quot;
        }
   }
     elasticsearch {
      hosts =&gt; ['http://10.10.10.76:9200']
      index =&gt; &quot;%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}&quot;
      user =&gt; elastic
      password =&gt; xxx
    }
  }
  </code></pre>
  <p>After all deployment done, can only see system indices but data stream is not created.</p>
  <pre><code>[2021-08-28T12:42:50,103][ERROR][logstash.outputs.elasticsearch][main] Invalid data stream configuration, following parameters are not supported: {&quot;template&quot;=&gt;&quot;/tmp/pingaccess_template&quot;, &quot;ilm_pattern&quot;=&gt;&quot;000001&quot;, &quot;template_name&quot;=&gt;&quot;pingaccess_template&quot;, &quot;ilm_rollover_alias&quot;=&gt;&quot;pingaccess&quot;, &quot;ilm_policy&quot;=&gt;&quot;testpolicy&quot;}
  
  [2021-08-28T12:42:50,547][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=&gt;&quot;main&quot;, :exception=&gt;#&lt;LogStash::ConfigurationError: Invalid data stream configuration: [&quot;template&quot;, &quot;ilm_pattern&quot;, &quot;template_name&quot;, &quot;ilm_rollover_alias&quot;, &quot;ilm_policy&quot;]&gt;, :backtrace=&gt;[&quot;/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.2-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:57:in `check_data_stream_config!'&quot;
  
  [2021-08-28T12:42:50,702][ERROR][logstash.agent           ] Failed to execute action {:id=&gt;:main, :action_type=&gt;LogStash::ConvergeResult::FailedAction, :message=&gt;&quot;Could not execute action: PipelineAction::Create&lt;main&gt;, action_result: false&quot;, :backtrace=&gt;nil}
  </code></pre>
  <p>error is saying parameters like <code>template&quot;=&gt;&quot;/tmp/pingaccess_template&quot;, &quot;ilm_pattern&quot;=&gt;&quot;000001&quot;, &quot;template_name&quot;=&gt;&quot;pingaccess_template&quot;, &quot;ilm_rollover_alias&quot;=&gt;&quot;pingaccess&quot;, &quot;ilm_policy&quot;=&gt;&quot;testpolicy&quot;</code> are not valid but in below link they are mentioned.</p>
  <p><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams" rel="nofollow noreferrer">https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams</a></p>