{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/5343","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/5343/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/5343/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/5343/events","html_url":"https://github.com/influxdata/influxdb/issues/5343","id":126198830,"number":5343,"title":"out of memory: select * from <series> where time > now() - 1m in over million rows","user":{"login":"brenol","id":7342697,"avatar_url":"https://avatars.githubusercontent.com/u/7342697?v=3","gravatar_id":"","url":"https://api.github.com/users/brenol","html_url":"https://github.com/brenol","followers_url":"https://api.github.com/users/brenol/followers","following_url":"https://api.github.com/users/brenol/following{/other_user}","gists_url":"https://api.github.com/users/brenol/gists{/gist_id}","starred_url":"https://api.github.com/users/brenol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brenol/subscriptions","organizations_url":"https://api.github.com/users/brenol/orgs","repos_url":"https://api.github.com/users/brenol/repos","events_url":"https://api.github.com/users/brenol/events{/privacy}","received_events_url":"https://api.github.com/users/brenol/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-01-12T15:03:57Z","updated_at":"2016-01-17T17:26:27Z","closed_at":null,"body":"Hi,\r\n\r\nWe have a little instance running up in a 2-core / 8gb RAM machine here. We have a series with over a million writes, with constant writes (80k/hour ish).\r\n\r\nInflux is dying of OOM all the time when querying with a minute long search, not sure what we're doing wrong.\r\n\r\nRequests are being written with the write endpoint and the mem usage during writes is constantly in 3-4GB range.\r\nWe are writing 8 tags and 2 fields and we're using tsm1 engine, using nightly, config file is as follows:\r\n\r\n```toml\r\n### Welcome to the InfluxDB configuration file.\r\n\r\n# Once every 24 hours InfluxDB will report anonymous data to m.influxdb.com\r\n# The data includes raft id (random 8 bytes), os, arch, version, and metadata.\r\n# We don't track ip addresses of servers reporting. This is only used\r\n# to track the number of instances running and the versions, which\r\n# is very helpful for us.\r\n# Change this option to true to disable reporting.\r\nreporting-disabled = false\r\n\r\n###\r\n### [meta]\r\n###\r\n### Controls the parameters for the Raft consensus group that stores metadata\r\n### about the InfluxDB cluster.\r\n###\r\n\r\n[meta]\r\n  dir = \"/var/lib/influxdb/meta\"\r\n  hostname = \"localhost\"\r\n  bind-address = \":8088\"\r\n  retention-autocreate = true\r\n  election-timeout = \"1s\"\r\n  heartbeat-timeout = \"1s\"\r\n  leader-lease-timeout = \"500ms\"\r\n  commit-timeout = \"50ms\"\r\n  cluster-tracing = false\r\n\r\n  # If enabled, when a Raft cluster loses a peer due to a `DROP SERVER` command,\r\n  # the leader will automatically ask a non-raft peer node to promote to a raft\r\n  # peer. This only happens if there is a non-raft peer node available to promote.\r\n  # This setting only affects the local node, so to ensure if operates correctly, be sure to set\r\n  # it in the config of every node.\r\n  raft-promotion-enabled = true\r\n\r\n###\r\n### [data]\r\n###\r\n### Controls where the actual shard data for InfluxDB lives and how it is\r\n### flushed from the WAL. \"dir\" may need to be changed to a suitable place\r\n### for your system, but the WAL settings are an advanced configuration. The\r\n### defaults should work for most systems.\r\n###\r\n\r\n[data]\r\n  dir = \"/var/lib/influxdb/data\"\r\n\r\n  # Controls the storage engine used for new shards. Engines available are b1,\r\n  # bz1, and tsm1. b1 was the original default engine from 0.9.0 to 0.9.2. bz1\r\n  # has been the default engine since 0.9.3. tsm1 was introduced in 0.9.5 and is\r\n  # currently EXPERIMENTAL. Consequently, data written into the tsm1 engine may\r\n  # need to be wiped between upgrades.\r\n  engine =\"tsm1\"\r\n\r\n  # The following WAL settings are for the b1 storage engine used in 0.9.2. They won't\r\n  # apply to any new shards created after upgrading to a version > 0.9.3.\r\n  max-wal-size = 104857600 # Maximum size the WAL can reach before a flush. Defaults to 100MB.\r\n  wal-flush-interval = \"10m\" # Maximum time data can sit in WAL before a flush.\r\n  wal-partition-flush-delay = \"2s\" # The delay time between each WAL partition being flushed.\r\n\r\n  # These are the WAL settings for the storage engine >= 0.9.3\r\n  wal-dir = \"/var/lib/influxdb/wal\"\r\n  wal-logging-enabled = true\r\n  data-logging-enabled = true\r\n\r\n  # When a series in the WAL in-memory cache reaches this size in bytes it is marked as ready to\r\n  # flush to the index\r\n  # wal-ready-series-size = 25600\r\n\r\n  # Flush and compact a partition once this ratio of series are over the ready size\r\n  # wal-compaction-threshold = 0.6\r\n\r\n  # Force a flush and compaction if any series in a partition gets above this size in bytes\r\n  # wal-max-series-size = 2097152\r\n\r\n  # Force a flush of all series and full compaction if there have been no writes in this\r\n  # amount of time. This is useful for ensuring that shards that are cold for writes don't\r\n  # keep a bunch of data cached in memory and in the WAL.\r\n  wal-flush-cold-interval = \"10m\"\r\n\r\n  # Force a partition to flush its largest series if it reaches this approximate size in\r\n  # bytes. Remember there are 5 partitions so you'll need at least 5x this amount of memory.\r\n  # The more memory you have, the bigger this can be.\r\n  wal-partition-size-threshold = 20971520\r\n\r\n  # Whether queries should be logged before execution. Very useful for troubleshooting, but will\r\n  # log any sensitive data contained within a query.\r\n  # query-log-enabled = true\r\n\r\n  # Settings for the TSM engine\r\n\r\n  # CacheMaxMemorySize is the maximum size a shard's cache can\r\n  # reach before it starts rejecting writes.\r\n  cache-max-memory-size = 524288000\r\n\r\n  # CacheSnapshotMemorySize is the size at which the engine will\r\n  # snapshot the cache and write it to a TSM file, freeing up memory\r\n  # cache-snapshot-memory-size = 26214400\r\n\r\n  # CacheSnapshotWriteColdDuration is the length of time at\r\n  # which the engine will snapshot the cache and write it to\r\n  # a new TSM file if the shard hasn't received writes or deletes\r\n  # cache-snapshot-write-cold-duration = \"1h\"\r\n\r\n  # MinCompactionFileCount is the minimum number of TSM files\r\n  # that need to exist before a compaction cycle will run\r\n  # compact-min-file-count = 3\r\n\r\n  # CompactFullWriteColdDuration is the duration at which the engine\r\n  # will compact all TSM files in a shard if it hasn't received a\r\n  # write or delete\r\n  # compact-full-write-cold-duration = \"24h\"\r\n\r\n  # MaxPointsPerBlock is the maximum number of points in an encoded\r\n  # block in a TSM file. Larger numbers may yield better compression\r\n  # but could incur a performance peanalty when querying\r\n  # max-points-per-block = 1000\r\n\r\n###\r\n### [hinted-handoff]\r\n###\r\n### Controls the hinted handoff feature, which allows nodes to temporarily\r\n### store queued data when one node of a cluster is down for a short period\r\n### of time.\r\n###\r\n\r\n[hinted-handoff]\r\n  enabled = true\r\n  dir = \"/var/lib/influxdb/hh\"\r\n  max-size = 1073741824\r\n  max-age = \"168h\"\r\n  retry-rate-limit = 0\r\n\r\n  # Hinted handoff will start retrying writes to down nodes at a rate of once per second.\r\n  # If any error occurs, it will backoff in an exponential manner, until the interval\r\n  # reaches retry-max-interval. Once writes to all nodes are successfully completed the\r\n  # interval will reset to retry-interval.\r\n  retry-interval = \"1s\"\r\n  retry-max-interval = \"1m\"\r\n\r\n  # Interval between running checks for data that should be purged. Data is purged from\r\n  # hinted-handoff queues for two reasons. 1) The data is older than the max age, or\r\n  # 2) the target node has been dropped from the cluster. Data is never dropped until\r\n  # it has reached max-age however, for a dropped node or not.\r\n  purge-interval = \"1h\"\r\n\r\n###\r\n### [cluster]\r\n###\r\n### Controls non-Raft cluster behavior, which generally includes how data is\r\n### shared across shards.\r\n###\r\n\r\n[cluster]\r\n  shard-writer-timeout = \"5s\" # The time within which a remote shard must respond to a write request. \r\n  write-timeout = \"10s\" # The time within which a write request must complete on the cluster.\r\n\r\n###\r\n### [retention]\r\n###\r\n### Controls the enforcement of retention policies for evicting old data.\r\n###\r\n\r\n[retention]\r\n  enabled = true\r\n  check-interval = \"30m\"\r\n\r\n###\r\n### [shard-precreation]\r\n###\r\n### Controls the precreation of shards, so they are available before data arrives.\r\n### Only shards that, after creation, will have both a start- and end-time in the\r\n### future, will ever be created. Shards are never precreated that would be wholly\r\n### or partially in the past.\r\n\r\n[shard-precreation]\r\n  enabled = true\r\n  check-interval = \"10m\"\r\n  advance-period = \"30m\"\r\n\r\n###\r\n### Controls the system self-monitoring, statistics and diagnostics.\r\n###\r\n### The internal database for monitoring data is created automatically if\r\n### if it does not already exist. The target retention within this database\r\n### is called 'monitor' and is also created with a retention period of 7 days\r\n### and a replication factor of 1, if it does not exist. In all cases the\r\n### this retention policy is configured as the default for the database.\r\n\r\n[monitor]\r\n  store-enabled = true # Whether to record statistics internally.\r\n  store-database = \"_internal\" # The destination database for recorded statistics\r\n  store-interval = \"10s\" # The interval at which to record statistics\r\n\r\n###\r\n### [admin]\r\n###\r\n### Controls the availability of the built-in, web-based admin interface. If HTTPS is\r\n### enabled for the admin interface, HTTPS must also be enabled on the [http] service.\r\n###\r\n\r\n[admin]\r\n  enabled = true\r\n  bind-address = \":8083\"\r\n  https-enabled = false\r\n  https-certificate = \"/etc/ssl/influxdb.pem\"\r\n\r\n###\r\n### [http]\r\n###\r\n### Controls how the HTTP endpoints are configured. These are the primary\r\n### mechanism for getting data into and out of InfluxDB.\r\n###\r\n\r\n[http]\r\n  enabled = true\r\n  bind-address = \":8086\"\r\n  auth-enabled = false\r\n  log-enabled = true\r\n  write-tracing = false\r\n  pprof-enabled = false\r\n  https-enabled = false\r\n  https-certificate = \"/etc/ssl/influxdb.pem\"\r\n\r\n###\r\n### [[graphite]]\r\n###\r\n### Controls one or many listeners for Graphite data.\r\n###\r\n\r\n[[graphite]]\r\n  enabled = false\r\n  # database = \"graphite\"\r\n  # bind-address = \":2003\"\r\n  # protocol = \"tcp\"\r\n  # consistency-level = \"one\"\r\n  # name-separator = \".\"\r\n\r\n  # These next lines control how batching works. You should have this enabled\r\n  # otherwise you could get dropped metrics or poor performance. Batching\r\n  # will buffer points in memory if you have many coming in.\r\n\r\n  # batch-size = 1000 # will flush if this many points get buffered\r\n  # batch-pending = 5 # number of batches that may be pending in memory\r\n  # batch-timeout = \"1s\" # will flush at least this often even if we haven't hit buffer limit\r\n  # udp-read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.\r\n\r\n  ## \"name-schema\" configures tag names for parsing the metric name from graphite protocol;\r\n  ## separated by `name-separator`.\r\n  ## The \"measurement\" tag is special and the corresponding field will become\r\n  ## the name of the metric.\r\n  ## e.g. \"type.host.measurement.device\" will parse \"server.localhost.cpu.cpu0\" as\r\n  ## {\r\n  ##     measurement: \"cpu\",\r\n  ##     tags: {\r\n  ##         \"type\": \"server\",\r\n  ##         \"host\": \"localhost,\r\n  ##         \"device\": \"cpu0\"\r\n  ##     }\r\n  ## }\r\n  # name-schema = \"type.host.measurement.device\"\r\n\r\n  ## If set to true, when the input metric name has more fields than `name-schema` specified,\r\n  ## the extra fields will be ignored.\r\n  ## Otherwise an error will be logged and the metric rejected.\r\n  # ignore-unnamed = true\r\n\r\n###\r\n### [collectd]\r\n###\r\n### Controls the listener for collectd data.\r\n###\r\n\r\n[collectd]\r\n  enabled = false\r\n  # bind-address = \"\"\r\n  # database = \"\"\r\n  # typesdb = \"\"\r\n\r\n  # These next lines control how batching works. You should have this enabled\r\n  # otherwise you could get dropped metrics or poor performance. Batching\r\n  # will buffer points in memory if you have many coming in.\r\n\r\n  # batch-size = 1000 # will flush if this many points get buffered\r\n  # batch-pending = 5 # number of batches that may be pending in memory\r\n  # batch-timeout = \"1s\" # will flush at least this often even if we haven't hit buffer limit\r\n  # read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.\r\n\r\n###\r\n### [opentsdb]\r\n###\r\n### Controls the listener for OpenTSDB data.\r\n###\r\n\r\n[opentsdb]\r\n  enabled = false\r\n  # bind-address = \":4242\"\r\n  # database = \"opentsdb\"\r\n  # retention-policy = \"\"\r\n  # consistency-level = \"one\"\r\n  # tls-enabled = false\r\n  # certificate= \"\"\r\n  # log-point-errors = true # Log an error for every malformed point.\r\n\r\n  # These next lines control how batching works. You should have this enabled\r\n  # otherwise you could get dropped metrics or poor performance. Only points\r\n  # metrics received over the telnet protocol undergo batching.\r\n\r\n  # batch-size = 1000 # will flush if this many points get buffered\r\n  # batch-pending = 5 # number of batches that may be pending in memory\r\n  # batch-timeout = \"1s\" # will flush at least this often even if we haven't hit buffer limit\r\n\r\n###\r\n### [[udp]]\r\n###\r\n### Controls the listeners for InfluxDB line protocol data via UDP.\r\n###\r\n\r\n[[udp]]\r\n  enabled = false\r\n  # bind-address = \"\"\r\n  # database = \"udp\"\r\n  # retention-policy = \"\"\r\n\r\n  # These next lines control how batching works. You should have this enabled\r\n  # otherwise you could get dropped metrics or poor performance. Batching\r\n  # will buffer points in memory if you have many coming in.\r\n\r\n  # batch-size = 1000 # will flush if this many points get buffered\r\n  # batch-pending = 5 # number of batches that may be pending in memory\r\n  # batch-timeout = \"1s\" # will flush at least this often even if we haven't hit buffer limit\r\n  # read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.\r\n \r\n  # set the expected UDP payload size; lower values tend to yield better performance, default is max UDP size 65536\r\n  # udp-payload-size = 65536\r\n\r\n###\r\n### [continuous_queries]\r\n###\r\n### Controls how continuous queries are run within InfluxDB.\r\n###\r\n\r\n[continuous_queries]\r\n  log-enabled = true\r\n  enabled = true\r\n  # run-interval = \"1s\" # interval for how often continuous queries will be checked if they need to run\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/172354837","html_url":"https://github.com/influxdata/influxdb/issues/5343#issuecomment-172354837","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/5343","id":172354837,"user":{"login":"brenol","id":7342697,"avatar_url":"https://avatars.githubusercontent.com/u/7342697?v=3","gravatar_id":"","url":"https://api.github.com/users/brenol","html_url":"https://github.com/brenol","followers_url":"https://api.github.com/users/brenol/followers","following_url":"https://api.github.com/users/brenol/following{/other_user}","gists_url":"https://api.github.com/users/brenol/gists{/gist_id}","starred_url":"https://api.github.com/users/brenol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brenol/subscriptions","organizations_url":"https://api.github.com/users/brenol/orgs","repos_url":"https://api.github.com/users/brenol/repos","events_url":"https://api.github.com/users/brenol/events{/privacy}","received_events_url":"https://api.github.com/users/brenol/received_events","type":"User","site_admin":false},"created_at":"2016-01-17T17:26:27Z","updated_at":"2016-01-17T17:26:27Z","body":"@beckettsean \r\n\r\nWe have also tried with disabled HTTP writes, and database keeps OOMing upon query. I'll try to build a test script to see if the problem can be reproduced.\r\n\r\nThanks!"}}