title: Cannot connect to GCP Memorystore from GCP Dataflow
body: <p>I'm attempting to use GCP Memorystore to handle session ids for a event streaming job running on GCP Dataflow. The job fails with a timeout when trying to connect to Memorystore:</p>
  
  <pre><code>redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host 10.0.0.4:6379
      at redis.clients.jedis.Connection.connect(Connection.java:207)
      at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:101)
      at redis.clients.jedis.Connection.sendCommand(Connection.java:126)
      at redis.clients.jedis.Connection.sendCommand(Connection.java:117)
      at redis.clients.jedis.Jedis.get(Jedis.java:155)
  </code></pre>
  
  <p>My Memorystore instance has these properties:</p>
  
  <pre><code>Version is 4.0
  Authorized network is default-auto
  Master is in us-central1-b. Replica is in us-central1-a.
  Connection properties: IP address: 10.0.0.4, Port number: 6379 
  &gt; gcloud redis instances list --region us-central1
  INSTANCE_NAME  VERSION    REGION       TIER         SIZE_GB  HOST      PORT  NETWORK       RESERVED_IP  STATUS  CREATE_TIME
  memorystore    REDIS_4_0  us-central1  STANDARD_HA  1        10.0.0.4  6379  default-auto  10.0.0.0/29  READY   2019-07-15T11:43:14
  </code></pre>
  
  <p>My Dataflow job has these properties:</p>
  
  <pre><code>runner: org.apache.beam.runners.dataflow.DataflowRunner
  zone: us-central1-b
  network: default-auto
  &gt; gcloud dataflow jobs list   
  JOB_ID                                    NAME                        TYPE       CREATION_TIME        STATE      REGION
  2019-06-17_02_01_36-3308621933676080017   eventflow                   Streaming  2019-06-17 09:01:37  Running    us-central1
  </code></pre>
  
  <p>My "default" network could not be used since it is a legacy network, which Memorystore would not accept. I failed to find a way to upgrade the default network from legacy to auto and did not want to delete the existing default network since this would require messing with production services. Instead I created a new network "default-auto" of type auto, with the same firewall rules as the default network. The one I believe is relevant for my Dataflow job is this:</p>
  
  <pre><code>Name: default-auto-internal
  Type: Ingress
  Targets: Apply to all   
  Filters: IP ranges: 10.0.0.0/20
  Protocols/ports: 
    tcp:0-65535
    udp:0-65535
    icmp
  Action: Allow
  Priority: 65534
  </code></pre>
  
  <p>I can connect to Memorystore using "telnet 10.0.0.4 6379" from a Compute Engine instance.</p>
  
  <p>Things I have tried, which did not change anything:
  - Switched Redis library, from Jedis 2.9.3 to Lettuce 5.1.7
  - Deleted and re-created the Memorystore instance</p>
  
  <p>Is Dataflow not supposed to be able to connect to Memorystore, or am I missing something?</p>