title: How can I avoid PG::InFailedSqlTransaction when adding new columns?
body: <p>We are running two rails applications against the same database.  When we deploy, we typically deploy to App A, then App B, restarting all rails processes during the deploy.  App A runs on 7 servers with at least 20 processes connections to the database.  App B runs on 4 servers with at least 8 connections to the database. </p>
  
  <p>Today, when we deployed App A, we added a column to an existing table:</p>
  
  <pre><code>change_table :organizations do |t|
    t.integer :users_count, default: 0
  end
  </code></pre>
  
  <p>We expected this to be fine: its new column on an existing table and it has a default.  Shortly after the migration ran, a number of errors showed up, from both App A (before it was restarted) and App B (before it was deployed to).  </p>
  
  <p>These errors were:</p>
  
  <pre><code>FATAL ActiveRecord::StatementInvalid error: PG::InFailedSqlTransaction:
  ERROR:  current transaction is aborted, commands ignored until end of 
  transaction block
  </code></pre>
  
  <p>In the postgres log, I have 58 errors like this:</p>
  
  <pre><code>postgres[12283]: ERROR:  cached plan must not change result type
  postgres[12283]: STATEMENT:  SELECT  "organizations".* FROM 
    "organizations" WHERE "organizations"."id" = $1 LIMIT $2
  </code></pre>
  
  <p>This repeats a number of times and goes away after all deploys have finished and all processes restarted.</p>
  
  <p>It appeared that <a href="https://github.com/rails/rails/issues/12330" rel="nofollow noreferrer">Rails bug #12330</a> and <a href="https://github.com/rails/rails/pull/22170" rel="nofollow noreferrer">Rails PR 22170</a> addressed this in Rails 5.0, but I have that commit and am still seeing this error.  </p>
  
  <p>Relevant software versions</p>
  
  <ul>
  <li>Rails 5.0.2</li>
  <li>PG 0.19.0</li>
  <li>Postgres 9.5</li>
  </ul>
  
  <p>One comment on <a href="https://github.com/rails/rails/issues/12330" rel="nofollow noreferrer">Rails bug #12330</a> suggests that I have to add the columns with null defaults.  Another suggests performing multiple deploys, one to disable prepare statements, then another to perform the migration and and re-enable prepared statements. </p>
  
  <p>Is there away to avoid this?  It clears up when we restart the servers, but I feel like I'm missing something - like only using nullable columns perhaps that would avoid these errors all together.  This doesn't happen on every deploy and I don't know how to reproduce it - but this wasn't the first time it has happened.</p>