<p>whilst 'hardening' the accounts - namely removing or toning down accounts with editor permissions on the projects I removed editor from what appears to be the kubernetes account that container engine uses on the back end of <code>gcloud</code> commands.</p>

<p>Once you remove the last role from an account it vanishes - hard lesson to learn!
<code>
Removed editor
serviceAccount:386242358897@cloudservices.gserviceaccount.com
</code></p>

<ul>
<li>It meant I initially couldn't deploy because it couldn't access container registry.</li>
<li>So I deleted the cluster and recreated expecting the account to get recreated. That failed due to insufficient permissions.</li>
<li>so I manually removed the compute instances (it wouldn't have permissions to recreate them), then templates and then the cluster.</li>
<li><p>As the UI now thinks you have no clusters it looks like you are back to the beginning. So I ran my scripts and they failed.</p>

<p>ERROR: (gcloud.container.clusters.create) Opetion [https://container.googleapis.com/v1/projects/xxxx/zones/europe-west2-b/operations/operation-xxxx'
 startTime: u'2017-10-17T17:59:41.515667863Z'
 status: StatusValueValuesEnum(DONE, 3)
 statusMessage: u'Deploy error: "Not all instances running in IGM. Expect 1. Current actions &amp;{Abandoning:0 Creating:0 CreatingWithoutRetries:0 Deleting:0 None:0 Recreating:1 Refreshing:0 Restarting:0 Verifying:0 ForceSendFields:[] NullFields:[]}. Errors [<a href="https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/gke-xxxx-default-pool-xxxx:PERMISSIONS_ERROR]" rel="nofollow noreferrer">https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/gke-xxxx-default-pool-xxxx:PERMISSIONS_ERROR]</a>".'
 targetLink: u'<a href="https://container.googleapis.com/v1/projects/xxxx/zones/europe-west2-b/clusters/xxxx" rel="nofollow noreferrer">https://container.googleapis.com/v1/projects/xxxx/zones/europe-west2-b/clusters/xxxx</a>'
 zone: u'europe-west2-b'>] finished with error: Deploy error: "Not all instances running in IGM. Expect 1. Current actions &amp;{Abandoning:0 Creating:0 CreatingWithoutRetries:0 Deleting:0 None:0 Recreating:1 Refreshing:0 Restarting:0 Verifying:0 ForceSendFields:[] NullFields:[]}. Errors [<a href="https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/xxxx:PERMISSIONS_ERROR]" rel="nofollow noreferrer">https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/xxxx:PERMISSIONS_ERROR]</a>".
Updated property [container/cluster].</p></li>
</ul>

<p>when I try to create through UI I get this</p>

<p><code>
Permission denied (HTTP 403): Google Compute Engine: Required 'compute.zones.get' permission for 'projects/xxxx/zones/us-central1-a'
</code></p>

<p>Have done a number on it!
My problem is that I don't see a way of giving permissions back to whatever account it is trying to use (as I cannot see that account <em>if</em> it exists) nor can I see how to attach a new service account with permissions that are needed to whatever is doing the work under the hood.</p>

<p>UPDATE:</p>

<p>So ...</p>

<p>I recreated the account at the organisation level. Gave it service account role there because you cannot modify the domain of the accounts at project level.</p>

<p>I have then modified that at the project level to have editor permissions.</p>

<p>This means i can deploy a cluster but ... still cannot create load balancer - insufficient permissions</p>

<pre><code>Error creating load balancer (will retry): Error getting LB for service default/bot: googleapi: Error 403: Required
 'compute.forwardingRules.get' permission for 'projects/xxxx/regions/europe-west2/forwardingRules/xxxx', forbidden
</code></pre>

<p>the user having the problem this time is:
<code>service-xxx@container-engine-robot.iam.gserviceaccount.com</code></p>

<p>So ...</p>

<p>I played with recreating accounts etc. Eventually got Kubernetes working again.
A week later tried to use datastore and discovered that AppEngine was dead beyond dead.</p>

<p>The only recourse was to start a new project from scratch.</p>

<p>The answer to this question is (some may laugh at its self evidence, but we are all in a rush at some point).</p>

<h1>DO NOT CREATE USER ACCOUNTS OR GIVE THEM PERMISSIONS BEYOND WHAT THEY NEED BECAUSE DELETING THEM LATER IS REALLY NOT WORTH THE RISK.</h1>

<p>Thankyou for listening :D</p>
