user151: We've started using the 1.35.x node docker images in preparation for Vasil. I always start with the testnet before deploying to production. In our testnet deployment, the new images will not work. While synching with the chain, they hang up. If we restart them, we get a communication error. You can see these below:
Step 1: run node docker image from scratch (all relevant volumes are first deleted)
b8e82085:cardano.node.ChainDB:Notice:35] [2022-08-14 11:41:24.19 UTC] Chain extended, new tip: c7380283205467161f2a587da610ea79b482347ecc9a459f9ecf3030c03e5a6a at slot 63417270
[b8e82085:cardano.node.ChainDB:Error:35] [2022-08-14 11:41:24.48 UTC] Invalid block 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 at slot 63419637: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))
[b8e82085:cardano.node.ChainDB:Info:35] [2022-08-14 11:41:24.48 UTC] Valid candidate ce9c9431882310df559099e2bfa3e6eb31c95606d83ffc05568df7f863d4db60 at slot 63419598

We let the system run for 12 hours after that last line. No further activity occured. The node was hung up.
Step 2: restart the docker image
Since the node was hung up, we killed the container and started it up again. We left the volume mounts intact. The node then threw the following after boilerplate initalization:
[973dad53:cardano.node.DnsSubscription:Error:233] [2022-08-14 16:57:26.08 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Application Exception: 3.131.32.242:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))))
[973dad53:cardano.node.ErrorPolicy:Warning:119] [2022-08-14 16:57:26.08 UTC] IP 3.131.32.242:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))))))) 200s 200s
[973dad53:cardano.node.ChainDB:Info:102] [2022-08-14 16:57:34.32 UTC] Took ledger snapshot DiskSnapshot {dsNumber = 63343083, dsSuffix = Nothing} at 1f46697e362676f795cfe4608217c805bde9830fea2c304381cb6c9853b1a074 at slot 63343083
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.78 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a202:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.78 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 3.131.32.242:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.80 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a202:0:1:0:3]:3001
[973dad53:cardano.node.DnsSubscription:Notice:266] [2022-08-14 16:57:35.80 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 3.133.220.20:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a201:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:267] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 54.199.108.71:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt End, destination 3.131.32.242:3001 outcome: ConnectSuccessLast
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 3.133.220.20:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 54.199.108.71:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s

It seems to be complaining about connection with Cardano relays. This doesn't make sense to us since we if we repeat the above process it will start synchronizing again but eventually fail in the same way which suggests communication with the relays is possible. This machine has a stable internet connection. In fact it's running on the same machine as our production system which is working fine (still on an older version of the docker image).
We have tried 1.35.1 and 1.35.3. They both do the same thing.
These nodes are connected to the testnet.
Does anyone know why we can't get this node running? We're out of ideas for what could be wrong.

user252: 
We have tried 1.35.1 and 1.35.3.

You should ONLY use node 1.35.2 on testnet. 1.35.1 produced unforeseen invalid block on testnet network, which was fixed on node 1.35.2 and rules updated to discard the invalid blocks (to prevent fork) at node 1.35.3. Thus, the testnet network is in a state where you cannot use 1.35.3 or above, and 1.35.1 would produce invalid blocks.
The only node version that will be usable on that network is 1.35.2. IO have announced they will be moving to new networks (that have already been spun) soon, you can follow those via environments page. As regards docker, I dont use it myself but if these configs are not readily available, you can mount them on.

151: <p>We've started using the <code>1.35.x</code> node docker images in preparation for Vasil. I always start with the testnet before deploying to production. In our testnet deployment, the new images will not work. While synching with the chain, they hang up. If we restart them, we get a communication error. You can see these below:</p>
<p><strong>Step 1: run node docker image from scratch</strong> (all relevant volumes are first deleted)</p>
<pre><code>b8e82085:cardano.node.ChainDB:Notice:35] [2022-08-14 11:41:24.19 UTC] Chain extended, new tip: c7380283205467161f2a587da610ea79b482347ecc9a459f9ecf3030c03e5a6a at slot 63417270
[b8e82085:cardano.node.ChainDB:Error:35] [2022-08-14 11:41:24.48 UTC] Invalid block 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 at slot 63419637: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash &quot;eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605&quot;])))))])})))))))
[b8e82085:cardano.node.ChainDB:Info:35] [2022-08-14 11:41:24.48 UTC] Valid candidate ce9c9431882310df559099e2bfa3e6eb31c95606d83ffc05568df7f863d4db60 at slot 63419598
</code></pre>
<p>We let the system run for 12 hours after that last line. No further activity occured. The node was hung up.</p>
<p><strong>Step 2: restart the docker image</strong></p>
<p>Since the node was hung up, we killed the container and started it up again. We left the volume mounts intact. The node then threw the following after boilerplate initalization:</p>
<pre><code>[973dad53:cardano.node.DnsSubscription:Error:233] [2022-08-14 16:57:26.08 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Application Exception: 3.131.32.242:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash &quot;eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605&quot;])))))])})))))))))
[973dad53:cardano.node.ErrorPolicy:Warning:119] [2022-08-14 16:57:26.08 UTC] IP 3.131.32.242:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash &quot;eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605&quot;])))))])})))))))))))) 200s 200s
[973dad53:cardano.node.ChainDB:Info:102] [2022-08-14 16:57:34.32 UTC] Took ledger snapshot DiskSnapshot {dsNumber = 63343083, dsSuffix = Nothing} at 1f46697e362676f795cfe4608217c805bde9830fea2c304381cb6c9853b1a074 at slot 63343083
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.78 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Unsupported remote target address [2a05:d014:e00:a202:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.78 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Connection Attempt Start, destination 3.131.32.242:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.80 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Unsupported remote target address [2a05:d014:e00:a202:0:1:0:3]:3001
[973dad53:cardano.node.DnsSubscription:Notice:266] [2022-08-14 16:57:35.80 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Connection Attempt Start, destination 3.133.220.20:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.83 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Unsupported remote target address [2a05:d014:e00:a201:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:267] [2022-08-14 16:57:35.83 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Connection Attempt Start, destination 54.199.108.71:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.83 UTC] Domain: &quot;relays-new.cardano-testnet.iohkdev.io&quot; Connection Attempt End, destination 3.131.32.242:3001 outcome: ConnectSuccessLast
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 3.133.220.20:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = &quot;Parallel connection cancelled&quot;, seStack = []}))) 1s
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 54.199.108.71:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = &quot;Parallel connection cancelled&quot;, seStack = []}))) 1s
</code></pre>
<p>It seems to be complaining about connection with Cardano relays. This doesn't make sense to us since we if we repeat the above process it will start synchronizing again but eventually fail in the same way which suggests communication with the relays is possible. This machine has a stable internet connection. In fact it's running on the same machine as our production system which is working fine (still on an older version of the docker image).</p>
<p>We have tried <code>1.35.1</code> and <code>1.35.3</code>. They both do the same thing.
These nodes are connected to the testnet.</p>
<p>Does anyone know why we can't get this node running? We're out of ideas for what could be wrong.</p>
 252: <blockquote>
<p>We have tried 1.35.1 and 1.35.3.</p>
</blockquote>
<p>You should ONLY use node 1.35.2 on testnet. 1.35.1 produced unforeseen invalid block on testnet network, which was fixed on node 1.35.2 and rules updated to discard the invalid blocks (to prevent fork) at node 1.35.3. Thus, the testnet network is in a state where you cannot use 1.35.3 or above, and 1.35.1 would produce invalid blocks.</p>
<p>The only node version that will be usable on that network is 1.35.2. IO have announced they will be moving to new networks (that have already been spun) soon, you can follow those via <a href="https://book.world.dev.cardano.org/environments.html" rel="nofollow noreferrer">environments page</a>. As regards docker, I dont use it myself but if these configs are not readily available, you can mount them on.</p>
