{"action":"created","issue":{"url":"https://api.github.com/repos/dealii/dealii/issues/8729","repository_url":"https://api.github.com/repos/dealii/dealii","labels_url":"https://api.github.com/repos/dealii/dealii/issues/8729/labels{/name}","comments_url":"https://api.github.com/repos/dealii/dealii/issues/8729/comments","events_url":"https://api.github.com/repos/dealii/dealii/issues/8729/events","html_url":"https://github.com/dealii/dealii/issues/8729","id":491843799,"node_id":"MDU6SXNzdWU0OTE4NDM3OTk=","number":8729,"title":"PETSc issue: Convergence failure with hp-adaptivity","user":{"login":"marcfehling","id":18285973,"node_id":"MDQ6VXNlcjE4Mjg1OTcz","avatar_url":"https://avatars0.githubusercontent.com/u/18285973?v=4","gravatar_id":"","url":"https://api.github.com/users/marcfehling","html_url":"https://github.com/marcfehling","followers_url":"https://api.github.com/users/marcfehling/followers","following_url":"https://api.github.com/users/marcfehling/following{/other_user}","gists_url":"https://api.github.com/users/marcfehling/gists{/gist_id}","starred_url":"https://api.github.com/users/marcfehling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcfehling/subscriptions","organizations_url":"https://api.github.com/users/marcfehling/orgs","repos_url":"https://api.github.com/users/marcfehling/repos","events_url":"https://api.github.com/users/marcfehling/events{/privacy}","received_events_url":"https://api.github.com/users/marcfehling/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2019-09-10T19:01:32Z","updated_at":"2019-09-11T14:41:59Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I continued to prepare scaling plots with parallel hp-adaptivity for my graduation after tracing back some issues to `hypre` in #8565. But I encountered a different issue with `PETSc`, which also occurs with the 'compatible' version of `hypre`: The `SolverCG` object does not converge by solving the Poisson equation from `step-27`, even when using `PreconditionJacobi`. I am still working with the same scenario introduced in #8565.\r\n\r\nI'll post some logs from some runs on our `Jureca Booster` supercomputer below, so that you may get a picture yourself. By checking the logs for the failing `PETSc` cases, I was questioning whether `SolverControl` has a hard-coded maximum number of iterations set to `10000`?\r\n\r\nHowever, using `SolverCG` from `Trilinos` with `ML` as an AMG preconditioner, there are no convergence issues so far. The only concern I have right now is that the number of iterations explode with increasing number of degrees of freedom (and constraints). I should maybe provide better suited properties to the `AdditionalData` object in case of hp-adaptivity, e.g. enabling `higher_order_elements` since we do not solely work with linear elements anymore. @bangerth @tjhei @kronbichler -- Maybe you have some suggestions in this regard, since I don't have that much experience picking the right combination of solver and preconditioner...\r\n\r\nAt least for now, I will continue to use `Trilinos` for my investigations and come back to this issue later, when the thesis is handed in. I will start by checking if this is also an issue in the non-hp case, so that it is maybe related to the domain rather than to hp-adaptivity.\r\n\r\n---\r\n\r\n# Logs\r\n\r\n## PETSc\r\n\r\n### Code:\r\n```\r\nSolverControl solver_control(dof_handler.n_dofs(), 1e-8);\r\nLinearAlgebraPETSc::SolverCG  cg(solver_control, mpi_communicator);\r\n\r\nLinearAlgebraPETSc::MPI::PreconditionAMG                 preconditioner;\r\nLinearAlgebraPETSc::MPI::PreconditionAMG::AdditionalData data;\r\ndata.symmetric_operator = true;\r\n\r\npreconditioner.initialize(system_matrix, data);\r\ncg.solve(system_matrix,\r\n         completely_distributed_solution,\r\n         system_rhs,\r\n         preconditioner);\r\n```\r\n\r\n### `PreconditionAMG` & `data.symmetric_operator = true;`\r\n```\r\nRunning with PETSc on 384 MPI rank(s)...\r\nCycle 0:\r\n   Number of active cells      : 49152\r\n   Number of degrees of freedom: 198144\r\n   Number of constraints       : 3792\r\n   Solved in 8 iterations.\r\nCycle 1:\r\n   Number of active cells      : 58818\r\n   Number of degrees of freedom: 297682\r\n   Number of constraints       : 12471\r\n   Solved in 11 iterations.\r\nCycle 2:\r\n   Number of active cells      : 63213\r\n   Number of degrees of freedom: 424413\r\n   Number of constraints       : 37948\r\n   Solved in 12 iterations.\r\nCycle 3:\r\n   Number of active cells      : 67929\r\n   Number of degrees of freedom: 560387\r\n   Number of constraints       : 77877\r\n   Solved in 21 iterations.\r\nCycle 4:\r\n   Number of active cells      : 75966\r\n   Number of degrees of freedom: 738436\r\n   Number of constraints       : 117773\r\n   Solved in 33 iterations.\r\nCycle 5:\r\n   Number of active cells      : 78216\r\n   Number of degrees of freedom: 899142\r\n   Number of constraints       : 155587\r\n   Solved in 59 iterations.\r\nCycle 6:\r\n   Number of active cells      : 80556\r\n   Number of degrees of freedom: 1068303\r\n   Number of constraints       : 188171\r\n--------------------------------------------------------\r\nAn error occurred in line <148> of file </p/project/cias-7/fehling1/dealii/source/lac/petsc_solver.cc> in function\r\n    void dealii::PETScWrappers::SolverBase::solve(const dealii::PETScWrappers::MatrixBase &, dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::PreconditionerBase &)\r\nThe violated condition was:\r\n    false\r\nAdditional information:\r\nIterative method reported convergence failure in step 5. The residual in the last step was 0.0212258.\r\n\r\nThis error message can indicate that you have simply not allowed a sufficiently large number of iterations for your iterative solver to converge. This often happens when you increase the size of your problem. In such cases, the last residual will likely still be very small, and you can make the error go away by increasing the allowed number of iterations when setting up the SolverControl object that determines the maximal number of iterations you allow.\r\n\r\nThe other situation where this error may occur is when your matrix is not invertible (e.g., your matrix has a null-space), or if you try to apply the wrong solver to a matrix (e.g., using CG for a matrix that is not symmetric or not positive definite). In these cases, the residual in the last iteration is likely going to be large.\r\n--------------------------------------------------------\r\n```\r\n\r\n### `PreconditionAMG` & `data.symmetric_operator = false;`\r\n```\r\nRunning with PETSc on 384 MPI rank(s)...\r\nCycle 0:\r\n   Number of active cells      : 49152\r\n   Number of degrees of freedom: 198144\r\n   Number of constraints       : 3792\r\n   Solved in 8 iterations.\r\nCycle 1:\r\n   Number of active cells      : 58818\r\n   Number of degrees of freedom: 297682\r\n   Number of constraints       : 12471\r\n   Solved in 11 iterations.\r\nCycle 2:\r\n   Number of active cells      : 63213\r\n   Number of degrees of freedom: 424413\r\n   Number of constraints       : 37948\r\n   Solved in 12 iterations.\r\nCycle 3:\r\n   Number of active cells      : 67929\r\n   Number of degrees of freedom: 560387\r\n   Number of constraints       : 77877\r\n   Solved in 21 iterations.\r\nCycle 4:\r\n   Number of active cells      : 75966\r\n   Number of degrees of freedom: 738436\r\n   Number of constraints       : 117773\r\n   Solved in 33 iterations.\r\nCycle 5:\r\n   Number of active cells      : 79296\r\n   Number of degrees of freedom: 906384\r\n   Number of constraints       : 155140\r\n--------------------------------------------------------\r\nAn error occurred in line <148> of file </p/project/cias-7/fehling1/dealii/source/lac/petsc_solver.cc> in function\r\n    void dealii::PETScWrappers::SolverBase::solve(const dealii::PETScWrappers::MatrixBase &, dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::PreconditionerBase &)\r\nThe violated condition was: \r\n    false\r\nAdditional information: \r\nIterative method reported convergence failure in step 10000. The residual in the last step was 3.32354e-05.\r\n\r\nThis error message can indicate that you have simply not allowed a sufficiently large number of iterations for your iterative solver to converge. This often happens when you increase the size of your problem. In such cases, the last residual will likely still be very small, and you can make the error go away by increasing the allowed number of iterations when setting up the SolverControl object that determines the maximal number of iterations you allow.\r\n\r\nThe other situation where this error may occur is when your matrix is not invertible (e.g., your matrix has a null-space), or if you try to apply the wrong solver to a matrix (e.g., using CG for a matrix that is not symmetric or not positive definite). In these cases, the residual in the last iteration is likely going to be large.\r\n--------------------------------------------------------\r\n```\r\n\r\n### `PreconditionJacobi`\r\n```\r\nRunning with PETSc on 384 MPI rank(s)...\r\nCycle 0:\r\n   Number of active cells      : 49152\r\n   Number of degrees of freedom: 198144\r\n   Number of constraints       : 3792\r\n   Solved in 720 iterations.\r\nCycle 1:\r\n   Number of active cells      : 58935\r\n   Number of degrees of freedom: 298551\r\n   Number of constraints       : 13110\r\n   Solved in 1206 iterations.\r\nCycle 2:\r\n   Number of active cells      : 63627\r\n   Number of degrees of freedom: 428033\r\n   Number of constraints       : 40082\r\n   Solved in 1788 iterations.\r\nCycle 3:\r\n   Number of active cells      : 72234\r\n   Number of degrees of freedom: 603781\r\n   Number of constraints       : 79870\r\n   Solved in 2445 iterations.\r\nCycle 4:\r\n   Number of active cells      : 78540\r\n   Number of degrees of freedom: 778115\r\n   Number of constraints       : 125120\r\n   Solved in 3180 iterations.\r\nCycle 5:\r\n   Number of active cells      : 84201\r\n   Number of degrees of freedom: 973346\r\n   Number of constraints       : 172439\r\n   Solved in 4057 iterations.\r\nCycle 6:\r\n   Number of active cells      : 87828\r\n   Number of degrees of freedom: 1189531\r\n   Number of constraints       : 229537\r\n   Solved in 5711 iterations.\r\nCycle 7:\r\n   Number of active cells      : 94614\r\n   Number of degrees of freedom: 1452985\r\n   Number of constraints       : 289226\r\n--------------------------------------------------------\r\nAn error occurred in line <148> of file </p/project/cias-7/fehling1/dealii/source/lac/petsc_solver.cc> in function\r\n    void dealii::PETScWrappers::SolverBase::solve(const dealii::PETScWrappers::MatrixBase &, dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::VectorBase &, const dealii::PETScWrappers::PreconditionerBase &)\r\nThe violated condition was: \r\n    false\r\nAdditional information: \r\nIterative method reported convergence failure in step 10000. The residual in the last step was 1.36075e-08.\r\n\r\nThis error message can indicate that you have simply not allowed a sufficiently large number of iterations for your iterative solver to converge. This often happens when you increase the size of your problem. In such cases, the last residual will likely still be very small, and you can make the error go away by increasing the allowed number of iterations when setting up the SolverControl object that determines the maximal number of iterations you allow.\r\n\r\nThe other situation where this error may occur is when your matrix is not invertible (e.g., your matrix has a null-space), or if you try to apply the wrong solver to a matrix (e.g., using CG for a matrix that is not symmetric or not positive definite). In these cases, the residual in the last iteration is likely going to be large.\r\n--------------------------------------------------------\r\n```\r\n\r\n# Trilinos\r\n\r\n## `PreconditionAMG` & default `AdditionalData`\r\n```\r\nRunning with Trilinos on 384 MPI rank(s)...\r\nCycle 0:\r\n   Number of active cells      : 49152\r\n   Number of degrees of freedom: 198144\r\n   Number of constraints       : 3792\r\n   Solved in 18 iterations.\r\nCycle 1:\r\n   Number of active cells      : 58758\r\n   Number of degrees of freedom: 297499\r\n   Number of constraints       : 12573\r\n   Solved in 29 iterations.\r\nCycle 2:\r\n   Number of active cells      : 63168\r\n   Number of degrees of freedom: 424183\r\n   Number of constraints       : 37944\r\n   Solved in 41 iterations.\r\nCycle 3:\r\n   Number of active cells      : 68919\r\n   Number of degrees of freedom: 572452\r\n   Number of constraints       : 75721\r\n   Solved in 51 iterations.\r\nCycle 4:\r\n   Number of active cells      : 77154\r\n   Number of degrees of freedom: 751833\r\n   Number of constraints       : 114273\r\n   Solved in 60 iterations.\r\nCycle 5:\r\n   Number of active cells      : 80418\r\n   Number of degrees of freedom: 920478\r\n   Number of constraints       : 152049\r\n   Solved in 114 iterations.\r\nCycle 6:\r\n   Number of active cells      : 84798\r\n   Number of degrees of freedom: 1103671\r\n   Number of constraints       : 186474\r\n   Solved in 412 iterations.\r\nCycle 7:\r\n   Number of active cells      : 88983\r\n   Number of degrees of freedom: 1292478\r\n   Number of constraints       : 217475\r\n   Solved in 892 iterations.\r\nCycle 8:\r\n   Number of active cells      : 91470\r\n   Number of degrees of freedom: 1491980\r\n   Number of constraints       : 247361\r\n   Solved in 2022 iterations.\r\nCycle 9:\r\n   Number of active cells      : 95511\r\n   Number of degrees of freedom: 1729013\r\n   Number of constraints       : 293411\r\n   Solved in 5087 iterations.\r\nCycle 10:\r\n   Number of active cells      : 106944\r\n   Number of degrees of freedom: 2063872\r\n   Number of constraints       : 366391\r\n   Solved in 13309 iterations.\r\n```"},"comment":{"url":"https://api.github.com/repos/dealii/dealii/issues/comments/530412626","html_url":"https://github.com/dealii/dealii/issues/8729#issuecomment-530412626","issue_url":"https://api.github.com/repos/dealii/dealii/issues/8729","id":530412626,"node_id":"MDEyOklzc3VlQ29tbWVudDUzMDQxMjYyNg==","user":{"login":"marcfehling","id":18285973,"node_id":"MDQ6VXNlcjE4Mjg1OTcz","avatar_url":"https://avatars0.githubusercontent.com/u/18285973?v=4","gravatar_id":"","url":"https://api.github.com/users/marcfehling","html_url":"https://github.com/marcfehling","followers_url":"https://api.github.com/users/marcfehling/followers","following_url":"https://api.github.com/users/marcfehling/following{/other_user}","gists_url":"https://api.github.com/users/marcfehling/gists{/gist_id}","starred_url":"https://api.github.com/users/marcfehling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcfehling/subscriptions","organizations_url":"https://api.github.com/users/marcfehling/orgs","repos_url":"https://api.github.com/users/marcfehling/repos","events_url":"https://api.github.com/users/marcfehling/events{/privacy}","received_events_url":"https://api.github.com/users/marcfehling/received_events","type":"User","site_admin":false},"created_at":"2019-09-11T14:41:59Z","updated_at":"2019-09-11T14:41:59Z","author_association":"CONTRIBUTOR","body":"> Two quick comments:\r\n> You are using CG, which means your system and preconditioner need to be SPD. If they are not, CG can fail at any time (break down) or not converge. So you need to use `data.symmetric_operator=true`, otherwise PETSc might use a smoother or coarse solver that is not symmetric.\r\n> \r\n> Secondly, I would suggest experimenting with GMRES first as it is typically the most robust method and does not require for the system to be SPD. For timings you don't want to do that, but maybe one of the constraints breaks positive definiteness. If GMRES works but CG doesn't you know what the problem is...\r\n\r\nSorry, I missed that comment...\r\n\r\nI tried `GMRES` with `Trilinos` on that case to check if there are any performance differences, but there really is none. Haven't tried that with `PETSc` yet -- Will do!"}}
{"actor":{"display_login":"marcfehling"}}