{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/10711","repository_url":"https://api.github.com/repos/hazelcast/hazelcast","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/10711/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/10711/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/10711/events","html_url":"https://github.com/hazelcast/hazelcast/issues/10711","id":234026812,"number":10711,"title":"IScheduledExecutorService deserializes tasks with Spring dependencies before the ApplicationContext is available","user":{"login":"jeiroa","id":12732000,"avatar_url":"https://avatars3.githubusercontent.com/u/12732000?v=4","gravatar_id":"","url":"https://api.github.com/users/jeiroa","html_url":"https://github.com/jeiroa","followers_url":"https://api.github.com/users/jeiroa/followers","following_url":"https://api.github.com/users/jeiroa/following{/other_user}","gists_url":"https://api.github.com/users/jeiroa/gists{/gist_id}","starred_url":"https://api.github.com/users/jeiroa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeiroa/subscriptions","organizations_url":"https://api.github.com/users/jeiroa/orgs","repos_url":"https://api.github.com/users/jeiroa/repos","events_url":"https://api.github.com/users/jeiroa/events{/privacy}","received_events_url":"https://api.github.com/users/jeiroa/received_events","type":"User","site_admin":false},"labels":[{"id":789013630,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Priority:%20Medium","name":"Priority: Medium","color":"87ffbf","default":false},{"id":793782133,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Source:%20Community","name":"Source: Community","color":"3afc37","default":false},{"id":137752575,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"109def","default":false},{"id":1740692,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Enhancement","name":"Type: Enhancement","color":"5319e7","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/37","html_url":"https://github.com/hazelcast/hazelcast/milestone/37","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/37/labels","id":531321,"number":37,"title":"Backlog","description":"This milestone is for grouping the issues which we want to consider in the far future","creator":{"login":"gurbuzali","id":1741680,"avatar_url":"https://avatars3.githubusercontent.com/u/1741680?v=4","gravatar_id":"","url":"https://api.github.com/users/gurbuzali","html_url":"https://github.com/gurbuzali","followers_url":"https://api.github.com/users/gurbuzali/followers","following_url":"https://api.github.com/users/gurbuzali/following{/other_user}","gists_url":"https://api.github.com/users/gurbuzali/gists{/gist_id}","starred_url":"https://api.github.com/users/gurbuzali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gurbuzali/subscriptions","organizations_url":"https://api.github.com/users/gurbuzali/orgs","repos_url":"https://api.github.com/users/gurbuzali/repos","events_url":"https://api.github.com/users/gurbuzali/events{/privacy}","received_events_url":"https://api.github.com/users/gurbuzali/received_events","type":"User","site_admin":false},"open_issues":274,"closed_issues":168,"state":"open","created_at":"2014-01-09T13:44:04Z","updated_at":"2018-02-19T11:42:50Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2017-06-06T21:14:50Z","updated_at":"2018-02-19T11:42:50Z","closed_at":"2018-02-19T11:42:50Z","author_association":"NONE","body":"  I am developing a project that has to run in a Glassfish server using Hazelcast 3.8.2 and Spring 4.1.2. The cluster has two members, there are no Hazelcast clients, Java version is OpenJDK Runtime Environment (IcedTea 2.5.1) (7u65-2.5.1-5~deb7u1) 64bits on a Debian GNU/Linux 7.6 (wheezy), kernel 3.2.0-4-amd64.\r\n\r\n  I have 4 tasks that are executed periodically by a IScheduledExecutorService (scheduleAtFixedRate method). These tasks have Spring services dependencies injected as transient attributes.\r\n\r\n  I have created StreamSerializer classes for all tasks to serialize the non-transient attributes in its write method. The transient attributes are reinjected in the read method accessing the ApplicationContext that is stored in a Singleton POJO, populated when the ContextRefreshedEvent is triggered by Spring just when the application has been started in the server.\r\n\r\n  Everything works fine when one Hazelcast node is running but when I start the second node I get NullPointerExceptions because it seems that Hazelcast tries to balance the tasks before the Spring ApplicationContext has been started.\r\n\r\n  This log is from the server when the first node is started:\r\n```\r\n2017-06-06 17:13:04.649 UTC || INFO  || [admin-thread-pool-57048(2) ] || [Connector] - Initialising connector ...\r\n2017-06-06 17:13:07.008 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - Received a ContextRefreshedEvent\r\n2017-06-06 17:13:07.008 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - ApplicationContext is now available\r\n2017-06-06 17:13:07.009 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - ContextRefreshedEvent processed\r\n2017-06-06 17:13:07.009 UTC || INFO  || [app.starter ] || [ConnectorStarter] - Starting connector...\r\n2017-06-06 17:13:08.029 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - Scheduled Task Manager start triggered.\r\n2017-06-06 17:13:08.029 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - BEGIN Scheduled Task Manager start triggered\r\n2017-06-06 17:13:08.029 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - BEGIN Scheduled Tasks initialisation\r\n2017-06-06 17:13:08.029 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - Getting fetchEventsFeedsScheduledExecutorService\r\n2017-06-06 17:13:08.032 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - Getting fetchEventsScheduledExecutorService\r\n2017-06-06 17:13:08.032 UTC || INFO  || [app.starter ] || [ScheduledTaskManager] - Creating schedulers\r\n...\r\n```\r\n\r\n  As you can see, the ApplicationContext is set at the beginning and then the schedulers are created so tasks work as expected. But when I start the second node I see these logs:\r\n```\r\n2017-06-06 17:13:45.378 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - BEGIN deserialize\r\n2017-06-06 17:13:45.379 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [FetchMarketPricesTaskSerializer] - BEGIN deserialize\r\n2017-06-06 17:13:45.379 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [FetchMarketPricesTaskSerializer] - FixtureMarketClient is null\r\n2017-06-06 17:13:45.379 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [FetchMarketPricesTaskSerializer] - CacheService is null\r\n2017-06-06 17:13:45.380 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [FetchMarketPricesTaskSerializer] - END deserialize\r\n2017-06-06 17:13:45.381 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - WorkerTaskManager is null\r\n2017-06-06 17:13:45.381 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - END deserialize\r\n2017-06-06 17:13:45.383 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - BEGIN deserialize\r\n2017-06-06 17:13:45.383 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [FetchFeedsTaskSerializer] - BEGIN deserialize\r\n2017-06-06 17:13:45.383 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [FetchFeedsTaskSerializer] - FeedClient is null\r\n2017-06-06 17:13:45.383 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [FetchFeedsTaskSerializer] - CacheService is null\r\n2017-06-06 17:13:45.384 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [FetchFeedsTaskSerializer] - END deserialize\r\n2017-06-06 17:13:45.384 UTC || WARN  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - WorkerTaskManager is null\r\n2017-06-06 17:13:45.384 UTC || INFO  || [hz.app.partition-operation.thread-0 ] || [SendTaskToWorkerTaskSerializer] - END deserialize\r\n2017-06-06 17:13:45.402 UTC || INFO  || [hz.app.cached.thread-4 ] || [SendTaskToWorkerTask] - BEGIN SendTaskToWorkerTask(FetchFeedsTask{enabled: true}, FETCH_EVENTS_FEEDS_EXECUTOR)\r\n2017-06-06 17:13:45.402 UTC || ERROR || [hz.app.cached.thread-4 ] || [SendTaskToWorkerTask] - Error executing the task\r\njava.lang.NullPointerException\r\n...\r\n2017-06-06 17:13:54.704 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - Received a ContextRefreshedEvent\r\n2017-06-06 17:13:54.704 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - ApplicationContext is now available\r\n2017-06-06 17:13:54.704 UTC || INFO  || [admin-thread-pool-57048(2) ] || [ConnectorStarter] - ContextRefreshedEvent processed\r\n2017-06-06 17:13:54.705 UTC || INFO  || [app-starter ] || [ConnectorStarter] - Starting connector...\r\n2017-06-06 17:13:54.899 UTC || ERROR || [hz.app.cached.thread-4 ] || [SendTaskToWorkerTask] - Error executing the task\r\njava.lang.NullPointerException\r\n2017-06-06 17:13:55.673 UTC || ERROR || [hz.app.cached.thread-4 ] || [SendTaskToWorkerTask] - Error executing the task\r\njava.lang.NullPointerException\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - Scheduled Task Manager start triggered.\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - BEGIN Scheduled Task Manager start triggered\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - BEGIN Scheduled Tasks initialisation\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - Getting fetchEventsFeedsScheduledExecutorService\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - Getting fetchEventsScheduledExecutorService\r\n2017-06-06 17:13:55.732 UTC || INFO  || [app-starter ] || [ScheduledTaskManager] - Creating schedulers\r\n...\r\n```\r\n\r\n  It seems the first thing the second node does is balancing the tasks from the other node in the hz.app.partition-operation.thread-0 thread, deserializing them in the appropriate StreamSerializer,  but transient Spring services to be injected are null (FixtureMarketClient, CacheService, WorkerTaskManager, and FeedClient) because the Spring ApplicationContext is not available yet because it is not set until 2017-06-06 17:13:54.704 UTC. All NullPointerExceptions logged after are because those services are null.\r\n\r\n  I have also tried using @SpringAware annotation in tasks to inject the Spring services but what it happens then is that the second node gets stuck and the server never starts up. I would say it is waiting for the context to be available but as it is not, it enters in a deadlock."},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/366665426","html_url":"https://github.com/hazelcast/hazelcast/issues/10711#issuecomment-366665426","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/10711","id":366665426,"user":{"login":"burakcelebi","id":928875,"avatar_url":"https://avatars1.githubusercontent.com/u/928875?v=4","gravatar_id":"","url":"https://api.github.com/users/burakcelebi","html_url":"https://github.com/burakcelebi","followers_url":"https://api.github.com/users/burakcelebi/followers","following_url":"https://api.github.com/users/burakcelebi/following{/other_user}","gists_url":"https://api.github.com/users/burakcelebi/gists{/gist_id}","starred_url":"https://api.github.com/users/burakcelebi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burakcelebi/subscriptions","organizations_url":"https://api.github.com/users/burakcelebi/orgs","repos_url":"https://api.github.com/users/burakcelebi/repos","events_url":"https://api.github.com/users/burakcelebi/events{/privacy}","received_events_url":"https://api.github.com/users/burakcelebi/received_events","type":"User","site_admin":false},"created_at":"2018-02-19T11:42:50Z","updated_at":"2018-02-19T11:42:50Z","author_association":"MEMBER","body":"Closing this as the recommended solution has been provided. Please let us know if you need further help. Thanks!"}}
{"actor":{"display_login":"burakcelebi"}}