Troubleshooting Common Problems

Solve common problems may be encountered using GemFire XD.

ClassNotFoundException after Upgrade

After upgrading a GemFire XD member, you may receive a ClassNotFoundException when starting a member, referencing package names that begin with com.vmware.sqlfire or com.pivotal.sqlfire. These exceptions occur if you fail to stop and remove older procedure or listener implementations that use package names from the SQLFire product. Stop and remove these objects before upgrading to GemFire XD. See Performing a Manual Upgrade Using the RPM Distribution or Performing a Manual Upgrade Using the ZIP Distribution for more information..

Member Startup Problems

When you start GemFire XD members, startup delays can occur if specific disk store files on other members are unavailable. This is part of the normal startup behavior and is designed to help ensure data consistency. For example, consider the following startup message for a locator ("locator2):
GemFire XD Locator pid: 23537 status: waiting
Continuing the startup by booting the server1 data store yields:
Logs generated in /pivotal/server1/gfxdserver.log
My persistent id:

  DiskStore ID: aa77785a-0f03-4441-84f7-6eb6547d7833
  Location: /

Members with potentially new data:
  DiskStore ID: f417704b-fff4-4b99-81a2-75576d673547
  Location: /

The above messages for data stores and locators may indicate that some members were not started. If the indicated disk store persistence files are available on the missing member, simply start that member and allow the running members to recover. For example, in the above system you would simply start locator1 and allow locator2 and server1 to synchronize their data.

To avoid this type of delayed startup and recovery:
  1. When possible, shut down data store members (shut-down-all) after disk stores have been synchronized in the system. Shut down remaining locator members after the data stores have stopped.
  2. Make that that sure all persistent members are restarted properly. See Recovering from a ConflictingPersistentDataException for more information.
  3. If a member cannot be restarted and it is preventing other data stores from starting, use revoke-missing-disk-store to revoke the disk stores that are preventing startup. This can cause some loss of data if the revoked disk store actually contains recent changes to the data dictionary or to table data. The revoked disk stores cannot be added back to the system later. If you revoke a disk store on a member you will need to delete the associated disk files from that member in order to start it again. Only use the revoke-missing-disk-store command as a last resort.

Recovering from a ConflictingPersistentDataException

If you receive a ConflictingPersistentDataException during startup, it indicates that you have multiple copies of some persistent data and GemFire XD cannot determine which copy to use. Normally GemFire XD uses metadata to automatically determine which copy of persistent data to use. Each member persists, along with the data dictionary or table data, a list of other members that have the data and whether their data is up to date.

A ConflictingPersistentDataException happens when two members compare their metadata and find that it is inconsistent—they either don’t know about each other, or they both believe that the other member has stale data. The following are some scenarios that can cause a ConflictingPersistentDataException.

Independently-created copies

Trying to merge two independently-created distributed systems into a single distributed system causes a ConflictingPersistentDataException. There are a few ways to end up with independently-created systems:

  • Configuration problems may cause GemFire XD members connect to different locators that are not aware of each other. To avoid this problem, ensure that all locators and data stores always specify the same, complete list of locator addresses at startup (for example, locators=locator1[10334],locator2[10334],locator3[10334]). Consider using a single file for common configuration items, as described in Steps to Plan and Configure a GemFire XD Deployment.
  • All persistent members in a system may be shut down, after which a brand new set of different persistent members attempts to start up.

Trying to merge independent systems by pointing all members to the same set of locators then results in a ConflictingPersistentDataException.

GemFire XD cannot merge independently-created data for the same table. Instead, you need to export the data from one of the systems and import it into the other system. See Exporting and Importing Data with GemFire XD.

Starting new members first

Starting a brand new member with no persistent data before starting older members that have persistent data can cause a ConflictingPersistentDataException.

This can happen by accident if you shut down the system, then add a new member to the startup scripts, and finally start all members in parallel. In this case, the new member may start first. If this occurs, the new member creates an empty, independent copy of the data before the older members start up. When the older members start, the situation is similar to that described above in “Independently-created copies.”

In this case, the fix is simply to move aside or delete the (empty) persistence files for the new member, shut down the new member, and finally restart the older members. After the older members have fully recovered, restart the new member.

A network split, with enable-network-partition-detection set to false

With enable-network-partition-detection set to true, GemFire XD detects a network partition and shuts down members to prevent a "split brain." In this case no conflicts should occur when the system is restored.

However, if enable-network-partition-detection is false, GemFire XD cannot prevent a "split brain" after a network partition. Instead, each side of the network partition records that the other side of the partition has stale data. When the partition is healed and persistent members are restarted, they find a conflict because each side believes the other side's members are stale.

In some cases it may be possible to choose between sides of the network partition and keep only the data from one side of the partition. Otherwise you may need to salvage data and import it into a fresh system.

Resolving a ConflictingPersistentDataException


First, determine if there is one part of the system that you can recover. For example, if you just added some new members to the system, try to start up without including those members. For the remaining members, use the data extractor tool to extract data from the persistence files and import it into a running system. See Recovering Data from Disk Stores.

Connection Problems





WAN Replication Problems



Preventing Disk Full Errors


You can prevent disk file errors using the following techniques:


    Pre-allocation is governed by the following system properties:




Recovering from Disk Full Errors


  4. Restart the member.

See Handling Missing Disk Stores.