How do I copy a SOW from one AMPS instance to another?


Use the spark command line tool. In current versions of AMPS, spark is included in the AMPS distribution.

The instructions in this FAQ are useful for a one-time transfer, for example, loading sample data into a QA environment for testing. For production use, AMPS replication is the recommended approach for low-latency delivery of messages from instance to instance as described in the AMPS User's Guide.

These instructions assume that your data does not contain embedded newlines: if your data does contain newlines, we recommend writing a simple program that simply does a SOW query on one instance and publishes the results to the other instance or using the -file option in more recent versions of spark.

Copy from a SOW File

To recreate a SOW from a server that is no longer running, use amps_sow_dump to extract the records from the SOW and use spark to publish those records into the destination AMPS instance.

This process assumes that the records do not contain embedded newlines, since amps_sow_dump uses newline characters to delimit messages.

$ amps_sow_dump  /path/to/sow/file.sow > records.txt
$ spark publish –server new_server –topic new_topic –type message_type –file records.txt

Copying from a SOW topic in a running instance

To recreate a SOW on a server that is currently running, use the spark sow command to retrieve records from the source topic, and use the spark publish command to publish records to the destination topic.

$ spark sow -server old_server topic old_topic -type message_type \
| spark publish –server new_server –topic new_topic –type message_type

Notice that, because this is an operation on a running instance of AMPS, any updates that are made to the SOW topic after the sow command runs will not be published to the new topic.

If the messages are binary or contain embedded delimiter characters, and the number of records can be easily stored on the system where the command is being run, you can use the -file flag to store messages to a compressed file, and then republish the messages from that file.

$ spark sow -server old_server topic old_topic -type message_type -file
$ spark publish –server new_server –topic new_topic –type message_type -file


Keywords: sow synchronization, move sow, copy sow, synchronize sow, move messages

Have more questions? Submit a request


  • Avatar
    Adam Mclaurin

    Dirk, what about bringing down Server A and manually copying the files to Server B? Of course, the configs would have to be consistent, but it would work, right?

  • Avatar
    Yip, Lophing

    Does the amps_sow_dump needs to be run be a user with the right entitlement or running it with the application ID will do?

  • Avatar
    Dirk Myers

    Moving files should work.

    There are some reasons why I consider moving a file to be a little less desirable than publishing messages in the general case (for the specific case you have in mind, it may be the right thing to do):

    1) You need to be using the same major/minor version of AMPS as the AMPS instance that created the source file. (Or use amps_upgrade as necessary to prepare the file for the new instance.)

    2) It requires taking both the source and destination instance down -- copying a file out from under a running AMPS instance isn't guaranteed to work (for example, if there's a record partially-written at the exact point in time you do the copy).

    3) If the SOW is removed again, the SOW will be out of sync again. Copying the SOW file only moves the SOW, and doesn't synchronize the SOW with the transaction log. Republishing the state directly gets to a known state that will synchronize and recover to that state at that point.

    4) There's a little more possibility for error involved in moving files around, having a file system you can mount on both instances, and getting them to the right spot for the new instance, etc.

  • Avatar
    Dirk Myers

    amps sow dump (omitting underscores to work around comment autoformatting) operates on the SOW file, so the AMPS entitlement system isn't involved.

    The user that runs amps sow dump needs to have read permission to the SOW file.

  • Avatar
    Yip, Lophing

    The user ID that runs "spark publish" would need the right entitlement?
    If so is there a spark Window version? Can the user use the client api to publisher the data from ampssowdump ?

  • Avatar
    Dirk Myers

    Yes, the user that runs spark publish must have permission to publish to the destination instance.

    The spark included with AMPS 4.0 and above is a Java application, and should run fine on Windows.

    If your installation uses authentication that requires an Authenticator (for example, it uses challenge-response), you will also need to provide spark with an Authenticator that implements that authentication (basically, the same thing an AMPS Java client application needs to have to authenticate to AMPS).

    And, yes, you could also use one of the AMPS client APIs to publish the data from the dump to AMPS, and that will also work just fine.

Powered by Zendesk