Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Design question: handling hundreds of state machines in a Web context

by Anonymous Monk
on Jan 02, 2013 at 16:24 UTC ( #1011309=perlquestion: print w/ replies, xml ) Need Help??





Comment on Design question: handling hundreds of state machines in a Web context
by Corion (Pope) on Jan 02, 2013 at 16:36 UTC

    You'll need to specify in more detail what you mean by accuracy. I assume that "accuracy" relates to the workflow(s) in progress, that is, the state of each state machine. Personally, I would at least write a continous log of each state change of each machine to be able to easily replay/restore a crashed session. Personally, I'm really fond of pushing the problem of keeping (shared) state to a database, so I would at least store the state and possibly also the log of the transition(s) of each state machine in database tables.

    One system that sounds a bit like what you're doing is Deliantra, a MORPG written by Marc Lehmann, the author of AnyEvent (among other things). I think it supports a fairly large number of clients and think its overall architecture is likely worth investigating.

      "Accuracy" means that if a machine should move from state A to state B, that transition takes place.

        Sure, but at what point do you accept the legal responsibility for the transition with regards to your customer, and how will you handle recovery? Is it OK to replay (a series of) transitions if you keep the state in memory and write the state to disk every five minutes, while keeping a transition log? Do you want/need two-phase commit, where you send a confirmation for each processed transition?

        How consistent needs the overall state of the system to be? Is it OK if all transitions for client A were processed but only the first half of the transitions for client B were processed? What if A and B own ultiple machines? What is the processing order of the transitions? Is it OK to process transitions in parallel across different threads? Is it OK to reorder transitions for a single state machine? Can a transition be cancelled or can it time out?

        Most of these guarantees should be answered by a proper messaging system, like IBM MQSeries or maybe ZeroMQ nowadays, or alternatively by having all clients write directly to a database. I think you will need one, but I don't have much experience with the advantages or disadvantages of such queue systems.

by RichardK (Priest) on Jan 02, 2013 at 18:34 UTC

    I agree with Corion, store your state in an ACID database then state changes can happen in transactions. I'd choose postgresql but YMMV.

    From what you've described, I don't think you'd get much advantage from a queueing system but without seeing a full & detailed spec it's difficult to tell :)

by BrowserUk (Pope) on Jan 02, 2013 at 19:21 UTC
    Our primary concerns are performance and accuracy.

    The 'state' of any individual user is simply their ID, their current workflow ID, and the step they are currently on.

    A workflow is its ID, and (making a few assumptions), a list of templates representing the appropriate form for each step.

    Each of these can be stored in a table index by ID. When a user logs in, requesting two such small pieces of data directly by their promary index should be absolutely no problem for any RDBMS worthy of the name; even with 1000s of concurrent users. (Who will spend most of their logged on time staring at the screen or typing, the load on the DB will be minimal.) So performance should be no problem.

    As for accuracy, once the user clicks submit on any given step, there are a few possibilities:

    1. The transmission gets lost in transit.

      Assuming you aren't using too much client side trickery, the browser will time out and the user should be able to click the back button and attempt to re-submit.

      The user expectations of reliability should not be high if they fail to sucessfully submit as the intervening internet is not your responsibility.

    2. The submit completes, but your webserver crashes before the state is saved.

      Use a good webserver and test your cgis thoroughly.

    3. The webserver extracts the formdata and issues the DB update, but the DB server crashes.

      Use a reliable DB server.

    4. The update happens but is subsequently lost due DB server crash or disk failure.

      This is bread and butter DB stuff. Use a reliable RDBMS.

    In essence, on the basis of your description, this sounds very similar -- in terms of data flows -- to any multi-step ordering/shopping basket/booking process already actioned on a million websites. Ie. nothing extraordinary.

    Use reliable tools and don't get too complicated with either your client-side or server side processing.

    In the absence of evidence, opinion is indistinguishable from prejudice.
by sundialsvc4 (Monsignor) on Jan 02, 2013 at 21:07 UTC






Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1011309]
Front-paged by Arunbear
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-07-14 02:06 GMT
Find Nodes?
    Voting Booth?

    When choosing user names for websites, I prefer to use:

    Results (254 votes), past polls