as one skilled in the art will appreciate , embodiments of the present invention may be embodied as , among other things : a method , system , or computer - program product . accordingly , the embodiments may take the form of a hardware embodiment , a software embodiment , or an embodiment combining software and hardware . in one embodiment , the present invention takes the form of a computer - program product that includes computer - useable instructions embodied on one or more computer - readable media . computer - readable media include both volatile and nonvolatile media , removable and nonremovable media , and contemplates media readable by a database , a switch , and various other network devices . network switches , routers , and related components are conventional in nature , as are means of communicating with the same . by way of example , and not limitation , computer - readable media comprise computer - storage media and communications media . computer - storage media , or machine - readable media , include media implemented in any method or technology for storing information . examples of stored information include computer - useable instructions , data structures , program modules , and other data representations . computer - storage media include , but are not limited to ram , rom , eeprom , flash memory or other memory technology , cd - rom , digital versatile discs ( dvd ), holographic media or other optical disc storage , magnetic cassettes , magnetic tape , magnetic disk storage , and other magnetic storage devices . these memory components can store data momentarily , temporarily , or permanently . communications media typically store computer - useable instructions — including data structures and program modules — in a modulated data signal . the term “ modulated data signal ” refers to a propagated signal that has one or more of its characteristics set or changed to encode information in the signal . an exemplary modulated data signal includes a carrier wave or other transport mechanism . communications media include any information - delivery media . by way of example but not limitation , communications media include wired media , such as a wired network or direct - wired connection , and wireless media such as acoustic , infrared , radio , microwave , spread - spectrum , and other wireless media technologies . combinations of the above are included within the scope of computer - readable media . fig1 is a block diagram of a system 100 that includes components such as client 102 , scoring manager 104 , scoring engine 106 , and primitive library 108 . each component includes a communication interface . the communication interface may be an interface that can allow a component to be directly connected to any other component or allows the component to be connected to another component over network 110 . network 110 can include , for example , a local area network ( lan ), a wide area network ( wan ), cable system , telco system , or the internet . in an embodiment , a component can be connected to another device via a wireless communication interface through the network 110 . client 102 may be or can include a desktop computer , a laptop computer or other mobile computing device , a network - enabled cellular telephone ( with or without media capturing / playback capabilities ), a server , a wireless email client , or other client , machine or device , or any combination of the above , to perform various tasks including web browsing , search , electronic mail ( email ) and other tasks , applications and functions . client 102 may additionally be any portable media device such as digital still camera devices , digital video cameras ( with or without still image capture functionality ), media players such as personal music players and personal video players , and any other portable media device , or any combination of the above . scoring manager 104 is utilized for administering and scoring constructed response items for a user . the scoring manager 104 may also be utilized to generate individual expert systems to represent the scoring knowledge for a single constructed response item . the scoring manager 104 additionally may be configured to refine each expert system and test it against a broad range of student responses . in an embodiment , scoring manager 104 is a server external to client 102 . in another embodiment , scoring manger 104 may be an application that resides and is executable within client 102 . as shown , scoring engine 106 and primitive library 108 are components that reside within scoring manager 104 . in other embodiments , one or more of the scoring engine 106 and primitive library 108 may be external to the scoring manager 104 . the scoring engine 106 is a component that receives a user &# 39 ; s response to a question and evaluates the response against a scoring rubric . the scoring engine 106 may include or have access to the library of primitives 108 . in an embodiment , the primitive library 108 may include the calculation of distances , slopes , comparisons of strings and numbers , and other basic operations . in order to make the scoring engine 106 general so that it can support a very large range of items , the primitive library 108 may be low - level and higher order predicates may be created from the primitive library 108 . in other embodiments , complex predicates may be added to the primitive library . in an embodiment , in using the primitive library 108 , the language for representing a scoring rubric may enable the library functions to reference elements including , but not limited to , object sets , objects , attributes of objects , as well as transformations of any of these elements . fig2 is a screenshot of an exemplary user interface ( ui ) 200 for collecting student responses . the ui 200 may be presented through an application that is part of the client 102 and / or the scoring manager 104 . in an embodiment , through the ui 200 , the invention uses items developed for user responses collected on a cartesian grid to illustrate points . in other embodiments , the invention can be applied to user responses to other types of items using other response modes . in an embodiment , the ui 200 is referred to as an interactive grid ( ig ). a broad range of different item types can be presented in the ig . the ui 200 may be used to ensure that user responses are collected with a consistent mechanism that creates and transmits a data structure to a scoring engine . a user response may comprise a set of objects , each of which may have one or more attributes . for example , the ui can produce a collection of objects that may include points , line segments connecting points , geometric objects comprised of connected line segments , and user - defined atomic objects , such as the weights 202 on the left palette in fig2 . each object may be characterized by an ordered set of points . for example , the lines on the bottom - center of the weights 202 may be an example of an ordered set of points . in an embodiment , the ui 200 can return a data structure containing these objects to the scoring engine 106 . in an embodiment , objects have properties that include , but are not limited to , locations , names , labels , and values . in another embodiment , the ui 200 can be configured to capture natural language where the object set may include elements of a semantic network derived from a parse of the text provided by the user . alternatively , the ui 200 can be configured to capture input from an equation editor representing sequences of symbols as the initial set of objects . moreover , in other embodiments , an application to test proficiency with a computer program may capture menu commands , keyboard input , or mouse events as the set of objects . however , this list is intended to be exemplary rather than exhaustive . in an embodiment , a scoring rubric may be defined in three sequential stages : a binding stage , in which references to elements are established ; an assertion stage , in which assertions about elements are evaluated and stored ; and a scoring stage , in which a score is assigned based on the values of the results of the assertions . xml - based language may be used for implementing these stages for the ui responses . fig3 illustrates exemplary binding statements used in the binding stage . fig3 presents two exemplary binding statements . the first , “ selectobjectset ” binds a subset of the input set to the variable “ s1 .” the first binding statement collects all of the objects that have at least one side ( numberofsides gt 0 ). the symbol “@” is bound sequentially to each object in the input set . the second statement in fig3 , “ bind ,” creates an additional binding , associating the symbol “ s1count ” with the number of elements contained in set “ s1 .” the symbol “$” dereferences the previously bound variable “ s1 .” an assertion is a predicate that is either true or false . the assertion further is an atomic unit from which scoring rubrics can be built . each assertion can be named for later reference in the scoring stage . fig4 illustrates exemplary assertions according to an embodiment of the invention . in fig4 , the first two assertions dereference the previously bound integer “ s1count ” and assert that it is ( respectively ) equal to four and less than four . these assertions are named by the user , for example , “ fourobjects ” and “ fewerobjects .” the third assertion references another previously bound integer , which is the count of objects in another set meeting some other set of conditions . in this example , the third assertion is named “ fourgood ” and is true if the value of “ s3count ” is 4 . in the scoring stage , named assertions are collected in a set of and - or trees , one tree for each numeric score point . an exemplary snippet from a scoring specification for a three - point item appears in fig5 according to an embodiment of the invention . in this case , full credit is assigned if there are four objects represented and all four meet whatever criteria was used to construct set “ s3 ,” and thus leading to the bound variable “ s3count ” above . the representation of annotated and - or trees is well known in the computer science art . in an embodiment , the internal representation used is a set of nodes , in which each node has a list of children , each of which can be an and node , an or node , or an assertion node . the resulting internal representation of the binding , assertion , and scoring trees comprises an answer set that includes an expert system embodying the knowledge of the scoring rubric for a particular item . the scoring rubric may be written directly in the specification language or authoring tools may be developed to help test developers specify the rubrics . in some embodiments , tools may be domain specific . fig6 is a block diagram of an exemplary method 600 for scoring user responses . in an embodiment , the three - stage answer set is applied to the set of elements returned by the ui . one practical value of method 600 is that this process facilitates the use of a low - level library of primitives which can reduce or eliminate the need for any programming when defining a very broad range of new items or item types . in an embodiment , method 600 can integrate the assertion and scoring stage into one stage . at operation 602 , a user response is captured as a collection of objects with attributes . in an embodiment , the response is captured through a ui such as ui 200 ( fig2 ). at operation 604 , a component binds the variables identified in a binding tree . in an embodiment , the component is a scoring engine such as scoring engine 106 ( fig1 ). at operation 606 , results ( true or false ) are stored for each named assertion . at operation 608 , starting with the highest possible score , a scoring tree is evaluated , stopping when the subtree associated with a score evaluates to true . various implementations also provide an enhanced method of “ rangefinding ” which refines expert systems and tests them against a broad range of student responses . rangefinding is a committee process in which subject - matter experts agree on appropriate scores for sample examinee responses . during rangefinding , a small sample of items , often in the range of 25 - 100 , are reviewed by committees to test the application of the scoring rubrics . during this process , refinements are made to the rubric , and sample papers are selected to train scoring staff on the accurate scoring of responses to the item . however , improvements are needed for enhancing the rangefinding process . the invention provides such improvements . for example , decisions of the rangefinding committee can be expressed formally as assertions in the language used to define the scoring rubrics . formalizing the committee results as a series of explicit rules improves the accuracy of scoring , and would likely lead to more reliable scoring even when scoring is done by human scorers . furthermore , committee decisions can be systematically tested against the full set of field - test data to locate unintended consequences of the proposed new rules . fig7 is a block diagram of an exemplary method 700 for refining a scoring rubric . at operation 702 , items are field tested and are scored either in real time or after data collection . at operation 704 , a sample of responses are identified for transmission to a rangefinding committee . in an embodiment , the sample of responses may be selected by combing a small random sample with student responses selected to represent the work of otherwise high - or low - performing students . this may be done because otherwise high - performing students may score poorly on the item , or otherwise low - performing students may score well on the item . at operation 706 , items and corresponding scores are provided to the rangefinding committee . in an embodiment , the rangefinding committee is trained in the formal specifications of the scoring rubric . in instances where the committee reaches a consensus that a score is incorrect , at operation 708 , one or more piles or principles are identified that differentiates the correct score from the incorrect scores . at operation 710 , a modification to the scoring rubric , corresponding to the indentified rules , is provided . at operation 712 , the identified rules for modifying the scoring rubric are applied to field test responses in order to identify any unintended consequences of the new rules . in an embodiment , this may be done by identifying scores that changed under the new rules and evaluating those changes . at operation 714 , a consensus on whether to fully implement the new rules is achieved based on the modification to the formal scoring rubric . in an embodiment , the consensus is achieved after the committee reviews a new sample of responses for which the revision resulted in a change of scores and determines that the changes are limited to those intended . referring to fig8 , a method for use in defining a scoring rubric that may be used , by way of example and not by way of limitation , with the various systems described above includes five steps . the first step allows the user to create sets of items by specifying their properties . in one implementation , this is accomplished through a series of menu selections . in one instance , the user selects properties or functions from categories of functions or properties ( e . g ., location , geometric properties , etc .). properties might be specified in response to questions or by visually drawing ( locations for example ) on a representation of the answer grid . in the second step , the user names sets , properties of sets , elements of the created sets , and / or properties of the elements . again , the properties may be arranged according to their type or functions and accessed through menus , icons , or hand - entered . users may want to create sets based on properties of other sets or their elements . for example , a rubric may require information about all objects within a given distance of another object ( say , object a ). users might first select all object a as in the first step ( set a ), then bind the location of the desired element of set a to a variable or set of variables . those variables could then be used to create another set ( say , set b ), which consists of all objects within a certain radius of object a . the process then allows users to make assertions about sets or bound variables ( step 4 ). these true / false assertions might be selected from a menu or categorized tree or other interface . individual assertions may be combined using boolean logic to create complex assertions . each assertion must evaluate to either true or false . for example , an assertion might assert that a single object is within the prescribed radius of object a , or that single object a is found in set a , or that both conditions are true . in step 4 scores are associated with sets of assertions . an interface might represent the set of assertions as a tree , with nodes representing basic logical operators and leaves representing previously defined assertions . for example , if assertion 1 is true and assertion 2 is true , then the student receives a 1 . various interfaces might facilitate this specification . in the final step , the system allows the user to test the rubric and refine it . in a good implementation , this provides information about all bound variables and sets , and provides a mechanism for the user to go back and refine the rubric . one implementation of a tool for defining constructed responses is described on pages 11 through 35 below . furthermore , the attached appendix describes an implementation of a tool for defining constructed responses . while particular embodiments of the invention have been illustrated and described in detail herein , it should be understood that various changes and modifications might be made to the invention without departing from the scope and intent of the invention . the embodiments described herein are intended in all respects to be illustrative rather than restrictive . alternate embodiments will become apparent to those skilled in the art to which the present invention pertains without departing from its scope . from the foregoing it will be seen that this invention is one well adapted to attain all the ends and objects set forth above , together with other advantages , which are obvious and inherent to the system and method . it will be understood that certain features and sub - combinations are of utility and may be employed without reference to other features and sub - combinations .