the preferred embodiments of the present invention will now be described with reference to fig1 - 4 of the drawings . reference will now be made in detail to embodiment of the present invention . such embodiments are provided by way of explanation of the present invention , which is not intended to be limited thereto . in fact , those of ordinary skill in the art may appreciated upon reading the present specification and viewing the present drawings that various modifications and variations can be made thereto . although the illustrative embodiment will be generally described in the context of an application program running on a personal computer , those skilled in the art will recognize that the present invention may be implemented in conjunction with operating system programs or with other types of program modules for other types of computers . furthermore , those skilled in the art will recognize that the present invention may be implemented in a stand - alone or in a distributed computing environment . in a distributed computing environment , program modules may be physically located in different local and remote memory storage devices . execution of the program modules may occur locally in a stand - alone manner or remotely in a client server manner . examples of such distributed computing environments include local area networks and the internet . the detailed description that follows is represented largely in terms of processes and symbolic representations of operations by conventional computer components , including a processing unit ( a processor ), memory storage devices , connected display devices , and input devices . furthermore , these processes and operations may utilize conventional computer components in a heterogeneous distributed computing environment , including remote file servers , compute servers , and memory storage devices . each of these conventional distributed computing components is accessible by the processor via a communication network . the processes and operations performed by the computer include the manipulation of signals by a processor and the maintenance of these signals within data structures resident in one or more memory storage devices . for the purposes of this discussion , a process is generally conceived to be a sequence of computer - executed steps leading to a desired result . these steps usually require physical manipulations of physical quantities . usually , though not necessarily , these quantities take the form of electrical , magnetic , or optical signals capable of being stored , transferred , combined , compared , or otherwise manipulated . it is convention for those skilled in the art to refer to representations of these signals as bits , bytes , words , information , elements , symbols , characters , numbers , points , data , entries , objects , images , files , or the like . it should be kept in mind , however , that these and similar terms are associated with appropriate physical quantities for computer operations , and that these terms are merely conventional labels applied to physical quantities that exist within and during operation of the computer . it should also be understood that manipulations within the computer are often referred to in terms such as creating , adding , calculating , comparing , moving , receiving , determining , identifying , populating , loading , executing , etc . that are often associated with manual operations performed by a human operator . the operations described herein are machine operations performed in conjunction with various input provided by a human operator or user that interacts with the computer . in addition , it should be understood that the programs , processes , methods , etc . described herein are not related or limited to any particular computer or apparatus . rather , various types of general purpose machines may be used with the program modules constructed in accordance with the teachings described herein . similarly , it may prove advantageous to construct a specialized apparatus to perform the method steps described herein by way of dedicated computer systems in specific network architecture with hard - wired logic or programs stored in nonvolatile memory , such as read - only memory . referring now to the drawings , aspects of the present invention and the illustrative operating environment will be described . fig1 and the following discussion are intended to provide a brief , general description of a suitable computing environment in which the invention may be implemented . referring now to fig1 , an illustrative environment for implementing the invention includes a conventional personal computer 10 , including a processing unit 2 , a system memory , including read only memory ( rom ) 4 and random access memory ( ram ) 8 , and a system bus 5 that couples the system memory to the processing unit 2 . the read only memory ( rom ) 4 includes a basic input / output system 6 ( bios ), containing the basic routines that help to transfer information between elements within the personal computer 10 , such as during start - up . the personal computer 100 further includes a hard disk drive 18 and an optical disk drive 22 , e . g ., for reading a cd - rom disk or dvd disk , or to read from or write to other optical media . the drives and their associated computer - readable media provide nonvolatile storage for the personal computer 10 . although the description of computer - readable media above refers to a hard disk , a removable magnetic disk and a cd - rom or dvd - rom disk , it should be appreciated by those skilled in the art that other types of media are readable by a computer , such as magnetic cassettes , flash memory cards , digital video disks , bernoulli cartridges , and the like , may also be used in the illustrative operating environment . a number of program modules may be stored in the drives and ram 8 , including an operating system 14 and one or more application programs 11 , such as a program for browsing the world - wide - web , such as www browser 12 . such program modules may be stored on hard disk drive 18 and loaded into ram 8 either partially or fully for execution . a user may enter commands and information into the personal computer 10 through a keyboard 28 and pointing device , such as a mouse 30 . other control input devices ( not shown ) may include a microphone , joystick , game pad , satellite dish , scanner , or the like . these and other input devices are often connected to the processing unit 10 through an input / output interface 20 that is coupled to the system bus , but may be connected by other interfaces , such as a game port , universal serial bus , or firewire port . a display monitor 26 or other type of display device is also connected to the system bus 5 via an interface , such as a video display adapter 16 . in addition to the monitor , personal computers typically include other peripheral output devices ( not shown ), such as speakers or printers . the personal computer 100 may be capable of displaying a graphical user interface on monitor 26 . the personal computer 10 may operate in a networked environment using logical connections to one or more remote computers , such as a host computer 40 . the host computer 40 may be a server , a router , a peer device or other common network node , and typically includes many or all of the elements described relative to the personal computer 10 . the lan 36 may be further connected to an internet service provider 34 (“ isp ”) for access to the internet 38 . in this manner , www browser 12 may connect to host computer 40 through lan 36 , isp 34 , and the internet 38 . such networking environments are commonplace in offices , enterprise - wide computer networks , intranets and the internet , and are connected to the lan 36 through a network interface unit 24 . when used in a wan networking environment , the personal computer 10 typically includes a modem 32 or other means for establishing communications through the internet service provider 34 to the internet . the modem 32 , which may be internal or external , is connected to the system bus 105 via the input / output interface 20 . it will be appreciated that the network connections shown are illustrative and other means of establishing a communications link between the computers may be used . the operating system 14 generally controls the operation of the previously discussed personal computer 100 , including input / output operations . in the illustrative operating environment , the invention is used in conjunction with microsoft corporation &# 39 ; s “ windows 98 ” operating system and a www browser 12 , such as microsoft corporation &# 39 ; s internet explorer or netscape corporation &# 39 ; s internet navigator , operating under this operating system . however , it should be understood that the invention can be implemented for use in other operating systems , such as microsoft corporation &# 39 ; s “ windows 3 . 1 ,” “ windows 95 ”, “ windows nt ” , “ windows 2000 ”, “ windows xp ”, and “ windows vista ” operating systems , ibm corporation &# 39 ; s “ os / 2 ” operating system , sunsoft &# 39 ; s “ solaris ” operating system used in workstations manufactured by sun microsystems , “ linux ” and the operating systems used in “ macintosh ” computers manufactured by apple computer , inc . likewise , the invention may be implemented for use with other www browsers known to those skilled in the art . host computer 40 is also connected to the internet 38 , and may contain components similar to those contained in personal computer 10 described above . additionally , host computer 40 may execute an application program for receiving requests for www pages , and for serving such pages to the requestor , such as www server 42 . according to an embodiment of the present invention , www server 42 may receive requests for www pages 50 or other documents from www browser 12 . in response to these requests , www server 42 may transmit www pages 50 comprising hyper - text markup language (“ html ”) or other markup language files , such as active server pages , to www browser 12 . likewise , www server 42 may also transmit requested data files 48 , such as graphical images or text information , to www browser 12 . www server may also execute scripts 44 , such as cgi or perl scripts , to dynamically produce www pages 50 for transmission to www browser 12 . www server 42 may also transmit scripts 44 , such as a script written in javascript , to www browser 12 for execution . similarly , www server 42 may transmit programs written in the java programming language , developed by sun microsystems , inc ., to www browser 12 for execution . as will be described in more detail below , aspects of the present invention may be embodied in application programs executed by host computer 42 , such as scripts 44 , or may be embodied in application programs executed by computer 10 , such as java applications 46 . those skilled in the art will also appreciate that aspects of the invention may also be embodied in a stand - alone application program . our invention is a process for improving search engine results and is not dependent on any particular search engine , since our process only requires the search engine to produce a list of possible results when given a query . the server is the computer program ( s ) and any additional support that manages the improvement of search results and mediates the interaction between the user , the search engine and the human agents that carry out the improvement . the human agents are any humans that register with the server to improve results , possibly in return for some form of credit . so our server implements the crucial function of providing an interface and process to connect the user with a pool of readily available and qualified human agents in order to improve search engine results . after each task , our server may audit the performance of the human agents in order to assess the quality of their work , and then requalify or disqualify a human agent based on this audit and a record of their past performance . our best mode embodiment consists of using as the knowledge - base the world wide web and a web - based search engine . the results of a search engine are web pages given as a list containing the uri ( uniform resource identifiers ) of a result and possibly a fragment of summary text . this allows an agent to access and assess the contents of the result , mediated by the server and annotated with a number of constrained options to record their assessment . the following detailed description refers to steps in fig2 through 4 . in step 110 , the process begins with a user with a need that they believe can be fulfilled by some results that are available in a knowledge - base , such as the web , and who intends to use our server to get improved results for their search . in step 112 , the user phrases a query in the form of one or more natural language terms for a search engine . these query terms serve as part of the criteria and our interface also allows a user to specify in more detail additional and more precise criteria . these criteria specify what results would qualify as improved results . by “ results ” it is meant either summary results , i . e . the listing of results received from the search engine , or the combination of the results received from the search engine and the actual pages associated therewith . thus , a menu can be presented to the user allowing them to select criteria as results that “ are not advertisements ” or “ are suitable for viewing by children ” or “ provide definitions ” or “ are from well - known / trusted sources . 38 in the optional step 120 , the server determines if the query and criteria sufficiently match human - improved results from previous searches that have been cached . in this case , in optional step 122 the user has the option of accepting the results that are in the cache . if this is the case , the server uses the cached results as the improved results in the process , and skips to step 318 , although from there on any steps involving credit may be not taken . if the results are not cached on the server or the cache is not used , in step 124 ( optional ) the server asks for any additional constraints for the task . these constraints are not the criteria for each individual improved result as given in step 112 , but constraints for the entire result improvement task itself , such as the maximum time the task must be completed in or the knowledge bases or search engines to be queried or the minimum number of improved results that the user wants . in step 126 the server combines the query , the criteria , and any additional data needed by the human agents , such as the date and time of the user request , to create the instructions for the result improvement task . then in step 130 ( optional ) the user specifies if they are to pay an additional surcharge of credit for the completion of the result improvement task . since the server would then present the result improvement tasks to human agents , an additional surcharge would encourage human agents to prioritize filtering the search results of the particular user . note that for some embodiments of this process the surcharge may be mandatory , and in others not possible at all . if the user answers “ yes ,” then in the optional step 132 the server proceeds if necessary to take the details of the user and the exact value of the surcharge so that it may be taken into consideration in step 134 . then in step 134 the server determines the credit , which may be nothing , to be given to the human agents for completing an assessment for the result improvement task of the user . then in step 136 the server submits the query to one or more search engines with respect to one or more knowledge bases . in step 138 the search engines return a list of potential results . in step 140 , if the numbers of potential results are of such kind or size that the server determines it would be beneficial to have the single potential result list divided among multiple human agents , then the server may cap and / or divide the potential result list into smaller portions for distribution to human agents . in step 210 the server constructs annotation forms by annotating the potential result list ( s ) with the criteria that the user has specified . in our best mode embodiment , this would consist of adding to each result in the potential result list the options needed to assess the potential result with regard to each of the possible values of the criteria given by the user through input mechanisms such as radio buttons and check boxes . for example , if the user wants to find only video files , a simple check - box would be added to the annotation form to allow the agent to denote whether the result is a video file or not . if , in addition the user wanted the results to be rated for relevance as to whether they were about hurricanes , a control with an appropriate range of relevance options could be added to the annotation form . these input mechanisms on the annotation form will be used by the human agents to record their assessments of each result . in step 212 , the server determines how many human agents are needed in total for the entire result improvement task , taking into account any division of the task into smaller portions and the amount of redundancy required to provide for quality assurance . in step 214 , the server announces the tasks and the credit reward to the pool of human agents , indicating how many agents are wanted for each task . in step 216 , one or more human agents chooses one of the tasks . note that each such task , as explained in step 140 , may only cover a portion of the original potential result list returned by search engine . the server will offer the tasks until the requisite number of human agents have chosen each task , although it will allow the agents to improve the results asynchronously . in a simplified example , if five hundred results were returned , and the user wants twenty improved results , the server may automatically divide the list into five groups of one hundred results , and then fifteen agents might be needed , three for each group of results , with the agreement among the three agents to be used as a method of auditing their quality . therefore , the server will continue to offer the tasks until fifteen agents have signed up . in step 218 , the appropriate annotation form is displayed to each of the human agents that chose the corresponding task in step 216 . note that from step 220 up until step 250 there may be multiple human agents following the process in parallel , although from step 220 to step 250 we will refer to “ the human agent ” as one of the agents committed to this process . step 222 signals the beginning of the process , encompassing step 222 through step 230 , of examination by the human agents of each entry in the annotation form . in step 222 , for each unannotated result , the human agent uses the annotation form to record the assessment regarding the criteria given by the user . as given by step 230 , the human agent continues this until there are no unannotated results ( or , as noted in step 220 , they may choose to perform only a subset of the annotations ). for example , if the user is looking for web pages relevant to a certain subject , a web page may be marked as either relevant or irrelevant to the subject in the criteria via a checkbox in the annotation form . to determine this the human agent may be presented with the summary text alone , or may have access to the contents that can be accessed via the uri . optionally in step 232 the human agent may then rank the annotated result list . the completed annotation form is then returned to the server in step 234 . in step 240 , the server determines whether or not the human agent has completely filled in the annotation form . if not ( a “ no ” to step 240 ), in optional step 250 credit is deducted from the human agent . then in step 310 the server combines the annotated result lists from each of the human agents who chose the user &# 39 ; s result improvement task . the server takes into account variance or discrepancies in the human agent &# 39 ; s annotations about whether or not particular results fit the user &# 39 ; s criteria . also , if the original potential result list was divided into portions for the human agents ( in step 140 ), the server combines the results from each human agent who chose the task . in step 312 , the server determines if , for any reason , there are insufficient results of the necessary quality . if so , it returns to step 210 to construct new annotation forms and recruit further human agents to make up the necessary additional results . the improved results are then optionally re - ranked in step 316 by the server , again taking into account any variance or discrepancies in the rankings of the human agents . then the improved results are displayed for the user to browse in step 318 . note that the server may incorporate advertisements and other data in the display of the improved results . in optional step 320 , the user may then be given the opportunity to judge whether or not the results in the improved results are satisfactory and whether or not the task constraints have been fulfilled , and this is reported to the server . in this step , the user may judge whether or not each of the improved results actually fulfills their criteria . if the user judges the improved results to be less than fully satisfactory or their task constraints not fulfilled , as given by “ no ” at step 320 , in the optional step 322 , the credit given to the responsible human agent can be reduced . for example , one of the human agents may have returned a result that does not to the user fulfill the criteria , and this result was put in the improved results by the server . if the user notifies the server that this result did not fit the criteria , since the server records which human agent or agents was responsible for the incorrect annotation in the improved results , it will deduct credit those agents . the user can also tell the system if they believe their constraints were not met . in another example , a human agent could be too slow in annotating the results , and so also lose credit . this information is used by server to audit the human agents in order to maintain a high quality pool of human agents and to bar unsatisfactory agents from participating in the process . in step 324 , optionally the users may add additional metadata , such as the use of natural language tags or commentary , to their result list . then in optionally step 326 the server can cache the improved results and any optional or necessary metadata , taking into account whether or not the user was satisfied by the results . then in the optional step 328 , taking any deductions given in steps 250 and 322 into account , the server audits the human agents that performed the result improvement task in order to determine their suitability for participating in the result improvement process on another occasion . finally , in the optional step 330 the human agents are rewarded with credit if they qualify and the process ends . when one of the criteria , perhaps the only one , that a search result must satisfy to be useful is that it be suitable for delivery via a particular medium or device , such as a small screen , a low - bandwidth connection or a screen - reader , an alternative embodiment of the invention is appropriate , in which 1 ) preparation of material for the human agents includes simulating the effects of the required medium and / or device and 2 ) bulk processing of popular queries is done using the techniques described above but without individual user input , with the results made available via a portal . for example , this embodiment might be used to provide a portal with up - to - date mobile - phone - suitable search results for the top 1000 celebrity names . hierarchical link directories are an alternative to search engines for users seeking information about particular subjects . creating and maintaining such directories is difficult and expensive . an alternative embodiment of the invention addresses this problem by ( semi -) automatically generating queries from planned or existing directory path names and using human agents nominated by the directory owner in the procedure described above . although this invention has been described with a certain degree of particularity , it is to be understood that the present disclosure has been made only by way of illustration and that numerous changes in the details of construction and arrangement of parts may be resorted to without departing from the spirit and the scope of the invention .