Will this be on the test? (Part 2)

Posted: September 16, 2009 by Twitchdoctor in Game AI
Tags: , , , ,













  1. TwinHits says:

    For all I know, this is the point of your next installment, but what would be a more accurate way to test AI? If the problem as you described, judges evaluating the ability of an AI to fool a human and the test being a romanticized version of human behavior, how can this problem be removed?

    I’d suggest throwing a giant LAN party and making a few of the participants AI instead of humans. As you pointed out oh so cleverly, players are always on the lookout for possibly enhanced play styles. You can tell if the AI is successful by how many players lash out in a fit of rage, yelling “HAX!” all the way.

  2. twitchdoctor says:

    Of course, players also lash out in fits of rage and yell “HAX” at humans as well. . . 😉

    I think it would be possible to design a much better version of the both the Loebner prize and the botprize. I bet I’m not the first to point out how methodologically suspect these tests are. I just don’t think there’s any interest in doing so. The way it stands now, we all get to feel warm and fuzzy about how special our supposedly non-replicable human characteristics are, sigh about how AI research still has a long way to go, and life rolls merrily along.

    Before you design a test you need to decide on what it is you are testing for. These are not simply problematic tests, they are testing for a kind of AI that I’m not sure is that useful overall and certainly not for games. However your idea of the LAN party (which is certainly a realistic gaming environment) does connect with something that Incanter and I have been talking about: the role of complex environments in attributing intelligence to systems.

  3. waltersthegreat says:

    Forgive the following nerdy tangent, but I feel that some conversations are incomplete without Star Trek references. A very famous test appeared in appeared in the second film (Star Trek II: The Wrath of Khan)—the Kobiyashi Maru test. This test was defined by its impossibility—its purpose was to make cadets feel fear and experience indecision while facing certain death. It was not a winnable simulation because it was not meant to evaluate a cadet’s ability to make the right decisions and win—it was meant to evaluate the cadet’s character in the face of adversity.

    After failing to beat the Kobiyashi Maru test, cadets voiced their concerns to instructors that the test did not sufficiently showcase or evaluate their ability to command a starship. That frustration is similar to what we are feeling about these Turing tests. We want the tests to motivate the production of better AI to supplement the development of better games, but what the Loebner test really motivates is the production of an AI that as Twitchdoctor asserts, really may not even be useful or have a place in the games we play.

    As consumers of games that we want to be intelligently artificial (brilliant title for this blog by the way), we play the role of the frustrated cadet—frustrated because the tests are evaluating the wrong things. In their position to do wonders for AI development, these tests examine the traits unrelated to our (gamers’) concerns. Just as the cadet wants to show off his or her ‘booksmart’ knowledge of how to respond to specific scenarios, we want to see AI developments suited to the gaming community. And I think we’re rightly disappointed that these tests are examining the criteria that aren’t really associated with gaming—especially because they conduct the tests under the guise that they will somehow advance AI developments for games.

    Before I become too harsh a critic, I should say that like everyone else, I’m interested to see the outcome of the test when it’s conducted in a more game-like setting (moving to first-person shooters instead of simulated conversations). Still though, the methods of judging these AI tests are questionable at best, and if the tests were truly conducted in order to better AI for games, I agree that it’s more than reasonable to question these methods until they’re changed.

    Someone may find a use for an AI entity that is essentially human, but that kind of AI (as Twitchdoctor suggests,) isn’t necessarily important to gaming. It’s not that the Turing tests are useless, it’s not that the Kobiyashi Maru test doesn’t teach an important lesson, it’s that these tests pretend to be something that they aren’t—they are preceded by an expectation that they don’t fulfill.


  4. twitchdoctor says:

    You are right, no conversation would be complete without such a reference! I really like the Kobayashi Maru example in connection with this discussion for a couple of reasons. It is a test, true, but it is, more importantly, a simulation. Simulations can obviously function as tests (airline pilots regularly undergo different kinds of certification in simulators) so they can have evaluative components. But the main goal is for the subject taking the simulation to learn things about his or her own behavior. In Wrath of Khan Lieutenant Saavik is annoyed, as you point out, because she expects that this will be a conventional “abilities and skills” test. However it turns out to be something more nebulous: a test of character. The important difference between this kind of test and the Loebner and botprizes is that the people running the simulation are pretty clear about what the test is supposed to do and what the results will mean.

    I also liked how they used this test again in the most recent Star Trek movie, where we learn the full details of the legend of Kirk beating the test. Now from one point of view he cheats. But what he does is simply broaden the frame of the test. He interprets the challenge as being not just the simulation itself, but the need to beat the structure that has created and is administering the simulation. As the movie demonstrates, by “cheating” Kirk in fact fulfills the purpose of the test: he demonstrates something essential about his character when faced with overwhelming odds and the certainty of death. The expected outcome was acquiescence in the face of certain destruction; Kirk’s response is to assume that the test is rigged and to look for a way to un-rig it, as it were. (To throw in another nerdy reference, its kind of a Han Solo moment: “Never tell me the odds!”)

    If the Loebner prize and the Botprize were to be reworked, I think it would require just this kind of broadening/shifting of the frame. Think how interesting, it would be to have an AI that “cheats.” I’m not even sure what that would mean.

Leave a Reply


WordPress.com Logo


Google photo


Twitter picture


Facebook photo


Connecting to %s