External/Tools/AS2API/parse/lexer.rb
# This is a naive lexer implementation that considers input line-by-line,
# with special cases to handle multiline tokens (strings, comments).
# spacial care must be taken to declaire tokens in the 'correct' order (as
# the fist match wins), and to cope with keyword/identifier ambiguity
# (keywords have '\b' regexp-lookahead appended)
This is a naive lexer implementation that considers input line-by-line,
with special cases to handle multiline tokens (strings, comments).
spacial care must be taken to declaire tokens in the 'correct' order (as
the fist match wins), and to cope with keyword/identifier ambiguity
(keywords have '\b' regexp-lookahead appended)
This is a naive lexer implementation that considers input line - by - line with special cases to handle multiline tokens ( strings comments ) . spacial care must be taken to declaire tokens in the ' correct ' order ( as the fist match wins ) and to cope with keyword / identifier ambiguity ( keywords have ' \ b ' regexp - lookahead appended )
def build_lexer(target_class)
    text = <<-EOS
      def fill
        line = StringScanner.new(@io.readline)
        until line.eos?
    EOS
    @matches.each_with_index do |token_match, index|
      re, lex_method, tok_class = token_match
      text << "if line.scan(/#{re}/)\n"
      if tok_class
      	text << "  emit(#{lex_method.to_s}(:#{tok_class.to_s}, line, @io))\n"
      else
      	text << "  emit(#{lex_method.to_s}(line, @io))\n"
      end
      text << "  next\n"
      text << "end\n"
    end
    text << <<-EOS
          # no previous regexp matched,
          parse_error(line.rest)
        end
      end
    EOS
    target_class.class_eval(text)
  end
def build_lexer ( target_class ) text = <<-EOS 
      def fill
        line = StringScanner.new(@io.readline)
        until line.eos?
     EOS @matches . each_with_index do | token_match , index | re , lex_method , tok_class = token_match text << "if line.scan(/#{re}/)\n" if tok_class text << "  emit(#{lex_method.to_s}(:#{tok_class.to_s}, line, @io))\n" else text << "  emit(#{lex_method.to_s}(line, @io))\n" end text << "  next\n" text << "end\n" end text << <<-EOS 
          # no previous regexp matched,
          parse_error(line.rest)
        end
      end
     EOS target_class . class_eval ( text ) end
This is a naive lexer implementation that considers input line-by-line,
with special cases to handle multiline tokens (strings, comments).
This is a naive lexer implementation that considers input line - by - line with special cases to handle multiline tokens ( strings comments ) .
target_class
<not_specific>