vendor/gems/BlueCloth-1.0.0/lib/bluecloth.rb
### Break the HTML source in +str+ into a series of tokens and return
### them. The tokens are just 2-element Array tuples with a type and the
### actual content. If this function is called with a block, the type and
### text parts of each token will be yielded to it one at a time as they are
### extracted.
Break the HTML source in +str+ into a series of tokens and return
them. The tokens are just 2-element Array tuples with a type and the
actual content. If this function is called with a block, the type and
text parts of each token will be yielded to it one at a time as they are
extracted.
Break the HTML source in + str + into a series of tokens and return them . The tokens are just 2 - element Array tuples with a type and the actual content . If this function is called with a block the type and text parts of each token will be yielded to it one at a time as they are extracted .
def tokenize_html( str )
		depth = 0
		tokens = []
		@scanner.string = str.dup
		type, token = nil, nil
		until @scanner.empty?
			@log.debug "Scanning from %p" % @scanner.rest
			if (( token = @scanner.scan(MetaTag) ))
				type = :tag
			elsif (( token = @scanner.scan(HTMLTagOpenRegexp) ))
				tagstart = @scanner.pos
				@log.debug " Found the start of a plain tag at %d" % tagstart
				depth = 1
				type = :tag
				while depth.nonzero?
					chunk = @scanner.scan( HTMLTagPart ) or
						raise "Malformed tag at character %d: %p" % 
							[ tagstart, token + @scanner.rest ]
					@log.debug "  Found another part of the tag at depth %d: %p" % [ depth, chunk ]
					token += chunk
					depth += ( token[-1, 1] == '>' ? -1 : 1 )
					@log.debug "  Depth is now #{depth}"
				end
			else
				@log.debug " Looking for a chunk of text"
				type = :text
				token = @scanner.scan_until( /[^<]+/m )
			end
			@log.debug " type: %p, token: %p" % [ type, token ]
			if block_given?
				yield( type, token )
			end
			tokens << [ type, token ]
		end
		return tokens
	end
def tokenize_html ( str ) depth = 0 tokens = [ ] @scanner . string = str . dup type , token = nil , nil until @scanner . empty? @log . debug "Scanning from %p" % @scanner . rest if ( ( token = @scanner . scan ( MetaTag ) ) ) type = :tag elsif ( ( token = @scanner . scan ( HTMLTagOpenRegexp ) ) ) tagstart = @scanner . pos @log . debug " Found the start of a plain tag at %d" % tagstart depth = 1 type = :tag while depth . nonzero? chunk = @scanner . scan ( HTMLTagPart ) or raise "Malformed tag at character %d: %p" % [ tagstart , token + @scanner . rest ] @log . debug "  Found another part of the tag at depth %d: %p" % [ depth , chunk ] token += chunk depth += ( token [ - 1 , 1 ] == '>' ? - 1 : 1 ) @log . debug "  Depth is now #{depth}" end else @log . debug " Looking for a chunk of text" type = :text token = @scanner . scan_until ( / [^<]+ /m ) end @log . debug " type: %p, token: %p" % [ type , token ] if block_given? yield ( type , token ) end tokens << [ type , token ] end return tokens end
Break the HTML source in +str+ into a series of tokens and return
them.
Break the HTML source in + str + into a series of tokens and return them .
# Match comments and PIs without nesting # Do nested matching for HTML tags # Start the token with the opening angle # Scan the rest of the tag, allowing unlimited nested <>s. If # the scanner runs out of text before the tag is closed, raise # an error. # Scan either an opener or a closer # If the last character of the token so far is a closing # angle bracket, decrement the depth. Otherwise increment # it for a nested tag. # Match text segments # Scan forward, always matching at least one character to move # the pointer beyond any non-tag '<'. # If a block is given, feed it one token at a time. Add the token to # the token list to be returned regardless.
str
<not_specific>