lib/middlemac/private.rb
#--------------------------------------------------------
# Analyzes the content of the given resource for words
# and their frequencies. Omits words in the respective
# stopwords .plist file. If the resource contains any
# `exactmatch` entries, they will be added with high
# weights.
#
# @param [Sitemap::Resource] resource The resource for
#   which to get the word frequency list.
# @return [Hash] The word frequency list.
# @!visibility private
#--------------------------------------------------------
Analyzes the content of the given resource for words
and their frequencies. Omits words in the respective
stopwords .plist file. If the resource contains any
`exactmatch` entries, they will be added with high
weights.
Analyzes the content of the given resource for words and their frequencies . Omits words in the respective stopwords . plist file . If the resource contains any ` exactmatch ` entries they will be added with high weights .
def word_frequencies_impl(resource)
    return nil if resource == @resource
    @resource = resource
    return nil unless iso_code = resource.hb_locale
    lorem_words = %w(alias consequatur aut perferendis sit voluptatem accusantium doloremque aperiam eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo aspernatur aut odit aut fugit sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt neque dolorem ipsum quia dolor sit amet consectetur adipisci velit sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem ut enim ad minima veniam quis nostrum exercitationem ullam corporis nemo enim ipsam voluptatem quia voluptas sit suscipit laboriosam nisi ut aliquid ex ea commodi consequatur quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae et iusto odio dignissimos ducimus qui blanditiis praesentium laudantium totam rem voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident sed ut perspiciatis unde omnis iste natus error similique sunt in culpa qui officia deserunt mollitia animi id est laborum et dolorum fuga et harum quidem rerum facilis est et expedita distinctio nam libero tempore cum soluta nobis est eligendi optio cumque nihil impedit quo porro quisquam est qui minus id quod maxime placeat facere possimus omnis voluptas assumenda est omnis dolor repellendus temporibus autem quibusdam et aut consequatur vel illum qui dolorem eum fugiat quo voluptas nulla pariatur at vero eos et accusamus officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae itaque earum rerum hic tenetur a sapiente delectus ut aut reiciendis voluptatibus maiores doloribus asperiores repellat)
    my_path = File.dirname(File.expand_path(__FILE__))
    stopwords = File.expand_path(File.join(my_path, '..', '..', 'resources', 'stopwords', "#{iso_code}.plist"))
    stopwords = open_binary_stoplist( stopwords )  + lorem_words
    rendered_resource = "<h1>#{resource.data.title}</h1>" + resource.render(layout: false)
    doc = Nokogiri::HTML.fragment(rendered_resource)
    doc.search('style,script').each(&:remove)
    counter = WordsCounted.count( doc.text, :exclude => [stopwords, ->(t){t.length < 4}] )
    tokens = counter.token_frequency.to_h
    resource&.data&.keywords&.each do |k|
      tokens[k] = tokens[k] ? tokens[k] + 10 : 10
    end
    resource&.data&.exactmatch&.each do |k|
      tokens[k] = tokens[k] ? tokens[k] + 100 : 100
    end
    tokens
  end
def word_frequencies_impl ( resource ) return nil if resource == @resource @resource = resource return nil unless iso_code = resource . hb_locale lorem_words = %w( alias consequatur aut perferendis sit voluptatem accusantium doloremque aperiam eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo aspernatur aut odit aut fugit sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt neque dolorem ipsum quia dolor sit amet consectetur adipisci velit sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem ut enim ad minima veniam quis nostrum exercitationem ullam corporis nemo enim ipsam voluptatem quia voluptas sit suscipit laboriosam nisi ut aliquid ex ea commodi consequatur quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae et iusto odio dignissimos ducimus qui blanditiis praesentium laudantium totam rem voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident sed ut perspiciatis unde omnis iste natus error similique sunt in culpa qui officia deserunt mollitia animi id est laborum et dolorum fuga et harum quidem rerum facilis est et expedita distinctio nam libero tempore cum soluta nobis est eligendi optio cumque nihil impedit quo porro quisquam est qui minus id quod maxime placeat facere possimus omnis voluptas assumenda est omnis dolor repellendus temporibus autem quibusdam et aut consequatur vel illum qui dolorem eum fugiat quo voluptas nulla pariatur at vero eos et accusamus officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae itaque earum rerum hic tenetur a sapiente delectus ut aut reiciendis voluptatibus maiores doloribus asperiores repellat ) my_path = File . dirname ( File . expand_path ( __FILE__ ) ) stopwords = File . expand_path ( File . join ( my_path , '..' , '..' , 'resources' , 'stopwords' , "#{iso_code}.plist" ) ) stopwords = open_binary_stoplist ( stopwords ) + lorem_words rendered_resource = "<h1>#{resource.data.title}</h1>" + resource . render ( layout : false ) doc = Nokogiri :: HTML . fragment ( rendered_resource ) doc . search ( 'style,script' ) . each ( & :remove ) counter = WordsCounted . count ( doc . text , :exclude => [ stopwords , -> ( t ) { t . length < 4 } ] ) tokens = counter . token_frequency . to_h resource &. data &. keywords &. each do | k | tokens [ k ] = tokens [ k ] ? tokens [ k ] + 10 : 10 end resource &. data &. exactmatch &. each do | k | tokens [ k ] = tokens [ k ] ? tokens [ k ] + 100 : 100 end tokens end
Analyzes the content of the given resource for words
and their frequencies.
Analyzes the content of the given resource for words and their frequencies .
# Prevent possible recursion when we render this resource, below. # Give extra weight to tokens that are in the data.keywords. # Give a lot of extra weight to tokens thare in the data.exactmatch.
The word frequency list.
The word frequency list .
Hash
resource
<not_specific>
<not_specific>
resource The resource for which to get the word frequency list.
resource The resource for which to get the word frequency list .
!visibility
None