rubyCode/forwardIndexGenerator.rb
#This function takes the processed data and path to the file where it will store
#generate an sql file which contains data for forward Index.Sql file was generated
#instead of using the ruby mysql gem because of the massive performance improvement
#achieved by executing the generated sql file versus using mysql gem (80x faster in our case)
#This function also calculates the number of hits for each word in each document
This function takes the processed data and path to the file where it will store
generate an sql file which contains data for forward Index.Sql file was generated
instead of using the ruby mysql gem because of the massive performance improvement
achieved by executing the generated sql file versus using mysql gem (80x faster in our case)
This function also calculates the number of hits for each word in each document
This function takes the processed data and path to the file where it will store generate an sql file which contains data for forward Index . Sql file was generated instead of using the ruby mysql gem because of the massive performance improvement achieved by executing the generated sql file versus using mysql gem ( 80x faster in our case ) This function also calculates the number of hits for each word in each document
def finalize(rows, reservedFiles)
  nhits = Hash.new 0
  rows.each do |row|
    nhits[row.docID.to_s + row.wordID.to_s] += 1
  end
  rows.each do |row|
    row.nHits = nhits[row.docID.to_s + row.wordID.to_s]
  end
  Push to database
  This method below works but it is very expensive
  sqlStatementInsert = $connection.prepare 'INSERT INTO forwardIndex(docID, wordID, nHits, hit) VALUES(?,?,?,?)'
  rows.each do |row|
    sqlStatementInsert.execute(row.docID, row.wordID, row.nHits, row.hit)
  end
  So instead we generate our own sql file which then can be executed for quicker insertion
  GenSQL.generate(rows, reservedFiles)
end
def finalize ( rows , reservedFiles ) nhits = Hash . new 0 rows . each do | row | nhits [ row . docID . to_s + row . wordID . to_s ] += 1 end rows . each do | row | row . nHits = nhits [ row . docID . to_s + row . wordID . to_s ] end GenSQL . generate ( rows , reservedFiles ) end
This function takes the processed data and path to the file where it will store
generate an sql file which contains data for forward Index.Sql file was generated
instead of using the ruby mysql gem because of the massive performance improvement
achieved by executing the generated sql file versus using mysql gem (80x faster in our case)
This function also calculates the number of hits for each word in each document
This function takes the processed data and path to the file where it will store generate an sql file which contains data for forward Index . Sql file was generated instead of using the ruby mysql gem because of the massive performance improvement achieved by executing the generated sql file versus using mysql gem ( 80x faster in our case ) This function also calculates the number of hits for each word in each document
#Calculate nhits # #Push to database #This method below works but it is very expensive #sqlStatementInsert = $connection.prepare 'INSERT INTO forwardIndex(docID, wordID, nHits, hit) VALUES(?,?,?,?)' #rows.each do |row| #  sqlStatementInsert.execute(row.docID, row.wordID, row.nHits, row.hit) #end #So instead we generate our own sql file which then can be executed for quicker insertion
rows
<not_specific>
reservedFiles
<not_specific>