Take the 2-minute tour ×

Is there a way to remove the BOM from a UTF-8 encoded file?

I know that all of my JSON files are encoded in UTF-8, but the data entry person who edited the JSON files saved it as UTF-8 with the BOM.

When I run my Ruby scripts to parse the JSON, it is failing with an error. I don't want to manually open 58+ JSON files and convert to UTF-8 without the BOM.

share|improve this question
possible duplicate of How to avoid tripping over UTF-8 BOM when reading files –  knut Mar 27 '12 at 10:15
add comment

3 Answers

With ruby 1.9.2 you can use the mode r:bom|utf-8

This should work (I haven't test it in combination with json):

json = nil #define the variable outside the block to keep the data
File.open('file.txt', "r:bom|utf-8"){|file|
  json = JSON.parse(file.read)

It doesn't matter, if the BOM is available in the file or not.

share|improve this answer
one-liner for ruby (takes file paths on STDIN): ruby -ne 'File.open($_.strip, "r:bom|utf-8") {|f| content=f.read(f.size); f.close; File.open($_.strip, "w").write(content) } –  Markus Strauss Oct 4 '12 at 17:18
add comment
up vote 6 down vote accepted

So, the solution was to do a search and replace on the BOM via gsub! I forced the encoding of the string to UTF-8 and also forced the regex pattern to be encoded in UTF-8.

I was able to derive a solution by looking at http://self.d-struct.org/195/howto-remove-byte-order-mark-with-ruby-and-iconv and http://blog.grayproductions.net/articles/ruby_19s_string

def read_json_file(file_name, index)
  content = ''
  file = File.open("#{file_name}\\game.json", "r") 
  content = file.read.force_encoding("UTF-8")

  content.gsub!("\xEF\xBB\xBF".force_encoding("UTF-8"), '')

  json = JSON.parse(content)

  print json
share|improve this answer
I would recommend sub!(/^\xEF\xBB\xBF/, '') as the BOM occures only once at the beginning of the string. –  Jan Nov 26 '12 at 10:08
I would actually suggest String#delete! over String#gsub!(..., '') –  Farley Knight Dec 18 '13 at 21:28
add comment

You can also specify encoding with the File.read and CSV.read methods, but you don't specify the read mode.

File.read(path, :encoding => 'bom|utf-8')
CSV.read(path, :encoding => 'bom|utf-8')
share|improve this answer
add comment

Your Answer


