Take the 2-minute tour ×

I'm reading data from a remote source, and occassionally get some characters in another encoding. They're not important.

I'd like to get get a "best guess" utf-8 string, and ignore the invalid data.

Main goal is to get a string I can use, and not run into errors such as:

  • Encoding::UndefinedConversionError: "\xFF" from ASCII-8BIT to UTF-8:
  • invalid byte sequence in utf-8
share|improve this question
You posted an answer and deleted it. That answer looked useful. Can you tell us why it isn't correct/helpful? –  Joachim Sauer Oct 24 '11 at 6:47
Undeleted and commented upon. –  Jordan Feldstein Oct 24 '11 at 16:27
add comment

5 Answers

up vote 13 down vote accepted

I thought this was it:

string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")

will replace all knowns with '?'.

To ignore all unknowns, :replace => '':



I'm not sure this is reliable. I've gone into paranoid-mode, and have been using:

string.encode("UTF-8", ...).force_encoding('UTF-8')

Script seems to be running, ok now. But I'm pretty sure I'd gotten errors with this earlier.

Edit 2:

Even with this, I continue to get intermittant errors. Not every time, mind you. Just sometimes.

share|improve this answer
Did you ever figure this out? I'm trying to determine if iconv can be safely replaced with String#encode in 1.9.3 (since it warns you about deprecations), but I can't tell from your comment if this is the case. –  Matt Green Mar 2 '12 at 15:00
Note for newbies: the ellipses in the edit refer to repeated code - the full line is: string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "").force_encoding('UTF-8') –  Ben Feb 18 '13 at 16:11
add comment

String#chars or String#each_char can be also used.

# Table 3-8. Use of U+FFFD in UTF-8 Conversion
# http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf)
str = "\x61"+"\xF1\x80\x80"+"\xE1\x80"+"\xC2"

p [
  'abcd' == str.chars.collect { |c| (c.valid_encoding?) ? c : '' }.join,
  'abcd' == str.each_char.map { |c| (c.valid_encoding?) ? c : '' }.join

String#scrub can be used since Ruby 2.1.

p [
  'abcd' == str.scrub(''),
  'abcd' == str.scrub{ |c| '' }
share|improve this answer
If you really need iconv since Ruby 2.0, consider iconv gem. –  masakielastic Jun 8 '13 at 9:12
Thanks @masakielastic, iconv gem fixed the problem. –  abhidsm Aug 16 '13 at 12:08
We have issues with content pasted in from word a lot and based on the answer above this has helped a lot: def find_invalid_chars(s) s.chars.select {|c| !c.valid_encoding? } end –  rainkinz Jan 24 at 14:35
add comment

This works great for me:

share|improve this answer
add comment

To ignore all unknown parts of the string that aren't correctly UTF-8 encoded the following (as you originally posted) almost does what you want.


The caveat is that encode doesn't do anything if it thinks the string is already UTF-8. So you need to change encodings, going via an encoding that can still encode the full set of unicode characters that UTF-8 can encode. (If you don't you'll corrupt any characters that aren't in that encoding - 7bit ASCII would be a really bad choice!) So go via UTF-16:

string.encode('UTF-16', :invalid => :replace, :replace => '').encode('UTF-8')
share|improve this answer
add comment

With a bit of help from @masakielastic I have solved this problem for my personal purposes using the #chars method.

The trick is to break down each character into its own separate block so that ruby can fail.

Ruby needs to fail when it confronts binary code etc. If you don't allow ruby to go ahead and fail its a tough road when it comes to this stuff. So I use the String#chars method to break the given string into an array of characters. Then I pass that code into a sanitizing method that allows the code to have "microfailures" (my coinage) within the string.

So, given a "dirty" string, lets say you used File#read on a picture. (my case)

dirty = File.open(filepath).read    
clean_chars = dirty.chars.select do |c|
  rescue ArgumentError
clean = clean_chars.join("")

def num_or_letter?(char)
  if char =~ /[a-zA-Z0-9]/
  elsif char =~ Regexp.union(" ", ".", "?", "-", "+", "/", ",", "(", ")")

allowing the code to fail somewhere along in the process seems to be the best way to move through it. So long as you contain those failures within blocks you can grab what is readable by the UTF-8-only-accepting parts of ruby

share|improve this answer
add comment

Your Answer


