Question

我用开放式的言语阅读一个网页,该网页的编码为:o-8859-1。当我读到该页的内容时,开放式返回了ASCII-8BIT编码的插图。

open("http://www.nigella.com/recipes/view/DEVILS-FOOD-CAKE-5310") {|f| p f.content_type, f.charset, f.read.encoding }
 => ["text/html", "iso-8859-1", #<Encoding:ASCII-8BIT>]

我对此表示猜疑,是因为网页具有无效力的星号(或特性)x92,http://en.wikipedia.org/wiki/ISO/IEC_8859-1 。

我需要将网页储存为“utf-8”编码文档。任何关于如何处理编码不准确的网页的想法。我可以抓住这一例外情况,试图猜测正确的编码,但这似乎很麻烦,而且容易发生错误。

Answer 1

ASCII-8BIT is an alias for BINARY
open-uri does a funny thing: if the file is less than 10kb (or something like that), it returns a String and if it s bigger then it returns a StringIO. That can be confusing if you re trying to deal with encoding issues.

如果卷宗数量庞大,我建议人工装入插图:

require  uri 
require  net/http 
require  net/https 

uri = URI.parse url_to_file

http = Net::HTTP.new(uri.host, uri.port)
if uri.scheme ==  https 
  http.use_ssl = true
  # possibly useful if you see ssl errors
  # http.verify_mode = ::OpenSSL::SSL::VERIFY_NONE
end
body = http.start { |session| session.get uri.request_uri }.body

Then you can use the https://rubygems.org/gems/ensure-encoding gem

require  ensure/encoding 
utf8_body = body.ensure_encoding( UTF-8 , :external_encoding => :sniff, :invalid_characters => :transcode)

I have been pretty happy with ensure-encoding... we use it in production at http://data.brighterplanet.com

Note that you can also say :invalid_characters => :ignore instead of :transcode.

Also, if you know the encoding somehow, you can pass :external_encoding => ISO-8859-1 instead of :sniff

友情链接