细说python字符编码(三)

本文介绍编码中的html实体(编码)

背景

继昨天在整理好之前的两篇字符编码笔记之后,本来觉得豁然开朗,对编码这东西应该不会再遇到什么难题了
谁知道今天早上刚回来工作又遇到一种奇怪的编码方式 …(⊙_⊙;)…

html编码

之前讨论过的都是属于字符串编码,从网页上抓取到的数据一般也是unicode字符,但这回莫名其妙在数据库中看到了大量 &#xx; 字符如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[ti:Saving All My Love For You]

[ar:Whitney Houston]

[al:237436]

[offset:0]

[00:00.10]
Saving All My Love For You - Whitney Houston

[00:00.20]Written by :Michael Masser|Gerry Goffin

[00:00.30]

[00:14.65]A few stolen moments is all that we share

[00:22.09]You've got your family and they need you there

[00:28.87]Though I've tried to resist being last on your list

[00:36.27]But no other man's gonna do

[00:43.29]So I'm saving all my love for you

[00:54.94]It's not very easy living all alone

[01:01.67]My friends try and tell me find a man of my own

[01:08.91]But each time I try I just break down and cry

[01:16.14]Cause I'd rather be home feeling blue

[01:23.35]So I'm saving all my love for you

[01:31.72]You used to tell me we'd run away together

[01:39.07]Love gives you the right to be free

[01:45.30]You said be patient just wait a little longer

[01:52.58]But that's just an old fantasy

[02:00.26]I've got to get ready  just a few minutes more

[02:06.86]Gonna get that old feeling when you walk through that door

[02:14.07]For tonight is the night for feeling alright

[02:21.44]We'll be making love the whole night through

[02:28.59]So I'm saving all my love

[02:32.19]Yes I'm saving all my love

[02:35.80]Yes I'm saving all my love for you

[02:47.71]No other woman  is gonna love you more

[02:54.05]Cause tonight is the night that I''m feeling alright

[03:01.38]We'll be making love the whole night through

[03:08.42]So I'm saving all my love

[03:12.23]Yeah I''''m saving all my love

[03:15.85]Yes I''''m saving all my love for you

[03:27.06]For you  for you

这是一份歌词页面,在其中发现大量的字符如下

1

   '

在网上查阅后知道这是html编码,并且其中有如下对应关系

1
2
3
4
5

 --- 换行 "\n"
  --- 空格 " "
' --- 撇号 "'"

图片1
图片2
更详细的可以在这里查找,我们这里暂时只有用到ascii的html编码,后面必然会带有一个冒号,注意一些中文的html编码可能并不带有后面那个冒号

乱码解决

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/usr/bin/env python
#!coding:utf-8
import sys
reload(sys)
sys.setdefaultencoding('utf8')
import HTMLParser
import chardet
def html2utf8(html):
str_html = html
t = HTMLParser.HTMLParser()
str_utf8 = t.unescape(str_html).encode('utf8')
return str_utf8
if __name__=="__main__":
html = r"'"
print html2utf8(html)
print chardet.detect(html)
print chardet.detect(html2utf8(html))
#>>> '
#>>> {'confidence': 1.0, 'language': '', 'encoding': 'ascii'}
#>>> {'confidence': 1.0, 'language': '', 'encoding': 'ascii'}

使用HTMLParser.HTMLParser()的unescape方法,之后再将得到的unicode字符encode成utf8
最后得到的结果如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[ti:Saving All My Love For You]
[ar:Whitney Houston]
[al:237436]
[offset:0]
[00:00.10]Saving All My Love For You - Whitney Houston
[00:00.20]Written by :Michael Masser|Gerry Goffin
[00:00.30]
[00:14.65]A few stolen moments is all that we share
[00:22.09]You've got your family and they need you there
[00:28.87]Though I've tried to resist being last on your list
[00:36.27]But no other man's gonna do
[00:43.29]So I'm saving all my love for you
[00:54.94]It's not very easy living all alone
[01:01.67]My friends try and tell me find a man of my own
[01:08.91]But each time I try I just break down and cry
[01:16.14]Cause I'd rather be home feeling blue
[01:23.35]So I'm saving all my love for you
[01:31.72]You used to tell me we'd run away together
[01:39.07]Love gives you the right to be free
[01:45.30]You said be patient just wait a little longer
[01:52.58]But that's just an old fantasy
[02:00.26]I've got to get ready just a few minutes more
[02:06.86]Gonna get that old feeling when you walk through that door
[02:14.07]For tonight is the night for feeling alright
[02:21.44]We'll be making love the whole night through
[02:28.59]So I'm saving all my love
[02:32.19]Yes I'm saving all my love
[02:35.80]Yes I'm saving all my love for you
[02:47.71]No other woman is gonna love you more
[02:54.05]Cause tonight is the night that I''m feeling alright
[03:01.38]We'll be making love the whole night through
[03:08.42]So I'm saving all my love
[03:12.23]Yeah I''''m saving all my love
[03:15.85]Yes I''''m saving all my love for you
[03:27.06]For you for you"}

以utf8的格式保存的,这样的字符串保证到什么都不会出现乱码,okay

End

作为一个数据提供者和可能的消费者,提供可靠的数据源是很重要的,打好基础才能使后面的路更顺利~