吴裕雄--天生自然python学习笔记：beautifulsoup库的使用

时间：2022-09-30 00:09:49
Beautiful Soup 库简介

Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。

Beautiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时，Beautiful Soup就不能自动识别编码方式了。然后，你仅仅需要说明一下原始编码方式就可以了。

Beautiful Soup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度。
from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.title)#选择title标签中的内容

print(type(soup.title))#title标签的类型

print(soup.head)#head标签中的内容

print(soup.p)#第一个p标签中的内容
<title>The Dormouse's story</title>

<class 'bs4.element.Tag'>

<head><title>The Dormouse's story</title></head>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
soup.标签名可以选择所需要的标签中的内容，返回结果是Tag类型。它查找的是在所有内容中的第一个符合要求的标签，例如上面的

的内容。

如果要查询所有的标签，在后面进行介绍。
获取名称

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.title.name)
title
获取属性

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print (soup.p.attrs['name'])

print(soup.p['name'])
dromouse

dromouse
获取内容

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print (soup.p.string)
The Dormouse's story
嵌套选择

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print (soup.head.title.string)
The Dormouse's story
获取子节点

.contents方法：

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.p.contents)
[<b>The Dormouse's story</b>]
.content 属性可以将tag的子节点以列表的方式输出，输出方式为列表，我们可以用列表索引来获取它的某一个元素，如：

print(soup.head.contents[0])

#返回结果：<title>The Dormouse's story</title>

.children方法：

print(soup.head.children)

#返回结果：<listiterator object at 0x7f71457f5710>

他返回一个迭代器对象，可以通过遍历获得其中的子节点，如：

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.head.children)

for child in  soup.body.children:

    print (child)
<list_iterator object at 0x0000019BAF29BE48>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>
获取子孙节点

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for child in soup.body.descendants:

    print (child)
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<b>The Dormouse's story</b>

The Dormouse's story

<p class="story">Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>

 Elsie

,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

Lacie

 and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

Tillie

;

and they lived at the bottom of a well.

<p class="story">...</p>

...
以上代码可以获取到head标签的所有子孙节点中的内容。

.descendants返回的是一个迭代器的类型，可以通过遍历输出所有的结果。
获取多个内容

.strings 获取多个内容，不过需要遍历获取，比如下面的例子

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for string in soup.strings:

    print(repr(string))
"The Dormouse's story"

'\n'

'\n'

"The Dormouse's story"

'\n'

'Once upon a time there were three little sisters; and their names were\n'

',\n'

'Lacie'

' and\n'

'Tillie'

';\nand they lived at the bottom of a well.'

'\n'

'...'

'\n'
发现将所有标签的内容返回了出来，但是也有许多空的标签，也会进行返回，如果想过滤空行，可以使用.stripped_strings 函数。

.stripped_strings 输出的字符串中可能包含了很多空格或空行,使用 .stripped_strings 可以去除多余空白内容

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for string in soup.stripped_strings:

    print(repr(string))
"The Dormouse's story"

"The Dormouse's story"

'Once upon a time there were three little sisters; and their names were'

','

'Lacie'

'and'

'Tillie'

';\nand they lived at the bottom of a well.'

'...'
获取父节点

.parent 属性 获取此标签的上一层标签

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

p = soup.p

print (p.parent.name)
body
获取祖先节点

.parents 属性

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

content = soup.head.title.string

for parent in  content.parents:

    print (parent.name)
title

head

html

[document]
获取兄弟节点

.next_sibling 获取此标签后面的节点

.previous_sibling 获取此标签前面的节点

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print (soup.p.next_sibling)

#       实际该处为空白

print (soup.p.prev_sibling)

#None   没有前一个兄弟节点，返回 None

print (soup.p.next_sibling.next_sibling)
None

<p class="story">Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>
兄弟节点可以理解为和本节点处在统一级的节点，.next_sibling 属性获取了该节点的下一个兄弟节点，.previous_sibling 则与之相反，如果节点不存在，则返回 None

注意：实际文档中的tag的 .next_sibling 和 .previous_sibling 属性通常是字符串或空白，因为空白或者换行也可以被视作一个节点，所以得到的结果可能是空白或者换行
获取前后节点

.next_element 获取此标签后一个节点

.previous_element 获取此标签前一个节点

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print (soup.head.next_element)
<title>The Dormouse's story</title>
与 .next_sibling .previous_sibling 不同，它并不是针对于兄弟节点，而是在所有节点，不分层次。

比如 head 节点为：

<head><title>The Dormouse's story</title></head>

那么它的下一个节点便是 title，它是不分层次关系的
获取所有前后节点

.next_elements 获取后面的所有节点

.previous_elements获取前面的所有节点

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for element in soup.body.next_elements:

    print(repr(element))
'\n'

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<b>The Dormouse's story</b>

"The Dormouse's story"

'\n'

<p class="story">Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

'Once upon a time there were three little sisters; and their names were\n'

<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>

' Elsie '

',\n'

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>

'Lacie'

' and\n'

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

'Tillie'

';\nand they lived at the bottom of a well.'

'\n'

<p class="story">...</p>

'...'

'\n'
通过 .next_elements 和 .previous_elements 的迭代器就可以向前或向后访问文档的解析内容,就好像文档正在被解析一样
标准选择器

find_all( name , attrs , recursive , text , **kwargs )

可以根据标签名，属性，内容查找文档。
name 参数

可以传入一个标签名，会返回该标签名下的内容。

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.find_all('a'))

print(type(soup.find_all('p')))
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

<class 'bs4.element.ResultSet'>
也可以通过层层嵌套的方法获取输出：

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for p in soup.find_all('p'):

    print(p.find_all('a'))
attrs参数

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.find_all(attrs={'name':'dromouse'}))

#也可以使用以下方法，输出结果相同

print(soup.find_all(class_='title'}))
[<p class="title" name="dromouse"><b>The Dormouse's story</b></p>]

[<p class="title" name="dromouse"><b>The Dormouse's story</b></p>]
text属性

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.find_all(text='Lacie'))
['Lacie']
返回的是它里面的内容，但是在做元素查找时没有什么用处，但是做元素匹配时会有用处。
limit 参数

find_all() 方法返回全部的搜索结构,如果文档树很大那么搜索会很慢.如果我们不需要全部结果,可以使用 limit 参数限制返回结果的数量.效果与SQL中的limit关键字类似,当搜索到的结果数量达到 limit 的限制时,就停止搜索返回结果.

文档树中有3个tag符合搜索条件,但结果只返回了2个,因为我们限制了返回数量。
soup.find_all("a", limit=2)

# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,

#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]
find( name , attrs , recursive , text , **kwargs )

它与 find_all() 方法唯一的区别是 find_all() 方法的返回结果是值包含一个元素的列表,而 find() 方法直接返回结果，也就是说，find方法只会返回一个元素。

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.find('p'))
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
其他用法一样的标准选择器

（1）find_parents() find_parent()

find_all() 和 find() 只搜索当前节点的所有子节点,孙子节点等. find_parents() 和 find_parent() 用来搜索当前节点的父辈节点,搜索方法与普通tag的搜索方法相同,搜索文档搜索文档包含的内容

（2）find_next_siblings() find_next_sibling()

这2个方法通过 .next_siblings 属性对当 tag 的所有后面解析的兄弟 tag 节点进行迭代, find_next_siblings() 方法返回所有符合条件的后面的兄弟节点,find_next_sibling() 只返回符合条件的后面的第一个tag节点

（3）find_previous_siblings() find_previous_sibling()

这2个方法通过 .previous_siblings 属性对当前 tag 的前面解析的兄弟 tag 节点进行迭代, find_previous_siblings() 方法返回所有符合条件的前面的兄弟节点, find_previous_sibling() 方法返回第一个符合条件的前面的兄弟节点

（4）find_all_next() find_next()

这2个方法通过 .next_elements 属性对当前 tag 的之后的 tag 和字符串进行迭代, find_all_next() 方法返回所有符合条件的节点, find_next() 方法返回第一个符合条件的节点

（5）find_all_previous() 和 find_previous()

这2个方法通过 .previous_elements 属性对当前节点前面的 tag 和字符串进行迭代, find_all_previous() 方法返回所有符合条件的节点, find_previous()方法返回第一个符合条件的节点
CSS选择器

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

print(soup.select('.story .sister'))

print(soup.select('p a'))

print(soup.select('#link2'))

print(soup.select('p')[0])
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

[<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
第一个打印的结果为class属性为‘story’的标签中class属性为‘sister’的元素，使用class进行选择时，名称前需要加一个"."。

第二个打印结果为p标签中a标签的内容，与第一个返回结果相同。

第三个打印结果为id为link2标签的内容，在使用id标签进行选择时需要在名字前面加"#"符号。

第四个打印的是第一个p标签的内容。
进行层层迭代的输出

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for p in soup.select('p'):

    print(p.select('a'))
[]

[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

[]
获取属性

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for p in soup.select('p'):

    print(p['class'])
['title']

['story']

['story']
获取内容

from bs4 import BeautifulSoup

html = """

<html><head><title>The Dormouse's story</title></head>

<body>

<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.</p>

<p class="story">...</p>

"""

soup = BeautifulSoup(html,'lxml')

for p in soup.select('a'):

    print(p.get_text)
<bound method Tag.get_text of <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>>

<bound method Tag.get_text of <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>>

<bound method Tag.get_text of <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>>
获取到了p标签中a标签的所有内容
														
		
吴裕雄--天生自然python学习笔记：beautifulsoup库的使用的更多相关文章
	

    

  1. 
吴裕雄--天生自然python学习笔记：Beautiful Soup 4&period;2&period;0模块
		
    Beautiful Soup 是一个可以从HTML或XML文件中提取数据的Python库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup会帮你节省数小时 ...
    
		
    2. 
								
  2. 
吴裕雄--天生自然python学习笔记：pandas模块强大的数据处理套件
		
    用 Python 进行数据分析处理,其中最炫酷的就属 Pa ndas 套件了 . 比如,如果我 们通过 Requests 及 Beautifulsoup 来抓取网页中的表格数据 , 需要进行较复 杂的 ...
    
		
    3. 
						
  3. 
吴裕雄--天生自然python学习笔记：python 用pyInstaller模块打包文件
		
    要想在没有安装 Python 集成环境的电脑上运行开发的 Python 程序,必须把 Python 文件打包成 .exe 格式的可执行 文件. Python 的打包工作 PyInstaller 提供了 ...
    
		
    4. 
						
  4. 
吴裕雄--天生自然python学习笔记：python 文件批量查找
		
    在多个文本文件中查找 我们首先来学习文本文件的查找字符 . 我们通过 os.walk 扩大查找范围, 查找指定目录和子目录下的文件. 应用程序总览 读取 当 前目录及子目录下的所有 PY 和 txt  ...
    
		
    5. 
						
  5. 
吴裕雄--天生自然python学习笔记：python爬虫与网页分析
		
    我们所抓取的网页源代码一般都是 HTML 格式的文件,只要研究明白 HTML 中 的标签( Tag )结构,就很容易进行解析并取得所需数据 . HTML 网页结构 HTML 网 页是由许多标签( Ta ...
    
		
    6. 
						
  6. 
吴裕雄--天生自然python学习笔记：WEB数据抓取与分析
		
    Web 数据抓取技术具有非常巨大的应用需求及价值, 用 Python 在网页上收集数据,不仅抓取数据的操作简单, 而且其数据分析功能也十分强大. 通过 Python 的时lib 组件中的 urlpar ...
    
		
    7. 
						
  7. 
吴裕雄--天生自然python学习笔记：python通过&OpenCurlyDoubleQuote;任务计划程序”实现定时自动下载或更新运行 PM2&period;5 数据抓取程序数据
		
    在 Windows 任务计划程序中,设置每隔 30 分钟自动抓取 PM2.5 数据,井保存 在 SQLite 数据库中 . import sqlite3,ast,requests,os from bs ...
    
		
    8. 
						
  8. 
吴裕雄--天生自然python学习笔记：python 用pygame模块动画一让图片动起来
		
    动画是游戏开发中不可或缺的要素,游戏中的角色只有动起来才会拥有“生命”, 但动画处理也是最让游戏开发者头痛的部分.Pygame 包通过不断重新绘制绘图窗口,短短几行代码就可以让图片动起来! 动画处理程 ...
    
		
    9. 
						
  9. 
吴裕雄--天生自然python学习笔记：python 用pygame模块游戏开发
		
    游戏开发在软件开发领域占据了非常重要的位直.游 戏开发需要用到的技术相当广泛,除了多媒体.图片.动 画的处理外,程序设计更是游戏开发的核心内容. Py game 是为了让 Python 能够进行游戏开 ...
    
		
    10. 
		


随机推荐
	

    

  1. 
javascript练习-定义子类
			
    function defineSubclass(superclass, //父类的构造函数 constructor, //新的子类的构造函数 methods, //实例方法:复制至原型中 static ...
    
			
    2. 
						
  2. 
WWW压缩解压缩
			
    unity的WWW参考文档:http://game.ceeger.com/Script/WWW/WWW.html 在unity中把资源打包成Assetbundle其实把资源通过 LZMA 压缩成二进制 ...
    
			
    3. 
						
  3. 
OD&colon; GS Bypasing via SEH &sol; &period;data
			
    通过 SEH 绕过 GS 保护 GS 机制没对 SEH 提供保护,所以可心通过攻击异常来绕过 GS. 实验环境为: VMware : Windows sp4, 此版本无 SafeSEH 的影响 Vis ...
    
			
    4. 
						
  4. 
GUI RedHat7中常用的一些设置
			
    GUI RedHat7中常用的一些设置... ------------------ RedHat7更改桌面背景 ================= 更改RedHat7的分辨率 ============ ...
    
			
    5. 
						
  5. 
stringify在苹果电脑下的值不能为空
			
     sessionStorage.channel = JSON.stringify(  );苹果的safari不接受stringify里面为空 火桑飘零ご 2018/1/25 20:21:49 wind ...
    
			
    6. 
						
  6. 
学习致用九---centos7&period;2&plus;vim vundle
			
    目的,安装vim插件,vundle   Vundle是Vim的插件管理插件 YouCompleteMe 简称 YCM 1.安装vundle: git clone https://github.com/ ...
    
			
    7. 
						
  7. 
Codeforces 988F Rain and Umbrellas&lpar;DP&rpar;
			
    题目链接:http://codeforces.com/contest/988/problem/F 题目大意: 有三个整数a,n,m,a是终点坐标,给出n个范围(l,r)表示这块区域下雨,m把伞(p,w ...
    
			
    8. 
						
  8. 
苹果Mac OS X系统十三年视觉变化发展史
			
    1Mac OS 9 一个普通的桌面操作系统 经过多个测试版本后,苹果终于正式公布OS X 10.10 Yosemite操作系统.苹果称这个第11版的OS X系统是自从2001年问世以来在视觉效果上变化 ...
    
			
    9. 
						
  9. 
WordPress插件：自定义登录注册插件：DX Login Register
			
    众所周知,wordpress自带的注册系统比较简单,需要接收邮件密码才能完成.不过对于国内的站长来说,会碰到不少麻烦.首先个人站长一般都使用虚拟主机,有不少还是使用国外的,你的服务器不一定会提供邮件发 ...
    
			
    10. 
						
  10. 
2017中国大学生程序设计竞赛 - 网络选拔赛  HDU 6153 A Secret KMP，思维
			
    题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=6153 题意:给了串s和t,要求每个t的后缀在在s中的出现次数,然后每个次数乘上对应长度求和. 解法:关 ...
    
			
    11. 
			
相关文章