之前 br>之后的BeautifulSoup Parse Text

I have this code trying to parse search results from a grant website (please find the URL in the code, I can't post the link yet until my rep is higher), the "Year"and "Amount Award" after tags and before tags.

我有这个代码试图从授权网站解析搜索结果(请在代码中找到URL,我不能发布链接,直到我的代表更高),标签之后的“年份”和“金额奖励”标签。

Two questions:

1) Why is this only returning the 1st table?

1)为什么这只返回第一张桌子?

2) Any way I can get the text that is after the (i.e. Year and Amount Award strings) and (i.e. the actual number such as 2015 and $100000)

2)我可以通过任何方式获得(即年份和金额奖励字符串)之后的文本(即2015年和100000美元等实际数字)

Specifically:

<td valign="top">
				            <b>Year: </b>2014<br>
				            <b>Award Amount: </b>$84,907				                                				                                				        </td>

Here is my script:

这是我的脚本:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'http://www.ned.org/wp-content/themes/ned/search/grant-search.php?' \
    'organizationName=&region=ASIA&projectCountry=China&amount=&fromDate=&toDate=&' \
    'projectFocus%5B%5D=&search=&maxCount=25&orderBy=Year&start=1&sbmt=1'

r = requests.get(url)

html_content = r.text

soup = BeautifulSoup(html_content, "html.parser")

tables = soup.find_all('table')


data = {
        'col_names': [],
        'info' : [],
        'year_amount':[]
        }


index = 0

for table in tables:
    rows = table.find_all('tr')[1:]
    for row in rows:
        cols = row.find_all('td')
        data['col_names'].append(cols[0].get_text())
        data['info'].append(cols[1].get_text())
        try:
            data['year_amount'].append(cols[2].get_text())
        except IndexError:
            data['year_amount'].append(None)
    grant_df = pd.DataFrame(data)
    index += 1
    filename = 'grant ' + str(index) + '.csv'
    grant_df.to_csv(filename)

1 个解决方案

#1

I would suggest approaching the table parsing in a different manner. All of the information is available in the first row of each table. So you can parse the text of the row like:

我建议以不同的方式处理表解析。所有信息都可在每个表的第一行中找到。所以你可以解析行的文本,如:

Code:

text = '\n'.join([x.strip() for x in rows[0].get_text().split('\n')
                  if x.strip()]).replace(':\n', ': ')
data_dict = {k.strip(): v.strip() for k, v in
             [x.split(':', 1) for x in text.split('\n')]}

How?:

This takes the text and

这需要文本和

1. splits it on newlines

将它拆分为换行符

2. removes any blank lines

删除任何空行

3. removes any leading/trailing space

删除任何前导/尾随空格

4. joins the lines back together into a single text

将行重新连接成一个文本

5. joins any line ending in : with the next line

以下一行加入以:结尾的任何行

Then:

1. split the text again by newline

换行再次拆分文本

2. split each line by :

将每一行拆分为:

3. strip any whitespace of ends of text on either side of :

剥离任何一侧文本末尾的任何空格:

4. use the split text as key and value to a dict

使用拆分文本作为键和dict的值

Test Code:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'http://www.ned.org/wp-content/themes/ned/search/grant-search.php?' \
      'organizationName=&region=ASIA&projectCountry=China&amount=&' \
      'fromDate=&toDate=&projectFocus%5B%5D=&search=&maxCount=25&' \
      'orderBy=Year&start=1&sbmt=1'

r = requests.get(url)
soup = BeautifulSoup(r.text, "html.parser")

data = []
for table in soup.find_all('table'):
    rows = table.find_all('tr')
    text = '\n'.join([x.strip() for x in rows[0].get_text().split('\n')
                      if x.strip()]).replace(':\n', ': ')
    data_dict = {k.strip(): v.strip() for k, v in
                 [x.split(':', 1) for x in text.split('\n')]}

    if data_dict.get('Award Amount'):
        data.append(data_dict)
grant_df = pd.DataFrame(data)
print(grant_df.head())

Results:

  Award Amount                                        Description  \
0      $84,907  To strengthen the capacity of China's rights d...   
1     $204,973  To provide an effective forum for free express...   
2      $48,000  To promote religious freedom in China. The org...   
3      $89,000  To educate and train civil society activists o...   
4      $65,000  To encourage greater public discussion, transp...   

            Organization Name Project Country                Project Focus  \
0                         NaN  Mainland China                  Rule of Law   
1  Princeton China Initiative  Mainland China       Freedom of Information   
2                         NaN  Mainland China                  Rule of Law   
3                         NaN  Mainland China  Democratic Ideas and Values   
4                         NaN  Mainland China                  Rule of Law   

  Project Region                                      Project Title  Year  
0           Asia             Empowering the Chinese Legal Community  2014  
1           Asia  Supporting Free Expression and Open Debate for...  2014  
2           Asia  Religious Freedom, Rights Defense and Rule of ...  2014  
3           Asia     Education on Civil Society and Democratization  2014  
4           Asia        Promoting Democratic Policy Change in China  2014