使用Python从HTML中提取可读文本？

I know about utils like html2text, BeautifulSoup etc. but the issue is that they also extract javascript and add it to the text making it tough to separate them.

我知道像html2text，BeautifulSoup等的utils，但问题是他们也提取javascript并将其添加到文本中，这使得分离它们变得困难。

htmlDom = BeautifulSoup(webPage)

htmlDom.findAll(text=True)

Alternately,

交替，

from stripogram import html2text
extract = html2text(webPage)

Both of these extract all the javascript on the page as well, this is undesired.

这两个都提取了页面上的所有javascript，这是不受欢迎的。

I just wanted the readable text which you could copy from your browser to be extracted.

我只是想要提取您可以从浏览器中复制的可读文本。

4 个解决方案

#1

If you want to avoid extracting any of the contents of script tags with BeautifulSoup,

如果您想避免使用BeautifulSoup提取脚本标记的任何内容，

nonscripttags = htmlDom.findAll(lambda t: t.name != 'script', recursive=False)

will do that for you, getting the root's immediate children which are non-script tags (and a separate htmlDom.findAll(recursive=False, text=True) will get strings that are immediate children of the root). You need to do this recursively; e.g., as a generator:

将为您做到这一点，让root的直接子项是非脚本标记（并且单独的htmlDom.findAll（recursive = False，text = True）将获得作为根的直接子项的字符串）。你需要递归地做这件事;例如，作为发电机：

def nonScript(tag):
    return tag.name != 'script'

def getStrings(root):
   for s in root.childGenerator():
     if hasattr(s, 'name'):    # then it's a tag
       if s.name == 'script':  # skip it!
         continue
       for x in getStrings(s): yield x
     else:                     # it's a string!
       yield s

I'm using childGenerator (in lieu of findAll) so that I can just get all the children in order and do my own filtering.

我正在使用childGenerator（代替findAll），以便我可以让所有的孩子按顺序完成并自己进行过滤。

#2

you can remove script tags in beautiful soup, something like:

你可以在漂亮的汤中删除脚本标签，例如：

for script in soup("script"):
    script.extract()

Removing Elements

删除元素

#3

Using BeautifulSoup, something along these lines:

使用BeautifulSoup，有以下几点：

def _extract_text(t):
    if not t:
        return ""
    if isinstance(t, (unicode, str)):
        return " ".join(filter(None, t.replace("\n", " ").split(" ")))
    if t.name.lower() == "br": return "\n"
    if t.name.lower() == "script": return "\n"
    return "".join(extract_text(c) for c in t)
def extract_text(t):
    return '\n'.join(x.strip() for x in _extract_text(t).split('\n'))
print extract_text(htmlDom)

#4

Try it out:

试试看：

http://code.google.com/p/boilerpipe/

http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html/

#1