X路径Python错误- 'list'对象没有属性'xpath'

I am trying to do a Web-Crawling project, and I am keep on getting an error code called "'list' object has no attribute 'xpath'"/

我正在尝试做一个web爬行项目，我不断地得到一个名为“list”的错误代码，对象没有属性“xpath”/

This is the code so far:

这是迄今为止的代码:

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

headlines = headlineNode.xpath('/a')
len(headlines)

I want to extraxt the headlines from CNBC Home, but I am keep on getting an error saying

我想从CNBC电视台的头条新闻中脱颖而出，但我一直在说错话

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-d9f54582133c> in <module>()
----> 1 headlines = headlineNode.xpath('/a')
      2 len(headlines)

AttributeError: 'list' object has no attribute 'xpath'

Is there anyway to fix this?

有办法解决这个问题吗?

2 个解决方案

#1

The reason you are getting this error is because xpath always results a list of elements and xpath can be called on elements.

出现这个错误的原因是xpath总是会产生一个元素列表，可以对元素调用xpath。

So to get expected results, what you can do is iterate over the headlineNode list and then call the xpath to the element nodes such as node.xpath('/a')

为了得到预期的结果，可以遍历headlineNode列表，然后调用元素节点的xpath，比如node.xpath('/a')

The following code will give you the list of all the url you needed as a list. You can do something like this :

下面的代码将为您提供作为列表所需的所有url的列表。你可以这样做:

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

url_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    if url_node : 
        url_list.append(url + url_node[0].strip())
print(url_list)
len(url_list)

This will result in :

这将导致:

['http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'http://www.cnbc.com/2017/03/28/us-markets.html', 'http://www.cnbc.com/2017/03/28/european-markets-investors-shrug-off-trump-concerns.html', 'http://www.cnbc.com/2017/03/27/gold-steady-as-dollar-edges-up-focus-on-trump-agenda.html', 'http://www.cnbc.com/2017/03/27/weaker-dollar-lifts-oil-futures-but-soaring-us-output-weighs.html', 'http://www.cnbc.com/2017/03/28/us-treasurys-edge-lower-as-investors-eye-data-auctions.html', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/28/markets-could-see-heartbreak-if-brexit-is-hard-to-do.html', 'http://www.cnbc.com/2017/03/28/former-google-exec-heres-how-to-know-its-time-to-fire-an-employee.html', 'http://www.cnbc.com/2017/03/28/chinas-secret-plan-to-crush-spacex-and-the-us-space-program.html', 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'http://www.cnbc.com/2017/03/28/following-tuesdays-rally-stocks-will-grind-higher-into-2018-analyst-says.html', 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'http://www.cnbc.com/2017/03/28/toshibas-westinghouse-to-file-for-us-bankruptcy-sources.html', 'http://www.cnbc.com/2017/03/28/early-reports-of-damage-in-wake-of-australias-cyclone-debbie.html', 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'http://www.cnbc.com/2017/03/28/rodrigo-duterte-turns-72-a-look-at-his-controversial-presidency-so-far-commentary.html', 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'http://www.cnbc.com/2017/03/28/congress-clears-way-for-isps-to-sell-browsing-history.html', 'http://www.cnbc.com/2017/03/28/british-pm-may-to-fire-starting-gun-on-brexit.html', 'http://www.cnbc.com/2017/03/28/brexit-is-like-arthritis-it-doesnt-kill-you-but-it-hits-you-says-former-boe-member.html', 'http://www.cnbc.com/2017/03/27/florida-preparesfor-*svisit.html', 'http://www.cnbc.com/2017/03/28/us-heading-for-recession-after-2-years-of-unsustainable-growth-economist-says.html', 'http://www.cnbc.com/2017/03/28/this-is-what-immigration-means-to-the-us-economy-in-two-charts.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'http://www.cnbc.com/2017/03/28/next-iphone-is-last-hurrah-for-smartphones-apple.html', 'http://www.cnbc.com/2017/03/28/south-africa-speculates-over-possible-firing-of-finance-minister.html', 'http://www.cnbc.com/2017/03/28/snap-was-a-drunk-at-the-wheel-but-now-looks-like-a-very-real-business-fund-manager-says.html', 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'http://www.cnbc.com/2017/03/28/siemens-medical-solutions-wins-41-billion-us-defense-contract.html', 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'http://www.cnbc.com/2017/03/28/trump-to-meet-egyptian-president-at-white-house-on-april-3.html', 'http://www.cnbc.com/2017/03/28/tesla-is-rejecting-a-racial-harassment-lawsuit-filed-by-an-assembly-line-worker.html', 'http://www.cnbc.com/2017/03/28/jpmorgan-says-markets-are-underestimating-the-upside-of-asian-banks.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/https://www.facebook.com/cnbcinternational/', 'http://www.cnbc.com/https://twitter.com/cnbcworld', 'http://www.cnbc.com/http://www.linkedin.com/company/2966126', 'http://www.cnbc.com/http://www.youtube.com/user/CNBCInternational', 'http://www.cnbc.com/http://www.cnbc.com/id/28295763', 'http://www.cnbc.com/pre-markets/', 'http://www.cnbc.com/http://www.cnbc.com/world-markets-heat-map/', 'http://www.cnbc.com/dow-components/', 'http://www.cnbc.com/http://data.cnbc.com/quotes/aapl%2Caxp%2Cba%2Ccat%2Ccsco%2Ccvx%2Cdd%2Cdis%2Cge%2Cgs%2Chd%2Cibm%2Cintc%2Cjnj%2Cjpm%2Cko%2Cmcd%2Cmmm%2Cmrk%2Cmsft%2Cnke%2Cpfe%2Cpg%2Ctrv%2Cunh%2Cutx%2Cv%2Cvz%2Cwmt%2Cxom', 'http://www.cnbc.com/sectors/', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'http://www.cnbc.com/2017/03/28/iran-says-russia-can-use-its-military-bases-on-case-by-case-basis.html', 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'http://www.cnbc.com/2017/03/28/hootsuite-ceo-talks-social-media-and-trump.html', 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'http://www.cnbc.com/2017/03/28/dow-trump-dollar-kushner-stoxx-wallstreet.html', 'http://www.cnbc.com/2017/03/27/trump-brexit-healthcare-russia-dow-stoxx.html', 'http://www.cnbc.com/2017/03/24/trump-healthcare-markets-us-dow-stocks.html', 'http://www.cnbc.com/2017/03/23/trump-london-terrorism-markets-stocks-dow-ftse-may-uk-us.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000603723', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000602217', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000600643', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000598102', 'http://www.cnbc.com/2017/03/26/trader-poll-which-will-garner-the-most-market-attention.html', 'http://www.cnbc.com/2017/03/20/trader-poll-what-are-chinas-greatest-challenges.html', 'http://www.cnbc.com/2017/03/12/trader-poll-is-the-fed-definitely-going-to-hike-rates-at-this-meeting.html', 'http://www.cnbc.com/2017/03/06/trader-poll-has-the-us-dollar-got-higher-to-go.html', 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'http://www.cnbc.com/advertorial/2017/03/09/breathing-new-life-into-brands.html', 'http://www.cnbc.com/2017/03/23/make-sure-youre-not-replaced-by-a-machine-microsoft-us-chief-marketer.html', 'http://www.cnbc.com/2017/03/23/online-ads-need-to-stop-being-intrusive-eurosport.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595689', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595687', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595688', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595690', 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'http://www.cnbc.com/2017/03/23/engineers-develop-solar-powered-skin-which-could-transform-lives.html', 'http://www.cnbc.com/2017/03/23/palm-oil-its-uses-and-ways-to-limit-its-impact.html', 'http://www.cnbc.com/advertorial/2017/03/13/access-to-energy-when-business-issues-meet-social-impacts.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/2017/03/28/amazon-to-acquire-souqcom-a-large-e-commerce-site-in-the-middle-east.html', 'http://www.cnbc.com/advertorial/2016/12/13/a-new-era-of-intelligent-machinery.html', 'http://www.cnbc.com/advertorial/2016/12/05/the-reimagining-of-cities.html', 'http://www.cnbc.com/advertorial/2016/11/22/could-mining-asteroids-save-earth.html', 'http://www.cnbc.com/advertorial/2016/11/02/smarter-ships-safer-seas.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/2017/03/28/risk-of-correction-quite-high-but-wont-kill-bull-market-says-goldman-sachs-strategist.html', 'http://www.cnbc.com/2017/03/28/samsung-galaxy-s8-apple-iphone-8-launch.html', 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'http://www.cnbc.com/2017/03/28/facebook-stole-these-features-from-snapchat.html', 'http://www.cnbc.com/2017/03/28/cisco-separating-some-key-networking-software-from-hardware--report.html', 'http://www.cnbc.com/2017/03/28/uber-releases-diversity-report-and-repudiates-its-hard-charging-attitude.html', 'http://www.cnbc.com/2017/03/28/cramers-charts-explain-why-general-mills-looks-ripe-for-a-run.html', 'http://www.cnbc.com/2017/03/28/cramer-why-it-might-be-time-for-you-to-unfollow-trumps-twitter-feed.html', 'http://www.cnbc.com/2017/03/28/blackrock-plans-major-changes-to-its-active-stockpicking-business.html', 'http://www.cnbc.com/2017/03/28/darden-shares-pop-4-percent-on-strong-earnings-beat-cheddars-acquisition.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605169', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605171', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605046', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605164', 'http://www.cnbc.com/2017/03/28/nunes-seems-like-the-wrong-person-to-lead-trump-russia-probe-commentary.html', 'http://www.cnbc.com/2017/03/28/donald-trump-is-making-his-russia-scandal-about-clintons-commentary.html', 'http://www.cnbc.com/2017/03/28/self-made-millionaire-shares-3-tricks-for-raising-successful-kids.html', 'http://www.cnbc.com/2017/03/28/president-trump-is-directing-his-anger-at-the-wrong-people-commentary.html']

To add title also to the result_list, you can modify it something like this :

要将title添加到result_list中，您可以对它进行如下修改:

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

result_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    title = node.xpath('./a/text()')
    if (url_node and title) : 
        result_list.append({'url' : url + url_node[0].strip(),
            'title' : title[0].strip()})
print(result_list)
len(result_list)

This will result in :

这将导致:

[{'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/us-markets.html', 'title': 'Dow snaps 8-day losing streak as consumer confidence surges; financials lead'}, {'url': 'http://www.cnbc.com/2017/03/28/european-markets-investors-shrug-off-trump-concerns.html', 'title': 'European stocks close higher as investors shrug off Trump concerns; Wolseley up 5%'}, {'url': 'http://www.cnbc.com/2017/03/27/gold-steady-as-dollar-edges-up-focus-on-trump-agenda.html', 'title': 'Gold rises as investors focus on Trump agenda'}, {'url': 'http://www.cnbc.com/2017/03/27/weaker-dollar-lifts-oil-futures-but-soaring-us-output-weighs.html', 'title': 'US crude settles at $48.37, rising 64 cents with Libyan output disruption'}, {'url': 'http://www.cnbc.com/2017/03/28/us-treasurys-edge-lower-as-investors-eye-data-auctions.html', 'title': 'US Treasurys mostly higher as investors digest data, 5-year note auction'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/markets-could-see-heartbreak-if-brexit-is-hard-to-do.html', 'title': 'Markets could see heartbreak if Brexit is hard to do'}, {'url': 'http://www.cnbc.com/2017/03/28/former-google-exec-heres-how-to-know-its-time-to-fire-an-employee.html', 'title': "Former Google exec: Here's how to know it's time to fire an employee"}, {'url': 'http://www.cnbc.com/2017/03/28/chinas-secret-plan-to-crush-spacex-and-the-us-space-program.html', 'title': "China's secret plan to crush SpaceX and the US space program"}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as dangerous as Greece's financial meltdown: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'How this tiny country went from nuclear problem child to role model'}, {'url': 'http://www.cnbc.com/2017/03/28/following-tuesdays-rally-stocks-will-grind-higher-into-2018-analyst-says.html', 'title': "Following Tuesday's rally, stocks will 'grind higher' into 2018, analyst says"}, {'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'title': "These three start-ups show why it's so hard to build products in America"}, {'url': 'http://www.cnbc.com/2017/03/28/japans-tepid-retail-sales-raise-concerns-about-consumption-growth.html', 'title': "Japan's tepid retail sales raise concerns about consumption, growth"}, {'url': 'http://www.cnbc.com/2017/03/28/toshibas-westinghouse-to-file-for-us-bankruptcy-sources.html', 'title': "Toshiba's Westinghouse to file for US bankruptcy: Sources"}, {'url': 'http://www.cnbc.com/2017/03/28/early-reports-of-damage-in-wake-of-australias-cyclone-debbie.html', 'title': "Early reports of damage in wake of Australia's Cyclone Debbie"}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/rodrigo-duterte-turns-72-a-look-at-his-controversial-presidency-so-far-commentary.html', 'title': 'As Rodrigo Duterte turns 72, a look at his controversial presidency so far'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/congress-clears-way-for-isps-to-sell-browsing-history.html', 'title': 'Congress just cleared the way for internet providers to sell your web browsing history'}, {'url': 'http://www.cnbc.com/2017/03/28/british-pm-may-to-fire-starting-gun-on-brexit.html', 'title': 'British PM May to fire starting gun on Brexit'}, {'url': 'http://www.cnbc.com/2017/03/28/brexit-is-like-arthritis-it-doesnt-kill-you-but-it-hits-you-says-former-boe-member.html', 'title': u'Brexit is like arthritis, it doesn\u2019t kill you but it hits you, says former BOE member'}, {'url': 'http://www.cnbc.com/2017/03/27/florida-preparesfor-*svisit.html', 'title': u"Florida authorities prep\xa0for Xi *'s\xa0visit"}, {'url': 'http://www.cnbc.com/2017/03/28/us-heading-for-recession-after-2-years-of-unsustainable-growth-economist-says.html', 'title': 'US heading for recession after 2 years of unsustainable growth, economist says'}, {'url': 'http://www.cnbc.com/2017/03/28/this-is-what-immigration-means-to-the-us-economy-in-two-charts.html', 'title': 'This is what immigration means to the U.S. economy in two charts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u'Jack Dorsey brushes off calls to quit as CEO, says will do \u2018whatever it takes\u2019 for Twitter, Square to succeed'}, {'url': 'http://www.cnbc.com/2017/03/28/next-iphone-is-last-hurrah-for-smartphones-apple.html', 'title': 'The next iPhone is the last hurrah for smartphones: Fund manager'}, {'url': 'http://www.cnbc.com/2017/03/28/south-africa-speculates-over-possible-firing-of-finance-minister.html', 'title': 'South Africa faces uncertain wait amid speculation finance minister is to be fired'}, {'url': 'http://www.cnbc.com/2017/03/28/snap-was-a-drunk-at-the-wheel-but-now-looks-like-a-very-real-business-fund-manager-says.html', 'title': "Snap was a 'drunk at the wheel' but now looks like a very real business, fund manager says"}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia and rebranding with the name of its Jewish founders'}, {'url': 'http://www.cnbc.com/2017/03/28/siemens-medical-solutions-wins-41-billion-us-defense-contract.html', 'title': 'Siemens Medical Solutions wins $4.1 billion US defense contract'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/28/trump-to-meet-egyptian-president-at-white-house-on-april-3.html', 'title': 'Trump to meet Egyptian president at White House on April 3'}, {'url': 'http://www.cnbc.com/2017/03/28/tesla-is-rejecting-a-racial-harassment-lawsuit-filed-by-an-assembly-line-worker.html', 'title': 'Tesla rejects racial harassment lawsuit filed by assembly line worker'}, {'url': 'http://www.cnbc.com/2017/03/28/jpmorgan-says-markets-are-underestimating-the-upside-of-asian-banks.html', 'title': 'JPMorgan says markets are underestimating the upside of Asian banks'}, {'url': 'http://www.cnbc.com/https://www.facebook.com/cnbcinternational/', 'title': 'Facebook'}, {'url': 'http://www.cnbc.com/https://twitter.com/cnbcworld', 'title': 'Twitter'}, {'url': 'http://www.cnbc.com/http://www.linkedin.com/company/2966126', 'title': 'LinkedIn'}, {'url': 'http://www.cnbc.com/http://www.youtube.com/user/CNBCInternational', 'title': 'Youtube'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/id/28295763', 'title': 'RSS'}, {'url': 'http://www.cnbc.com/pre-markets/', 'title': 'Pre-markets'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/world-markets-heat-map/', 'title': 'World stock markets heat map'}, {'url': 'http://www.cnbc.com/dow-components/', 'title': 'Dow 30: Real-time quotes'}, {'url': 'http://www.cnbc.com/http://data.cnbc.com/quotes/aapl%2Caxp%2Cba%2Ccat%2Ccsco%2Ccvx%2Cdd%2Cdis%2Cge%2Cgs%2Chd%2Cibm%2Cintc%2Cjnj%2Cjpm%2Cko%2Cmcd%2Cmmm%2Cmrk%2Cmsft%2Cnke%2Cpfe%2Cpg%2Ctrv%2Cunh%2Cutx%2Cv%2Cvz%2Cwmt%2Cxom', 'title': 'Dow 30: After-hours quotes'}, {'url': 'http://www.cnbc.com/sectors/', 'title': 'Sector watch'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as perilous as Greece's financial crisis: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/iran-says-russia-can-use-its-military-bases-on-case-by-case-basis.html', 'title': "Iran says Russia can use its military bases 'on case by case basis'"}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'Nuclear nonproliferation: Lessons from Myanmar'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/hootsuite-ceo-talks-social-media-and-trump.html', 'title': 'Companies need top-down approach to social media: Hootsuite CEO'}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/dow-trump-dollar-kushner-stoxx-wallstreet.html', 'title': 'Reaction: Global markets stepped past Trump wobble on Obamacare'}, {'url': 'http://www.cnbc.com/2017/03/27/trump-brexit-healthcare-russia-dow-stoxx.html', 'title': 'Reaction: Risk-off hits markets after Trump fails to pass Obamacare hurdle'}, {'url': 'http://www.cnbc.com/2017/03/24/trump-healthcare-markets-us-dow-stocks.html', 'title': 'Reaction: Wall Street stocks move higher as investors countdown to health-care vote'}, {'url': 'http://www.cnbc.com/2017/03/23/trump-london-terrorism-markets-stocks-dow-ftse-may-uk-us.html', 'title': 'Reaction: London police name attacker in Westminster terrorist incident'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000603723', 'title': "Weekly Agenda: From EU visas to 'Fearless Girl'"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000602217', 'title': 'Weekly Agenda: From the Canadian budget to the Boao Forum'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000600643', 'title': "Weekly Agenda: From Merkel's US visit to the FOMC meeting"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000598102', 'title': "Weekly Agenda: From 'mini' EU summit to US daylight saving"}, {'url': 'http://www.cnbc.com/2017/03/26/trader-poll-which-will-garner-the-most-market-attention.html', 'title': 'Trader poll: Which will garner the most market attention?'}, {'url': 'http://www.cnbc.com/2017/03/20/trader-poll-what-are-chinas-greatest-challenges.html', 'title': u'Trader poll: What are China\u2019s greatest challenges?'}, {'url': 'http://www.cnbc.com/2017/03/12/trader-poll-is-the-fed-definitely-going-to-hike-rates-at-this-meeting.html', 'title': 'Trader Poll: Is the Fed definitely going to hike rates at this meeting?'}, {'url': 'http://www.cnbc.com/2017/03/06/trader-poll-has-the-us-dollar-got-higher-to-go.html', 'title': 'Trader poll: Has the US dollar got higher to go?'}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia; using name of its Jewish founders'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/09/breathing-new-life-into-brands.html', 'title': 'Breathing new life into brands'}, {'url': 'http://www.cnbc.com/2017/03/23/make-sure-youre-not-replaced-by-a-machine-microsoft-us-chief-marketer.html', 'title': "Make sure you're not replaced by a machine: Microsoft US chief marketer"}, {'url': 'http://www.cnbc.com/2017/03/23/online-ads-need-to-stop-being-intrusive-eurosport.html', 'title': 'Online ads need to stop being intrusive, especially around 2018 Olympics: Eurosport MD'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595689', 'title': 'Why naming a product is so important'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595687', 'title': u'\u2018Building up a sofa\u2019 that\u2019s fits customer needs: Matthew\u2019s brand proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595688', 'title': u'A men\u2019s \u2018grooming brand with an organic twist\u2019: Connor\u2019s proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595690', 'title': 'Why passion is the true driver of a successful entrepreneur'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/23/engineers-develop-solar-powered-skin-which-could-transform-lives.html', 'title': "Engineers develop solar powered 'skin' which could transform lives"}, {'url': 'http://www.cnbc.com/2017/03/23/palm-oil-its-uses-and-ways-to-limit-its-impact.html', 'title': 'Palm oil: Its uses and ways to limit its impact'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/13/access-to-energy-when-business-issues-meet-social-impacts.html', 'title': 'Access to energy: When business issues meet social impacts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u"Jack Dorsey: I'll do \u2018whatever it takes\u2019 for Twitter, Square to succeed"}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'title': 'Square launches in UK, CEO Jack Dorsey says on the lookout for AI firms to acquire'}, {'url': 'http://www.cnbc.com/2017/03/28/amazon-to-acquire-souqcom-a-large-e-commerce-site-in-the-middle-east.html', 'title': 'Amazon reaches deal to acquire Middle East e-commerce site Souq.com'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/13/a-new-era-of-intelligent-machinery.html', 'title': 'A new era of intelligent machinery'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/05/the-reimagining-of-cities.html', 'title': 'The reimagining of cities'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/22/could-mining-asteroids-save-earth.html', 'title': 'Could mining asteroids save Earth?'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/02/smarter-ships-safer-seas.html', 'title': 'At sea, every hour is rush hour. Are autonomous ships the solution?'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO says'}]

You can refer to documentation of scraping and lxml for more understanding.

您可以参考有关抓取和lxml的文档以了解更多信息。

#2

As your error says, you are trying to invoke .xpath(..) on a list

如您的错误所示，您正在尝试在列表中调用.xpath(.. ..)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)
# ^ headlineNode is a list, I guess you know this.

Now,

现在,

headlines = headlineNode.xpath('/a')
                       #^ and you are invoking .xpath on a list.

Fix:

解决办法:

You need to say:

你需要说:

headlines = headlineNode[0].xpath('/a')

Consider renaming your variable name to headlineNodes to avoid confusion, or say: headlineNode = doc.xpath(..)[0] (this might raise an Exception though).

考虑将变量名重命名为headlineNodes以避免混淆，或者可以说:headlineNode = doc.xpath(.. .)[0](这可能会引发异常)。