I am trying to scrape a website but I don't get some of the elements, because these elements are dynamically created.
我试图刮一个网站,但我没有得到一些元素,因为这些元素是动态创建的。
I use the cheerio in node.js and My code is below.
我在node.js中使用cheerio,我的代码在下面。
var request = require('request');
var cheerio = require('cheerio');
var url = "http://www.bdtong.co.kr/index.php?c_category=C02";
request(url, function (err, res, html) {
var $ = cheerio.load(html);
$('.listMain > li').each(function () {
console.log($(this).find('a').attr('href'));
});
});
This code returns empty response, because when the page is loaded, the <ul id="store_list" class="listMain">
is empty.
此代码返回空响应,因为在加载页面时,
-
为空。
The content has not been appended yet.
内容尚未附加。
How can I get these elements using node.js? How can I scrape pages with dynamic content?
如何使用node.js获取这些元素?如何用动态内容抓取页面?
3 个解决方案
#1
18
Here you go;
干得好;
var phantom = require('phantom');
phantom.create(function (ph) {
ph.createPage(function (page) {
var url = "http://www.bdtong.co.kr/index.php?c_category=C02";
page.open(url, function() {
page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
page.evaluate(function() {
$('.listMain > li').each(function () {
console.log($(this).find('a').attr('href'));
});
}, function(){
ph.exit()
});
});
});
});
});
#2
11
Use the new npm module x-ray, with a pluggable web driver x-ray-phantom.
使用新的npm模块X射线,带有可插拔的网络驱动程序x-ray-phantom。
Examples in the pages above, but here's how to do dynamic scraping:
上面几页中的示例,但这里是如何进行动态抓取:
var phantom = require('x-ray-phantom');
var Xray = require('x-ray');
var x = Xray()
.driver(phantom());
x('http://google.com', 'title')(function(err, str) {
if (err) return done(err);
assert.equal('Google', str);
done();
})
#3
7
Check out GoogleChrome/puppeteer
查看GoogleChrome / puppeteer
Headless Chrome Node API
无头Chrome节点API
It makes scraping pretty trivial. The following example will scrape the headline over at npmjs.com (assuming .npm-expansions
remains)
它使得刮擦非常简单。以下示例将在npmjs.com上删除标题(假设.npm-expansions仍然存在)
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.npmjs.com/');
const textContent = await page.evaluate(() => {
return document.querySelector('.npm-expansions').textContent
});
console.log(textContent); /* No Problem Mate */
browser.close();
})();
evaluate
will allow for the inspection of the dynamic element as this will run scripts on the page.
evaluate将允许检查动态元素,因为这将在页面上运行脚本。
#1
18
Here you go;
干得好;
var phantom = require('phantom');
phantom.create(function (ph) {
ph.createPage(function (page) {
var url = "http://www.bdtong.co.kr/index.php?c_category=C02";
page.open(url, function() {
page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
page.evaluate(function() {
$('.listMain > li').each(function () {
console.log($(this).find('a').attr('href'));
});
}, function(){
ph.exit()
});
});
});
});
});
#2
11
Use the new npm module x-ray, with a pluggable web driver x-ray-phantom.
使用新的npm模块X射线,带有可插拔的网络驱动程序x-ray-phantom。
Examples in the pages above, but here's how to do dynamic scraping:
上面几页中的示例,但这里是如何进行动态抓取:
var phantom = require('x-ray-phantom');
var Xray = require('x-ray');
var x = Xray()
.driver(phantom());
x('http://google.com', 'title')(function(err, str) {
if (err) return done(err);
assert.equal('Google', str);
done();
})
#3
7
Check out GoogleChrome/puppeteer
查看GoogleChrome / puppeteer
Headless Chrome Node API
无头Chrome节点API
It makes scraping pretty trivial. The following example will scrape the headline over at npmjs.com (assuming .npm-expansions
remains)
它使得刮擦非常简单。以下示例将在npmjs.com上删除标题(假设.npm-expansions仍然存在)
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.npmjs.com/');
const textContent = await page.evaluate(() => {
return document.querySelector('.npm-expansions').textContent
});
console.log(textContent); /* No Problem Mate */
browser.close();
})();
evaluate
will allow for the inspection of the dynamic element as this will run scripts on the page.
evaluate将允许检查动态元素,因为这将在页面上运行脚本。