python - Scrapy pipeline to export csv file in the right format -


i made improvement according suggestion alexce below. need picture below. each row/line should 1 review: date, rating, review text , link.

i need let item processor process each review of every page.
takefirst() takes first review of page. 10 pages, have 10 lines/rows in picture below.

enter image description here

spider code below:

import scrapy amazon.items import amazonitem  class amazonspider(scrapy.spider):    name = "amazon"    allowed_domains = ['amazon.co.uk']    start_urls = [     'http://www.amazon.co.uk/product-reviews/b0042eu3a2/'.format(page)      page in xrange(1,114)  ]  def parse(self, response):     sel in response.xpath('//*[@id="productreviews"]//tr/td[1]'):         item = amazonitem()         item['rating'] = sel.xpath('div/div[2]/span[1]/span/@title').extract()         item['date'] = sel.xpath('div/div[2]/span[2]/nobr/text()').extract()         item['review'] = sel.xpath('div/div[6]/text()').extract()         item['link'] = sel.xpath('div/div[7]/div[2]/div/div[1]/span[3]/a/@href').extract()          yield item

i started scratch , following spider should run with

scrapy crawl amazon -t csv -o amazon.csv --loglevel=info

so opening csv-file spreadsheet shows me

enter image description here

hope helps :-)

import scrapy  class amazonitem(scrapy.item):     rating = scrapy.field()     date = scrapy.field()     review = scrapy.field()     link = scrapy.field()  class amazonspider(scrapy.spider):      name = "amazon"     allowed_domains = ['amazon.co.uk']     start_urls = ['http://www.amazon.co.uk/product-reviews/b0042eu3a2/' ]      def parse(self, response):          sel in response.xpath('//table[@id="productreviews"]//tr/td/div'):              item = amazonitem()             item['rating'] = sel.xpath('./div/span/span/span/text()').extract()             item['date'] = sel.xpath('./div/span/nobr/text()').extract()             item['review'] = sel.xpath('./div[@class="reviewtext"]/text()').extract()             item['link'] = sel.xpath('.//a[contains(.,"permalink")]/@href').extract()             yield item          xpath_next_page = './/table[@id="productreviews"]/following::*//span[@class="paging"]/a[contains(.,"next")]/@href'         if response.xpath(xpath_next_page):             url_next_page = response.xpath(xpath_next_page).extract()[0]             request = scrapy.request(url_next_page, callback=self.parse)             yield request

Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more