Python crawler is not crawling all sites -


i'm trying links websites through code 

import requests bs4 import beautifulsoup def get_links(max_pages):     page = 1     while page <= max_pages:         address = 'http://hamariweb.com/mobiles/nokia_mobile-phones1.aspx?page=' + str(page)         source_code = requests.get(address)         plain_text = source_code.text         soup = beautifulsoup(plain_text)         link in soup.findall('a', {'class': 'textclass8pt'}):             href = link.get("href")             print(href)          page += 1  get_links(3)

and it's giving expected output. when tried this

address = 'http://propakistani.pk/category/cellular/page/' + str(page)

for link in soup.findall('a', {'class': 'aa_art_hdng'}):

it's showing error

typeerror: getresponse() got unexpected keyword argument 'buffering'

i tried site time neighter showed output nor error. why it's showing proper output different sites? there problem code? please me. thanks

there no tag match condition soup.findall('a', {'class': 'textclass8pt'})

try following

demo:

import requests bs4 import beautifulsoup  def get_links(max_pages):     page = 1      while page <= max_pages:         address = 'http://propakistani.pk/category/cellular/page/' + str(page)         source_code = requests.get(address)         plain_text = source_code.text         soup = beautifulsoup(plain_text)         link in soup.findall('a'):             href = link.get("href")             print(href)          page += 1  get_links(3)

or

there a tag class value aa_loop_h2a e.g. <a class="aa_loop_h2a" href="http://propakistani.pk/2015/04/20/mobile-data-usage-in-pakistan-grows-600-during-2014/" title="mobile data usage in pakistan grows 600% during 2014">mobile data usage in pakistan grows 600% during 2014</a>

so try soup.findall('a', {'class': 'aa_loop_h2a'}) condition.


Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more