python - How can I assign a value to a null node in Xpath? -


this parts need crawl using scrapy xpath:

<tr class="o"><td>alabama</td><td><code>us.al</code></td><td><code>us01</code></td><td>ala.</td><td>-6~</td><td class="n">4,779,736</td><td class="n">133,916</td><td class="n">51,705</td><td>2</td><td>montgomery</td><td>alabamian</td><td>350-369</td></tr> <tr class="e"><td>alaska</td><td><code>us.ak</code></td><td><code>us02</code></td><td></td><td>-9~</td><td class="n">710,231</td><td class="n">1,530,700</td><td class="n">591,007</td><td>6</td><td>juneau</td><td>alaskan</td><td>995-999</td></tr>

my xpath expression is:

response.xpath('//tr[@class="o" or @class="e"][2]/descendant::*').extract()

but there null node in "alaska". <td> node after <code> "us02". not happen in alabama.

when use expression:

response.xpath('//tr[@class="o" or @class="e"][2]/descendant::*/text()').extract()

to extract text, null node ignored.

but have comply format. how can set null node space?

by way, have better solution crawl page in scrapy?

http://www.statoids.com/uus.html

i here explicit possible , data in "by column" fashion:

for state in response.xpath('//tr[@class="o" or @class="e"]'):     item = state()     item["hasc"] = state.xpath(".//td[2]/code/text()").extract()     ...     yield item

where state item class. note extract() return list. using item loader takefirst or join processor have string values inside item fields.


Comments

Post a Comment

Popular posts from this blog

java - Spring Data JPA: Why findOne(id) executing delete query internally? -

i have 1 application using @manytomany annotation link 2 different entities 1 parent entity. not sure if have implemented properly. when execute .findone() on parent entity's repository find it, @ first fetch data along executing delete queries internally delete record mapping table (a table jpa creates after adding @manytomany annotation.). following generated sql queries , code snippet: java class @entity @table(name = "inventory") public class inventory { @id long id; // many other properties @manytomany(fetch=fetchtype.lazy) private list<stream> streams; @manytomany(fetch=fetchtype.lazy) private list<subject> subjects; } there no other annotation i've put in stream , subject entity class link inventory class. when run app, create following tables: inventory stream subject inventory_stream inventory_subject (inventory_stream , inventory_subject, 2 tables created automatically, don't k...
Read more

python - Mongodb How to add addtional information when aggregating? -

i beginner , wrote line pipeline works, wanna add other information output, screen name , or number of tweets.i tried add under $group gave me syntax error everytime here pipeline: def make_pipeline(): # complete aggregation pipeline pipeline = [ { '$match': { "user.statuses_count": {"$gt":99 }, "user.time_zone": "brasilia" } }, { "$group": { "_id": "$user.id", "followers": { "$max": "$user.followers_count" } } }, { "$sort": { "followers": -1 } }, { "$limit" : 1 } ]; i using on example : { "_id" : objectid("5304e2e3cc9e684aa98bef97"), "text" : "first week of school on :p", "in_reply_to_status_id" : null, "retweet_count" : null, ...
Read more

java - Incorrect order of records in M-M relationship in hibernate -

Image
i have 4 records in bridge table in mysql database. using @manytomany in hibernate, accurate order of employees incorrect incorrect of projects. following required material figure out doing mistake. this source. @entity @table(name="project") public class projectbean { @id @generatedvalue @column(name="project_id") private int projectid; @column(name="title") private string projecttitle; @manytomany(mappedby="projects") private collection<employeebean> employees; public projectbean() { employees = new arraylist<employeebean>(); } //getters & setters } @entity @table(name="employee") public class employeebean { @id @generatedvalue @column(name="employee_id") private int employee_id; @column(name="first_name") private string ...
Read more