r - Parse multiple XBRL files stored in a zip file -


i have downloaded multiple zip files website. each zip file contains multiple html , xml extension files (~ 100k in each).

it possible manually extract files , parse them. however, able within r (if possible)

example file (sorry bit big) using code previous question - download 1 zip file

library(xml)  pth <- "http://download.companieshouse.gov.uk/en_monthlyaccountsdata.html" doc <- htmlparse(pth)  myfiles <- doc["//a[contains(text(),'accounts_monthly_data')]", fun = xmlattrs][[1]] fileurls <- file.path("http://download.companieshouse.gov.uk", myfiles) [[1]]  dir.create("temp", "hmrccache") download.file(fileurls, destfile = file.path("temp", myfiles))

i can parse files using xbrl package if manually extract them. can done follows

library(xbrl)      inst <- file.path("temp", "prod224_0004_00000121_20130630.html") out <- xbrldoall(inst, cache.dir="temp/hmrccache", prefix.out=null, verbose=t)

i struggling how extract these files zip folder , parse each , say, in loop using r, without manually extracting them. tried making start, don't know how progress here. advice.

# names of files lst <- unzip(file.path("temp", myfiles), list=true) dim(lst) # 118626  # unzip  , extract first file nms <- lst$name[1] # prod224_0004_00000121_20130630.html lst2 <- unz(file.path("temp", myfiles), filename=nms)

i using windows 8.1

r version 3.1.2 (2014-10-31)

platform: x86_64-w64-mingw32/x64 (64-bit)

using suggestion karsten in comments, unzipped files temporary directory, , parsed each file. used snow package speed things up. 

  # parse 1 zip file start   fls <- list.files(temp)[[1]]    # unzip    tmp <- tempdir()   lst <- unzip(file.path(temp, fls), exdir=tmp)    # parse first 10 records   inst <- lst[1:10]    # start parse - in parallel   cl <- makecluster(parallel::detectcores())   clustercall(cl, function() library(xbrl))    # start   st <- sys.time()    out <- parlapply(cl, inst, function(i)                                    xbrldoall(i,                                              cache.dir="temp/hmrccache",                                              prefix.out=null, verbose=t) )    stopcluster(cl)    sys.time() - st

(i not sure using tempdir() correctly seems save large amounts of data local\temp directory - welcome comments if have approached incorrectly, thanks).


Comments

Post a Comment

Popular posts from this blog

java - Spring Data JPA: Why findOne(id) executing delete query internally? -

i have 1 application using @manytomany annotation link 2 different entities 1 parent entity. not sure if have implemented properly. when execute .findone() on parent entity's repository find it, @ first fetch data along executing delete queries internally delete record mapping table (a table jpa creates after adding @manytomany annotation.). following generated sql queries , code snippet: java class @entity @table(name = "inventory") public class inventory { @id long id; // many other properties @manytomany(fetch=fetchtype.lazy) private list<stream> streams; @manytomany(fetch=fetchtype.lazy) private list<subject> subjects; } there no other annotation i've put in stream , subject entity class link inventory class. when run app, create following tables: inventory stream subject inventory_stream inventory_subject (inventory_stream , inventory_subject, 2 tables created automatically, don't k...
Read more

python - Mongodb How to add addtional information when aggregating? -

i beginner , wrote line pipeline works, wanna add other information output, screen name , or number of tweets.i tried add under $group gave me syntax error everytime here pipeline: def make_pipeline(): # complete aggregation pipeline pipeline = [ { '$match': { "user.statuses_count": {"$gt":99 }, "user.time_zone": "brasilia" } }, { "$group": { "_id": "$user.id", "followers": { "$max": "$user.followers_count" } } }, { "$sort": { "followers": -1 } }, { "$limit" : 1 } ]; i using on example : { "_id" : objectid("5304e2e3cc9e684aa98bef97"), "text" : "first week of school on :p", "in_reply_to_status_id" : null, "retweet_count" : null, ...
Read more

java - Incorrect order of records in M-M relationship in hibernate -

Image
i have 4 records in bridge table in mysql database. using @manytomany in hibernate, accurate order of employees incorrect incorrect of projects. following required material figure out doing mistake. this source. @entity @table(name="project") public class projectbean { @id @generatedvalue @column(name="project_id") private int projectid; @column(name="title") private string projecttitle; @manytomany(mappedby="projects") private collection<employeebean> employees; public projectbean() { employees = new arraylist<employeebean>(); } //getters & setters } @entity @table(name="employee") public class employeebean { @id @generatedvalue @column(name="employee_id") private int employee_id; @column(name="first_name") private string ...
Read more