gensim - combining LSA/LSI with Naive Bayes for document classification -


i'm new gensim package , vector space models in general, , i'm unsure of what should lsa output.

to give brief overview of goal, i'd enhance naive bayes classifier using topic modeling improve classification of reviews (positive or negative). here's great paper i've been reading has shaped ideas left me still confused implementation..

i've got working code naive bayes--currently, i'm using unigram bag of words features , labels either positive or negative.

here's gensim code

from pprint import pprint # pretty printer import gensim gs  # tutorial sample documents docs = ["human machine interface lab abc computer applications",               "a survey of user opinion of computer system response time",               "the eps user interface management system",               "system , human system engineering testing of eps",               "relation of user perceived response time error measurement",               "the generation of random binary unordered trees",               "the intersection graph of paths in trees",               "graph minors iv widths of trees , quasi ordering",               "graph minors survey"]   # stoplist removal, tokenization stoplist = set('for of , in'.split()) # each document: lowercase document, split whitespace, , add words not in stoplist texts texts = [[word word in doc.lower().split() if word not in stoplist] doc in docs]   # create dict dict = gs.corpora.dictionary(texts) # create corpus corpus = [dict.doc2bow(text) text in texts]  # tf-idf tfidf = gs.models.tfidfmodel(corpus) corpus_tfidf = tfidf[corpus]  # latent semantic indexing 10 topics lsi = gs.models.lsimodel(corpus_tfidf, id2word=dict, num_topics =10)  in lsi.print_topics():     print

here's output

0.400*"system" + 0.318*"survey" + 0.290*"user" + 0.274*"eps" + 0.236*"management" + 0.236*"opinion" + 0.235*"response" + 0.235*"time" + 0.224*"interface" + 0.224*"computer" 0.421*"minors" + 0.420*"graph" + 0.293*"survey" + 0.239*"trees" + 0.226*"paths" + 0.226*"intersection" + -0.204*"system" + -0.196*"eps" + 0.189*"widths" + 0.189*"quasi" -0.318*"time" + -0.318*"response" + -0.261*"error" + -0.261*"measurement" + -0.261*"perceived" + -0.261*"relation" + 0.248*"eps" + -0.203*"opinion" + 0.195*"human" + 0.190*"testing" 0.416*"random" + 0.416*"binary" + 0.416*"generation" + 0.416*"unordered" + 0.256*"trees" + -0.225*"minors" + -0.177*"survey" + 0.161*"paths" + 0.161*"intersection" + 0.119*"error" -0.398*"abc" + -0.398*"lab" + -0.398*"machine" + -0.398*"applications" + -0.301*"computer" + 0.242*"system" + 0.237*"eps" + 0.180*"testing" + 0.180*"engineering" + 0.166*"management"

any suggestions or general comments appreciated.

just started working on same problem, svm instead, afaik after training model need this:

new_text = 'here document' text_bow = dict.doc2bow(new_text) vector = lsi[text_bow]

where vector topic distribution in document, length equal number of topics choose training, 10 in case. need represent documents topic distributions , feed them classification algorithm.

p.s. know it's kind of old question, keep seeing in google results every time searching )


Comments

Post a Comment

Popular posts from this blog

java - Spring Data JPA: Why findOne(id) executing delete query internally? -

i have 1 application using @manytomany annotation link 2 different entities 1 parent entity. not sure if have implemented properly. when execute .findone() on parent entity's repository find it, @ first fetch data along executing delete queries internally delete record mapping table (a table jpa creates after adding @manytomany annotation.). following generated sql queries , code snippet: java class @entity @table(name = "inventory") public class inventory { @id long id; // many other properties @manytomany(fetch=fetchtype.lazy) private list<stream> streams; @manytomany(fetch=fetchtype.lazy) private list<subject> subjects; } there no other annotation i've put in stream , subject entity class link inventory class. when run app, create following tables: inventory stream subject inventory_stream inventory_subject (inventory_stream , inventory_subject, 2 tables created automatically, don't k...
Read more

python - Mongodb How to add addtional information when aggregating? -

i beginner , wrote line pipeline works, wanna add other information output, screen name , or number of tweets.i tried add under $group gave me syntax error everytime here pipeline: def make_pipeline(): # complete aggregation pipeline pipeline = [ { '$match': { "user.statuses_count": {"$gt":99 }, "user.time_zone": "brasilia" } }, { "$group": { "_id": "$user.id", "followers": { "$max": "$user.followers_count" } } }, { "$sort": { "followers": -1 } }, { "$limit" : 1 } ]; i using on example : { "_id" : objectid("5304e2e3cc9e684aa98bef97"), "text" : "first week of school on :p", "in_reply_to_status_id" : null, "retweet_count" : null, ...
Read more

java - Incorrect order of records in M-M relationship in hibernate -

Image
i have 4 records in bridge table in mysql database. using @manytomany in hibernate, accurate order of employees incorrect incorrect of projects. following required material figure out doing mistake. this source. @entity @table(name="project") public class projectbean { @id @generatedvalue @column(name="project_id") private int projectid; @column(name="title") private string projecttitle; @manytomany(mappedby="projects") private collection<employeebean> employees; public projectbean() { employees = new arraylist<employeebean>(); } //getters & setters } @entity @table(name="employee") public class employeebean { @id @generatedvalue @column(name="employee_id") private int employee_id; @column(name="first_name") private string ...
Read more