python - how to transform pandas dataframe for insertion via executemany() statement? -


i have big pandas dataframe - 50 or headers , few hundred thousand rows of data - , i'm looking transfer data database using ceodbc module. using pyodbc , using simple execute statement in loop taking ridiculously long (1000 records per 10 minutes)...

i'm trying new module , trying introduce executemany() although i'm not quite sure what's meant sequence of parameters in:

    cursor.executemany("""insert table.name(a, b, c, d, e, f)  values(?, ?, ?, ?, ?), sequence_of_parameters)

should constant list working through each header like

    ['asdas', '1', '2014-12-01', 'true', 'asdasd', 'asdas', '2',  '2014-12-02', 'true', 'asfasd', 'asdfs', '3', '2014-12-03', 'false', 'asdasd']
  • where example of 3 rows

or format that's needed?

as related question, how can go converting regular pandas dataframe format?

thanks!

i managed figure out in end. if have pandas dataframe want write database using ceodbc module used, code is:

(with all_data dataframe) map dataframe values string , store each row tuple in list of tuples

for r in all_data.columns.values:     all_data[r] = all_data[r].map(str)     all_data[r] = all_data[r].map(str.strip)    tuples = [tuple(x) x in all_data.values]

for list of tuples, change null value signifiers - have been captured strings in conversion above - null type can passed end database. issue me, might not you.

string_list = ['nat', 'nan', 'nan', 'none']  def remove_wrong_nulls(x):     r in range(len(x)):         i,e in enumerate(tuples):             j,k in enumerate(e):                 if k == x[r]:                     temp=list(tuples[i])                     temp[j]=none                     tuples[i]=tuple(temp)  remove_wrong_nulls(string_list)

create connection database

cnxn=ceodbc.connect('driver={someodbcdriver};dbcname=xxxxxxxxxxx;uid=xxxxxxx;pwd=xxxxxxx;quietmode=yes;', autocommit=false) cursor = cnxn.cursor()

define function turn list of tuples new_list further indexing on list of tuples, chunks of 1000. necessary me pass data database sql query not exceed 1mb.

def chunks(l, n):     n = max(1, n)     return [l[i:i + n] in range(0, len(l), n)]  new_list = chunks(tuples, 1000)

define query.

query = """insert xxxxxxxxxxxx("xxxxxxxxxx", "xxxxxxxxx", "xxxxxxxxxxx") values(?,?,?)"""

run through the new_list containing list of tuples in groups of 1000 , perform executemany. follow committing , closing connection , that's :)

for in range(len(new_list)):     cursor.executemany(query, new_list[i]) cnxn.commit() cnxn.close()

Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more