Limit to the number of results from a Dataflow input file pattern glob? -


update:

we've seen these 400-class errors re:

com.google.api.client.googleapis.json.googlejsonresponseexception: 400 bad request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "request payload exceeds allowable limit: 50000.", "reason" : "badrequest" } ], "message" : "request payload exceeds allowable limit: 50000.", "status" : "invalid_argument" } @ com.google.api.client.googleapis.json.googlejsonresponseexception.from(googlejsonresponseexception.java:145) at

on glob resolves to:

total: 60 objects, 8405391 bytes (8.02 mib)

and have been experiencing increased variability of input globs hitting limit on past several days.

--

recently we've had observations of job failure when filepattern specs derive large numbers of files passed input dataflow jobs. examples of messages produced in these scenarios are:

apr 29, 2015, 9:22:51 (5dd3e79031bdcc45): com.google.api.client.googleapis.json.googlejsonresponseexception: 400 bad request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "request payload exceeds allowable limit: 50000.", "reason" : "badrequest" } ], "message" : "request payload exceeds allowable limit: 50000.", "status" : "invalid_argument" } @ com.google.api.client.googleapis.json.googlejsonresponseexception.from(googlejsonresponseexception.java:145) @ com.google.api.client.googleapis.services.json.abstractgooglejsonclientrequest.newexceptiononerror(abstractgooglejsonclientrequest.java:113) @ com.google.api.client.googleapis.services.json.abstractgooglejsonclientrequest.newexceptiononerror(abstractgooglejsonclientrequest.java:40) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest$1.interceptresponse(abstractgoogleclientrequest.java:321) @ com.google.api.client.http.httprequest.execute(httprequest.java:1049) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.executeunparsed(abstractgoogleclientrequest.java:419) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.executeunparsed(abstractgoogleclientrequest.java:352) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.execute(abstractgoogleclientrequest.java:469) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworkerharness$dataflowworkunitclient.reportworkitemstatus(dataflowworkerharness.java:273) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworker.reportstatus(dataflowworker.java:209) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworker.dowork(dataflowworker.java:157) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworker.getandperformwork(dataflowworker.java:95) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworkerharness$workerthread.call(dataflowworkerharness.java:139) @ com.google.cloud.dataflow.sdk.runners.worker.dataflowworkerharness$workerthread.call(dataflowworkerharness.java:124) @ java.util.concurrent.futuretask.run(futuretask.java:266) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745)  9:22:51  failed task going retried.

we've had success job parallelization in response this, wondering if there hard limit or quota somewhere being run into. retried tasks inevitably fail after maximum # of retries reached causing job fail.

thanks!

sal

the dataflow service has been updated handle larger requests of type, , should no longer produce issue.


Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more