yarn - Apache Spark: setting executor instances does not change the executors -


i have apache spark application running on yarn cluster (spark has 3 nodes on cluster) on cluster mode.

when application running spark-ui shows 2 executors (each running on different node) , driver running on third node. want application use more executors tried adding argument --num-executors spark-submit , set 6.

spark-submit --driver-memory 3g --num-executors 6 --class main.application --executor-memory 11g --master yarn-cluster myjar.jar <arg1> <arg2> <arg3> ...

however, number of executors remains 2.

on spark ui can see parameter spark.executor.instances 6, intended, , somehow there still 2 executors.

i tried setting parameter code 

sparkconf.set("spark.executor.instances", "6")

again, can see parameter set 6, still there 2 executors.

does know why couldn't increase number of executors?

yarn.nodemanager.resource.memory-mb 12g in yarn-site.xml

increase yarn.nodemanager.resource.memory-mb in yarn-site.xml

with 12g per node can launch driver(3g) , 2 executors(11g).

node1 - driver 3g (+7% overhead)

node2 - executor1 11g (+7% overhead)

node3 - executor2 11g (+7% overhead)

now requesting executor3 of 11g , no node has 11g memory available.

for 7% overhead refer spark.yarn.executor.memoryoverhead , spark.yarn.driver.memoryoverhead in https://spark.apache.org/docs/1.2.0/running-on-yarn.html


Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more