scala - How to compare every element in the RDD with every other element in the RDD ? -


i'm trying perform k nearest neighbor search using spark.

i have rdd[seq[double]] , i'm planing return rdd[(seq[double],seq[seq[double]])] actual row , list of neighbors

val out = data.map(row => {     val neighbours = data.top(num = 3)(new ordering[seq[double]] {       override def compare(a:seq[double],b:seq[double]) = {         euclideandistance(a,row).compare(euclideandistance(b,row))*(-1)       }     })   (row,neighbours.toseq) })

and gives following error on spark submit

15/04/29 21:15:39 warn tasksetmanager: lost task 0.0 in stage 1.0 (tid 2, 192.168.1.7): org.apache.spark.sparkexception: rdd transformations , actions can invoked driver, not inside of other transformations; example, rdd1.map(x => rdd2.values.count() * x) invalid because values transformation , count action cannot performed inside of rdd1.map transformation. more information, see spark-5063.

i understand nesting rdd not possible how perform such operations can compare every element in rdd every other element in rdd

something should it.

import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext._ import org.apache.spark.sparkconf import org.apache.spark.rdd.rdd  val conf = new sparkconf().setappname("spark-scratch").setmaster("local") val sco= new sparkcontext(conf)  // k number of nearest neighbors required  val k = 3  // generate 5 rows of two-dimensional coordinates val rows = list.fill(5)(list.fill(2)(math.random)) val datardd = sco.parallelize(rows, 1)  // no need sqrt we're comparing them def euclidean(a:list[double], b:list[double]) =   (a zip b) map {case (x:double, y:double) => (x-y)*(x-y)} sum  // pairs val pairs = datardd.cartesian(datardd)  // case class keep things bit neater // neighbor, , distance current point case class entry(neighbor: list[double], dist:double)  // map second element element , distance first val pairswithdist = pairs.map {case (x, y) => (x, entry(y, euclidean(x,y)))}  // merge row of pairswithdist resultrow point def mergeone(u: list[entry], v:entry) = (v::u).sortby{_.dist}.take(k)  // merge 2 results different partitions def mergelist(u: list[entry], v:list[entry]) = (u:::v).sortby{_.dist}.take(k)  val nearestneighbors = pairswithdist                       .aggregatebykey(list[entry]())(mergeone, mergelist)

Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more