java - Prevent Duplicate Daily Report Emails being sent from Google App Engine -


we have problem. our customers complaining getting duplicate emails in in-box. days 5 or 6 instances of exact same email @ exact same time. don't understand why. code has been re-written @ least once problem persists.

i'll try explain this... it's bit complicated :o(

every night (early morning) want send our users daily report containing usage stats. have cron job:

<cron>     <url>/redacted/report/url</url>     <description>send out daily reports active subscribers</description>     <schedule>every 2 hours</schedule> </cron>

the cron job hits servlet method:

protected void doget(httpservletrequest req, httpservletresponse resp) throws servletexception, ioexception {     accountfilter filter = accountfilter.forwebsafename(req.getparameter("filter"));     createtasks(filter, null); }

which calls createtasks method null cursor:

private void createtasks(accountfilter accountfilter, string cursor) {     try {         pagedresults<account> pagedaccounts = accountrepository.getaccounts(accountfilter.getfilter(), 50, cursor);         createtaskbatch(pagedaccounts);          // if there still more results in cursor, send cursor servlet's dopost method don't hit request time limit         if (pagedaccounts.getcursor() != null) {             getqueue(queue_name).add(withurl(worker_url).param(cursor_key, pagedaccounts.getcursor()).param(filter_key, accountfilter.getwebsafename()));         }     } catch(exception ex) {         logger.log(level.warning, "problem creating daily report task batch filter " + accountfilter.getwebsafename(), ex);     } }

which grabs 50 accounts , iterates on them creating new queued jobs emails should sent @ time. there code explcitely check last report sent timestamp , update timestamp before creating new queued task. should err on side of not sending report rather sending duplicates:

private void createtaskbatch(pagedresults<account> pagedaccounts) {     // gae datastore query might return duplicate results?!     list<account> list = pagedaccounts.getresults();     set<account> noduplicates = new hashset<>(list);     int dups = list.size() - noduplicates.size();     if ( dups > 0 ){         logger.warning ("accounts paged results contained " + dups + " duplicates!");     }     (account account : noduplicates) {         try {             if (lastreportsentover12hoursago(account)) {                 list<parent> parents = parentrepository.getverifiedparentsforaccount(account.getid());                 if (eitherparentsubscribed(parents)) {                     list<accountuser> users = accountuserrepository.listusers(account.getid());                     list<device> devices = getuserdevices(account, users);                     if (!devices.isempty()) {                         datetimezone tz = getmostcommontimezone(devices);                         if ( null == tz ){                             logger.warning("no timezone found account: " + account.getid() );                         }                         else{                             // send in morning report contains previous day's stats                             if (now(tz).gethourofday() < 7) {                                 // mark sent because queue might not processed while                                 // , next cursor set might contain of same accounts                                 accountrepository.markreportsent(account.getid(), now());                                 getqueue(queue_name).add(withurl(dailyreportservlet.worker_url).param(dailyreportservlet.account_id, account.getid()).param(dailyreportservlet.common_timezone, tz.getid()));                             }                         }                     }                 }             }         } catch(exception ex) {             logger.log(level.warning, "problem creating daily report task " + account.getid(), ex);         }     } }

the servlet post method takes care of handling follow pages of results via cursor method:

public void dopost(httpservletrequest req, httpservletresponse resp) throws ioexception {     accountfilter accountfilter = accountfilter.forwebsafename(req.getparameter(filter_key));     logger.log(level.info, "dopost hit task queue filter " + accountfilter.getwebsafename());     string cursor = req.getparameter(cursor_key);     createtasks(accountfilter, cursor); }

there servlet handles each report task , creates email contents , calls send on com.sendgrid.sendgrid class.

the eventual consistency in datastore seems candidate should resolved within few seconds , don't see how account both number of customers complaining , number of duplicates customers see.

help! ideas? being dumb somewhere?

updated

for clarity... email send task queue ends in method catch exceptions , reports them us. don't see exception duplicate cases:

private void sendreport(account account, datetimezone tz) throws ioexception, entitynotfoundexception {     try {             boolean sent = false;             map<string, object> root = buildemaildata(account, tz);             (parent parent : parentrepository.getverifiedparentsforaccount(account.getid())) {                 if (parent.getemailpreferences().issubscribedreports()) {                     emailbuilder.send(account, parent, root, "report", emailsender.notification);                     sent = true;                 }             }             if ( sent ){                 accountrepository.markreportsent(account.getid(), now());             }     } catch (exception ex) {         string message = "problem building report email account " + account.getid();         logger.log(level.warning, message, ex);;         new teamnotificationevent( message + " : exception: " + ex.getmessage()).fire();         throw new ioexception(message, ex);     } }

update 2 after adding debug logging

i see 2 posts in @ same time same task queue same cursor:

09:35:08.397 2015-04-30 200 0 b 3.78s /ws/notification/daily-report-task-creator 0.1.0.2 - - [30/apr/2015:01:35:08 -0700] "post /ws/notification/daily-report-task-creator http/1.1" 200 0 "http://screentimelabs.appspot.com/ws/notification/daily-report-task-creator" "appengine-google; (+http://code.google.com/appengine)" "screentimelabs.appspot.com" ms=3782 cpu_ms=662 queue_name=dailyreports task_name=8168414365365326983 instance=00c61b117c33a909790f0d1882657e04f40b2c7e app_engine_release=1.9.20 09:35:04.618 com.screentime.service.taskqueue.reports.dailyreporttaskcreatorservlet createtasks: createtasks called filter: active cursor: e-abaiico2oqc35zy3jlzw50aw1lbgfic3incxihqwnjb3vudciaamfybw8ua2fya2thaw5lbkbnbwfpbc5jb20miaiafa

09:35:08.432 2015-04-30 200 0 b 8.84s /ws/notification/daily-report-task-creator 0.1.0.2 - - [30/apr/2015:01:35:08 -0700] "post /ws/notification/daily-report-task-creator http/1.1" 200 0 "http://screentimelabs.appspot.com/ws/notification/daily-report-task-creator" "appengine-google; (+http://code.google.com/appengine)" "screentimelabs.appspot.com" ms=8837 cpu_ms=1348 queue_name=dailyreports task_name=50170612326424582061 instance=00c61b117c2bffe8de313e96fea8aeb813f4b20f app_engine_release=1.9.20 trace_id=7e5c0348382e66cf4e2c6ba400529fb7 09:34:59.608 com.screentime.service.taskqueue.reports.dailyreporttaskcreatorservlet createtasks: createtasks called filter: active cursor: e-abaiico2oqc35zy3jlzw50aw1lbgfic3incxihqwnjb3vudciaamfybw8ua2fya2thaw5lbkbnbwfpbc5jb20miaiafa

searching 1 particular account id see these requests:

09:35:08.397 2015-04-30 200 0 b 3.78s /ws/notification/daily-report-task-creator

09:35:08.432 2015-04-30 200 0 b 8.84s /ws/notification/daily-report-task-creator

09:35:08.443 2015-04-30 200 0 b 6.73s /ws/notification/daily-report-task-creator

09:35:10.541 2015-04-30 200 0 b 4.03s /ws/notification/daily-report-task-creator

09:35:10.690 2015-04-30 200 0 b 11.09s /ws/notification/daily-report-task-creator

09:35:13.678 2015-04-30 200 0 b 862ms /ws/notification/daily-report-worker

09:35:13.829 2015-04-30 500 0 b 1.21s /ws/notification/daily-report-worker

09:35:14.677 2015-04-30 200 0 b 1.56s /ws/notification/daily-report-worker

09:35:14.961 2015-04-30 200 0 b 346ms /ws/notification/daily-report-worker

some have repeated cursor values.

i make guess because dont see task queue code. not handling errors correctly in task queue. if task queue finishes error, gae re-queue it. if emails sent, task still run again. need way remember processed in task queue retry wont reprocess those.


Comments

Post a Comment

Popular posts from this blog

php - failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request -

this code i'm using find latitude , longitude of place: function coords($address){ echo $url="http://maps.googleapis.com/maps/api/geocode/json?address=".$address; $json = file_get_contents(urlencode($url)); $data = json_decode($json, true); print_r($data['results'][0]['geometry']['location']['lat']); print_r($data['results'][0]['geometry']['location']['lng']); } however returns warning: failed open stream: http request failed! http/1.0 400 bad request if use above code in simple procedure works fine not in function. note: have tried curl_init()... method produced same result? instead of encoding whole url encode address part. following code work fine $add='jamshoro phase 2'; $url="http://maps.googleapis.com/maps/api/geocode/json?address=".urlencode($add); $json = file_get_contents($url); $data = json_decode($json, true); print_r($data['r
Read more

java - How to filter a backspace keyboard input -

i have code gives message when key number pressed: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochararray().length - 1]; if (!(ch >= '0' && ch <= '9')) { system.out.println("the char entered not number"); t.consume(); } } }); now if press on wrong button , press backspace remove it, error message. how can add backspace input if statement? i have found answer: txtvalueofclause.addeventfilter(keyevent.key_typed, new eventhandler<keyevent>() { @override public void handle(keyevent t) { char ar[] = t.getcharacter().tochararray(); char ch = ar[t.getcharacter().tochar
Read more

java - Show Soft Keyboard when EditText Appears -

i have code in android app, alert dialog , keyboard, , when dialog shown want keyboard appear. not , not sure why? here code: final alertdialog.builder alert = new alertdialog.builder(this); alert.settitle("title"); alert.setmessage("message"); final edittext input = new edittext(this); input.setinputtype(inputtype.type_class_number); alert.setview(input); input.setonfocuschangelistener(new view.onfocuschangelistener() { @override public void onfocuschange(view v, boolean hasfocus) { if (hasfocus) { log.i(tag, "in focus"); inputmethodmanager inputmethodmanager = (inputmethodmanager) getsystemservice(context.input_method_service); inputmethodmanager.showsoftinput(input, inputmethodmanager.show_implicit); } } }); but keyboard not appear though edit text in focus, , tell appear why, doesn't come up? thanks in advance.
Read more