Restoring cassandra from snapshot -


so did of test run/disaster recovery practice deleting table , restoring in cassandra via snapshot on test cluster have built.

this test cluster has 4 nodes, , used node restart method after truncating table in question, nodes shutdown, commitlog directories cleared, , current snapshot data copied table directory each node. afterwards, brought each node up. following documentation ran repair on each node, followed refresh on each node.

my question is, why necessary me run repair on each node afterwards assuming none of nodes down except when shut them down perform restore procedure? (in test instance small amount of data , took little time repair, if happened in our production environment repairs take 12 hours perform huge issue in disaster scenario).

and assume running repair unnecessary on single node instance, correct?

just trying figure out purpose of running repair , subsequent refresh is.

what repair?

repair 1 of cassandra's main anti-entropy mechanisms. ensures nodes have latest version of data. reason takes 12 hours (this normal way) is expensive operation -- io , cpu intensive -- generate merkel trees data, compare them merkel trees other nodes, , stream missing / outdated data.

why run repair after restoring snapshots

repair gives consistency baseline. example: if snapshots weren't taken @ exact same time, have chance of reading stale data if you're using cl 1 , hit replica restored older snapshot. repair ensures replicas date latest data available.

tl;dr:

repairs take 12 hours perform huge issue in disaster scenario).

while repair running, you'll have risk of reading stale data if snapshots don't have same exact data. if old snapshots, gc_grace may have passed tombstones giving higher risk of zombie data if tombstones aren't propagated across cluster.

related side rant - when run repair?

the coloquial definition of term repair seems imply system broken. think "i have run repair? must have done wrong un-repaired state!" not true. repair normal maintenance operation cassandra. in fact, should running repair @ least every gc_grace seconds ensure data consistency , avoid zombie data (or use opscenter repair service).

in opinion, should have called antientropymaintenence or cassandraoilchange or rather repair : )


Comments

Post a Comment

Popular posts from this blog

java - Spring Data JPA: Why findOne(id) executing delete query internally? -

i have 1 application using @manytomany annotation link 2 different entities 1 parent entity. not sure if have implemented properly. when execute .findone() on parent entity's repository find it, @ first fetch data along executing delete queries internally delete record mapping table (a table jpa creates after adding @manytomany annotation.). following generated sql queries , code snippet: java class @entity @table(name = "inventory") public class inventory { @id long id; // many other properties @manytomany(fetch=fetchtype.lazy) private list<stream> streams; @manytomany(fetch=fetchtype.lazy) private list<subject> subjects; } there no other annotation i've put in stream , subject entity class link inventory class. when run app, create following tables: inventory stream subject inventory_stream inventory_subject (inventory_stream , inventory_subject, 2 tables created automatically, don't k...
Read more

python - Mongodb How to add addtional information when aggregating? -

i beginner , wrote line pipeline works, wanna add other information output, screen name , or number of tweets.i tried add under $group gave me syntax error everytime here pipeline: def make_pipeline(): # complete aggregation pipeline pipeline = [ { '$match': { "user.statuses_count": {"$gt":99 }, "user.time_zone": "brasilia" } }, { "$group": { "_id": "$user.id", "followers": { "$max": "$user.followers_count" } } }, { "$sort": { "followers": -1 } }, { "$limit" : 1 } ]; i using on example : { "_id" : objectid("5304e2e3cc9e684aa98bef97"), "text" : "first week of school on :p", "in_reply_to_status_id" : null, "retweet_count" : null, ...
Read more

java - Incorrect order of records in M-M relationship in hibernate -

Image
i have 4 records in bridge table in mysql database. using @manytomany in hibernate, accurate order of employees incorrect incorrect of projects. following required material figure out doing mistake. this source. @entity @table(name="project") public class projectbean { @id @generatedvalue @column(name="project_id") private int projectid; @column(name="title") private string projecttitle; @manytomany(mappedby="projects") private collection<employeebean> employees; public projectbean() { employees = new arraylist<employeebean>(); } //getters & setters } @entity @table(name="employee") public class employeebean { @id @generatedvalue @column(name="employee_id") private int employee_id; @column(name="first_name") private string ...
Read more