tag:blogger.com,1999:blog-7682331042447026332024-03-14T04:53:24.275-07:00The Scale-Out BlogCreating robust applications using open source databases and commodity hardwareRobert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.comBlogger121125tag:blogger.com,1999:blog-768233104244702633.post-78899175693549732792014-10-29T08:00:00.000-07:002014-10-29T08:07:00.653-07:00An Ending and a Beginning: VMware Has Acquired Continuent<div class="MsoNormal">
As of today, Continuent is part of VMware. We are absolutely
over the moon about it.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
You can read more about the news on the <a href="http://blogs.vmware.com/vcloud/2014/10/vmware-acquires-continuent.html">VMware vCloud blog</a> by
Ajay Patel, our new boss. There’s also an official post on our <a href="http://continuent-tungsten.blogspot.com/2014/10/vmware-acquires-continuent.html">Continuent company blog</a><span style="font-size: 11px;">.</span> In a nutshell the Continuent team is joining the <span style="mso-fareast-font-family: "Times New Roman";">VMware Cloud Services Division</span>. We will continue to
improve, sell, and support our Tungsten products and work on innovative
integration into VMware’s product line. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
So why do I feel exhilarated about joining VMware? There are three reasons. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoListParagraphCxSpFirst" style="margin-bottom: 6.0pt; margin-left: .25in; margin-right: 0in; margin-top: 0in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="mso-fareast-font-family: "Times New Roman";"><span style="mso-list: Ignore;">1.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Continuent is joining a world-class company that
is the leader in virtualization and cloud infrastructure solutions. Even
better, VMware understands the value of data to businesses.<span style="mso-spacerun: yes;"> </span>They share our vision of managing an
integrated fabric of standard DBMS platforms, both in public clouds as well as in
local data centers. It is a great home to advance our work for many years to
come. <o:p></o:p><br />
<br /></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-bottom: 6.0pt; margin-left: .25in; margin-right: 0in; margin-top: 0in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="mso-fareast-font-family: "Times New Roman";"><span style="mso-list: Ignore;">2.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->We can continue to support our existing users
and make Tungsten even better. I know many of you have made big decisions to
adopt Continuent technology that would affect your careers if they turned out
badly. We now have more resources and a mandate to grow our product line. We
will be able to uphold our commitments to you and your businesses. <o:p></o:p><br />
<br /></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: .25in; mso-add-space: auto; mso-list: l0 level1 lfo1; text-indent: -.25in;">
<!--[if !supportLists]--><span style="mso-fareast-font-family: "Times New Roman";"><span style="mso-list: Ignore;">3.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->It’s
a great outcome for our team, which has worked for many years to make
Continuent Tungsten technology successful. This includes our investors at Aura in
Helsinki, who have been dogged in their support throughout our journey. <o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Speaking of the Continuent team…I am so proud of what all of
you have achieved.<span style="mso-spacerun: yes;"> </span>Today we are starting
a new chapter in our work together. See you at VMware!</div>
<div style="mso-element: comment-list;">
<div style="mso-element: comment;">
<div class="msocomtxt" id="_com_2" language="JavaScript">
<!--[if !supportAnnotations]--></div>
<!--[endif]--></div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com7tag:blogger.com,1999:blog-768233104244702633.post-57646335236799319922014-10-06T08:37:00.000-07:002014-10-06T09:58:10.477-07:00Exorcising the CAP DemonComputer science is like an enormous tool box you can rummage through whenever you have a problem to solve. Most of the tools are sturdy and practical, like algorithms for B-trees. Some are also elegant, like consistent hashing in Dynamo. Finally there are some tools that you never quite figure out even after years of reflection. That piece of steel you are looking at could be Excalibur. Or it could be a rusty knife.<br />
<br />
The <a href="http://en.wikipedia.org/wiki/CAP_theorem">CAP theorem</a> falls into the last category, at least for me. It was a major topic in the blogosphere a few years ago and Google Trends shows <a href="https://www.google.com/trends/explore#q=cap%20theorem&cmpt=q">steadily increasing interest in the term</a> since 2010. It's not my goal to explain CAP fully--a good informal description is <a href="http://ksat.me/a-plain-english-introduction-to-cap-theorem/">here</a> or you can just read the proof yourself. Instead I would like to talk about how I understand and use the CAP theorem today as well as how that understanding might evolve in the future.<br />
<br />
In a nutshell CAP puts a limit on how distributed database systems trade off data consistency and system availability. Eric Brewer originated the theorem as a conjecture in the late 1990s. Seth Gilbert and Nancy Lynch supplied a <a href="http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf">proof of the conjecture</a> in 2002. Brewer <a href="http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed">described it as follows in 2012</a>:<br />
<blockquote class="tr_bq">
<i>The CAP theorem states that any networked shared-data system can have at most two of three desirable properties:</i><br />
<ul>
<li><i>consistency (C) equivalent to having a single up-to-date copy of the data;</i></li>
</ul>
<ul>
<li><i>high availability (A) of that data (for updates); and</i></li>
</ul>
<ul>
<li><i>tolerance to network partitions (P).</i></li>
</ul>
</blockquote>
<blockquote class="tr_bq">
<ul style="background-color: white; border: 0px; clear: left; line-height: 21px; list-style-image: initial; list-style-position: initial; margin: 0px 0px 10px 10px; padding: 0px;">
</ul>
</blockquote>
My initial problem in understanding CAP was relating the proof to what happens in the real world, which is not especially easy. Network partitions are an example. Here's how the Gilbert/Lynch proof defines them in Section 2.3.<br />
<blockquote class="tr_bq">
<i>When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost. (And any pattern of message loss can be modeled as a temporary partition separating the communicating nodes at the exact instant the message is lost.)</i></blockquote>
<div>
So does this include an asymmetric communication failure? That's where a process on one host can see and send messages to a process on another host but the reverse is not true. This happens all the time in group communications for reasons that range from application software bugs to bad cabling and everything in between. Do you model the asymmetry as a sequence of temporary partitions? It's of course possible. But it feels a bit like using <a href="http://en.wikipedia.org/wiki/Deferent_and_epicycle">Ptolemaic astronomy with epicycles</a>. </div>
<div>
<br /></div>
<div>
Other people have made similar observations. Eric Brewer even wrote about the "nuances" of partitions in his 2012 retrospective. There are analogous problems with the other terms. There was enough public disagreement their meaning that I wrote a <a href="http://scale-out-blog.blogspot.com/2012/04/disproving-cap-theorem.html">"disproof" of CAP</a> a few years back as an April Fools Day joke. It depended on not being able to distinguish CA and CP choices in real systems. </div>
<div>
<br /></div>
<div>
That confusion is not a problem with the CAP theorem itself. Nobody has seriously challenged the proof. Instead, it's a matter of what logicians refer to as <a href="http://en.wikipedia.org/wiki/Interpretation_(logic)">interpretation</a>, which links a logical model to some domain of discourse so that you can draw valid conclusions about that domain. If you want to reason about real-world systems using the CAP theorem you must first ensure your systems really match the model. Otherwise it's like using a map of Oregon to drive between New York and Boston. The core difficulty is that the CAP theorem proof assumes binary properties whereas in reality properties like availability operate on a sliding scale. </div>
<div>
<br />
My other issue with CAP evaluation is what you might call a suitability problem. There are a lot of issues with operating distributed systems, and the 3-way trade-off is irrelevant to many of them. For instance, what happens when the network is behaving and you don't have to make pesky choices between availability and consistency? Let's look at some examples. </div>
<div>
<br /></div>
<div>
CAP defines consistency as <a href="http://en.wikipedia.org/wiki/Linearizability#Linearizability_versus_serializability">linearizability</a>, which means that transactions on different replicas look as if they all happened at once in a single place in a single unbroken series. Imagine driving around to different automated teller machines at a bank and making changes to your account balance or checking it. No matter which teller machine you visit next, it knows exactly what happened before and has the right balance amount. Or imagine a shopping cart on a website like Zappos.com. No matter how you jump around the website to select clothing or even if you fold up your laptop and fly to Paris, the items in your shopping care remain consistent without duplicate or missing selections. </div>
<div>
<br /></div>
<div>
You might say, well, not all systems work that way. You would be right, and that's the exactly the point. Real distributed systems do not always try to ensure linearizability. It turns out that many people, most particularly end users who ultimately pay for computer systems, conclude they don't really care so much about consistency of the sort CAP promises. Here are two different types of reasons: </div>
<div>
<br /></div>
<div>
<b>1. Linearized consistency is expensive.</b> Keeping active replicas up to date requires round trip messages between hosts, which can reduce transaction commit times by an order of magnitude or more. Users are allergic to slow response, regardless of any other benefits that slowness might bring. Daniel Abadi pointed out this latency problem some time ago in <a href="http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html">a great blog post on CAP</a> that is still excellent reading today. </div>
<div>
<br /></div>
<div>
<b>2. Linearized consistency is irrelevant for many applications. </b>Consider a measurement from a household thermometer or a text message from a cell phone. There is only one of each generated in a single location. Your servers either get them or they don't. Multiple copies are just that: replicas of the same thing. Conflicts don't exist. </div>
<div>
<br />
The share of immutable data from analytic systems like Hadoop and object stores like Amazon S3 is increasing rapidly, which means that there is an increasing number of applications for which CAP is not the only or even a major design consideration. It might be in the guts of the system but it's just one of many problems at that level and there may be multiple choices. The <a href="http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html">original Hadoop architecture</a> actually ignored CAP for one critical part of the system--the NameNode, which maps HDFS file names to storage, was a single point of failure.<br />
<br /></div>
<div>
Which brings us back to understanding CAP at a practical level. Is it Excalibur or just the rusty knife? At this point it feels like another tool in the toolbox that you use at the right time, albeit carefully. Imagine a band saw that does not have a very good guard on the blade. Here are my personal instructions for safe use. </div>
<div>
<br /></div>
<div>
<b>1. Use it for suitable problems</b>. The CAP theorem applies to a very specific problem involving systems that want to remain consistent and available across multiple networked hosts. If you design clusters or distributed databases, this is a relatively big deal. The trade-offs are real and you have to think about them. </div>
<div>
<br /></div>
<div>
For instance at <a href="http://www.continuent.com/">Continuent</a> we have some problems where the theorem is directly applicable. We build clusters that implement failover. We have to consider how to establish consensus while keeping the cluster available even when members lose messages or respond slowly. The CAP theorem guides you to manage this kind of problem rather than try to solve it using techniques that will not work, such as adding timeouts on messages. (Continuent Tungsten clusters are generally CP, in case you are wondering.) </div>
<div>
<br /></div>
<div>
<b>2. Avoid CAP where it does not obviously apply</b>. It is a tricky theorem to interpret correctly, and many applications are concerned with unrelated problems. I work a lot on transactional replication. There are no CAP issues in Tungsten Replicator. At the other end of the spectrum if you build systems that link multiple stores using replication, you likely have multiple CAP choices under the covers. That's a common pattern in complex applications. </div>
<div>
<br /></div>
<div>
It is therefore important to look with a jaundiced eye upon any product that claims to "beat CAP," like <a href="http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html">this widely read article</a>. This is just marketing hype. If your application matches the CAP theorem model, it applies and you are subject to the limitations. If the limitations don't seem to make sense you have not evaded them. You are either working on a problem to which CAP is not relevant or you made implicit CAP choices of which you are not aware. It is easy to make a fool of yourself by asserting otherwise. </div>
<div>
<br /></div>
<div>
<b>3. Other tools are important too.</b> CAP of course does not even cover all trade-offs in clusters. There are also many issues to consider when building distributed data systems that actually work. Latency, durability of data, monitoring, automation, reliability, ability to do zero-downtime maintenance, and security are critical. <u><i>Especially</i></u> security. That looks like the next big problem for a lot of existing distributed systems. </div>
<div>
<br />
Beyond these, don't stop thinking about CAP. It is one of those ideas that gets under your skin and really bugs you. In addition to Eric Brewer's 2012 article, Seth Gilbert and Nancy Lynch wrote a <a href="http://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf">follow-up perspective on the implications of CAP</a>, so even the originators are continuing to consider the problems. The long term value of CAP is that it has focused attention on a set of difficult data management problems and led to numerous productive ideas about how to manage them. The resulting evolution is not nearly finished. We will all continue to worry this bone for many years to come. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-22373228430775581182014-02-20T12:25:00.001-08:002014-02-20T12:25:33.341-08:00No Hadoop Fun for Me at SCaLE 12X :(I blogged a couple of weeks ago about <a href="http://scale-out-blog.blogspot.com/2014/02/fun-with-mysql-and-hadoop-at-scale-12x.html">my upcoming MySQL/Hadoop talk at SCaLE 12X</a>. Unfortunately I had to cancel. A few days after writing the article I came down with an eye problem that is fixed but prevents me from flying anywhere for a few weeks. That's a pity as I was definitely looking forward to attending the conference and explaining how Tungsten replicates transactions from MySQL into HDFS.<br />
<br />
Meanwhile, we are still moving at full steam with Hadoop-related work at Continuent, which is the basis for the next major replication release, <a href="https://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> 3.0.0. Binary builds and documentation will go up in a few days. There will also be many more public talks about Hadoop support, starting in April at <a href="http://www.percona.com/live/mysql-conference-2014">Percona Live 2014</a>. I hope you'll consider attending one of our talks there. It's a great conference.<br />
<br />
Since my SCaLE 12X talk won't be happening I would like to repeat the invitation to attend the <a href="http://www.continuent.com/news/live-webinars/1389-real-time-data-loading-from-mysql-to-hadoop">Continuent webinar on loading from MySQL to Hadoop</a> on Thursday February 27th. It's essentially the same talk, but no airplanes are involved.Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com1tag:blogger.com,1999:blog-768233104244702633.post-62185369839836603982014-02-17T20:00:00.000-08:002014-03-18T08:59:49.269-07:00Why Aren't All Data Immutable?Over the last few years there has been an increasing interest in immutable data management. This is a big change from the traditional update-in-place approach many database systems use today, where new values delete old values, which are then lost. With immutable data you record everything, generally using methods that append data from successive transactions rather than replacing them. In some DBMS types you can access the older values, while in others the system transparently uses the old values to solve useful problems like implementing eventual consistency.<br />
<br />
Baron Schwartz <a href="http://www.xaprb.com/blog/">recently pointed out</a> that it can be hard to get decent transaction processing performance based on append-only methods like <a href="http://guide.couchdb.org/draft/btree.html">append-only B-trees</a>. This is not a very strong argument against immutable data per se. Immutable data are already in wide use. It is actually surprising they have not made deeper inroads into online transaction processing, which is widely handled by relational DBMS servers like MySQL and Oracle. <br />
<br />
<b>Immutable Data Are Now Economically Feasible</b><br />
<br />
One reason for the popularity of update-in-place approaches is simple: storage used to be really expensive. This is no longer the case. Many applications can now afford to store the entire DBMS transaction log almost indefinitely. To illustrate, look at storage costs in Amazon Web Services. Applications running in Amazon have API-level access to practically unlimited replicated, long-term storage through services like <a href="http://aws.amazon.com/s3/">S3</a> and <a href="http://aws.amazon.com/glacier/">Glacier</a>. Amazon conveniently publishes prices that serve as good proxies for storage costs in general. Using these numbers, I worked up a <a href="https://docs.google.com/spreadsheet/ccc?key=0AtfIlRf7_aA6dFRfYWhkUGp6VEJJMWdnT3RDeUJLTEE&usp=sharing">simple spread sheet</a> that shows the cost of storing 7 years of transactions for a made-up business application. <br />
<br />
To start with, assume our sample app generates one thousand transactions per second at 1,000 bytes per transaction. This is not exceedingly busy by some standards but is relatively high for business systems that handle human-generated transactions. The main place you see numbers approaching this level is SaaS businesses that handle many customers on a single system. Our sample system generates about 205,591 gigabytes of data over seven years. <br />
<br />
<table cellpadding="0" cellspacing="0" dir="ltr" style="font-family: arial,sans,sans-serif; font-size: 13px; table-layout: fixed;"><colgroup><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col></colgroup><tbody>
<tr style="height: 17px;"><td style="border-bottom: 1px solid #000000; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Xacts/Sec</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Bytes/Xact</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Bytes/Sec</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">GB Generated in 1 Hour</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">GB Generated in 1 Day</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">GB Generated in 1 Month</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">GB Generated in 1 Year</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #000000; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">GB Generated in 7 Years</td></tr>
<tr style="height: 17px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,000</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,000</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">1,000,000</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">3.35</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">80.47</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">2,447.52</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">29,370.19</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: null; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">205,591.32</td></tr>
</tbody></table>
<br />
Amazon storage costs vary from $0.011/Gb/month for Glacier to $0.09/Gb/month for S3 with full redundancy. (These are numbers for the US-West region as of 29 December 2013.) Annual storage costs for 7 years of data are pretty hefty if you store uncompressed data. However, if you factor in compression--for example MySQL binlogs tend to compress around 90% in my experience--things start to look a lot better.<br />
<br />
<table cellpadding="0" cellspacing="0" dir="ltr" style="font-family: arial,sans,sans-serif; font-size: 13px; table-layout: fixed;"><colgroup><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col><col width="120"></col></colgroup><tbody>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td colspan="4" rowspan="1" style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Annual cost to store 7 years of data at different levels of compression</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #000000; border-left: 1px solid #ccc; border-right: 1px solid #ccc; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">0%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">20%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">40%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">60%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">70%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">80%</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: null; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">90%</td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Glacier</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$27,138.05</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$21,710.44</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$16,282.83</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$10,855.22</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$8,141.42</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$5,427.61</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$2,713.81</td></tr>
<tr style="height: 16px;"><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; direction: ltr; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: left; vertical-align: bottom;">S3 Reduce Redundancy</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$177,630.90</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$142,104.72</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$106,578.54</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$71,052.36</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$53,289.27</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$35,526.18</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$17,763.09</td></tr>
<tr style="height: 16px;"><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; direction: ltr; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: left; vertical-align: bottom;">S3 Standard</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$222,038.63</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$177,630.90</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$133,223.18</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$88,815.45</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$66,611.59</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$44,407.73</td><td style="border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-bottom-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: solid; border-right-width: 1px; padding-bottom: 0px; padding-left: 3px; padding-right: 3px; padding-top: 0px; text-align: right; vertical-align: bottom;">$22,203.86</td></tr>
</tbody></table>
<br />
The raw costs still look hefty to the untrained eye, but we need to factor in the <i><u>real</u></i> expense of operating this type of system. Here's a typical cost structure for a 3 node cluster (to ensure HA) with labor costs factored in and preserving 7 years of data. I have put in generously small IT overhead costs including software development, since the code has to come from somewhere. Under these assumptions long-term storage costs are less 10% of the yearly cost of operation. <br />
<br />
<table cellpadding="0" cellspacing="0" dir="ltr" style="font-family: arial,sans,sans-serif; font-size: 13px; table-layout: fixed;"><colgroup><col width="160"></col><col width="145"></col><col width="120"></col><col width="120"></col><col width="120"></col></colgroup><tbody>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #000000; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 1px solid #ccc; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Component</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #ccc; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">Cost</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid transparent; border-top: 1px solid #ccc; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">Percentage</td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; border-top: 1px solid #ccc; color: black; direction: ltr; font-weight: bold; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Notes</td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">3 i2.4xlarge instances</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">$46,306.68</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">20.09%</td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">(Heavy utilization reserved, 1 yr. term)</td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">3 support licenses</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">$15,000.00</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">6.51%</td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">(Support subscription costs * 3x) </td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">Raw dbadmin labor</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">$12,000.00</td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid transparent; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">5.21%</td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">(1 FTE/30 DBMS servers @ 120K per)</td></tr>
<tr style="height: 16px;"><td style="background-color: white; border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: left; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">Software dev/QA</td><td style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: right; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">$120,000.00</td><td style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid transparent; color: black; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: right; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">52.06%</td><td colspan="2" rowspan="1" style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; direction: ltr; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">(10 FTE/30 DBMS servers @ 120K per)</td></tr>
<tr style="height: 16px;"><td style="background-color: white; border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: left; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">Misc. overhead costs</td><td style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: right; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">$15,000.00</td><td style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid transparent; color: black; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: right; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">6.51%</td><td colspan="2" rowspan="1" style="background-color: white; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; font-family: arial,sans,sans-serif; font-size: 100.0%; font-style: normal; font-weight: normal; overflow: hidden; padding: 0px 3px 0px 3px; text-align: left; text-decoration: none; vertical-align: bottom; vertical-align: bottom; white-space: normal;">($5K per server)</td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #000000; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">S3 Storage</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">$22,203.86</td><td style="border-bottom: 1px solid #000000; border-right: 1px solid transparent; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;">9.63%</td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #000000; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;">(7 years of data, 90% compression)</td></tr>
<tr style="height: 16px;"><td style="border-bottom: 1px solid #ccc; border-left: 1px solid #ccc; border-right: 1px solid #ccc; color: black; direction: ltr; padding: 0px 3px 0px 3px; text-align: left; vertical-align: bottom;"><b>Total</b></td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;"><b>$230,510.54</b></td><td style="border-bottom: 1px solid #ccc; border-right: 1px solid transparent; color: black; padding: 0px 3px 0px 3px; text-align: right; vertical-align: bottom;"><b>100.00%</b></td><td colspan="2" rowspan="1" style="border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; padding: 0px 3px 0px 3px; vertical-align: bottom;"></td></tr>
</tbody></table>
<br />
Long storage costs for base transaction data can be far lower if any of the following hold:<br />
<ul>
<li>You generate fewer transactions per second or they are smaller. Many business apps produce far fewer transactions than my example. </li>
<li>You don't keep data for the full 7 years. Some of the analytic users I work with just keep a couple of years. </li>
<li>You are already paying archiving costs for backups, in which case the additional storage cost becomes a wash if you can stop using a separate backup system.</li>
<li>You add more external costs to the picture--running a real business that generates this level of transactions often takes far more people than are shown in my projection. </li>
</ul>
In these cases long term storage costs could be in the 1-2% range as a percentage of IT operating costs. Over time storage costs will decrease--<a href="http://blog.dshr.org/2012/10/storage-will-be-lot-less-free-than-it.html">though the rate of decline is hard to predict</a>--so each year the number systems able to afford preservation of complete transaction histories will corresponding increase. This is particularly true for business transactions, which tend to be human generated and subject to upper growth limits once businesses are fully automated. If you push data into Glacier, economically feasible retention periods can run to decades. This is far longer than most businesses (or more particularly their lawyers) even want to keep information around. <br />
<br />
There are still reasons for wanting an update-in-place model for OLTP systems, for example to keep as much of your working set as possible in RAM or on fast SSDs to keep response time low. But storage cost alone is no longer a major factor for a wide range of applications. This development is already affecting data management technology profoundly. Doug Cutting <a href="http://www.youtube.com/watch?feature=player_embedded&v=_WwuZI6AhN8">has pointed out</a> on numerous occasions that the downward cost trajectory of commodity storage was a key driver in the development of <a href="http://hadoop.apache.org/">Hadoop</a>. <br />
<br />
<b>Users Want Immutable Data</b><br />
<br />
Many organizations already keep long transaction histories to feed analytics by loading them into traditional data warehouses based on Teradata, Vertica, and the like. As soon as a practical method appeared to keep such data more economically, businesses began to adopt it quickly. That "method" is Hadoop. <br />
<br />
Hadoop has a fundamentally different approach to data management from relational and even many NoSQL systems. For one thing, immutable data are fundamental. The default processing model is that you write data but rarely change it once written. To illustrate, the <a href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual">HiveQL SQL dialect</a> does not even have UPDATE or DELETE statements. Instead, you overwrite entire tables or parts of them to make changes. This works because Hadoop organizes storage on cheap commodity hardware (HDFS) and provides a workable way to access data programmatically (MapReduce). <br />
<br />
Hadoop changes the data management cost model in other ways besides utilizing commodity hardware efficiently. With Hadoop you don't necessary define *any* data structures up front. Instead, you store transactions in native form and write programs to interpret them later on. If you need structure for efficient queries you add it through MapReduce and perhaps store it as a materialized view to make other queries more efficient. Hadoop eliminates a lot of the up-front effort (and risk) required to get transactions into a data warehouse. Instead, it defers those costs until you actually need to run specific analytics. Moreover by storing native transaction formats, you can answer new questions years later. That is a very powerful benefit. <br />
<br />
I have been working a lot with Hadoop over the last few months. It's a bear to use because it consists of a set of loosely integrated and rapidly evolving projects with weak documentation and lots of bugs. Even with these difficulties, the rising level of Hadoop adoption for analytics shows the underlying model has legs and that users want it. As Floyd Strimling pointed out a while ago on Twitter <a href="https://twitter.com/PlatenReport/status/408636716056969217">this genie is not going back in the bottle</a>. HDFS is becoming the default storage mechanism for vast quantities of data. <br />
<br />
<b>Immutable Data Management Looks Like a Good Bet</b><br />
<br />
One of the basic problems in discussing immutable data management is that there are different kinds of immutable data that persist at different timescales. Baron has a point that Couchbase, Datanomic, NuoDB, or whatever new DBMS implementation you choose are in some ways recapitulating solutions that existing RDBMS implementations reached long ago. But I also think that's not necessarily the right comparison when talking about immutable data, especially when you start to think about long retentions.<br />
<br />
The fact is that Oracle, MySQL, PostgreSQL, and the like do not utilize distributed commodity storage effectively and they certainly do not enable storage of the long tail transaction histories that many businesses clearly want for analytics. The best way to do that is to replicate transactions into HDFS and work on them there. That is hard even for MySQL, which has flexible and economical replication options. (We are working on making it easier to do at Continuent but that's another article. :) <br />
<br />
In my opinion a more useful criticism of the arriviste competitors of traditional OLTP systems is that they don't go <i><u>far enough</u></i> with immutable data and risk being outflanked by real-time transaction handling built on top of HDFS. Hadoop real-time work on projects like <a href="https://spark.incubator.apache.org/">Apache Spark</a> is for the time being is focused on analytics but OLTP support cannot be far behind. Moreover, there is a window to build competitors to HDFS that gets smaller as Hadoop becomes more entrenched. This seems more interesting than building stores that offer only incremental improvements over existing RDBMS implementations.<br />
<br />
Immutable data now permeate IT due to decreasing storage costs coupled with requirements for analytic processing. It's like the famous quote from <a href="http://en.wikiquote.org/wiki/William_Gibson">William Gibson</a>:<br />
<blockquote class="tr_bq">
The future is already here--it's just not very evenly distributed.</blockquote>
If you look at the big picture the arguments for database management based on immutable data seem pretty strong. It is hard to believe it won't be a persistent trend in DBMS design. Over the long term <i><u>mutable</u></i> data look increasingly like a special case rather than the norm. Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com3tag:blogger.com,1999:blog-768233104244702633.post-54913699533847404982014-02-07T17:07:00.000-08:002014-02-07T17:55:14.199-08:00Fun with MySQL and Hadoop at SCaLE 12XIt's my pleasure to be presenting at <a href="https://www.socallinuxexpo.org/scale12x">SCaLE 12X</a> on the subject of <a href="https://www.socallinuxexpo.org/scale12x/presentations/real-time-data-loading-mysql-hadoop">real-time data loading from MySQL to Hadoop</a>. This is the first public talk on work at Continuent that enables <a href="https://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> to move transactions from MySQL to HDFS (Hadoop Distributed File System). I will explain how replication to Hadoop works, how to set it up, and offer a few words on constructing views of MySQL data using tools like <a href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual">Hive</a>. <br />
<br />
As usual with replication everything we are doing on Hadoop replication is open source. Builds and documentation will be publicly available by the 21st of February, which is when the talk happens. Hadoop support is already in testing with Continuent customers, and we have confidence that we can handle basic loading cases already. That said, Hadoop is a complex beast with lots of use cases, and we need feedback from the community on how to make Tungsten loading support better. My colleagues and I plan to do a lot of talks about Hadoop to help community users get up to speed.<br />
<br />
Here is a tiny taste of what MySQL to Hadoop loading looks like. Most MySQL users are familiar with <a href="https://launchpad.net/ubuntu/+source/sysbench">sysbench</a>. Have you ever wondered what sysbench tables would look like in Hadoop? Let's use the following sysbench command to apply transactions to table db01.sbtest:<br />
<pre><code><b>sysbench --test=oltp --db-driver=mysql --mysql-host=logos1 --mysql-db=db01 \
--mysql-user=tungsten --mysql-password=secret \
--oltp-read-only=off --oltp-table-size=10000 \
--oltp-index-updates=4 --oltp-non-index-updates=2 --max-requests=200000 \
--max-time=900 --num-threads=5 run</b></code></pre>
This results in rows that look like the following in MySQL:<br />
<pre><code><b>mysql> select * from sbtest where id = 2841\G
*************************** 1. row ***************************
id: 2841
k: 2
c: 958856489-674262868-320369638-679749255-923517023-47082008-646125665-898439458-1027227482-602181769
pad: qqqqqqqqqqwwwwwwwwwweeeeeeeeeerrrrrrrrrrtttttttttt</b></code></pre>
After replication into Hadoop with Tungsten, we can crunch the log records using a couple of HiveQL queries to generate a point-in-time snapshot of the sbtest table on HDFS. By a point-in-time snapshot, I mean that a table that contains not only inserted data but also shows the results of subsequent update and delete operations on each row up to a particular point in time. We can now run the same query to see the data:
<br />
<pre><code><b>hive> select * from sbtest where id = 2841;
Total MapReduce jobs = 1
Launching Job 1 out of 1
...
Job 0: Map: 1 Cumulative CPU: 0.74 sec HDFS Read: 901196 HDFS Write: 158 SUCCESS
Total MapReduce CPU Time Spent: 740 msec
OK
2841 2 958856489-674262868-320369638-679749255-923517023-47082008-646125665-898439458-1027227482-602181769 qqqqqqqqqqwwwwwwwwwweeeeeeeeeerrrrrrrrrrtttttttttt</b></code></pre>
Tungsten does a lot more than just move transaction data, of course. It also provides tools to generate Hive schema, performs transformations on columns to make them match the limited HiveQL datatypes, and arranges data in a way that allows you generate materialized views for analytic usage (like the preceding example) with minimal difficulty.<br />
<br />
If you want to learn more about how Tungsten does all of this magic, please attend the talk. I hope to see you in Los Angeles.<br />
<br />
p.s., If you cannot attend SCaLE 12X, we will have a Continuent webinar on the same subject the following week. (Sign up <a href="http://www.continuent.com/news/live-webinars/1389-real-time-data-loading-from-mysql-to-hadoop">here</a>.)Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-55025712148985897832014-01-10T09:17:00.001-08:002014-01-10T09:17:02.024-08:00Why I Love Open SourceAnders Karlsson wrote about <a href="http://karlssonondatabases.blogspot.com/2014/01/some-myths-on-open-source-way-i-see-it.html">Some myths on Open Source, the way I see it</a> a few days ago. Anders' article is mostly focused on exploding the idea that open source magically creates high quality code. It is sad to say you do not have to look very far to see how true this is. <br />
<br />
While I largely agree with Anders' points, there is far more that could be said on this subject, especially on the benefits of open source. I love working on open source software. Here are three reasons that are especially important to me.<br />
<br />
<b>1.) Open source is a great way to disseminate technology to users. </b> In the best cases, it is this easy to get open source products up and running:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>$ sudo apt-get install software-i-want-to-use</b></span><br />
<br />
A lot of software companies (<a href="http://www.continuent.com/">mine</a> included) open source their software because it gets product into the hands of people who might pay money for it later. The strategy worked brilliantly for MySQL AB as Anders pointed out. MongoDB is repeating the tactic with what looks like equal success. There has been a lot of pointless argument over the years about whetherMySQL or MongoDB are "real databases." Being easy to get is just as critical to adoption as features like transactions and scalable performance.<br />
<br />
Open source is therefore even better for users, who can quickly decide if something works for them and provide feedback through communities about problems as well as suggested improvement. To the extent open source software has high quality, it originates in the tight feedback loop between software producers and their user communities. That in turn leads to faster innovation with fewer deviations from real user needs. In olden days we called this getting the requirements right. Open source projects often do it extraordinarily well.<br />
<br />
<b>2.) Open source allows like-minded communities of developers to create products that would otherwise never happen.</b> Linux became a dominant operating system in large part through the staggering scale of contributions enabled by exceptionally well-managed open source development. Linus Torvalds <a href="http://www.youtube.com/watch?v=84Sx0E13gAo&list=PLbzoR-pLrL6oVRP4F6Nz6K2DSkYzNramt&index=11">recently pointed out</a> that Linux kernel releases have patches from a thousand contributors or more. Thanks to the wide range of contributions, Linux operates on everything from tiny ARM processors to servers with over 200 cores. The development effort underlying the Linux ecosystem is huge when you include the kernel and all the packages that install over it. It dwarfs any comparable operating system effort I can think of. <br />
<br />
At the other end of the spectrum there are small but incredibly useful projects like <a href="http://curator.apache.org/">Apache Curator</a>. The Curator project currently has 8 project members, mostly from different companies, who collaborate to make <a href="http://zookeeper.apache.org/">Apache ZooKeeper</a> vastly easier to program. I doubt libraries like Curator would even exist without open source licenses and infrastructure like distributed source code management. Either would ZooKeeper, for that matter. <br />
<br />
Not every line of open source code is excellent or even above average. (I'm looking at you, <a href="http://hadoop.apache.org/">Hadoop</a>.) That said, open source projects are not so much about code but communities of developers who understand and are interested in solving a specific problem. Besides direct feedback from real users, this is the other prerequisite for creating truly great products. Clean code is helpful but not necessary.<br />
<br />
<b>3.) Open source means your creations can never be taken away from you.</b> In many creative endeavors work belongs to the people who employ you. It effectively disappears when you change jobs. Putting code on GitHub or code.google.com breaks that bond. Knowing that anything you create will always be accessible removes any hesitation about revealing your best ideas. I believe this is one of the drivers behind the flowering of creativity that infuses so many open source projects.<br />
<br />
At the same time working on open source software is not all peaches and cream. Building successful businesses on open source is hard, which limits the opportunities to work on it for a living. <br />
<br />
For instance, if most of the value of your product is in the software itself there is not much motivation for users to pay you. I think that's one reason mobile apps are by-and-large for pay or at least not open source. You need to find a business model that brings in enough money over time to fund the sort of concentrated engineering necessary to build robust software. Successful open source businesses often depend on finding the right markets or achieving network effects, and not all software can fit the pattern.<br />
<br />
The good news is that once you get the economics right it really wrong-foots your closed source competitors. <a href="http://www.redhat.com/">RedHat</a> has built a great business packaging and supporting open source for enterprises. They see open source as a competitive advantage that extends their market reach and speeds up innovation. An increasing number of companies producing DBMS software take the same view as they try to disrupt data management. Outside of enterprise software <a href="http://www.valvesoftware.com/">Valve Software</a> is <a href="http://readwrite.com/2014/01/08/steam-machines-valve-ps4-xbox-one#awesm=~oswBFoc7a4e18v">attacking proprietary gaming platforms</a> through open source.<br />
<br />
It's great to see the growing number of businesses based on open source development. When the model works it is incredibly satisfying. I guess this is a fourth reason why I love working on open source software.Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com2tag:blogger.com,1999:blog-768233104244702633.post-76663238115315094742013-03-28T13:34:00.000-07:002013-03-28T13:46:21.028-07:00See You at Percona Live 2013!<a href="http://www.percona.com/live/mysql-conference-2013">Percona Live 2013</a> is coming up fast. This is hands-down the best MySQL conference of the year, attended by a lot of people I really respect. Check the <a href="http://www.percona.com/live/mysql-conference-2013/program/speakers">speaker list</a> if you need some of their names. I will also be doing two talks myself.
<br />
<ul>
<li>9am Wednesday 24 April - <a href="http://www.percona.com/live/mysql-conference-2013/sessions/keynote-how-mysql-can-thrive-world-massive-data-hype">Keynote: How MySQL Can Thrive in the World of Massive Data Hype</a>. NoSQL solutions are oversold, but this is no reason for complacency in the MySQL community. There are new challenges in data management, and we need to solve them or become irrelevant. I will show some of the advances Continuent has on tap for MySQL-based applications and also point back to problems our experience shows must be solved within MySQL itself. </li>
<li>1pm Wednesday 24 April - <a href="http://www.percona.com/live/mysql-conference-2013/sessions/state-art-mysql-multi-master-replication">Session: State of the Art for MySQL Multi-Master Replication</a>. This talk will explain the fundamentals of multi-master operation and then trace the trade-offs of Tungsten, Galera, and other solutions. Thanks to excellent work on several products there is a lot of excitement about multi-master in 2013. My goal is to help listeners understand what applications are possible now as well as what we have the potential to achieve in the future. </li>
</ul>
<div>
I hope you will attend these talks. I am looking forward to meeting old friends at the conference and making new ones. </div>
<div>
<br /></div>
<div>
Incidentally, Percona Live sent me an email yesterday that you can get a 15% discount on the registration price using the code <b>KeySQL</b> when you sign up. At Continuent we are also offering free passes to customers who give us the best quotes about our software. However you get there, I really recommend this conference. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-33397767328665308342013-02-19T09:14:00.000-08:002013-02-19T09:14:47.285-08:00Data Fabric Design Patterns: Fabric ConnectorThis article is the third in a series on <a href="http://scale-out-blog.blogspot.com/2013/02/introducing-data-fabric-design-for.html">data fabric design</a> and introduces the fabric connector service design pattern. The <a href="http://scale-out-blog.blogspot.com/2013/02/data-fabric-design-patterns.html">previous article in this series</a> introduced the transactional data service design pattern, which defines individual data stores and is the building block for data fabrics based on SQL databases. The fabric connector builds on transactional data services and is another basic building block of fabric architecture. <br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Description and Responsibilities</b></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
Fabric connectors make a collection of DBMS servers look like a single server. The fabric connector presents what appears to be a data service API to applications. It routes each request to an appropriate physical server for whatever task the application is performing, hiding the fact that a data fabric can consist of dozens or even hundreds of servers. Applications cannot tell the difference between talking to the fabric connector and talking to a real DBMS server. We call this property <i>transparency</i>.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
Here are the responsibilities of a fabric connector. I will use the phrase proxying to refer to the first of these, and routing responsibilities to refer to the remaining three. </div>
</div>
<ol>
<li>Expose a data service interface to applications.</li>
<li>Route each application query to an appropriate DBMS server.</li>
<li>Balance load by distributing queries across multiple replicas, if available.</li>
<li>Switch to another server following a failure or if the DBMS becomes unavailable due to maintenance.</li>
</ol>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The following diagram shows the logical components of a fabric connector. The fabric connector sits between applications, transactional data services, and a fabric directory service. These are greyed out, as they are not part of the pattern.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQCAGlWqvar0iJ-GGctLFcfDhurG2rgZBVY7uMWs0dD3A8vy7xHI8rCpszz04wtXnzyEPpes0KExztlhwWeEW3An5B00TtKY4pgABynpziHEw2RxuGJj4dMF7yu2sFnXEwX6I2ocpg6Mg/s1600/Fabric-Connector.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQCAGlWqvar0iJ-GGctLFcfDhurG2rgZBVY7uMWs0dD3A8vy7xHI8rCpszz04wtXnzyEPpes0KExztlhwWeEW3An5B00TtKY4pgABynpziHEw2RxuGJj4dMF7yu2sFnXEwX6I2ocpg6Mg/s400/Fabric-Connector.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fabric Connector Design Pattern</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
</div>
</div>
</div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Fabric connectors contain two logical components. The <i>proxy</i> is responsible for routing queries and responses between applications and underlying data services. This can be a library layer, a separate server process, or a TCP/IP load balancer--anything that provides a transparent indirection layer. The <i>directory information </i>contains rules to route SQL queries correctly to the actual location of data. There is a <i>notification protocol</i> that permits connectors to receive updates about the fabric topology and confirm that they have responded to them.<br />
<br /></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Motivation</b></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
Connecting to data is a problem in large systems. Sharded data sets spread data across multiple services. Data services have different roles, such as master or slave. Services fail or go offline for maintenance. Services change roles, such as a master switching to a slave. Shards move between services to balance load and use storage more efficiently. Within short periods of time there may be significant variations in load across data services. Adding routing logic directly to applications in these cases adds complexity and can lead to a tangled mess for administrators.<br />
<br />
The fabric connector design pattern encapsulates logic to route connections from the application to DBMS servers. Hiding connection logic helps keep applications simple. It allows independent testing and tuning of the connection rules. That way you can have some assurance the logic actually works. You can also modify fabric behavior without modifying applications, for example to redistribute load more evening across replicas. <br />
<br />
<b>Related Design Patterns</b><br />
<br />
The fabric connector design pattern manages single application connections to data services, for example a <a href="http://scale-out-blog.blogspot.com/2013/02/data-fabric-design-patterns.html">transactional data service</a>. Transparency is the leitmotif of this pattern. It provides needed encapsulation for other data fabric design patterns and is particularly critical for sharded as well as fault tolerant data services. These will be covered in future articles on data fabric design.<br />
<br />
There are also other design patterns for data access. Here are two that should not be confused with fabric connectors.<br />
<ul>
<li><i>Federated query</i>. Federated query splits a SQL query into sub-queries that it routes to multiple underlying data services, then returns the results. Sharding products like <a href="http://www.dbshards.com/">DbShards</a> and <a href="http://code.google.com/p/shard-query/">shard-query</a> implement this pattern. It requires complex parsing, query optimization, and aggregation logic to do correctly and has varying levels of transparency. </li>
<li><i>MapReduce</i>. <a href="http://en.wikipedia.org/wiki/MapReduce">MapReduce</a> is a procedure for breaking queries into pieces that can run in parallel across large numbers of hosts by splitting the query into map operations to fetch data followed by reduce operations to aggregate results. It can work on any distributed data set, not just SQL. MapReduce implementations often eschew SQL features like joins and also can have a very different programming model from SQL. Their use is often non-transparent to SQL applications.</li>
</ul>
Finally, there is a very important pattern for the <i>fabric directory service</i>. This is a <a href="http://en.wikipedia.org/wiki/Directory_service">directory service</a> that maintains information about the desired topology of the fabric and its actual state. It can be implemented in forms ranging from a shared configuration file to network services in a distributed configuration manager like <a href="http://zookeeper.apache.org/">ZooKeeper</a>.<br />
<br />
I hope to add more complete descriptions for the latter three design patterns at some point in the future. For the current article, we will stick to simple connectivity. <br />
<br /></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Detailed Behavior</b></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
Fabric connectors are conceptually simple: route request from application to server, then transfer results back. Actual behavior can be quite complex. To give some perspective on the problem, here is a short Perl program for a SaaS application that logs order detail information in a table named sale, then reads the same data back. We will use the sample program to illustrate the responsibilities of this design pattern in detail.<br />
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;"><br /></span>
<span class="Apple-style-span" style="font-family: monospace; white-space: pre;">use DBI;</span><br />
<pre># Connect to server.
$dbh = DBI->connect("DBI:mysql:test;host=prodg23", "app", "s3cr3t5"
) || die "Could not connect to database: $DBI::errstr";
# Insert order using a transaction.
$dbh->{'AutoCommit'} = 0;
$dbh->do("INSERT INTO sale(order_id, cust_id, sku, amount) \
VALUES(2331, 9959, 353009, 24.99)");
$dbh->do("INSERT INTO sale(order_id, cust_id, sku, amount) \
VALUES(2331, 9959, 268122, 59.05)");
$dbh->commit();
# Select order back with an auto-commit read
$dbh->{'AutoCommit'} = 1;
$sth = $dbh->prepare("SELECT * FROM sale WHERE order_id=2331");
$sth->execute();
while( $href = $sth->fetchrow_hashref ) {
print "id : $$href{id} \n";
print "order_id: $$href{order_id} \n";
print "cust_id : $$href{cust_id} \n";
print "sku : $$href{sku} \n";
print "amount : $$href{amount} \n";
}
# Disconnect from server.
$dbh->disconnect();</pre>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
</div>
The first responsibility of the fabric connector design pattern is to provide a transparent interface for underlying data services. That means that our Perl program has to work as written--no extra changes. Here are just a few things a connector needs to do:<br />
<ol>
<li>Implement the DBMS connection protocol fully or pass it transparently to an underlying server. This includes handling authentication handshakes as well as setting base session context like client character sets. </li>
<li>Handle all standard features of query invocation and response, including submitting queries, returning automatically generated keys, and handling all supported datatypes in results. </li>
<li>Respect transaction boundaries so that the INSERT statements on the sales table are enclosed in a transaction in the DBMS and the SELECT statement is auto-commit (i.e., a single-statement transaction.) </li>
<li>Read back data written to the sales table. </li>
</ol>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
In addition to handling APIs protocols, fabric connectors need to avoid slowing down transaction processing as a result of proxying. Properly written connectors for the most part add minimal overhead, but there are at least two instances where this may not be the case for some implementations (such as network proxies). The first is establishing connections, a relatively expensive operation that occurs constantly in languages like PHP that do not use connection pools. The second is short primary key-lookup queries on small datasets, which tend to be memory-resident in the server and hence have quick access.</div>
<div>
<br />
One common reaction is to see such overhead as a serious problem and avoid the whole fabric connector approach. Yet the "tax" applications pay for proxying is not the whole story on performance. Fabric connectors can boost throughput by an order of magnitude by distributing load intelligently across replicas. To understand the real application overhead of a connector you therefore need to measure with a properly sized data set and take into account load-balancing effects. Test results on small data sets that reside fully in memory with no load balancing tend to be very misleading. </div>
<div>
<br /></div>
<div>
The remaining fabric connector design pattern responsibilities are closely related: route requests accurately to the correct service, load-balance queries across replicas within a service, and route around replicas that are down due to maintenance or failure. We call these routing responsibilities. They require information about the fabric topology, which is maintained in the connector's directory information. Here is a diagram of typical directory organization. </div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1kU6uT1ask-eX3_VS_HVHbfxOEM2fbVjiHBEWlT7j1MP9sI1b3ZLdKs5eNZQbJlPihg_zN2QI86zy_yobhKbUPJbMYxnY1k_btj67oTbL3SpO6nFCMtt8kQHLreQEKWeztwfQvKqfSME/s1600/directory-service.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="351" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1kU6uT1ask-eX3_VS_HVHbfxOEM2fbVjiHBEWlT7j1MP9sI1b3ZLdKs5eNZQbJlPihg_zN2QI86zy_yobhKbUPJbMYxnY1k_btj67oTbL3SpO6nFCMtt8kQHLreQEKWeztwfQvKqfSME/s400/directory-service.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fabric Directory Service Organization</td></tr>
</tbody></table>
<div>
Let's start with the responsibility to route requests to data services. A simple fabric connector implementation allows connections using a logical server name, such as group2, which the connector would translate to an actual DBMS server and port, such as prodg23:3306. A better fabric connector would allow applications use a customer name like "walmart" that matches what the application is doing. The connector would look up the location of customer data and connect automatically to the right server and even DBMS schema. This is especially handy for SaaS applications, which often shard data by customer name or some other simple identifier. </div>
<div>
<br /></div>
<div>
We could then change our program as follows to connect to the local host and look for the "walmart" schema. Under the covers, the fabric connector will connect to the prodg23 server and use the actual schema for that customer's data. </div>
<div>
<br /></div>
<div>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">use DBI;
# Connect to customer data.
$dbh = DBI->connect("DBI:mysql:walmart;host=localhost", "app", "s3cr3t5"
) || die "Could not connect to database: $DBI::errstr";
</pre>
</div>
<div>
This is a modest change that is very easy to explain and implement. It is a small price to pay for omitting complex logic to locate the correct server and schema that contains the data for this customer. </div>
<div>
<br /></div>
<div>
The next responsibility is to distribute data across replicas. This requires additional directory information, such as the DBMS server role (master vs. slave), current status (online or offline), and other relevant information like slave latency or log position. There are many ways to use this information effectively. Here are a few of the more interesting things we can do.</div>
<div>
<ol>
<li><b>Slave load balancing</b>. Allow applications to request a read-only connection, then route to the most up-to-date slave. This works well for applications such as <a href="http://drupal.org/node/310071">Drupal 7</a>, which is an application for website content management. Drupal 7 is <i>slave-enabled</i>, which means that it can use separate connections for read-only queries that can run on a replica. Many applications tuned to work with MySQL have similar features. </li>
<li><b>Session load balancing</b>. Track the log position for each application session and dispatch reads to slaves when they are caught up with the last write of the session. This is a good technique for SaaS applications that have large numbers of users spread across many schemas. It is one of the most effectively scaling algorithms for master/slave topologies. </li>
<li><b>Partitioning</b>. Split requests by schema across a number of multi-master data services. SQL requests for schema 1 go to server 1, requests for schema 2 to server 2, etc. Besides distributing load across replicas this technique also helps avoid deadlocks, which can become common in multi-master topologies if applications simultaneously update a small set of tables across multiple replicas. </li>
</ol>
</div>
<div>
Recalling our sample program, we could imagine a connector using session load balancing to write the sales table transaction to the master DBMS server, then sending the SELECT to a slave if it happened to be caught up for customer "walmart." No program changes are required for this behavior. </div>
<div>
<br /></div>
<div>
The final responsibility is to route traffic around offline replicas. This gets a bit complicated. We need not only state information but an actual <i>state model </i>for DBMS servers. There also needs to be a procedure to tell fabric connectors about a pending change as well as wait for them to reconfigure themselves. Returning to our sample program, it should be possible to execute the following transaction: </div>
<div>
<br /></div>
<div>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">$dbh->{'AutoCommit'} = 0;
$dbh->do("INSERT INTO sale(order_id, cust_id, sku, amount) \
VALUES(2331, 9959, 353009, 24.99)");
$dbh->do("INSERT INTO sale(order_id, cust_id, sku, amount) \
VALUES(2331, 9959, 268122, 59.05)");
$dbh->commit();</pre>
</div>
<div>
<br /></div>
<div>
then failover to a new master and execute:</div>
<div>
<br /></div>
<div>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">$dbh->{'AutoCommit'} = 1;
$sth = $dbh->prepare("SELECT * FROM sale WHERE order_id=2331");
$sth->execute();
</pre>
<pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">...</pre>
</div>
<div>
<br />
To do this properly we need to ensure that the connecter responds to updates in a timely fashion. We would not want to change fabric topology or take a DBMS server offline while connectors were still using it. The notification protocol that updates connector directory information has to ensure reconfiguration does not proceed until connectors are ready.<br />
<br />
Does every fabric connector have to work exactly this way? Not at all. So far, we have only been talking about responsibilities. There are many ways to implement them. To start with, fabric connectors do not even need to handle SQL. This is interesting in two ways. <br />
<br />
First, you can skip using the Perl DBI completely and use a specialized interface to connect to the fabric. We will see an example of this shortly. Second, the underlying store does not even need to be a SQL database at all. You can use the fabric connector design pattern for other types of stores, such as key-value stores that use the <a href="https://github.com/memcached/memcached/blob/master/doc/protocol.txt">memcached protocol</a>. This series of articles focuses on SQL databases, but the fabric connector design pattern is very general. <br />
<br />
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="font-weight: normal; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b><b>Implementations</b></b></div>
</div>
</div>
</div>
</div>
<br />
Here are a couple of off-the-shelf implementations that illustrate quite different ways to implement the fabric connector design pattern. <br />
<br />
1. <b>Tungsten Connector</b>. <a href="https://docs.continuent.com/wiki/display/TEDOC/Using+the+Tungsten+Connector">Tungsten Connector</a> is a Java proxy developed by <a href="http://www.continuent.com/">Continuent</a> that sits between applications and clusters of MySQL or PostgreSQL servers. It implements the MySQL and PostgreSQL network protocols faithfully, so that it appears to applications like a DBMS server. <br />
<br />
Tungsten Connector gets directory information from <a href="http://www.continuent.com/solutions">Tungsten clusters</a>. Tungsten clusters use a simple distributed consensus algorithm to keep directory data consistent across nodes even when there are failures or network outages--connectors can receive topology updates from any node in the cluster through a protocol that also ensures each connector acts on it when the cluster reconfigures itself. In this sense, Tungsten clusters implement the fabric directory service pattern described earlier.<br />
<br />
The directory information allows the connector to switch connections transparently between servers in the event of planned or even some unplanned failovers. It can also load balance reads automatically using <a href="https://docs.continuent.com/wiki/display/TEDOC/Reading+From+Slave+Data+Sources">a variety of polices</a> including the slave load balancing and session load balancing techniques described above. <br />
<br />
The big advantage of the network proxy approach is the high level of transparency for all applications. Here is a sample session with the out-of-the-box mysql utility that is part of MySQL distributions. In this sample, we check the DBMS host name using the MySQL <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">show variables</span> command. Meanwhile, a planned cluster failover occurs, followed by an unplanned failover.<br />
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<pre><code>mysql> show variables like 'hostname';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| hostname | prodg23 |
+---------------+---------+
1 row in set (0.00 sec)
</code></pre>
<pre><code><b>(Planned failover to prodg21 to permit upgrade on prodg23)
</b>mysql> show variables like 'hostname';
</code><span class="Apple-style-span" style="font-family: Times; white-space: normal;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><code>+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| hostname | prodg21 |
+---------------+---------+
</code></pre>
</span><code>1 row in set (0.01 sec)
</code></pre>
<pre><code><b>(Unplanned failure to prodg22)
</b>mysql> show variables like 'hostname';
</code><span class="Apple-style-span" style="font-family: Times; white-space: normal;"><pre style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><code>+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| hostname | prodg22 |
+---------------+---------+
</code></pre>
</span><code>1 row in set (4.82 sec)
</code></pre>
As this example shows, the session continues uninterrupted as the location of the server switches. These changes occur transparently to applications. The downside is that there is some network overhead due to the extra network hop through the Tungsten Connector, though of course load balancing of reads can more than repay the extra latency cost. Also, this type of connector is hard to build because of the complexity of the MySQL network API as well as the logic to transfer connections seamlessly between servers.<br />
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
2. <b>Gizzard</b>. <a href="https://github.com/twitter/gizzard/blob/master/README.markdown">Gizzard</a> is an open source sharding software developed by Twitter to manage links between Twitter users. The proxy part of the design pattern is implemented by middleware servers, which accept requests from clients using <a href="http://thrift.apache.org/">thrift</a>, a language-independent set of tools for building distributed services. For more on a particular application built on Gizzard, look at descriptions of Twitter's <a href="http://engineering.twitter.com/2010/05/introducing-flockdb.html">FlockDB</a> service. Gizzard servers give applications a simple API for data services, which fulfills the proxy responsibility of the fabric connector design pattern. <br />
<br />
Gizzard servers get directory information using <a href="https://github.com/twitter/gizzmo">gizzmo</a>. Gizzmo is a simple command line tool that maintains persistent copies of the Gizzard cluster topology and takes care of propagating changes out to individual Gizzard servers. For more on how Gizzmo works, look <a href="https://github.com/twitter/gizzmo/wiki/How-to-use-Gizzmo">here</a>. Using this information, Gizzard servers can locate data, route around down servers, and handle distribution of queries to replicas, which are the final three responsibilities of the fabric connector design pattern. <br />
<br />
The Gizzard architecture lacks the generality of the Tungsten Connector, because it requires clients to use a specific interface rather than general-purpose SQL APIs. It also introduces an extra network hop. On the other hand, it works extremely well for its intended use case of tracking relationships between Twitter users. This is because Gizzard deals with a simplified problem and also allows scalability through many Gizzard servers. Like the Tungsten Connector the network hop expense pays for itself due to the ability to load-balance across multiple replicas. <br />
<br />
Gizzard is a nice example of how the fabric connector design pattern does not have to be specifically about SQL. Gizzard clients do not use SQL, so the underlying store could be anything. Gizzard is specifically designed to work with a range of DBMS types. <br />
<br />
<b>Fabric Connector Implementation Trade-Offs</b><br />
<br />
General-purpose fabric connectors like the one used for Tungsten are hard to implement for a variety of reasons. This approach is really only practical if you have a lot of resources at your disposal or are doing it as a business venture like Continuent. You can still roll your own implementations. The Gizzard architecture nicely illustrates some of the trade-offs necessary to do so.<br />
<br />
1. <b>General vs. particular data service interfaces</b>. Implementing a simple data service interface, for example using thrift, eliminates the complexity of DBMS interfaces like those of MySQL or PostgreSQL. Rather than a thrift server you can also use a library within applications themselves. This takes out the network hop.<br />
<br />
2. <b>Automatic vs. manual failover</b>. Automatic failover requires connectors to respond to fabric topology changes in real time, which is a hard problem with a lot of corner cases. (Look <a href="http://en.wikipedia.org/wiki/Consensus_(computer_science)">here</a> if you disagree.) You can simplify things considerably by minimizing automated administration and instead orchestrate changes through scripts. <br />
<br />
3. <b>Generic vs. application-specific semantics</b>. Focusing on a particular application allows you to add features that are important for particular use cases. Gizzard supports shard migration. To make it tractable to implement Gizzard requires a simple update model in which transactions can be applied in any order.<br />
<br />
These and other simplifications make the fabric connector design pattern much easier to implement correctly. You can make the same sort of trade-offs for most applications. <br />
<br />
<b>Implementations to Avoid</b><br />
<b><br /></b>
Here are a couple of implementations for the fabric connector design pattern that you should avoid or at least approach warily. <br />
<br />
1. Virtual IP Addresses (VIPs). VIPs allow hosts to listen for traffic on multiple IP addresses. They are commonly used in many failover schemes, such as programs like heartbeat. VIPs do not have the intelligence to fulfill fabric connector responsibilities like load-balancing queries across replicas. They are subject to nasty split-brains, a subject I covered in detail as part of <a href="http://scale-out-blog.blogspot.com/2011/01/virtual-ip-addresses-and-their.html">an earlier article on this blog</a>. Finally, VIPs are not available in Amazon and other popular cloud environment. VIPs do not seem like a good implementation choice for data fabrics. <br />
<br />
2. SQL proxies. There are a number of software packages that solve the problem of proxying SQL queries, such as <a href="http://pgfoundry.org/projects/pgbouncer">PgBouncer</a> or <a href="http://dev.mysql.com/doc/refman/5.6/en/mysql-proxy.html">MySQL Proxy</a>. Many of them do this quite well, which means that they fulfill the first responsibility of the fabric connector design pattern. The problem is that they do not have a directory service. This means they do not fulfill the next three responsibilities to route queries effectively, at least out of the box. <br />
<br />
Unlike VIPs, SQL proxies can be a good starting point for fabric implementations. You need to add the directory information and notification protocol to make them work. It is definitely quite doable for specific cases, especially if you make the sort of trade-offs that Gizzard illustrates.<br />
<br />
<b>Conclusion and Takeaways</b></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b><br /></b>The fabric connector design pattern reduces the complexity of applications by encapsulating the logic required to connect to servers in a data fabric. There is a tremendous benefit to putting this logic in a separate layer that you can test and tune independently. Fabric connectors are more common than they appear at first because many applications implement the responsibilities within libraries or middleware servers that include embedded session management. Fabric connectors do not have to expose SQL interfaces or any other DBMS-specific interface, for that matter. <br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Fault-tolerant and sharded data service design patterns depend on fabric connectors to work properly and avoid polluting applications with complex logic to locate data. Products that implement these design patterns commonly include fabric connector implementations as well. You can evaluate them by finding out how well they fulfill the design pattern responsibilities. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
Off-the-shelf fabric connectors have the advantage that they are more general than something you can develop easily for yourself. If you decide to write your own fabric connector, you will need to consider some of the trade-offs like reducing automation or simplifying APIs in order to make the problem easier to solve. Regardless of the approach, you should allow time. The responsibilities are complicated and must be implemented with care. Fabric connectors that only work 99% of the time of are not much use in production environments. <br />
<br />
One final point about fabric connectors. Automated failover can make fabric connectors harder to implement and increase the risk that the fabric connector may write to the wrong replica. The difficulty of managing connectivity is one of the reasons many data management experts are very cautious about automatic failover. This problem <a href="http://scale-out-blog.blogspot.com/2012/09/automated-database-failover-is-weird.html">is tractable in my opinion</a>, but it is definitely a practical consideration in system design.<br />
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
My next article on data fabrics will cover the fault-tolerant data service design pattern. This design pattern depends on the fabric connector design pattern to hide replicas. I hope you will continue reading to find out about it. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
</div>
</div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-18677567508859733242013-02-06T21:56:00.000-08:002013-02-19T09:53:05.606-08:00Data Fabric Design Patterns: Transactional Data ServiceThis article is the second in a series on data fabric design and introduces the transactional data service design pattern. The <a href="http://scale-out-blog.blogspot.com/2013/02/introducing-data-fabric-design-for.html">previous article in this series</a> introduced data fabrics, which are collections of off-the-shelf DBMS servers that applications can connect to like a single server. They are implemented from data fabric design patterns, which are reusable arrangements of DBMS servers, replication, and connectivity. With this article we begin to look at individual design patterns in detail.<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Description and Responsibilities</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The transactional data service is a basic building block of data fabric architectures. A transactional data service is a DBMS server that processes transactions submitted by applications and stores data safely. Transactional data services have the following responsibilities:</div>
<ul>
<li>Store data transactionally and recover data faithfully to the last full transaction following failure. </li>
<li>Provide a network-accessible application interface for accessing data</li>
<li>Provide a reliable and reasonably quick method for backup and restore. </li>
<li>Maintain an accessible, serialized log of transactions. This enables replication between services. </li>
</ul>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The following diagram illustrates the moving parts of a transactional data service. In future diagrams we will just use the standard database symbol for the entire transactional data service, but for now we need to be able to see the contents. </div>
<div class="separator" style="clear: both; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; text-align: center;">
</div>
<div class="separator" style="clear: both; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnYfzyvwipWqyTzAq5PPwVPCX2eNT-ZJq2LPob0A7K49u8A5QLeipSnJMg-kF-AuMjom1wamBdN8MTe-KmqBOjpXULabmYraoT9_iNAg8sz5hRmYoorU0_u26aBumB6McDCMWtWzGKNfs/s1600/Transactional-Data-Service.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="158" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnYfzyvwipWqyTzAq5PPwVPCX2eNT-ZJq2LPob0A7K49u8A5QLeipSnJMg-kF-AuMjom1wamBdN8MTe-KmqBOjpXULabmYraoT9_iNAg8sz5hRmYoorU0_u26aBumB6McDCMWtWzGKNfs/s400/Transactional-Data-Service.jpg" width="400" /></a></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Motivation</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Durable storage of transactions is the most fundamental responsibility of database systems. It is difficult to build reliable applications if stored data can disappear or become corrupted because transactions were not committed before a crash. Both problems can cause data loss. Moreover, they can break replication links very badly if the DBMS server comes up in an inconsistent state, for example with some updates committed but others randomly rolled back. This condition affects not only the one server but potentially many others throughout the fabric. <br />
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The transactional data service therefore focuses on storing data safely and recovering to the last committed transaction after a restart. With this basic capability we can construct more complex services knowing that individual changes are unlikely to disappear or be recorded inconsistently. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Detailed Behavior</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Let's look in detail at the properties required for a successful transactional data service. Somewhat surprisingly, an off-the-shelf SQL DBMS does not necessarily fit the pattern, though it comes close. It is important to understand the differences. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The transactional store keeps data from getting lost and is the basis for recovery throughout the fabric. Transactional stores support commit and rollback with multi-statement transactions. In theory the transaction data service responsibility for data persistence matches MySQL/InnoDB and PostgreSQL behavior, both of which commit transactions safely in serial order. However, the reality is not quite that simple. <br />
<br />
Most DBMS allow applications to ignore transactions under certain conditions. This results in <i>wormholes</i>, which are violations in serial ordering of data. There are a number of table definition options in SQL that undo transactional consistency.<br />
<ul>
<li>(MySQL) <a href="http://dev.mysql.com/doc/refman/5.5/en/myisam-storage-engine.html">MyISAM table type</a>. MyISAM tables ignore transactions and commit immediately, even if the application later tries to roll back. The tables may also become corrupted by server crashes. </li>
<li>(MySQL) <a href="http://dev.mysql.com/doc/refman/5.5/en/memory-storage-engine.html">Memory table type</a>. These tables are maintained in memory only. They disappear on restart. </li>
<li>(PostgreSQL) <a href="http://www.postgresql.org/docs/9.2/static/sql-createtable.html">UNLOGGED tables</a>. Such tables are not logged and disappear on crash or unclean shutdown (thanks Frederico). </li>
</ul>
All of these allow data to disappear or become corrupted after a crash. However, there is a more subtle problem. If applications depend on these tables, transaction results may then depend on when the server last crashed or restarted, which in turn makes updates across replicas non-deterministic. Random updates create problems for data replication, which depends on replicas behaving identically when transactions are applied. It is important to avoid application dependencies on any feature that creates wormholes or you might not be able to use other design patterns.<br />
<br />
So is data loss always bad? Surprisingly, no. In some cases transactional stores can lose data <u>provided</u> that they do so by dropping the last transactions in serial order. It's as if the data just reverted to an earlier point in time. To understand why this might be OK, imagine three servers linked into a chain by asynchronous replication. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOhPsbEc15lv9eK8o1wnmxWZXijhMPwGu6LMGsVe3kR4Wac9RTMb6n3b61geTI-iPnb8uKyVZDxEd2TqMRLh1td3CsLU3XOqlgx1PL5y88qalNWN72Ysozh-YUMZTZ_SLUBMSW1g39e7I/s1600/Controlled-Data-Loss.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="110" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOhPsbEc15lv9eK8o1wnmxWZXijhMPwGu6LMGsVe3kR4Wac9RTMb6n3b61geTI-iPnb8uKyVZDxEd2TqMRLh1td3CsLU3XOqlgx1PL5y88qalNWN72Ysozh-YUMZTZ_SLUBMSW1g39e7I/s400/Controlled-Data-Loss.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Data loss is sometimes not a big deal</td></tr>
</tbody></table>
It is bad to lose data on the first server, especially if those data are lost before replicating transactions to the downstream replicas. However, data loss on the last server is fine. Assuming that server stores the replication restart point transactionally, it will just re-apply the missing transactions and catch up. This is exactly what happens when you restore a backup in slave in master/slave replication. <br />
<br />
Data loss on the second server also may not be a problem. It should restart replication and should generate identical transactions for itself as well as for replication to the last server. In both cases we assume that replication will handle these cases correctly and can replay missing transactions from logs. If so, you can not only tolerate such losses but even depend on recovering from them automatically.<br />
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Turning to the next responsibility of the transactional data service, the application interface may obviously include SQL via MySQL or PostgreSQL wire protocols. However, any consistent interface that is accessible over a network will do. The <a href="https://github.com/memcached/memcached/blob/master/doc/protocol.txt">memcached protocol</a> is also perfectly acceptable. Subsets of SQL such as stored procedures work quite well. Transactional data services are more general than SQL DBMS servers in this sense. Full SQL or even a subset is not a requirement. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Backup and restore are critical for data fabrics as they enable provisioning of new services as well as recovery of services that fail. You restore a backup and then let the service catch up using replication. Data fabrics can get along fine using a range of options from logical dumps of the DBMS (mysqldump or pgdump) to file system snapshots. Fabric backups just need to be transactionally consistent and movable to other hosts. <br />
<br />
Note that the backup required by the transaction data service design pattern is a less general form of backup than most businesses really require. Businesses may need to recover data after accidental deletion or to keep copies of information for many years for legal reasons. You can therefore use a general-purpose backup solution like <a href="http://www.zmanda.com/">Zmanda</a> or <a href="http://www.pgbarman.org/">Barman</a> provided it meets the fabric design pattern requirements. There's no need to do things twice. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Finally, the replication log is a serialized list of transactions to replicate to other hosts. Serialization enables the transactions to be replayed on another host and result in an identical copy. Generally speaking, data fabrics require <i>logical replication</i>, which applies changes to replicas using SQL statements on a live server. This is because other design patterns depend on being able to access and even write to the transactional data service when it is acting as a slave. Binary replication methods like disk block replication, such as <a href="http://www.drbd.org/">DRBD</a>, do not meet this requirement and therefore are of limited use in data fabrics. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Implementation</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
You can implement the transactional data service design pattern with any DBMS that meets the pattern responsibilities. That said, implementation details are very important. As we have seen, ensuring that DBMS servers live up to the responsibility to store transactions safely is a little harder than one might think. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>1. MySQL</b>. MySQL with InnoDB engine is generally a good choice. It has stable SQL APIs and a wide range of capable client libraries. However, MySQL must be correctly configured to maintain proper transactional guarantees. Here are three properties that should be in your my.cnf file to help ensure MySQL lives up to its responsibilities: </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Ensure durable flush to storage on transaction commit. </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">innodb_flush_log_at_trx_commit=1</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Synchronize binlog with committed transactions. </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sync_binlog=1</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Use InnoDB as default storage engine. (Unnecessary for MySQL 5.5 and above.) </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">default-table-type=InnoDB</span></div>
<div>
<br /></div>
<div>
There are a variety of good backup mechanisms for MySQL databases, including mysqldump (with --single-transaction, useful only for small data sets), <a href="http://www.percona.com/software/percona-xtrabackup">Percona XtraBackup</a>, and file system snapshots. Snapshots are especially good when using NetApp or other capable storage. NetApp snapshots can be restored in seconds and cost little in terms of performance overhead. </div>
<div>
<br /></div>
<div>
<b>2. PostgreSQL</b>. PostgreSQL with a trigger-based replication log, for example from Londiste or SLONY, and pgdump for backups is another good choice. PostgreSQL has unusually good trigger support for DML changes at least, and permits users to encode them in a number of languages. Be aware the PostgreSQL triggers do not capture DDL statements like CREATE TABLE, though.<br />
<br />
PostgreSQL is fully transactional out of the box and triggers create a fully serialized replication log. It does not have the problem that MySQL does with potentially unsafe table types like MyISAM. However, you need to set a couple of parameters to ensure safe operation. These ensure transactions are committed down to the storage level and prevent catastrophic corruption of the database and/or the WAL (write-ahead log). Since PostgreSQL defaults to these values, you mostly need to avoid turning them off.<br />
<div>
<br /></div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fsync = on # turns forced synchronization on or off</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">synchronous_commit = on # immediate fsync at commit</span><br />
<br />
Like MySQL, PostgreSQL SQL and APIs are stable and well-known. Pgdump also loads and restores data without difficulty for smallish data sets. For larger data sets file system snapshots work very well. </div>
<div>
<br />
Regardless of the DBMS type you choose, it is important to avoid application-level features that introduce wormholes, such as the PostgreSQL unlogged tables mentioned in the previous section. Generally speaking, you should only skip transactions if there is a very strong reason for doing so.<br />
<br /></div>
<div>
Do other database types work for this design pattern? Of course. You can also use a commercial DBMS like Oracle. Oracle fulfills the pattern responsibilities quite well, but is a bit more heavyweight than users want, particular when operating in the cloud. </div>
<div>
<br />
<b>And Hardware Implementation, Too...</b><br />
<br />
Even with a properly configured DBMS server you are still not completely out of the woods for data durability. Database servers generally use an <a href="http://linux.die.net/man/2/fsync">fsync()</a> or similar system call to flush data to storage. Unfortunately storage controller cards may just cache the data to be written in local RAM and return. That can fool the DBMS server into thinking transactions are safely stored when they actually are still sitting in memory on a controller card. The "committed" data will then vaporize in a host crash, which in turn can fatally corrupt both MySQL <i>and</i> PostgreSQL stores if you are very unlucky. Just a few bad blocks can cause very serious problems. <br />
<br />
Fortunately there is a cure to make data vaporization less likely. On local storage you can invest in <a href="http://serverfault.com/questions/114547/what-is-a-raid-controller-bbu-for">RAID with a battery-backed cache (BBU)</a>, which keeps power on for the cache even if the host fails completely. SANs and network attached storage tend to have this capability built in. (But check the specifications!) Battery backed cache also tends to be fast, since controllers can safely return from an fsync() operation as soon as the data to be written are in the on-board cache. Without this feature writes to storage can be painfully slow. <br />
<br />
One interesting question is how to handle cloud environments like Amazon. You just do not know how the storage actually works. (That's really the point of a cloud, after all.) Amazon provides SLAs for performance (example: <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html">EBS provisioned IOPS</a>), but there do not seem to be any SLAs about storage consistency. There is a lot to learn here and lessons that apply to Amazon may not necessarily apply to others. I suspect this will prompt some rethinking about data consistency--it's an interesting "what if" for transaction processing to suppose you cannot trust the underlying storage capabilities.<br />
<br />
Data loss occurs sooner or later in virtually all systems, but the good new is that you can make it uncommon. For more information check out data sources like <a href="http://www.amazon.com/High-Performance-MySQL-Optimization-Replication/dp/1449314287">this</a> and <a href="http://www.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X">this</a>. Also, other fabric design patterns like the Fault-Tolerant Data Service keep applications running when failures do occur and can even minimize the effects of data loss. See the upcoming article on that design pattern for more information. <br />
<br /></div>
<div>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>Implementations to Avoid</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Here are two examples that do not meet the transactional data service design pattern responsibilities or at least not fully. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<b>1. MySQL with MyISAM table type. </b> MyISAM does not support transactions and is not crash safe. You will lose data or incur downtime fixing problems. MyISAM does not belong in data fabrics. <br />
<b><span class="Apple-style-span" style="font-weight: normal;"><br /></span></b></div>
<div>
<b><span class="Apple-style-span">2. PostgreSQL with streaming replication.</span><span class="Apple-style-span" style="font-weight: normal;"> <a href="http://www.postgresql.org/docs/9.2/static/warm-standby.html#STREAMING-REPLICATION">Streaming replication</a> replicates log updates in real-time and has the added benefit of permitting queries on replicas. However, streaming replication does not allow you to write to replicates. It therefore does not support online schema maintenance or multi-master replication. It also does not help with heterogeneous replication. Streaming replication is therefore an unattractive choice, even though it is far simpler and works better for ensuring high availability than logical replication solutions like SLONY. </span></b></div>
<div>
<b><span class="Apple-style-span" style="font-weight: normal;"><br /></span></b></div>
<div>
<b><span class="Apple-style-span" style="font-weight: normal;">How do NoSQL stores fare in this design pattern? Let's pick on <a href="http://www.mongodb.org/">MongoDB</a>. MongoDB supports atomic commit to single BSON documents but does not support transactions across multiple documents. (Unless you think that the <a href="http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/">hacky two-phase commit</a> proposed by the MongoDB manual is an answer.) Atomic transactions are not one of the reasons people tend to choose NoSQL systems, so this is not surprising. </span></b><b><span class="Apple-style-span" style="font-weight: normal;">It means that MongoDB cannot handle the responsibilities of this design pattern. </span></b></div>
<div>
<b><span class="Apple-style-span" style="font-weight: normal;"><br /></span></b></div>
<div>
<div>
<b>Conclusion and Takeaways</b></div>
<div>
<b><br /></b>
The transactional data service design pattern can be implemented with a carefully configured SQL database. As we have seen, however, there are a number of details about what "carefully configured" really means. <br />
<br />
It is a good idea to use the transactional data service design pattern even if you are not planning to implement a data fabric. Systems grow. This pattern gives you the flexibility to build out later by adding other fabric design patterns, for example to introduce cross-site operation using the Multi-Site Data Server pattern. It also protects your data at multiple levels that include transactions as well as regular backups. Nobody really likes losing data if it can be avoided.<br />
<br />
Another important point: the transactional data service design pattern keeps your entire fabric working. Losing small amounts of data is typically just an inconvenience for users, especially if it does not occur too often. Broken replication on the other hand due to replicas that diverge or become corrupt after failures can lead to time-consuming administration and significant downtime to repair. The fabric is a network of servers. Poor configuration on one server can cause problems for multiple others. <br />
<br />
Finally, the biggest error people make with this design pattern is to neglect backups. There's something inherently human about it: backups are painful to test so most of us don't. My first and biggest IT mistake involved a problem with backups. It nearly caused a large medical records company to lose a week of data entry records. Put in backups and test them regularly to ensure they work. This is a theme that will recur in later articles. <br />
<br />
Speaking of which, the next article in this series will cover the <a href="http://scale-out-blog.blogspot.com/2013/02/data-fabric-design-patterns-fabric.html">fabric connector design pattern</a>. Stay tuned! </div>
</div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com4tag:blogger.com,1999:blog-768233104244702633.post-63863680279550707592013-02-05T09:23:00.000-08:002013-02-06T23:09:21.907-08:00Introducing Data Fabric Design for Commodity SQL DatabasesData management is undergoing a revolution. Many businesses now depend on data sets that vastly exceed the capacity of DBMS servers. Applications operate 24x7 in complex cloud environments using small and relatively unreliable VMs. Managers need to act on new information from those systems in real-time. Users want constant and speedy access to their data in locations across the planet. <br />
<br />
It is tempting to think popular SQL databases like MySQL and PostgreSQL have no place in this new world. They manage small quantities of data, lack scalability features like parallel query, and have weak availability models. One reaction is to discard them and adopt alternatives like Cassandra or MongoDB. Yet open source SQL databases have tremendous strengths: simplicity, robust transaction support, lightning fast operation, flexible APIs, and broad communities of users familiar with their operation. The question is how to design SQL systems that can meet the new requirements for data management. <br />
<br />
This article introduces an answer to that question: <i><u>data fabric design</u></i>. Data fabrics arrange off-the-shelf DBMS servers so that applications can connect to them as if they were a single database server. Under the covers a data fabric consists of a network of servers linked by specialized connectivity and data replication. Connectivity routes queries transparently from applications to DBMS servers. Replication creates replicas to ensure fault tolerance, distribute data across locations, and move data into and out of other DBMS types. The resulting lattice of servers can handle very large data sets and meet many other requirements as well.<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Data fabric design is a big topic, so I am going to spread the discussion over several articles. This first article provides a definition of data fabric architecture and introduces a set of design patterns to create successful data fabrics. In the follow-on articles I will explore each design pattern in detail. The goal is to make it possible for anyone with a background in database and application construction to design data management systems that operate not only today but far into the future. At the very least you should understand the issues behind building these systems. <br />
<br />
Some readers may see data fabric design as just another reaction to NoSQL. This would be a mistake. Building large systems out of small, reliable parts is a robust engineering approach that derives from ground-breaking work by <a href="http://www.amazon.com/Transaction-Processing-Concepts-Techniques-Management/dp/1558601902">Jim Grey</a>, <a href="http://cs.brown.edu/courses/cs227/papers/weaker/cidr07p15.pdf">Pat Helland</a>, and others dating back to the 1970s. Data fabrics consist of DBMS servers that you can look at and touch, whereas NoSQL systems tend to build storage, replication, and access into a single distributed system. It is an open question which approach is more complex or difficult to use. There are trade-offs and many systems actually require both of them. You can read this article and those that follow it, then make up your own mind about the proper balance. <br />
<br />
<b>Acknowledgements</b><br />
<br />
The data fabric concept is largely based on practical experience on <a href="http://www.continuent.com/solutions">Continuent Tungsten</a>. I am indebted to <a href="http://www.continuent.com/">Continuent</a> as well as our customers for the opportunity to work on this problem. I am likewise indebted to Ed Archibald, Continuent CTO, with whom I have worked for many years. Ed among other things came up with the data fabric moniker. Our interest in this topic goes back to shared experiences with <a href="http://manuals.sybase.com/onlinebooks/group-cnarc/cng1110e/osref/@Generic__BookTextView/281;hf=0">Sybase OpenServer</a> and <a href="http://www.sybase.com/products/businesscontinuity/replicationserver">Sybase Replication Server</a>, which were ground-breaking products in the fields of connectivity and replication. Two decades later we are still applying the insights gained from working on them. <br />
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>What Is a Data Fabric Architecture?</b><br />
<b><br /></b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Let's start the discussion of data fabrics with a practical problem. We want to design a SaaS application for <a href="http://en.wikipedia.org/wiki/Customer_relationship_management">customer relationship management</a> that will support up to a million users using a commodity open source DBMS like MySQL. What are the requirements?</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Obviously, our system must be able to scale over time to hundreds of app servers operating on hundreds of terabytes data. It must hide failures, maintenance, and schema upgrades on individual DBMS hosts. It must permit data to distribute across geographic regions. It must deliver and accept transactions from NoSQL, data warehouses, and commercial DBMS in real time. It must allow smooth technology upgrade and replacement. Finally, it must look as much like a single DBMS server to applications as possible. Here's a picture of what we want: </div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div class="separator" style="clear: both; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-bottom: 0.5em; margin-left: auto; margin-right: auto; padding-bottom: 6px; padding-left: 6px; padding-right: 6px; padding-top: 6px; text-align: center;"><tbody>
<tr><td style="text-align: center;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZz80lygBT-pANwTXMCd9UjNKEUCdZ_hTxB8yiNpArgYU70x0TIB7ZJa0BtaseAa-srOnfENGjIfwRFdNw6jDskoiu59O6-R91wW1AjCSkYe3NeRGUU0LEcvz4xLN6WnHg_54BlcCDZT8/s1600/Data-Fabric-Intro.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZz80lygBT-pANwTXMCd9UjNKEUCdZ_hTxB8yiNpArgYU70x0TIB7ZJa0BtaseAa-srOnfENGjIfwRFdNw6jDskoiu59O6-R91wW1AjCSkYe3NeRGUU0LEcvz4xLN6WnHg_54BlcCDZT8/s320/Data-Fabric-Intro.jpg" style="cursor: move;" width="320" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="font-size: 13px; padding-top: 4px; text-align: center;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Conceptual Data Fabric</div>
</td></tr>
</tbody></table>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The last requirement is especially important and goes to the essential nature of data fabric architecture. The fabric may contain dozens or even hundreds of servers but encapsulates their locations and number. Applications connect to the fabric the same way they connect to individual DBMS servers, a property we call <i>transparency</i>. Transparency permits developers to build applications using a DBMS on a laptop and push out code to production through increasingly capable test environments without changes in behavior. This is a potentially confusing requirement so let's look at a couple of examples.<br />
<br />
Transparency <i>does not</i> mean that you get access to all data all the time from everywhere. Say our sample application stores customer data across many servers. The following SQL query to list regions with SaaS users who have sales to their own customers greater than $100,000 would typically not work:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> select region, count(cust_id), sum(sales) as sales_total </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> from customer group by region having sales_total > 100000;</span><br />
<br />
This is not as big a problem as it sounds, as SaaS applications for the most part operate within a single SaaS user at a time and do not ask for data across users. Moreover, it is easy to understand that you need to do something special for this particular query, such as connect to all servers explicitly or load transactions into a data warehouse. Most SaaS designers are pretty comfortable with this limitation, which just makes explicit something that you know anyway. <br />
<br />
On the other hand transparency <i>does</i> mean that your application talks to what looks like a single server for individual SaaS user data. For instance the following sequence of commands on a single customer to insert a row and get the generated auto-increment key back <i><u>must</u></i> work in a fabric just as it does in a single DBMS server.<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> insert into customer(name, region, sales) values('bob', 'sfo', 10035.0);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Query OK, 1 row affected (0.00 sec)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> select last_insert_id();</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| last_insert_id() |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| 1 |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">1 row in set (0.00 sec)</span><br />
<br />
Selecting the last inserted ID is a standard idiom for adding rows to table with synthetic keys in MySQL. It is baked into widely used libraries like PHP mysqli. Change it and you break thousands of applications. This could happen if the fabric switched a DBMS connection across servers between these two commands. When operating on data for a single SaaS user fabric transparency needs to be as close to perfect as possible. <br />
<br />
Any architecture that meets the preceding requirements including transparency is a data fabric. The next topic is what is actually inside a fabric architecture. <br />
<br />
<b>What Are Data Fabrics Made of? </b><br />
<b><br /></b>
Combining individual components into lattices is hardly a new idea. Such compositions are common in fields from <a href="http://en.wikipedia.org/wiki/File:Double-Helix-Bridge.jpg">bridge-building</a> to art. One of my favorite examples is the famous arabesques of the <a href="http://en.wikipedia.org/wiki/Alhambra">Alhambra</a> in Granada, which combine simple motifs into patterns then combine those to create still more complex patterns that cover walls and ceilings throughout the palace. The resulting compositions are works of stunning beauty. </div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-bottom: 0.5em; margin-left: auto; margin-right: auto; padding-bottom: 6px; padding-left: 6px; padding-right: 6px; padding-top: 6px; text-align: center;"><tbody>
<tr><td style="text-align: center;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<a href="https://s3.amazonaws.com/blog-pictures/01-data-fabric/Alhambra-Arabesque-Atauriques.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="265" src="https://s3.amazonaws.com/blog-pictures/01-data-fabric/Alhambra-Arabesque-Atauriques.jpg" style="cursor: move;" width="400" /></a></div>
</td></tr>
<tr><td class="tr-caption" style="font-size: 13px; padding-top: 4px; text-align: center;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Detail of Arabesque from Alhambra, Spain (Source: Wikipedia)</div>
</td></tr>
</tbody></table>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Arabesque construction is far from random. Arabesques combine plant-like elements into geometric patterns. Only certain elements are allowed--there are typically no human representations--and only certain patterns work well. Arabesques are also expressed in a particular medium such as stone, plaster, tiles, or paint. <br />
<br />
Much as arabesques do, data fabrics combine very specific elements in a medium consisting of the following logical parts: </div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
</div>
<ol>
<li>A partitioned network of recoverable transactional stores (DBMS servers) connected by reliable messaging (replication). Partitioned means that not every service exchanges information with every other, in the same way that DBMS servers for different applications may be separate silos. </li>
<li>A routing layer that connects applications to relevant data based on simple hints provided by the application itself, such as the name of a schema, a primary key, or whether the current user transaction is a set of writes or an auto-commit read. </li>
</ol>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Stores, replication, and routing logic are the fundamental elements of fabric implementations. These are powerful tools for building very large applications. Let's take a quick tour, as their particular properties are critical for building systems that actually work. <br />
<br />
<i>Transactional stores</i> are<i> DBMS servers</i> that apply changes as atomic units that either commit or roll back as a whole. Transactional stores convert these changes into a <i>serial history</i>, which orders concurrent transactions in its log so that they can replay as if they had executed one after the other all by themselves. Serial ordering enables <i>recovery</i>, which is the procedure to bring data back cleanly to the last committed transaction after a crash or restart. These properties are at the heart of all relational DBMS systems today and are necessary for the fabric to operate. Non-transactional stores are close to useless in fabrics. Use them at your peril. <br />
<br />
<i>Reliable messaging </i>transmits changes between stores without losing them or applying them twice. The usual way to do this is through <i>replication</i>, which moves transactions automatically on commit and applies them to replicas. Replication needs to support transactions, apply them in serial order, and recover just as transactional stores do. Replication systems that fail to meet any of these three requirements do not cut the mustard and do not belong in data fabrics. <br />
<br />
Data replication implementations differ in at least three major ways. There is not really a "right approach" to replication--different methods work better for some applications than others.<br />
<br />
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>1. Synchronous vs. asynchronous.</i><b> </b>Synchronous replication moves transactions before reporting commit back to applications. It minimizes problems with data loss if a replica fails but may slow or block applications if replication is slow. It can also lead to deadlocks. Asynchronous replication moves transactions after commit. It does not block applications but leads to latency between replicas and may result in data loss if there is a failure before transactions replicate from a particular replica. </div>
</div>
</div>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
</div>
</div>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>2. Master-master vs. master/slave. </i>Master-master replication allows updates on any replica. It requires conflict management, typically either through locking in advance or fixing up after the fact (also known as <i>conflict resolution</i>). Master/slave replication requires applications to use a single master for updates. It works well with asynchronous replication, which is easier to implement. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>3. Log-based vs. trigger-based</i>. Log-based replication reads the database journal or some representation of it like the MySQL binlog. Log-based replication has lower impact on DBMS servers but can be very hard to implement correctly. Trigger-based replication uses triggers to capture transaction content, typically using transfer tables. Trigger-based replication is simpler to implement but adds load to DBMS servers, can add unwanted serialization points to transactions, and may not handle certain types of changes, such as DDL data. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
The routing layer directs application DBMS connections to specific copies of data. An application sees what appears to be a <i>single session</i> for each connection with consistent settings, such as character set encodings and session variables. Underneath the fabric may actually switch the connection across DBMS servers at opportune times. Transparency is absolutely critical. As mentioned previously, even tiny changes in session behavior breaks libraries that applications depend on to access data. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
There are many methods to route connections as well as transactions. As with replication, there is no "best" way. It all depends on your application and the type of environment in which you are operating. Here are four common approaches. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>1. Gateways. </i> A gateway is a proxy process that sits between the application and DBMS server. The gateway looks like a server to applications. It establishes connections on behalf of the applications, hence can perform very flexible transformations on data at the cost of some performance due to the double network hop the gateway introduces. Gateways are hard to implement initially, but good ones are relatively easy to program once they work properly. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>2. IP Routers. </i> Routing software switches IP packet paths between servers. It has the lowest overheard and potentially highest transparency but require substantial effort to implement correctly. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<i>3. Library wrappers. </i>Library wrappers re-implement standard interfaces like JDBC or Perl DBI, then route application requests through underlying connectors like the MySQL or PostgreSQL JDBC driver. Wrappers are relatively easy to implement and have excellent performance but do not handle traffic that goes outside the library. Compiled versions can have nasty library dependencies that introduce new forms of <a href="http://en.wikipedia.org/wiki/Dependency_hell">RPM hell</a> for users if you try to use them generally. <br />
<i><br /></i>
<i>4. Custom routing layers.</i> Any interface that your applications use to access data can implement routing logic. For instance, SOAP or JSON servers work perfectly well for this purpose. Internal data access libraries can also implement routing. This approach is specific to single applications, and like library wrappers does not cover other means of access. </div>
</div>
</div>
<br />
The biggest constraint for any routing method is to keep it simple. Fat routing layers with complex logic tend to introduce bugs as well as change database semantics. This in turn violates the requirement for transparency. <br />
<br />
Fabrics use the preceding elements over and over. As we build up data fabric designs it is helpful to use consistent notation for data stores, replication, and routing. The following summarizes the main notation that you will see in diagrams.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDjfiyj_X7Lt1HBsycai1ajqMx_ZmzuanEK7sFeSK_7WB_s6LiTjhJw5NewIiVs9aXfhe8rMfATzbkakVmoDj-ijk3AJMQB8_udxYiJ9uKaAPbaWxP636qrUabJOy87he8F0Z0uJGQqao/s1600/Data-Fabric-Notation.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDjfiyj_X7Lt1HBsycai1ajqMx_ZmzuanEK7sFeSK_7WB_s6LiTjhJw5NewIiVs9aXfhe8rMfATzbkakVmoDj-ijk3AJMQB8_udxYiJ9uKaAPbaWxP636qrUabJOy87he8F0Z0uJGQqao/s400/Data-Fabric-Notation.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Data Fabric Notation</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
At this point we understand that data fabrics consist of a specific set of elements. However, we have not provided any organization yet. The next question is therefore how to arrange them into real systems.<br />
<br />
<b>What Are the Design Patterns for Data Fabrics?</b><br />
<br />
Having a model of fabric elements is not the same as an actual implementation. We need to define how the elements are composed into a complete system that can be deployed. This is where design patterns come in.<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
<a href="http://en.wikipedia.org/wiki/Design_pattern">Design patterns</a> are reusable solutions that combine flexibly with each other to create large-scale architectures. In the case of data fabrics, design patterns offer guidelines to organize fabric elements in ways that work well for large data sets spread over multiple geographic locations. They serve the same function as the geometric arrangements that organize individual motifs in an arabesque.<br />
<br />
Data fabric design patterns arrange <i>data services. </i> A data service is an abstraction for a transactional store with a well-defined API. Design patterns either link data services in some way or create more capable data services from simpler ones. The ability to create new services in this way is called <a href="http://en.wikipedia.org/wiki/Service_composability_principle">service composability</a>. Composability is a fundamental attribute of fabric design patterns. It is the reason that fabric designs can handle very large data sets flexibly. Service composition is analogous to the patterns-within-patterns layout that allows arabesques to cover an entire building without seeming repetitive or boring.<br />
<br />
There are six design patterns that are particularly useful for SQL databases. The following diagram shows them using <i><u><span class="Apple-style-span" style="color: blue;">italized blue font</span></u></i> and illustrates how they organize the implementation of our sample CRM system. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje8OXUThjwJHQ_-BGEycpUJxZbI1IgDLsKhrNgfETERW1AvWAFebmW9MXE3irOWgJsrhOuso-TLEkvHwlTtbmLkLf34xKierafHJjOzVTnlEuorioZVign9IpT3a-fdJX4GEhq_C7isng/s1600/Full-Data-Fabric.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje8OXUThjwJHQ_-BGEycpUJxZbI1IgDLsKhrNgfETERW1AvWAFebmW9MXE3irOWgJsrhOuso-TLEkvHwlTtbmLkLf34xKierafHJjOzVTnlEuorioZVign9IpT3a-fdJX4GEhq_C7isng/s640/Full-Data-Fabric.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fully Implemented Data Fabric</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div class="separator" style="clear: both; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; text-align: center;">
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<div class="separator" style="clear: both; text-align: center;">
</div>
Here is a short description of each fabric design pattern. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<ul>
<li><b>Transactional Data Service</b> - A transactional store that can restart without losing data following failure. It has a well-defined API, a mechanism for backup, and a transaction log for replication. This is the building block for all other design patterns. </li>
<li><b>Fabric Connector</b> -- A routing method that hides the location of databases and enables data services consisting of multiple DBMS servers to look like a single server to applications. </li>
<li><b>Fault-Tolerant Data Service</b> -- A data service that protects against failures and enables zero-downtime maintenance using a set of redundant DBMS servers. </li>
<li><b>Sharded Data Service</b> -- A data service that handles large data sets by dividing them into shards distributed across a number of underlying data services</li>
<li><b>Multi-Site Data Service</b> -- A data service that enables data to spread across multiple geographic locations using data services on each site linked by replication</li>
<li><b>Real-Time Data Bridge</b> -- A link that enables real-time replication of data between heterogeneous data services, for example between MySQL and a data warehouse like Vertica. </li>
</ul>
You can implement fabric design patterns in many different ways. This means there are many possible implementations of data fabrics as well. As long as you meet the assumptions of each particular design pattern, the result is likely to work well. We will delve into some of the variations in the next few articles.<br />
<br />
<b>A Little Bit about Naming</b><br />
<br />
You may have encountered the term "data fabric" in products (<a href="http://nysetechnologies.nyx.com/en/data-technology/data-fabric-6-0">example here</a> or <a href="http://www.tervela.com/tervela-data-fabric">here</a>) or as a concept (<a href="http://wiki.tangosol.com/display/COH32UG/Provide+a+Queryable+Data+Fabric">a nice example here</a>). This is because the ideas behind fabrics are very general and apply to many types of data management systems, perhaps most notably <a href="http://en.wikipedia.org/wiki/Storage_area_network">storage area networks</a>. The name data fabric is very natural and seems to have occurred to many people independently, especially after storage vendors popularized it during the 1990s. <br />
<br />
In our case the term "data fabric" always means a data fabric architecture as I defined it a couple of sections back. <br />
<b><br /></b>
<b>Conclusion and More</b><br />
<br />
This has been a longish introduction but introduces terminology and sets the stage for looking at specific design patterns used to build data fabrics. Follow-on articles will look at individual design patterns in detail. <br />
<br />
One final thought: there are no doubt other ways of factoring design patterns for data fabrics. This particular set of patterns works very well for SQL databases, and I have seen them used successfully in many large systems over the years. I would argue they work well for other DBMS types as well but welcome your comments on other approaches. <br />
<br />
p.s., The next article covering the <a href="http://scale-out-blog.blogspot.com/2013/02/data-fabric-design-patterns.html">transactional data service</a> is now available. </div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com8tag:blogger.com,1999:blog-768233104244702633.post-57762280942467857352013-01-13T23:19:00.003-08:002013-01-13T23:19:47.639-08:00Replicating from MySQL to Amazon RDSThere have been a number of comments that <a href="http://aws.amazon.com/rds/">Amazon RDS</a> does not allow users access to MySQL replication capabilities (for example <a href="https://forums.aws.amazon.com/thread.jspa?messageID=178291&#178291">here</a> and <a href="https://forums.aws.amazon.com/message.jspa?messageID=221905#221909">here</a>). This is a pity. Replication is one of the great strengths of MySQL and the lack of it is a show-stopper for many users. As of the latest build of <a href="http://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> half of this problem is on the way to being solved. You can now set up real-time replication from an external MySQL master into an Amazon RDS instance.<br />
<br />
In the remainder of this article I will explain how to set up Tungsten replication to an Amazon RDS slave, then add a few thoughts about why this feature is useful along with some suggestions for improvement. To keep the article reasonably short I assume you understand the basics of installing Tungsten Replicator. If you need more information, check out the <a href="https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Documentation+Home">online documentation</a>.<br />
<br />
<b>Readying an RDS Test Instance</b><br />
<br />
Amazon RDS is Amazon's on-demand relational database. RDS supports several database types including MySQL, which I will use for this demonstration. Launching a new instance is simple. Login to the <a href="https://console.aws.amazon.com/console/home">Amazon AWS Console</a> using an account that has RDS enabled, then switch to the <a href="https://console.aws.amazon.com/rds">Amazon RDS Console</a>. Press the Launch a DB Instance button, whereupon a screen like the following appears:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4h1YtBBMWJGS5iaF7x9d68BGXa8xiLtm8dYzfGQclXoQUvVTvMHXHzBpPckGkiG_4W8U7wj4FqvXRCO-4-oxvgpNlEHjrEo0i0C-0z4-CGlU7E9UYOE5y3K7ej3GpCZj6Ysk8HbR6bbA/s1600/select-a-database-aws.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4h1YtBBMWJGS5iaF7x9d68BGXa8xiLtm8dYzfGQclXoQUvVTvMHXHzBpPckGkiG_4W8U7wj4FqvXRCO-4-oxvgpNlEHjrEo0i0C-0z4-CGlU7E9UYOE5y3K7ej3GpCZj6Ysk8HbR6bbA/s640/select-a-database-aws.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">RDS Database Selection</td></tr>
</tbody></table>
Press the Select button for MySQL Community Edition, which starts the configuration window. (You can also replicate into Oracle if you are up for a challenge. If you do this, please post what you did to get it to work!) Next fill out properties for MySQL.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkaTXZbvhZkHDrLfR0N9uNtPUAIsTkrCJhyCPgpY3fwMUi9SjqzOjijpqAa9_MSa1nAkiD5aBcbc_-lJlRB5-GM-Cxe4H8Uuhlbz0I3EMrh34wMrHOnHNi3fXCfFFABCWUms5mqubVC5A/s1600/configure-instance-details.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkaTXZbvhZkHDrLfR0N9uNtPUAIsTkrCJhyCPgpY3fwMUi9SjqzOjijpqAa9_MSa1nAkiD5aBcbc_-lJlRB5-GM-Cxe4H8Uuhlbz0I3EMrh34wMrHOnHNi3fXCfFFABCWUms5mqubVC5A/s640/configure-instance-details.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">MySQL Instance Configuration</td></tr>
</tbody></table>
Among other things, you create a master login and password. Note these carefully as you will need them to configure replication. Then continue for another couple of screens, at which point you can launch your instance. It takes about 10 minutes for new instances to spin up, after which you can see the instance properties in the AWS RDS Console. Here's a screen shot of my test instance. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMY-NPq88Pr0diaovwrJNsGJuVvId9wA5xOc6_dK2ygCa8bHA4XWSU8-k5mphiNXZDL16B7Gr0Q0eOur9jRlpZ5T-0rzhUh8-3d6vCBUKKfwxDa8ON4057VNWu7voodTUq27hGsy-S1Y8/s1600/rds-management-console.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMY-NPq88Pr0diaovwrJNsGJuVvId9wA5xOc6_dK2ygCa8bHA4XWSU8-k5mphiNXZDL16B7Gr0Q0eOur9jRlpZ5T-0rzhUh8-3d6vCBUKKfwxDa8ON4057VNWu7voodTUq27hGsy-S1Y8/s640/rds-management-console.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">AWS RDS Console</td></tr>
</tbody></table>
Once the instance is up, test access. This is of course necessary to prove the instance is running properly and that we can login from a remote location. Note the host name in the Endpoint field. This is a DNS entry for the new MySQL instance. Using that and the master login, we can now fire up the mysql client from a remote host where we plan to run replication. <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ mysql -utungsten -p -htest.c4villnbpuq1.us-west-1.rds.amazonaws.com</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Enter password: </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Welcome to the MySQL monitor. Commands end with ; or \g.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Your MySQL connection id is 2125</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Server version: 5.5.27-log Source distribution</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> show databases;</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+--------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| Database |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+--------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| information_schema |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| innodb |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| mysql |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| performance_schema |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| test |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+--------------------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">5 rows in set (0.01 sec)</span><br />
<br />
This looks quite good. We are now ready to install and start replication. <br />
<br />
Important note! If you have trouble connecting you may need to tweak your Amazon security group settings to open up ports, especially if you are replicating from non-Amazon locations. Amazon has a very cool feature that can guess your originating host IP and offer a <a href="http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing">CIDR address</a> that covers the port range from which you are operating. I used this when configuring my security groups.<br />
<br />
<b>Setting Up Tungsten Replication</b><br />
<br />
It is possible to set up replication from a MySQL master directly to Amazon RDS using a single Tungsten Replicator process. However, it is more versatile and simpler to set up two replicators: one to read from the MySQL master, and another replicator to apply to the Amazon RDS slave. I will therefore demonstrate this configuration.<br />
<br />
We will assume you have a MySQL master already running and that it meets <a href="https://docs.continuent.com/wiki/display/TEDOC/System+Requirements">prerequisites for running Tungsten</a>. Let's now grab the Tungsten code and install a master replicator. You can get fresh builds from the <a href="http://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/index.html">Tungsten Replicator builds page</a>. <br />
<br />
We will take a recent replicator build that contains the RDS changes, which are documented in <a href="http://code.google.com/p/tungsten-replicator/issues/detail?id=425">Issue 425</a>. Use 2.0.7 build 177 or later. The main improvement is to add a non-privileged slave mode that avoids invoking any of the operations forbidden by Amazon. Among other things Tungsten normally uses commands like 'SET SESSION SQL_LOG_BIN=0' to suppress writing to the binlog when it applies data on a slave. This command requires SUPER privilege, hence causes problems for underprivileged RDS logins.<br />
<br />
Unpack the code and install the master replicator in /opt/continuent. This is no different from installing a normal Tungsten master. My master host is logos1. Here are sample commands to pull the code and set up the master. The example shows the minimum options--if you have MySQL installed in a non-standard location or otherwise differ from a stock installation you may need to add additional options.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mkdir ~/staging</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">cd staging</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">wget --no-check-certificate https://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/tungsten-replicator-2.0.7-177.tar.gz</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tar -xf </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tungsten-replicator-2.0.7-177.tar.gz</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tungsten-replicator-2.0.7-177/tools/tungsten-installer \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-slave \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-host=logos1 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-user=tungsten \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-password=your_passord \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --service-name=aws \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --home-directory=/opt/continuent \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --cluster-hosts=logos1 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --start-and-report</span><br />
<br />
Next, set up the slave replicator. For convenience I am going to install the slave on a separate host, named logos2, to avoid port clashes between the two replicators. If you install on the same host, you'll need to install into another release directory location and use the --rmi-port and --thl-port options to avoid port overlaps. Here is the command to set up the Amazon RDS slave. Note that the tungsten-installer program can install code between hosts, which is an extremely useful feature.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tungsten-replicator-2.0.7-177/tools</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/tungsten-installer \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-slave \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --cluster-hosts=logos2 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-host=logos1 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-host=test.c4villnbpuq1.us-west-1.rds.amazonaws.com \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-user=tungsten \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-password=your_password \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --service-name=aws \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --slave-privileged-updates=false \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --home-directory=/opt/continuent \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --skip-validation-check=InstallerMasterSlaveCheck \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --skip-validation-check=MySQLPermissionsCheck \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --start-and-report</span><br />
<br />
You may see a few warnings during the RDS installation, as the tungsten-installer cannot verify some settings on the Amazon RDS host. These can be ignored. If everything goes well, you now have two replicators up and running. You can check the status of the master and slave using the trepctl command, as in:<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/opt/continuent/tungsten/tungsten-replicator/bin/trepctl -host logos1 status</span></div>
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/opt/continuent/tungsten/tungsten-replicator/bin/trepctl -host logos2 status</span></div>
</div>
<div>
<br /></div>
Both replicators should report that they are online. Now complete the exercise by proving that replication works between the replicators. We start by logging into the local MySQL instance, creating a new table in the test schema, and adding data.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ mysql -uroot test</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Welcome to the MySQL monitor. Commands end with ; or \g.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Your MySQL connection id is 231488</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Server version: 5.5.21-rel25.1-log Percona Server with XtraDB (GPL), Release rel25.1, Revision 234</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> create table foo(id int primary key);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Query OK, 0 rows affected (0.24 sec)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> insert into foo values (256);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Query OK, 1 row affected (0.00 sec)</span><br />
<div>
<br /></div>
<div>
Now login to the Amazon RDS instance and look for table foo. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ mysql -utungsten -p -htest.c4villnbpuq1.us-west-1.rds.amazonaws.com test</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Enter password: </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Reading table information for completion of table and column names</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">You can turn off this feature to get a quicker startup with -A</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Welcome to the MySQL monitor. Commands end with ; or \g.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Your MySQL connection id is 2161</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Server version: 5.5.27-log Source distribution</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> select * from foo;</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+-----+</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| id |</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+-----+</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">| 256 |</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">+-----+</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">1 row in set (0.01 sec)</span></div>
</div>
<div>
<br /></div>
<div>
Mission accomplished! We have real-time replication enabled from a MySQL master to Amazon RDS. At this point you can replicate more or less normally. There are some obvious limitations due to the fact that Amazon RDS is locked down and does not grant our login full privileges. </div>
<div>
<ol>
<li>Temp table replication may not work. Tungsten depends on being able to issue commands of the form "set @@session.pseudo_thread_id=23531" and the like. This prevents clashes between temp tables of the same name on different master sessions. You may need to enable row replication on the master, which suppresses temp table replication. (For other approaches, see my previous <a href="http://scale-out-blog.blogspot.com/2012/04/replication-is-bad-for-mysql-temp.html">article on temp tables and the binlog</a>.) </li>
<li>Any command that requires SUPER privilege will not work. As an obvious example, you will not be able to grant SUPER privilege to new accounts. Such commands will break replication. </li>
<li>All replicated commands go into the binlog, which is a potential performance drag and may slow down Amazon RDS slaves. Parallel replication may not help in this case, since committing to the binlog is a serialization point that blocks other transactions. This problem may be cured if Amazon picks up group commit fixes from MySQL 5.6 and/or MariaDB. </li>
</ol>
</div>
All things considered, however, these are minor inconveniences. Most applications should be able to replicate without difficulties, especially if the master transaction rate is not too high. <br />
<br />
<b>Configuring SSL for Connections to RDS</b><br />
<br />
In the previous demonstration I used a master host running outside Amazon. This means my test transactions traveled across the Internet, where they were visible to all and sundry along the way. To illustrate, we can run tcpdump and watch traffic as it goes by. <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ sudo tcpdump -A -vvv -s 256 host test.c4villnbpuq1.us-west-1.rds.amazonaws.com</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">...</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">logos2.46657 > ec2-54-241-56-140.us-west-1.compute.amazonaws.com.mysql: Flags [P.], cksum 0xb0f7 (incorrect -> 0x995d), seq 3012:3073, ack 3412, win 94, options [nop,nop,TS val 89630514 ecr 74223746], length 61</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">E..qw>@.@......n6.8..A..4..L...f...^.......</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.W.2.l..9....<b>insert into foo values (256) /* ___SERVICE___ = [aws] */</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">22:55:36.316049 IP (tos 0x8, ttl 51, id 26226, offset 0, flags [DF], proto TCP (6), length 63)</span><br />
<br />
If we were handling confidential data, exposing traffic like this to possible evildoers would be a serious problem. Fortunately, Amazon RDS supports SSL encrypted connections from clients. Here is how to use it with Tungsten. <br />
<br />
First, you need to get the Amazon RDS certificate, which is used to sign certificates for individual RDS instances. <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mkdir /opt/continuent/certs</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">cd /opt/continuent/certs</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">wget https://rds.amazonaws.com/doc/mysql-ssl-ca-cert.pem</span><br />
<br />
Next, you need to create a trust store that Java can access containing the certificates of signing authorities whom you trust. For this you will need the <a href="http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/keytool.html">Java keytool utility</a>, which is included in the JDK. If you are just using the Java runtime in production, you will need to generate the store on another host, then copy it over to your test hosts. I used the password "secret" in this example. <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ keytool -import -alias rds -file mysql-ssl-ca-cert.pem -keystore truststore</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Enter keystore password: </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Re-enter new password: </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Owner: CN=aws.amazon.com/rds/, OU=RDS, O=Amazon.com, L=Seattle, ST=Washington, C=US</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Issuer: CN=aws.amazon.com/rds/, OU=RDS, O=Amazon.com, L=Seattle, ST=Washington, C=US</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">...</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Trust this certificate? [no]: yes</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Certificate was added to keystore</span><br />
<div>
<br /></div>
<div>
We now need to tell the slave replicator about the truststore file using Java VM options. On the slave host, edit /opt/continuent/tungsten/tungsten-replicator/conf/wrapper.conf and add the extra options shown in <b><u>bold face</u></b>. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Java Additional Parameters</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">wrapper.java.additional.1=-Dreplicator.home.dir=../../tungsten-replicator/</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">wrapper.java.additional.2=-Dreplicator.log.dir=../../tungsten-replicator/log</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">wrapper.java.additional.3=-Dcom.sun.management.jmxremote</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><u><b>wrapper.java.additional.4=-Djavax.net.ssl.trustStore=/opt/continuent/certs/truststore</b></u></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><u><b>wrapper.java.additional.5=-Djavax.net.ssl.trustStorePassword=secret</b></u></span></div>
</div>
<div>
<br /></div>
<div>
The last step is to enable SSL encryption when applying data. We need to set an extra URL option on the drizzle JDBC driver to turn on SSL. For this we need to edit the static-svc.properties file that configures replication. In my example this file is located in /opt/continuent/tungsten/tungsten-replicator/conf/static-aws.properties. Open the file and look for the section that starts with APPLIERS. Edit the text to add additional urlOptions line as shown below. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">############</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># APPLIERS #</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">############</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms=com.continuent.tungsten.replicator.applier.MySQLDrizzleApplier</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.host=${replicator.global.db.host}</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.port=${replicator.global.db.port}</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.user=${replicator.global.db.user}</span></div>
<div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.password=${replicator.global.db.password}</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b><u>replicator.applier.dbms.urlOptions=?useSSL=true</u></b></span></div>
</div>
</div>
</div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.ignoreSessionVars=autocommit</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">replicator.applier.dbms.getColumnMetadataFromDB=true</span></div>
</div>
<div>
<br /></div>
<div>
Restart the replicator process (/opt/continuent/tungsten/tungsten-replicator/bin/replicator restart) and you will now be using SSL encryption. If we now look back at the tcpdump outout, it looks like garbage as the following example shows. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">...</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">23:46:52.370904 IP (tos 0x0, ttl 64, id 5899, offset 0, flags [DF], proto TCP (6), length 105)</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> logos2.44717 > ec2-54-241-56-140.us-west-1.compute.amazonaws.com.mysql: Flags [P.], cksum 0xb0ef (incorrect -> 0xd834), seq 3087:3140, ack 4838, win 102, options [nop,nop,TS val 89938121 ecr 74992771], length 53</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">E..i..@.@.r....n6.8......zSR.b.....f.......</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.\X..xL.....07..|..x.)...T..888.H/...iz...^.W8....'......<span class="Apple-tab-span" style="white-space: pre;"> </span>..J</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">...</span></div>
<div>
</div>
</div>
<div>
<br /></div>
<div>
This is much better. We are replicating to an Amazon RDS, and the transactions are safe from prying eyes. If you have gotten this far you are ready to try your own applications. </div>
<div>
<br /></div>
<div>
<b>Benefits of Replication into Amazon RDS</b></div>
<div>
<br /></div>
<div>
Amazon RDS is convenient thanks to its quick and simple setup, but the lack of replication is a severe limitation in building systems that need more than a single MySQL instance. In particular it makes it hard to integrate RDS into systems that consist of more than Amazon RDS itself. Adding the ability to replicate in real-time into RDS therefore has a number of benefits. The most obvious include using RDS to extend existing systems.<br />
<br /></div>
<div>
First, Amazon RDS can offer a quick way to add read capacity to existing MySQL applications. This is especially useful if you have a cluster, such as Tungsten, which handles your transaction processing and overall HA. You can now add Amazon RDS read slaves that you discard when no longer needed. Tungsten Replicator has a number of other useful features like the ability to read from a group of nodes, not just one, that make such topologies easy to set up and maintain. Clusters other than Tungsten will likewise benefit from this feature. </div>
<div>
<br />
Second, Amazon RDS is suitable for applications that do not need 24x7 high availability (limitations include slow failover, no online maintenance, no cross-cloud capabilities, etc.) You can now pull data from other sources and send them to Amazon RDS slaves for processing, which amounts to extending overall processing capacity. For example, you could use RDS to run back-office tasks using transactions replicated in from MySQL masters. Tungsten Replicator also replicates data from Oracle, so this is an additional source of transactions. </div>
<div>
<br /></div>
<div>
There are of course other ways to replicate data to and from RDS, for example using batch ETL tools like <a href="http://www.talend.com/">Talend</a>. However, these are not real-time and often require application changes to add timestamp columns or otherwise mark transactions that need to be extracted. Log-based replication as implemented by Tungsten is fast and has minimal impact on applications or MySQL itself. </div>
<div>
<br /></div>
<div>
<b>Thoughts about Further Improvements for Amazon RDS Replication</b></div>
<div>
<br />
On our side, i.e., at Continuent, we need to do more testing, add documentation, and fix problems as they arise. We are starting a beta test with one of our customers in the next few days, who incidentally was the same customer who requested this feature in the first place after hacking it for themselves. RDS also has some interesting provisioning capabilities that I would like to understand better. We are also adding options that eliminate the need for manual configuration of security settings. This will keep us busy for a while.<br />
<br />
Other improvements depend on changes to RDS itself. An obvious and huge improvement would be to permit replication <u><i>out</i></u> of Amazon RDS. Unfortunately, Tungsten needs a login that has REPLICATION_SLAVE privilege so that we can download binlog data. That privilege is not yet available to Amazon users. Once it is, Tungsten extraction will also work in very short order. We actually don't need other commands like START/STOP SLAVE or FLUSH LOGS--just ability to issue a <a href="http://dev.mysql.com/doc/internals/en/replication-protocol.html#com-binlog-dump">COM_BINLOG_EVENT</a> from a client connection and receive binlog records. I am sure other products would use this capability as well. (RDS developers, if you are listening here's an easy way to extend your product usability significantly...)</div>
<div>
<br /></div>
<div>
Replication is such a valuable feature of MySQL that Amazon RDS feels somewhat crippled without it. For this reason I would imagine that Amazon <i>will</i> open up additional capabilities in the future. Until then we will polish up replication from MySQL masters to Amazon RDS slaves, awaiting a time when we can add more features. </div>
<div>
<br /></div>
<div>
In the meantime, I hope you will try the new replication to Amazon RDS. As noted in this article you can grab the latest builds and try it yourself. Please report your experiences through either Continuent Support if you are a customer or the <a href="http://groups.google.com/group/tungsten-replicator-discuss">Tungsten discussion list</a> if you use the <a href="http://code.google.com/p/tungsten-replicator/">open source Tungsten Replicator</a>. I look forward to your feedback and suggestions for making Amazon RDS support better. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com19tag:blogger.com,1999:blog-768233104244702633.post-60435770298512012582013-01-09T22:59:00.000-08:002013-01-09T23:01:34.016-08:00Tungsten UniversityWe have started a new series of webinars at <a href="http://www.continuent.com/">Continuent</a> that we call Tungsten University. They provide education on <a href="http://www.continuent.com/solutions">Tungsten clustering and replication</a> in handy one-hour chunks. These are not sales pitches. Our goal is to provide accessible education about setting up and operating Tungsten without any marketing fluff. <br />
<div>
<br />
<div>
The first Tungsten University webinar entitled "<a href="http://www.continuent.com/news/live-webinars">Configure & provision Tungsten clusters</a>" will take place on Thursday January 17th at 10:00 PST. It will show you how to set up a cluster in Amazon EC2. There will be a repeat on January 22nd at 15:00 GMT. We usually record webinars, so you can look at them later as well. </div>
</div>
<div>
<br /></div>
<div>
You do not have to be a customer to attend these webinars, just interested in Tungsten. I hope users of our <a href="http://code.google.com/p/tungsten-replicator/">open source Tungsten Replicator</a> will attend, since we will have a number of presentations on replication. Here are some of the future webinar topics we are considering: </div>
<div>
<ul>
<li>Setting up, deploying, and upgrading Tungsten Replicator </li>
<li>Setting up multi-master and fan-in replication topologies</li>
<li>Configuring Tungsten Connector for transparent SQL routing and load balancing</li>
<li>Replication tips and tricks (such as how to improve performance and fix broken replicators)</li>
<li>Implementing zero-downtime maintenance and schema upgrade</li>
<li>Replicating between MySQL and Oracle</li>
<li>Loading MySQL data into a Vertica data warehouse</li>
</ul>
</div>
<div>
There will be more titles out soon, so watch our <a href="http://www.continuent.com/news/live-webinars">webinar list</a> and the announcements on the <a href="http://continuent-tungsten.blogspot.com/">Continuent Tungsten</a> blog. If there are topics that you would like to hear about please suggest them as comments on this blog or the official Continuent blog. </div>
<div>
<br /></div>
<div>
Meanwhile, I am looking forward to doing some of the replicator presentations and attending the talks on clustering. I mostly write code for the replicator, so talks about other parts of Tungsten tend to be learning experiences. The <a href="https://docs.continuent.com/wiki/display/TEDOC/Using+the+Tungsten+Connector">Tungsten Connector</a> is particularly interesting because it makes off-the-shelf database replicas look like a single DBMS server and transparently switches connections between them. It is our secret sauce for creating master/slave clusters. If you have not seen Tungsten Connector before, I recommend attending that talk when it comes up. I'm still amazed how well it functions even after working with it for a number of years. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-60113444877621435082013-01-01T14:58:00.002-08:002013-01-01T14:58:45.071-08:00Questions about MariaDB JDBC DriverThe <a href="http://blog.mariadb.org/monty-program-skysql-release-the-mariadb-client-library-for-c-and-mariadb-client-library-for-java-applications/">recent release of the MariaDB client libraries</a> has prompted <a href="http://www.xaprb.com/blog/2012/12/26/the-state-of-mysql-client-libraries/">questions</a> about their purpose as well as provenance. Colin Charles posted that <a href="http://www.bytebot.net/blog/archives/2012/12/31/a-few-great-weeks-for-mariadb">some of these would be answered in the very near future</a>. I have a couple of specific questions about the MariaDB JDBC driver, which I hope will be addressed at that time. <div>
<br /></div>
<div>
<div>
1.) <u>What is really in the MariaDB JDBC driver and how exactly does it differ from the drizzle JDBC driver?</u> What, if any, relation is there to Connector/J code? There is a <a href="https://mariadb.atlassian.net/browse/CONJ">JIRA project</a> but it contains only four bugs, hence is not very informative. The <a href="http://bazaar.launchpad.net/~maria-captains/mariadb-java-client/trunk/files">launchpad bzr history</a> shows detailed check-ins but not overall intent. </div>
<div>
<br /></div>
<div>
2.) <u>Why relicense from BSD to LGPL?</u> I have checked the class headers and so far as attributions are concerned everything seems to be done quite properly. However, the license change appears to prevent those of us currently using the drizzle JDBC driver from transferring code changes back to the drizzle driver. If so, that seems a little unneighborly. </div>
<div>
<br /></div>
<div>
<div>
Here is some background on the relationship between the drivers. The <a href="https://launchpad.net/mariadb-java-client">MariaDB JDBC client</a> is a fork of the <a href="https://github.com/krummas/DrizzleJDBC">BSD-licensed drizzle JDBC driver</a> originally developed by <a href="http://developian.blogspot.com/">Marcus Eriksson</a>, who continues to maintain the code. According to the bzr change history the code forked after <a href="http://bazaar.launchpad.net/~maria-captains/mariadb-java-client/trunk/revision/253">rev 253</a>, which was 24 April 2011. There are still many similarities in the Java classes. For instance, a number of classes in the org.mariadb.jdbc.internal.common package differ by little other than licensing headers and package names. The MariaDB code is now up to rev 375 and includes substantial changes that appear to be designed to bring the MariaDB JDBC driver closer to the capabilities of the <a href="http://dev.mysql.com/downloads/connector/j/">MySQL Connector/J driver</a>. </div>
</div>
<div>
<br /></div>
<div>
At <a href="http://www.continuent.com/">Continuent</a> we have a lively interest in the drizzle JDBC driver, as we adopted it for <a href="http://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> some time ago. The code had fewer bugs than Connector/J, which was attractive. More importantly, Marcus kindly accepted a patch from my colleague Stephane Giron (working as a Continuent employee) that made it easy for us to send queries using binary data rather than the usual Unicode data required by the JDBC standard. This fix allows Tungsten to replicate codesets and binary data correctly. We have since contributed a few other patches. Our modest contribution in part reflects the quality of the base code. </div>
<div>
<br /></div>
<div>
<div>
While waiting for answers I would like to commend Marcus as well as other drizzle contributors for their work. We are particularly indebted to Marcus for starting and continuing the drizzle JDBC project. Tungsten Replicator users have applied many trillions of transactions using the drizzle driver. If the MariaDB JDBC driver gains wide acceptance, the rest of the MySQL community owes Marcus Eriksson substantial thanks as well. </div>
</div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com2tag:blogger.com,1999:blog-768233104244702633.post-7768615589760462582012-12-27T09:25:00.002-08:002012-12-27T09:25:17.423-08:00The MySQL Community: Beleaguered or Better than Ever?<div>
The <a href="https://blog.mariadb.org/mariadb-foundation-to-safeguard-leading-open-source-database/">MariaDB Foundation</a> announcement spawned some interesting commentary about the state of open source databases. One recent headline cited the "<a href="http://www.infoworld.com/d/open-source-software/the-mariadb-foundation-turning-point-mysql-209168">beleaguered MySQL community</a>." Beleaguered is a delightful adjective. The OED tells us that it means beset, invested, or besieged. Much as I like the word, I do not think it is an accurate or useful description of the MySQL community. This article and <a href="http://gigaom.com/cloud/facebook-trapped-in-mysql-fate-worse-than-death/">others like it</a> miss the point of what is happening to MySQL and its users.<br />
<br /></div>
<div>
Let's start by disproving that the notion that the MySQL community is beleaguered. I don't know everyone who uses MySQL, but in my job I talk to numerous companies that have made sizable investments in MySQL and stand to lose big if they are wrong. They do not seem especially nervous.<br />
<br /></div>
<div>
1. <b>Nobody seriously questions MySQL viability.</b> I have yet to meet a manager with a substantial business on MySQL who is deeply worried about it disappearing or being ruined by Oracle. They are too busy working on software upgrades or keeping their sites running. The future of MySQL is well down the list of problems keeping them awake at night. </div>
<div>
<br /></div>
<div>
2. <b>MySQL meets or beats the immediate alternatives. </b> There is of course discussion about dropping MySQL for PostgreSQL but it is mostly <a href="http://developers.slashdot.org/story/12/08/18/0152237/is-mysql-slowly-turning-closed-source">idle talk</a>. I'm sure some companies have switched (actually in both directions), but I not seen a single customer migrate a working business app from MySQL to PostgreSQL. Once you get past the religion, it's clear MySQL and PostgreSQL are just too similar to supplant each other easily: reliable, row-based stores with single threaded SQL query engines that handle a few terabytes of data at most. Companies need far stronger reasons to switch to something new, especially given the large ecosystem and deep pool of MySQL expertise. </div>
<div>
<br />
3. <b>MySQL is not the only game in town.</b> Virtually every large web site I know uses at least one NoSQL store alongside MySQL. Column stores are increasingly common for data warehouses. Production Hadoop clusters are no longer a novelty. On the surface this might look like a failure of MySQL. What's really happening in many cases is that small businesses that started on MySQL are now large, profitable enterprises that require more than just economical OLTP. This is a mark of success, not a deficiency. <br />
<br />
If this is what beleaguered looks like I can't wait to see something that's actually successful. <br />
<br />
Turning the argument around, can we say that the MySQL community is better than it was? In at least one important way, yes. The community is now <u>multi-polar</u>. MySQL long benefitted from having a large community of open source users to find bugs, help focus development direction, and construct a wide range of robust tools like language bindings. However, innovation on MySQL itself was largely gated by a single company: MySQL AB. Multiple groups are now competing to improve MySQL, and it's a very good thing for users. Let me count the ways. <br />
<br />
There are three major versions of MySQL: <a href="http://www.mysql.com/">Oracle</a>, <a href="http://www.percona.com/software/percona-server">Percona</a>, and <a href="https://mariadb.org/">MariaDB</a>, not to mention cloud-only versions like <a href="http://aws.amazon.com/rds/">Amazon RDS</a>. There are at least four companies working directly on major upgrades to replication: <a href="https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Replication+Guide">Continuent</a>, <a href="http://dev.mysql.com/tech-resources/articles/mysql-5.6-replication.html">Oracle</a>, <a href="http://codership.com/">Codership</a>, and <a href="https://kb.askmonty.org/en/multi-source-replication/">Monty Program</a>. Oracle is continuing to make improvements in InnoDB like <a href="http://code.openark.org/blog/mysql/state-of-inndb-online-ddl-in-mysql-5-6-9-rc-good-news-included">online schema change</a> and <a href="http://www.blogger.com/"><span id="goog_407978987"></span>multi-core scaling<span id="goog_407978988"></span></a>, efforts that are complemented by Percona's <a href="http://www.mysqlperformanceblog.com/">persistent focus on all aspect of performance</a>. Aside from Amazon RDS, all of this work is available in open source, and there is an unusual degree of sharing across otherwise competitive groups. I could keep going for a while but to be frank there's so much it's hard to track all the improvements or give them their proper due.<br />
<br />
The MySQL community is therefore competitive in a way that did not exist a few years ago. That's good, because innovation in data management is no longer centered around the web-facing applications that MySQL helped enable. Businesses are grappling with massive data volumes that far exceed the capacity of single DBMS servers while simultaneously moving to Amazon or VMWare. There is a whole new set of problems such as deploying in unstable cloud environments, adjusting to <a href="http://martinfowler.com/articles/nosql-intro.pdf">polyglot persistence</a>, managing sharded data effectively, distributing data across multiple regions, and enabling real-time analytics on MySQL transactions. As a group, the MySQL community is well-positioned to address them. <br />
<br />
If there is a problem, it is how to keep a strong multi-polar community going for as long as possible. Competition creates uncertainty for users, because change is a given. <a href="http://en.wikipedia.org/wiki/Pointy-haired_Boss">Pointy-haired bosses</a> have to make decisions with incomplete information or even reverse them later. Competition is hard for vendors, because it is more difficult to make money in efficient markets. Competition even strikes against the vanity of community contributors, who have to try harder to get recognition. It is clear there will be pressures to make the community less competitive. They won't necessarily be from Oracle, which thrives on competition.<br />
<br /></div>
<div>
This gets back to the MariaDB Foundation reference that started this article. Anything that ensures long-term competitiveness and vitality of MySQL is good. Foundations in general seem well suited to this task. At Continuent we have already had some discussions about joining. So far we are undecided, for reasons that are somewhat similar to <a href="http://www.mysqlperformanceblog.com/2012/12/19/percona-and-the-mariadb-foundation/">Peter Zaitsev's comments on this subject</a>. If the MariaDB Foundation helps maintain a stable multi-polar community, we're in. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com11tag:blogger.com,1999:blog-768233104244702633.post-2715466543073633462012-12-10T18:18:00.000-08:002012-12-10T18:18:20.814-08:00Slides from Percona Live London and a RequestPercona hosted another excellent <a href="http://www.percona.com/live/london-2012/">Percona Live conference this past December 3-4</a> in London. It was my pleasure to deliver 3 talks including the first keynote following Peter Zaitsev. Percona does a great job of organizing these conferences--this year's London conference was well attended and in an excellent location in Kensington. My thanks to the entire Percona team for putting this together.<br />
<br />
Here are the slides for my talks in case you would like to see them.<br />
<br />
Keynote: <a href="https://continuent.box.com/s/a2rsn68rggu6snak4fxn">Future-Proofing MySQL for the World-Wide Data Revolution</a> -- Covering the greatly exaggerated death of MySQL and design patterns for robust MySQL systems that can last for decades<br />
<br />
Talk: <a href="https://continuent.box.com/s/vhvp15jkfy1ckpxds798">Why, What, and How of Data Warehouses for MySQL</a> -- Why you need a data warehouse for MySQL, some standard choices, and how to move data in real time from MySQL to Vertica. There was even a demo of sysbench data replicating into Vertica automatically. <br />
<br />
Talk: <a href="https://continuent.box.com/s/xhnn9n9l0d0557da4464">Multi-Master, Multi-Site MySQL Databases Made Easy with Continuent Tungsten</a> -- How to build clusters that span multiple sites using multi-master and primary/DR techniques. I did this with <a href="http://datacharmer.blogspot.com/">Giuseppe Maxia</a>, who did a couple of great demos along the way, including one that I found kind of terrifying to do in front of an audience. (It involved killing a lot of database servers, something Giuseppe does for a living.)<br />
<br />
Speaking of talks, there are many conferences with database tracks coming up in the next few months. If you have not done a talk on MySQL before I would encourage you to think about submitting for upcoming conferences like future <a href="http://www.percona.com/live/conferences">Percona Live events</a>, <a href="http://www.oscon.com/oscon2012">OSCON</a>, <a href="http://strataconf.com/strata2013">O'Reilly Strata</a>, or one of the many local meet-ups. There are many people doing interesting things with open source databases. It's great to hear your stories, so speak up!Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-2379112238587917892012-09-21T11:47:00.000-07:002012-09-21T13:04:19.850-07:00Data Fabrics and Other Tales: Percona Live and MySQL ConnectThe fall conference season is starting. I will be doing a number of talks including a keynote on <a href="http://www.percona.com/live/nyc-2012/sessions/future-proofing-mysql-world-wide-data-revolution">"future proofing" MySQL through the use of data fabrics</a>. Data fabrics allow you to build durable, long-lasting systems that take advantage of MySQL's strengths today but also evolve to solve future problems using fast-changing cloud and big data technologies. The talk brings together ideas that Ed Archibald (our CTO) and I have been working on for over two decades. I'm looking forward to rolling them out to a larger crowd. <br />
<br />
Here are the talks in calendar order. The first two are at <a href="http://www.oracle.com/mysqlconnect/learn/agenda/index.html">MySQL Connect 2012</a> in San Francisco on September 30th: <br />
<ul>
<li><a href="https://oracleus.activeevents.com/connect/sessionDetail.ww?SESSION_ID=11591">CON11591 - Managing Worldwide Data with MySQL and Continuent Tungsten</a> (Tech talk)</li>
<li><a href="https://oracleus.activeevents.com/connect/sessionDetail.ww?SESSION_ID=9319">CON9319 - Replicating from MySQL to Oracle Database and Back Again</a> (Tech talk)</li>
</ul>
<a href="http://www.oracle.com/mysqlconnect/index.html">MySQL Connect</a> is an add-on to Oracle Open World. You know the conference is big if they have to use 5-digit codes to keep track of talk titles. It's almost worth the price of admission <a href="http://oracle-magician.blogspot.com/2011/10/my-review-oracle-openworld-2011.html">to look at Larry Ellison's boat</a>. Well maybe not quite, but you get the idea.<br />
<br />
Next up is the <a href="http://www.percona.com/live/nyc-2012/">Percona Live MySQL Conference in New York</a> on October 2nd:
<br />
<ul>
<li><a href="http://www.percona.com/live/nyc-2012/sessions/future-proofing-mysql-world-wide-data-revolution">Future-Proofing MySQL for the Worldwide Data Revolution</a> (Keynote) </li>
<li><a href="http://www.percona.com/live/nyc-2012/sessions/solving-large-scale-database-administration-continuent-tungsten">Solving Large-Scale Database Administration with Tungsten</a> (Tech talk with Neil Armitage) </li>
</ul>
Percona has been doing an amazing job of organizing conferences. This will be my fifth one. The previous four were great. If you are in the New York area and like MySQL, sign up now. This conference is the single best way to get up to speed on state-of-the-art MySQL usage.Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com0tag:blogger.com,1999:blog-768233104244702633.post-27767299938321949952012-09-19T16:09:00.001-07:002012-09-19T16:09:42.912-07:00Database Failure Is Not the Biggest Availability ProblemThere have been a number of excellent articles about the pros and cons of automatic database failover triggered by <a href="http://www.xaprb.com/blog/2012/09/17/is-automated-failover-the-root-of-all-evil/">Baron's post</a> on the <a href="https://github.com/blog/1261-github-availability-this-week">GitHub database outage</a>. In the spirit of Peter Zaitsev's article "<a href="http://www.mysqlperformanceblog.com/2012/09/18/the-math-of-automated-failover/">The Math of Automated Failover</a>," it seems like a good time to point out that database failure is usually not the biggest source of downtime for websites or indeed applications in general. The real culprit is maintenance. <br />
<div>
<br /></div>
<div>
Here is a simple table showing availability numbers out to 5 nines and what they mean in terms of monthly down-time. <br />
<br />
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>JA</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";
mso-fareast-language:JA;}
</style>
<![endif]-->
<!--StartFragment-->
<table border="0" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="border-collapse: collapse; margin-left: -1.25pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184; width: 258px;">
<tbody>
<tr style="height: 15.0pt; mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td nowrap="" style="background: black; border-bottom: solid windowtext 1.0pt; border-left: solid black 1.0pt; border-right: none; border-top: solid black 1.0pt; height: 15.0pt; mso-border-bottom-alt: solid windowtext .5pt; mso-border-left-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<b><span style="color: white; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">Uptime<o:p></o:p></span></b></div>
</td>
<td nowrap="" style="background: black; border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: solid black 1.0pt; height: 15.0pt; mso-border-bottom-alt: solid windowtext .5pt; mso-border-right-alt: solid black .5pt; mso-border-top-alt: solid black .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<b><span style="color: white; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">Downtime per 30-Day Month<o:p></o:p></span></b></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 1;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">0.9<o:p></o:p></span></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">3 days<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 2;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">0.99<o:p></o:p></span></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">07:12:00<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 3;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">0.999<o:p></o:p></span></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">00:43:12<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 4;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">0.9999<o:p></o:p></span></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">00:04:20<o:p></o:p></span></div>
</td>
</tr>
<tr style="height: 15.0pt; mso-yfti-irow: 5; mso-yfti-lastrow: yes;">
<td nowrap="" style="border-top: none; border: solid windowtext 1.0pt; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 84.0pt;" valign="bottom" width="84">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">0.99999<o:p></o:p></span></div>
</td>
<td nowrap="" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; height: 15.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 174.0pt;" valign="bottom" width="174">
<div align="right" class="MsoNormal" style="text-align: right;">
<span style="color: black; font-family: Calibri; mso-fareast-font-family: "Times New Roman";">00:00:26<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Now let's do some math. We start with Peter's suggested number that the DBMS fails once a year. Let's also say you take a while to wake up (because it's the middle of the night and you don't like automatic failover), figure out what happened, and run your failover procedure. You are back online in an hour. Amortized over the year an hour of downtime is 5 minutes per month. Overall availability is close to 99.99% or four nines. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Five minutes per month is small potatoes compared to the time for planned maintenance. Let's say you allow yourself a one-hour maintenance window each month for DBMS schema changes, database version upgrades, and other work that takes the DBMS fully offline from applications. Real availability in this simple (and conservative) example is well below 99.9% or less than three nines. Maintenance accounts for over 90% of the downtime. The real key to improved availability is to be able to maintain the DBMS without taking applications offline. </div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
We have been very focused on the maintenance problem in Tungsten. Database replication is a good start for enabling rolling maintenance where you work on one replica at a time. In Tungsten the magic sauce is an intervening connectivity layer that can transparently switch connections between DBMS servers while applications are running. You can take DBMS servers offline and upgrade safely without bothering users. Managing planned failover in this way is easier to solve than providing bombproof automatic failover, I am happy to say. It is also considerably more valuable for many users. </div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com4tag:blogger.com,1999:blog-768233104244702633.post-84873253532500513212012-09-17T22:13:00.000-07:002012-09-17T22:22:35.175-07:00Automated Database Failover Is Weird but not EvilGithub had a <a href="https://github.com/blog/1261-github-availability-this-week">recent outage</a> due to malfunctioning automatic MySQL failover. Having worked on this problem for several years I felt sympathy but not much need to comment. Then Baron Schwartz wrote a short post entitled "<a href="http://www.xaprb.com/blog/2012/09/17/is-automated-failover-the-root-of-all-evil/">Is automated failover the root of all evil?</a>" OK, that seems worth a comment: it's not. Great title, though.<br />
<div>
<br />
<div>
Selecting automated database failover involves a trade-off between keeping your site up 24x7 and making things worse by having software do the thinking when humans are not around. When comparing outcomes of wetware vs. software it is worth remembering that humans are not at their best when woken up at 3:30am. Humans go on vacations, or their cell phones run out of power. Humans can commit devastating unforced errors due to inexperience. For these and other reasons, automated failover is the right choice for many sites even if it is not perfect. </div>
<div>
<br /></div>
<div>
Speaking of perfection, it is common to hear claims that automated database failover can never be trusted (such as <a href="http://support.rightscale.com/06-FAQs/FAQ_0059_-_How_do_I_set_up_autofailover_on_MySQL_databases%3F">this example</a>). For the most part such claims apply to particular implementations, not database failover in general. Even so, it is undoubtedly true that failover is complex and hard to get right. Here is a short list of things I have learned about failover from working on <a href="http://www.continuent.com/solutions/overview">Tungsten</a> and how I learned them. Tungsten clusters are master/slave, but you would probably derive similar lessons from most other types of clusters. </div>
<div>
<br /></div>
<div>
1. <b>Fail over once and only once</b>. Tungsten does so by electing a coordinator that makes decisions for the whole cluster. There are other approaches, but you need an algorithm that is provably sound. Good clusters stop when they cannot maintain the pre-conditions required for soundness, to which Baron's article alludes. (We got this right more or less from the start through work on other systems and reading lots of books about distributed systems.) </div>
<div>
<br /></div>
<div>
2. <b>Do not fail over unless the DBMS is truly dead</b>. The single best criterion for failure seems to be whether the DBMS server will accept a TCP/IP connection. Tests that look for higher brain function, such as running a SELECT, tend to generate false positives due to transient load problems like running out of connections or slow server responses. Failing over due to load is very bad as it can take down the entire cluster in sequence as load shifts to the remaining hosts. (We learned this through trial and error.) </div>
<div>
<br /></div>
<div>
3.<b> Stop if failover will not work or better yet don't even start.</b> For example, Tungsten will not fail over if it does not have up-to-date slaves available. Tungsten will also try to get back to the original pre-failover state when failover fails, though that does not always work. We get credit for trying, I suppose. (We also learned this through trial and error.) </div>
<div>
<br /></div>
<div>
4. <b>Keep it simple.</b> People often ask why Tungsten does not resubmit transactions that are in-flight when a master failover occurs. The reason is that there are many reasons why resubmission does not work on a new master and it is difficult to predict when such failures will occur. Imagine you were dependent on a temp table, for example. Resubmitting just creates more ways for failover to fail. Tungsten therefore lets connections break and puts the responsibility on apps to retry failed transactions. (We learned this from previous products that did not work.) </div>
<div>
<br /></div>
<div>
Even if you start out with such principles firmly in mind, new failover mechanisms tend to encounter a lot of bugs. They are hard to find and fix because failover is not easy to test. Yet the real obstacle to getting automated failover right is not so much bugs but the unexpected nature of the problems clusters encounter. There is a great quote from <a href="http://en.wikipedia.org/wiki/J._B._S._Haldane">J.B.S. Haldane</a> about the nature of the universe that also gives a flavor of the mind-bending nature of distributed programming: </div>
<blockquote class="tr_bq">
<i>My own suspicion is that the universe is not only queerer than we suppose, but queerer than we can suppose. </i></blockquote>
<div>
I can't count the number of times where something misbehaved in a way that would never have occurred to me without seeing it happen in a live system. That is why mature clustering products can be pretty good while young ones, however well-designed, are not. The problem space is just strange. </div>
<div>
<br /></div>
<div>
My sympathy for the Github failures and everyone involved is therefore heartfelt. Anyone who has worked on failover knows the guilt of failing to anticipate problems as well as as the sense of enlightenment that comes from understanding why they occur. Automated failover is not evil. But it is definitely weird. </div>
</div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com2tag:blogger.com,1999:blog-768233104244702633.post-7219357862936845332012-09-03T11:12:00.000-07:002012-09-03T11:12:04.987-07:00Life in the Amazon JungleIn late 2011 I attended a lecture by <a href="http://www.e-wilkes.com/john/work.html">John Wilkes</a> on Google compute clusters, which link thousands of commodity computers into huge task processing systems. At this scale hardware faults are common. Google puts a lot of effort into making failures harmless by managing hardware efficiently and using fault-tolerant application programming models. This is not just good for application up-time. It also allows Google to operate on cheaper hardware with higher failure rates, hence offers a competitive advantage in data center operation.<br />
<div>
<br /></div>
<div>
It's becoming apparent we all have to think like Google to run applications successfully in the cloud. At <a href="http://www.continuent.com/">Continuent</a> we run our IT and an increasing amount of QA and development on <a href="http://aws.amazon.com/">Amazon Web Services (AWS)</a>. During the months of July and August 2012 at least 3 of our EC2 instances were decommissioned or otherwise degraded due to hardware problems. One of the instances hosted our main website <a href="http://www.continuent.com/">www.continuent.com</a>. <br />
<br />
In Amazon failures are common and may occur with no warning. You have minimal ability to avoid them or some cases even understand the root causes. To survive in this environment, applications need to obey a new law of the jungle. Here are the rules as I understand them. </div>
<div>
<br /></div>
<div>
First, <u>build clusters of redundant services</u>. The www.continuent.com failure brought our site down for a couple of hours until we could switch to a backup instance. Redundant means up and ready to handle traffic now, not after a bridge call to decide what to do. We protect our MySQL servers by replicating data cross-region using Tungsten, but the website is an Apache server that runs on a separate EC2 instance. Lesson learned. Make <i>everything</i> a cluster and load balance traffic onto individual services so applications do not have to do anything special to connect. </div>
<div>
<br /></div>
<div>
Second, <u>make applications fault-tolerant</u>. Remote services can fail outright, respond very slowly, or hang. To live through these problems apply time-honored methods to create loosely coupled systems that degrade gracefully during service outages and repair themselves automatically when service returns. Here are some of my favorites. </div>
<div>
<div>
<ol>
<li>If your application has a feature that depends on a remote service and that service fails, turn off the feature but don't stop the whole application. </li>
<li>Partition features so that your applications operate where possible on data copies. Learn how to build caches that do not have distributed consistency problems. </li>
<li>Substitute message queues for synchronous network calls. </li>
<li>Set timeouts on network calls to prevent applications from hanging indefinitely. In Java you usually do this by putting the calls in a separate thread. </li>
<li>Use thread pools to limit calls to remote services so that your application does not explode when those services are unavailable or fail to respond quickly. </li>
<li>Add auto-retry so that applications reconnect to services when they are available again. </li>
<li>Add catch-all exception handling to deal with unexpected errors from failed services. In Java this means catching RuntimeException or even Throwable to ensure it gets properly handled. </li>
<li>Build in monitoring to report problems quickly and help you understand failures you have not previously seen. </li>
</ol>
</div>
</div>
<div>
Third, <u>revel in failure</u>. Netflix takes this to an extreme with <a href="http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html">Chaos Monkey</a>, which introduces failures in running systems. Another approach is to build scaffolding into applications so operations fail randomly. We use that technique (among others) to test clusters. In deployed database clusters I like to check regularly that any node can become the master and that you can recover any failed node. However, this is just the beginning. There are many, many ways that things can fail. It is better to provoke the problems yourself than have them occur for the first time when something bad happens in Amazon. </div>
<div>
<br /></div>
<div>
There is nothing new about these suggestions. That said, the Netflix approach exposes the difference between cloud operation and traditional enterprise computing. If you play the game applications will stay up 24x7 in this rougher landscape and you can tap into the flexible cost structure and rapid scaling of Amazon. The shift feels similar to using database transactions or eliminating GOTO statements--just something we all need to do in order to build better systems. There are big benefits to running in the cloud but you really need to step up your game to get them. </div>
Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com2tag:blogger.com,1999:blog-768233104244702633.post-38101852616679187562012-08-05T14:27:00.002-07:002012-08-05T14:27:40.818-07:00Is Synchronous Data Replication over WAN Really a Viable Strategy?Synchronous data replication over long distances has the sort of seductive appeal that often characterizes bad ideas. Why wouldn't you want every local credit card transaction simultaneously stored on the other side of the planet far away from earthquake, storms and human foolishness? The answer is simple: conventional SQL applications interact poorly with synchronous replication over wide area networks (WANs). <br />
<br />
I spent a couple of years down the synchronous replication rabbit hole in an earlier <a href="http://www.continuent.com/">Continuent</a> product. It was one of those experiences that make you a sadder but wiser person. This article digs into some of the problems with synchronous replication and shows why another approach, asynchronous multi-master replication, is currently a better way to manage databases connected by long-haul networks. <br />
<br />
<b>Synchronous Replication between Sites</b><br />
<br />
The most obvious problem with any form of synchronous replication is the hit on application performance. Every commit requires a round-trip to transfer and acknowledge receipt at the remote site, which in turn reduces single-thread transaction rates to the number of pings per second between sites. As a <a href="http://blog.9minutesnooze.com/performance-mysql-replication-high-latency/">nice article by Aaron Brown demonstrates</a>, you can show the effect easily using <a href="http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html">MySQL semi-synchronous replication</a> between hosts in Amazon regions. Aaron's experiment measured 11.5 transactions per second, or about <u style="font-style: italic;">100 times less</u> than single-thread performance on a local master between hosts with 85 millisecond latency. At that rate you would theoretically expect transaction throughput of ~11.7 transactions per second (1000 / 85 = 11.7), so the agreement between practice and theory is very close. It's great when science works out like this. <br />
<br />
You might argue that applications could tolerate the slow rate assuming it were at least constant. Sadly that's not the case for real systems. Network response varies enormously between sites in ways that are quite easy to demonstrate. <br />
<br />
To illustrate variability I set up an Amazon m1.small instance in the us-east-1 region (Virginia) and ran 24 hours of ping tests to instances in us-west-2 (Oregon) and ap-southeast-1 (Singapore). As the following graph shows, during a 4 hour period ping times to Singapore remain within a band but vary up to 10%. Ping times to Oregon on the other hand hover around 100ms but spike up randomly to almost 200ms. During these times, synchronous replication throughput would be cut by 50% to approximately 5 transactions per second. <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6OlcQCOwmDzrzWeSW8V0mBZsPNkExRlRCk5uQuM6I__tJANoW4rEeyoWf7bhg5pxhnLTObgnqW-Yq-3mvtohqcfDbOiZHdklLoXTpkRw2hHpnOp0uYVXrXNS3JgsPCaD8-uVND9bsfy4/s1600/Amazon-ping-times-from-us-east.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="299" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6OlcQCOwmDzrzWeSW8V0mBZsPNkExRlRCk5uQuM6I__tJANoW4rEeyoWf7bhg5pxhnLTObgnqW-Yq-3mvtohqcfDbOiZHdklLoXTpkRw2hHpnOp0uYVXrXNS3JgsPCaD8-uVND9bsfy4/s640/Amazon-ping-times-from-us-east.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Amazon ping times from us-east-1 to us-west-2 and ap-southeast-1 (240 minute interval)</td></tr>
</tbody></table>
Moreover, it's not just a question of network traffic. Remote VMs also become busy, which slows their response. To demonstrate, I ran two-minute sysbench CPU tests to saturate processing on the us-west-2 instance while observing the ping rate. Here is the command: <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sysbench --test=cpu --num-threads=10 --max-time=120 --cpu-max-prime=1000000 run</span><br />
<br />
As the next graph illustrates, CPU load has a unpredictable but substantial effect on ping times. As it happens, the ping variation in the previous graph may be due to resource contention on the underlying physical host. (Or it might really be network traffic--you never really know with Amazon.) <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnd4HXmEAlEhsoBL_gKWya9V0qEWkE-_Risiy7HdZnHhta-Nwxr0djMO-bP5Y79AtuTbKi1_p18uGxDilLDxu-JaJrUfva_PCn7rvMsFNipt52DS-pjrQdKXtAxMG7fe9aCmlkbgd2xb4/s1600/Amazon-ping-with-sysbench-on-us-west.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnd4HXmEAlEhsoBL_gKWya9V0qEWkE-_Risiy7HdZnHhta-Nwxr0djMO-bP5Y79AtuTbKi1_p18uGxDilLDxu-JaJrUfva_PCn7rvMsFNipt52DS-pjrQdKXtAxMG7fe9aCmlkbgd2xb4/s640/Amazon-ping-with-sysbench-on-us-west.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Effect of sysbench runs on ping times to US-West (20 minute interval)</td></tr>
</tbody></table>
Slow response on resource-bound systems is a problem that is familiar to anyone with experience with distributed systems, including systems where everything is on a single LAN. You cannot even count on clock ticks being delivered accurately within various types of virtual machines. The timing delays are magnified in WANs, as they already have high latency to begin with. Between busy hosts and network latency, it's reasonable to expect that at some point most systems would at least briefly experience single session transaction rates of <i><u>1 transaction per second or less</u></i>. Even with parallelized replication you would see substantial backups on the originating DBMS servers as commits begin to freeze. <br />
<br />
To complete the tale of woe, failures of various kinds can cause remote hosts to stop responding at all for periods of time that vary from seconds to days. Amazon is generally quite reliable but had <a href="http://gigaom.com/cloud/some-of-amazon-web-services-are-down-again/">two outages in the Virginia data center in June 2012 alone</a> that brought down applications from hours to days. If you replicate synchronously to a host affected by such an outage, your application just stops and you no longer store transactions at all, let alone securely. You need to turn off synchronous replication completely to stay available. <br />
<br />
So is synchronous replication really impossible between sites? Given the problems I just described it would be silly to set up MySQL semi-synchronous replication between over WAN for a real application. However, there are other ways to implement synchronous replication. Let's look at two of them. <br />
<br />
First, there is <a href="http://codership.com/products/mysql_galera">Galera</a>, which uses a distributed protocol called certification-based replication to agree on commit order between all cluster nodes combined with execution of non-conflicting transactions in parallel. Certification-based replication is a great algorithm in many ways, but Galera comes with some important practical limitations. First it replicates rows rather than statements. The row approach handles large transactions poorly, especially over distances, due to the large size of change sets. Also, not all workloads parallelize well, since transactions that conflict in any way must be fully serialized. Overall DBMS throughput may therefore reduce to the single-session throughput discussed above at unexpected times due to variations in workload. Finally, full multi-master mode between sites (as opposed to master/slave) is likely to be very problematic as nodes drop out of the cluster due to transient communication failures and require expensive reprovisioning. This is a general problem with group communications, which Galera depends on to order transactions. <br />
<br />
Second, there are theoretical approaches that claim many of the benefits of synchronous replication without killing throughput or availability. One example is the <a href="http://dbmsmusings.blogspot.com/2012/05/if-all-these-new-dbms-technologies-are.html">Calvin system developed by Daniel Abadi and others</a>, which seeks to achieve both strong transaction consistency and high throughput when operating across sites. The secret sauce in Calvin is that it radically changes the programming model to replicate what amount to transaction requests while forcing actual transaction processing to be under control of the Calvin transaction manager, which orders transaction order in advance across nodes. That should at least in principle reduce some of the unpredictability you may see in systems like Galera that do not constrain transaction logic. Unfortunately it also means a major rewrite for most existing applications. Calvin is also quite speculative. It will be some time before this approach is available for production systems and we can see whether it is widely applicable. <br />
<br />
There's absolutely a place for synchronous replication in LANs, but given the current state of the art it's hard to see how most applications can use effectively it to link DBMS servers over WAN links. In fact, the main issue with synchronous replication is the unpredictability it introduces into applications that must work with slow and unreliable networks. This is one of the biggest lessons I have learned at Continuent. <br />
<br />
<b>The Alternative: Asynchronous Multi-Master Replication</b><br />
<br />
So what are the alternatives? If you need to build applications that are available 24x7 with high throughput and rarely, if ever, lose data, you should consider high-speed local clusters linked by asynchronous multi-master replication between sites. Here is a typical architecture, which is incidentally a standard design pattern for <a href="http://www.continuent.com/solutions/overview">Tungsten</a>. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg86Wz4J7cD3Gd8-L1PQGwmnop5Ys4eEPPNi-svJC4w2GbjB6J5IVPy9j6gO9PTGi-mD3MU1blMKNRWdrzrHvuqef5c_Z9y7DuE5sUPVgRUSdh4srRK7k8W-QMxrFq0MjhcQ7FSgkA3khI/s1600/cluster-with-multi-master.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg86Wz4J7cD3Gd8-L1PQGwmnop5Ys4eEPPNi-svJC4w2GbjB6J5IVPy9j6gO9PTGi-mD3MU1blMKNRWdrzrHvuqef5c_Z9y7DuE5sUPVgRUSdh4srRK7k8W-QMxrFq0MjhcQ7FSgkA3khI/s400/cluster-with-multi-master.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Local clusters linked by asynchronous, multi-master replication</td></tr>
</tbody></table>
The big contrast between synchronous and and asynchronous replication between sites is that while both have downsides, you can minimize asynchronous multi-master problems using techniques that work now. Let's look at how async multi-master meets requirements and the possible optimizations.<br />
<ol>
<li><b>Performance</b>. Asynchronous replication solves WAN performance problem as completely as possible. To the extent that you use synchronous or near-synchronous replication technologies it is on local area networks, which are extremely fast and reliable, so application blocking is minimal. Meanwhile, long-haul replication can be improved by compression as well as parallelization, because WANs offer good bandwidth even if there is high end-to-end latency. </li>
<li><b>Data loss</b>. Speedy local replication, including synchronous and "near synchronous" methods, minimizes of data loss due to storage failures and configuration errors. Somewhat surprisingly you do not need fully synchronous replication for most systems even at the local level--that's a topic for a future blog article--but replication does need to be quite fast to ensure local replicas are up-to-date. Actually, one of the big issues for avoiding local data loss is to configure systems carefully (think sync_binlog=1 for MySQL, for example). </li>
<li><b>Availability</b>. Async multi-master systems have the delightful property that anything interrupts transaction flow between sites, replication just stops and then resumes when the problem is corrected. There's no failover and no moving parts. This is a major strength of the multi-master model. </li>
</ol>
So what are the downsides? Nothing comes for free, and there are at least two obvious issues. <br />
<ol>
<li><b>Applicability</b>. Not every application is compatible with asychronous multi-master. You will need to do work on most existing applications to implement multi-master and ensure you got it right. I touched on some of the MySQL issues <a href="http://scale-out-blog.blogspot.com/2012/04/if-you-must-deploy-multi-master.html">in an earlier article.</a> If multi-master is not a possibility, you may need the other main approach to cross-site replication: a <a href="http://scale-out-blog.blogspot.com/2011/08/system-of-record-approach-to-multi.html">system-of-record design</a> where applications update data on a single active site at any given time with other sites present for disaster recovery. (Tungsten also does this quite well, I should add.) </li>
<li><b>Data access</b>. While you might not lose data it's also quite likely you might not be able to access it for a while either. It's rare to lose a site completely but not uncommon for sites to be inaccessible for hours to days. The nice thing is that with a properly constructed multi-master application you will at least know that the data will materialize on all sites once the problem is solved. Meanwhile, relax and work on something else until the unavailable site comes back. </li>
</ol>
In the MySQL community local clusters linked by asynchronous multi-master are an increasingly common architecture for credit card payment gateways, which I mentioned at the beginning of this article. This is a telling point in favor of asynchronous cross-site replication, as credit card processors have a low tolerance for lost data. <br />
<div>
<br /></div>
<div>
Also, a great deal of current innovation in distributed data management is headed in the direction of asynchronous mechanisms. NoSQL systems (such as Cassandra) tend to use asynchronous replication between sites. There is interesting research afoot, for example in <a href="https://databeta.wordpress.com/2011/04/08/bud-bloom-under-development/">Joe Hellerstein's group at UC Berkeley</a>, to make asynchronous replication more efficient by accurately inferring cases where no synchronization is necessary. Like other research, this work is quite speculative, but the foundations are in use in operational systems today. </div>
<div>
<br /></div>
<div>
For now the challenge is to make the same mechanisms that NoSQL systems have jumped on work equally well for relational databases like MySQL. We have been working on this problem for the last couple of years at Continuent. I am confident we are well on the way to solutions that are as good as the best NoSQL offerings for distributed data management. </div>Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com5tag:blogger.com,1999:blog-768233104244702633.post-80270426002172494472012-06-03T23:58:00.001-07:002012-06-03T23:58:33.278-07:00MySQL to Vertica Replication, Part 2: Setup and OperationAs described in the <a href="http://scale-out-blog.blogspot.com/2012/06/mysql-to-vertica-replication-part-1.html">first article of this series</a>, Tungsten Replicator can replicate data from MySQL to Vertica in real-time. We use a new batch loading feature that applies transactions to data warehouses in very large blocks using COPY or LOAD DATA INFILE commands. This second and concluding article walks through the details of setting up and testing MySQL to Vertica replication. <br />
<br />
<div>
To keep the article reasonably short, I assume that readers are conversant with MySQL, Tungsten, and Vertica. Basic replication setup is not hard if you follow all the steps described here, but of course there are variations in every setup. For more information on Tungsten check out the <a href="http://code.google.com/p/tungsten-replicator/">Tungsten Replicator project at code.google.com site</a> well as current <a href="https://docs.continuent.com/wiki/display/TEDOC/Tungsten+Enterprise+Documentation+Home">Tungsten commercial documentation at Continuent</a>. </div>
<div>
<br /></div>
<div>
Now let's get replication up and running! </div>
<div>
<br />
<b>What Is the Topology? </b><br />
<br />
In this exercise we will set up Tungsten master/slave replication to move data from MySQL 5.1 to Vertical 5.1. Master/slave is the simplest topology to set up because you don't have to mix settings for different DBMS types in each service. To keep keep things simple, we will install Tungsten directly on the MySQL and Vertica hosts, which are named db1 and db2 respectively. Here is a diagram:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT2xYyBJlDbKGnBej7sGQTSAymAnzmx6PdqH7Dmvk3P9VVPi-eXc5MXeOVyEO24vd0TK644AjrfacKdEL2gsp5oRb8IUaQDc4e0bU-xJyRUogqkHHRSXATIg8JcdlToR6N_OKyA38ZRjc/s1600/mysql-to-vertica-master-slave.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT2xYyBJlDbKGnBej7sGQTSAymAnzmx6PdqH7Dmvk3P9VVPi-eXc5MXeOVyEO24vd0TK644AjrfacKdEL2gsp5oRb8IUaQDc4e0bU-xJyRUogqkHHRSXATIg8JcdlToR6N_OKyA38ZRjc/s400/mysql-to-vertica-master-slave.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
There are of course many other possible configurations. You can run replicators on separate hosts to reduce load on the DBMS servers. You can with a little patience set up direct replication using a Tungsten single replication service, which results in fewer processes to manage. Finally, you can use both direct as well as master/slave topologies to publish data from Tungsten 1.5 clusters. Tungsten clusters provide availability and scaling on the MySQL side. <br />
<a name='more'></a><br /></div>
<div>
<b>Preparing MySQL</b></div>
<div>
<br /></div>
<div>
To replicate heterogeneously, MySQL servers need to enable row-based replication. You therefore need to use MySQL 5.1 or higher. Tungsten supports all popular builds of MySQL 5.1 and 5.5, so you can pick your favorite. </div>
<div>
<br /></div>
<div>
Batch replication prints values in CSV files, so setting mismatches in character sets and timezones between MySQL, the OS platform, and Vertica will result in corrupted string and/or dates. Settling on UTF8 charset as server default and GMT as the default timezone solves these problems. This is important to get heterogeneous replication to work properly. </div>
<div>
<br /></div>
<div>
Here are sample my.cnf parameters to enable the recommended settings. </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Use row replication. </span></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">binlog-format=row</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Server timezone is GMT. </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">default-time-zone='+00:00'</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"># Tables default to UTF8. </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">character-set-server=utf8</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">collation-server=utf8_general_ci</span></div>
</div>
<div>
<br /></div>
<div>
Restart MySQL to pick up new settings. Beyond this you need to meet the usual requirements for installing Tungsten, such as defining a 'tungsten' user and ensuring replication is properly enabled. Check the Tungsten docs for more information. </div>
<div>
<br /></div>
<div>
<b>Preparing Vertica</b></div>
<div>
<br /></div>
<div>
Next, spin up Vertica. I used <a href="http://www.vertica.com/community">Vertica Community Edition</a> version 5.1.1 for these demos, but any recent production version should do. There is no special configuration for the Vertica instance at this point--unlike MySQL Tungsten works fine with Vertica default settings. </div>
<div>
<br /></div>
<div>
Once Vertica is started and you have created a database, you will need to login and create a schema to hold Tungsten catalogs. Here is an example: </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ vsql -Udbadmin -wsecret bigdata</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Welcome to vsql, the Vertica Analytic Database v5.1.1-0 interactive terminal.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Type: \h for help with SQL commands</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \? for help with vsql commands</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \g or terminate with semicolon to execute query</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \q to quit</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> create schema tungsten_mysql2vertica;</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CREATE SCHEMA</span></div>
<div>
<br />
Note the location of the JDBC driver in your Vertica release directory. For Vertica 5.1.1 this is located in the following directory: /opt/vertica/java/lib. It should have a name like vertica_5.1.1_jdk_5.jar or something similar. <br />
<br /></div>
<div>
<b>Downloading and Installing Tungsten</b></div>
<div>
<br /></div>
<div>
With MySQL and Vertica running, you can now install Tungsten. Let's first download and unpack the software in a staging directory. You should do this on the MySQL host as well as the Vertica host.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1$ mkdir ~/staging</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ cd ~/staging</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ wget --no-check-certificate https://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/tungsten-replicator-2.0.6-667.tar.gz</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ tar -xf tungsten-replicator-2.0.6-667.tar.gz</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ cd </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">tungsten-replicator-2.0.6-667</span><br />
<br />
Note that we use build 667 or later. This is necessary to pick up recent fixes to ensure compatibility with Vertica Version 5 JDBC drivers. You'll need to download from the <a href="http://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/index.html">Tungsten nightly build page</a> for now.<br />
<br />
Now set up the MySQL master. On the MySQL host, run the following installation command:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1$ tools/tungsten-installer --master-slave -a \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --service-name=mysql2vertica \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-host=db1 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --cluster-hosts=db1 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-user=msandbox \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-password=msandbox \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --home-directory=/opt/continuent \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --buffer-size=1000 \</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --java-file-encoding=UTF8 \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --java-user-timezone=GMT \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --mysql-use-bytes-for-string=false \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --svc-extractor-filters=colnames,pkey \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --property=replicator.filter.pkey.addPkeyToInserts=true \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --property=replicator.filter.pkey.addColumnsToDeletes=true \</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --start-and-report</span><br />
<div>
<br /></div>
<div>
This command has some special settings, highlighted in bold, to help with heterogeneous replication. </div>
<div>
<ol>
<li>The Java VM file encoding and timezone are UTF8 and GMT respectively. Standardizing is essential to avoid corrupting data in batch loads. </li>
<li>String values are translated to UTF8 rather than passed to slaves as bytes. </li>
<li>We insert filters to add column names and identify the primary key on tables including for INSERT operations. These are both required for batch loading to work correctly. </li>
</ol>
At the end of a successful master replicator installation you should see the following message: <br />
<br /></div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">INFO >> db1 >> .....</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Processing services command...</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">NAME VALUE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">---- -----</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLastSeqno: 0</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLatency : 0.973</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">role : master</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceName : mysql2vertica</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceType : local</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">started : true</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">state : ONLINE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Finished services command...</span></div>
<div>
<br /></div>
<div>
Next, let's turn to the Vertica slave. Before installing Tungsten on the Vertica host, we must copy in the JDBC driver to the replicator lib directory. Assuming you are in the unpacked Tungsten release, you can do this with a command like the following: </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db2$ cp /opt/vertica/java/lib/vertica_5.1.1_jdk_5.jar tungsten-replicator/lib</span></div>
<div>
<br /></div>
<div>
Now run the installation command for Vertica, which looks like the following:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db2$ tools/tungsten-installer --master-slave -a \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --service-name=mysql2vertica \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --cluster-hosts=db2 \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --master-host=db1 \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --datasource-type=vertica \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-user=dbadmin \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --datasource-password=secret \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --datasource-port=5433 \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --batch-enabled=true \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --batch-load-template=vertica \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --vertica-dbname=bigdata \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --buffer-size=25000 \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --java-file-encoding=UTF8 \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> --java-user-timezone=GMT \</b></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --skip-validation-check=InstallerMasterSlaveCheck \</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> --start-and-report</span></div>
<div>
<br /></div>
<div>
Vertica settings are fairly simple. Values that are different from a standard MySQL configuration are highlighted. JVM file encoding and timezones should match MySQL. Note also the very large buffer-size value. This means that Tungsten can commit up to 25,000 MySQL transactions in a single block on Vertica. I have tested values up to 100,000 without problems. </div>
<div>
<br /></div>
<div>
If the installation is successful, a message like the following appears at the end. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">INFO >> db2 >> .</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Processing services command...</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">NAME VALUE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">---- -----</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLastSeqno: -1</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLatency : -1.0</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">role : slave</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceName : mysql2vertica</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceType : local</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">started : true</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">state : ONLINE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Finished services command...</span></div>
</div>
<div>
<br /></div>
<div>
<b>Testing MySQL to Vertica Replication</b></div>
<div>
<br /></div>
<div>
We can check liveness quickly using a heartbeat on the MySQL master replicator. </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db1$ trepctl heartbeat</span></div>
<div>
<br /></div>
<div>
If everything is working, we then see the following on the Vertica slave replicator. </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">db2$ trepctl services</span></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Processing services command...</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">NAME VALUE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">---- -----</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLastSeqno: 1</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appliedLatency : 0.266</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">role : master</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceName : mysql2vertica</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">serviceType : local</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">started : true</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">state : ONLINE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Finished services command...</span></div>
</div>
<div>
<br /></div>
<div>
Next, let's try to replicate something. On MySQL, we need a table to hold some data, which we create as follows: </div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql -uroot test</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">...</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> create table simple_tab(id int primary key, </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> f_data varchar(100)) default charset=utf8;</span></div>
<div>
<br /></div>
<div>
Now let's create the same table in Vertica, plus a staging table. We will also have to create a schema 'test' on Vertica as well. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">vsql -Udbadmin -wsecret bigdata</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Welcome to vsql, the Vertica Analytic Database v5.1.1-0 interactive terminal.</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Type: \h for help with SQL commands</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \? for help with vsql commands</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \g or terminate with semicolon to execute query</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> \q to quit</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> create schema test;</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CREATE SCHEMA</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> create table test.simple_tab(</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> id int,</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> f_data varchar(100)</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> );</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CREATE TABLE</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> create table test.stage_xxx_simple_tab (</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> tungsten_seqno int,</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> tungsten_opcode char(1),</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> id int,</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> f_data varchar(100),</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata(> tungsten_row_id int);</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CREATE TABLE</span></div>
</div>
<div>
<br /></div>
<div>
Finally, we can try to move a row from one table to the other. Login to MySQL and insert a row: </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mysql> insert into test.simple_tab values(1, 'hello!');</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Query OK, 1 row affected (0.00 sec)</span></div>
</div>
<div>
<br /></div>
<div>
If we configured things properly, we will now see the following on the Vertica side. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">bigdata=> select * from test.simple_tab;</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> id | f_data </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">----+--------</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> 1 | hello!</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">(1 row)</span></div>
</div>
<div>
<br /></div>
<div>
At this point our topology is ready to start full-on data loading. </div>
<div>
<br /></div>
<div>
<b>Troubleshooting</b></div>
<div>
<br /></div>
<div>
It is common to run into errors when setting up heterogeneous replication--it's complicated. Don't forget to look at the replicator logs if 'trepctl status' does not show a meaningful error message. Here are two of the more common problems. </div>
<div>
<ol>
<li>MySQL writes to databases, whereas Vertica has a single database with schemas. If you write to database 'test' in MySQL it goes to schema 'test' on Vertica. It's easy to get confused if you are jumping back and forth. </li>
<li>Staging tables are easy to mess up. If you get the definition wrong, the Vertica slave will fail. In that case, drop the staging table and recreate it correctly. Then put the replicator back online. We plan to offer an automated tool to create staging tables in the future, but for now it is a little bit painful. </li>
<li>If you see dates that are off by a couple of hours, you likely did not configure Java timezones correctly. Make sure you have this correctly set. Refer to <a href="http://code.google.com/p/tungsten-replicator/wiki/Replicator_Batch_Loading#Time_Zones">this</a> for more information. </li>
</ol>
</div>
<div>
It is often very useful to look at the CSV files to debug problems. They are by default located in directory /tmp/staging/staging0. You can change the CSV file location in the Tungsten static-svc.properties file that controls the replication service configuration. Generally speaking, if there is a problem with loading you just fix it and put the replicator back online. <br />
<br />
If you run into problems that looks like a product issue, feel free to log a bug. The issue tracking system for replication is located <a href="http://code.google.com/p/tungsten-replicator/issues/list">here</a>. Before logging a bug, though, make sure it's really a Tungsten problem. Check out Giuseppe Maxia's <a href="http://datacharmer.blogspot.com/2011/12/how-to-submit-good-database-bug-report.html">hints on proper bug reporting</a>. You can submit questions on the <a href="https://groups.google.com/forum/?fromgroups#!forum/tungsten-replicator-discuss">Tungsten Replicator Discuss</a> group as well. And if you really get stuck, <a href="http://www.continuent.com/">Continuent</a> offers commercial support. </div>
<div>
<br /></div>
<div>
<b>Conclusion</b></div>
<div>
<br /></div>
<div>
This two-part series has provided a short introduction to setting up MySQL to Vertica replication. You can solve the real-time analytics problem used as an example in the articles as well as countless others that require loading data from MySQL to Vertica. </div>
<div>
<br /></div>
<div>
For more information about the detailed design of Tungsten batch loading, check out <a href="http://code.google.com/p/tungsten-replicator/wiki/Replicator_Batch_Loading">this wiki article</a>. More complete information will be posted in the Tungsten commercial documentation at <a href="http://www.continuent.com/">www.continuent.com</a> in the near future. Meanwhile, enjoy replicating to Vertica and send us feedback on the <a href="https://groups.google.com/forum/?fromgroups#!forum/tungsten-replicator-discuss">Tungsten Replicator Discuss</a> group. I look forward to hearing about your experiences. </div>Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com13tag:blogger.com,1999:blog-768233104244702633.post-47538733708039503302012-06-03T23:41:00.000-07:002012-06-04T00:00:09.806-07:00MySQL to Vertica Replication, Part 1: Enabling Real-Time Analytics with TungstenReal-time analytics allow companies to react rapidly to changing business conditions. Online ad services process click-through data to maximize ad impressions. Retailers analyze sales patterns to identify micro-trends and move inventory to meet them. The common theme is speed: moving lots of information without delay from operational systems to fast data warehouses that can feed reports back to users as quickly as possible. <br />
<br />
Real-time data publishing is a classic example of a <a href="http://en.wikipedia.org/wiki/Big_data">big data</a> replication problem. In this two-part article I will describe recent work on <a href="http://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> to move data out of MySQL into <a href="http://www.vertica.com/">Vertica</a> at high speed with minimal load on DBMS servers. This feature is known as <i>batch loading</i>. Batch loading enables not only real-time analytics but also any other application that depends on moving data efficiently from MySQL into a data warehouse. <br />
<br />
The first article works through the overall solution starting with replication problems for real-time analytics through a description of how Tungsten adapts real-time replication to data warehouses. If you are in a hurry to set up, just skim this article and jump straight to the implementation details in the <a href="http://scale-out-blog.blogspot.com/2012/06/mysql-to-vertica-replication-part-2.html">follow-on article</a>. <br />
<br />
<b>Replication Challenges for Real-Time Analytics</b><br />
<br />
To understand some of the difficulties of replicating to a data warehouse, imagine a hosted intrusion detection service that collects access log data from across the web and generates security alerts as well as threat assessments for users. The architecture for this application follows a pattern that is increasingly common in businesses that have to analyze large quantities of incoming data. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwmW2GcFs_yeBOp5iG06eyWRDJIadLY0iyUzlhXyZ7jXct5TBl8JcIJL0dsMmAjTC7dZc4vDM8GW7S0NQCd_0SsWaFlpicTspb5ajYpNBaR_cae5ERXbLt_JHyPdGsCiec3qp28biNDU4/s1600/sharded-mysql-to-vertica.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwmW2GcFs_yeBOp5iG06eyWRDJIadLY0iyUzlhXyZ7jXct5TBl8JcIJL0dsMmAjTC7dZc4vDM8GW7S0NQCd_0SsWaFlpicTspb5ajYpNBaR_cae5ERXbLt_JHyPdGsCiec3qp28biNDU4/s400/sharded-mysql-to-vertica.jpg" width="400" /></a></div>
<br />
Access log entries arrive through data feeds, whereupon an application server checks them to look for suspicious activity and commits results into a front-end DBMS tier of sharded MySQL servers. The front-end tier optimizes for a MySQL sweet spot, namely fast processing of a lot of small transactions.<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br />
<a name='more'></a><br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Next, MySQL data feed as quickly as possible into a Vertica cluster that generates reports to users. Vertica is a popular column store with data compression, advanced projections (essentially materialized views) and built-in redundancy. (For more on Vertica origins and column stores in general, <a href="http://db.csail.mit.edu/projects/cstore/">read this</a>.) The back-end DBMS tier optimizes for a Vertica sweet spot, namely fast parallel load and quick query performance. </div>
<div>
<br /></div>
<div>
There are many challenges in building any system that must scale to high numbers of transactions. Replicating from MySQL to Vertica is an especially thorny issue. Here is a short list of problems to overcome. </div>
<div>
<ol>
<li>Intrusion detection generates a lot of data. This type of application can generate aggregate peak rates of 100,000 updates <u style="font-style: italic;">per second</u> into the front-end DBMS tier. </li>
<li>Data warehouses handle normal SQL commands like INSERT, UPDATE or DELETE very inefficiently. You need to use batch loading methods like the Vertica COPY command rather than submitting individual transactions as they appear in the MySQL binlog. </li>
<li>Real applications generate not only INSERTS but also UPDATE and DELETE operations. You need to apply these in the correct order during batch loading or the data warehouse will quickly become inconsistent. </li>
<li>Both DBMS tiers are very busy, and whatever replication technique you use needs to reduce load as much as possible on both sides of the fence. </li>
</ol>
</div>
<div>
Until recently there were two obvious options for moving data between MySQL and Vertica. </div>
<div>
<ol>
<li>Use an <a href="http://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> tool like <a href="http://www.talend.com/index.php">Talend</a> to post batches extracted from MySQL to Vertica. </li>
<li>Write your own scripts to scrape data out of the binlog, process them with a fast scripting language like Perl, and load the result into Vertica. </li>
</ol>
ETL tools put load on MySQL to scan for changes and often require application changes, for example to add timestamps to detect updates. Home grown tools in addition to other limitations are difficult to maintain and deal poorly with corner cases unless very carefully tested. Both approaches also add latency to replication, which detracts from the real-time delivery goal. <br />
<br />
The summary, then, is that there is no simple way to provide anything like real-time reports to users when large volumes of data are involved. ETL and home-grown solutions tend to fall down on real-time transfer as well as the extra load they impose on already busy servers. That's where Tungsten comes in. </div>
<div>
<br /></div>
<b>Developing Tungsten Batch Loading for Data Warehouses</b><br />
<br />
Our first crack at replicating to data warehouses applied MySQL transactions to <a href="http://www.greenplum.com/">Greenplum</a> using the same approach used for MySQL--connect with a JDBC driver and apply row changes in binlog order as fast as possible. It was functionally correct but not very usable. Like many data warehouses, Greenplum processes individual SQL statements around 100 times slower than MySQL. To populate data at a reasonable speed you need to dump changes to <a href="http://tools.ietf.org/html/rfc4180">CSV</a> and insert them in batches using gpload, an extremely fast parallel loader for Greenplum. <br />
<br />
We did not add gpload support at that time, because it was obviously a major effort and we did not understand the implementation very well. However, I spent the next couple of months thinking about how to add CSV-based batch loading to Tungsten. The basic idea was to turn on MySQL row replication on the master and then apply updates to the data warehouse as follows: <br />
<ol>
<li>Accumulate a large number of transactions as rows in open CSV files. </li>
<li>Load the files to staging tables. </li>
<li>Merge the staging table contents into the base tables using SQL. </li>
</ol>
When a customer showed up needing fast replication into Vertica from MySQL we were therefore ready to develop batch loading and dived right in. It looked like a few weeks of work to get something ready for production deployment, but that estimate turned out to be quite optimistic. The implementation in fact took a good bit longer because of the complexities of CSV formats used by different DBMS services, problems with timezones, differences in SQL load command semantics, and the fact that when we started out we did not have an easy-to-setup method to test heterogeneous replication. Plus we needed to take time to create a proper installation. <br />
<br />
That said, most of the work was SMOP, or a simple matter of programming. After <strike>a few weeks</strike> six months we had fast, functional batch loading for Vertica as well as working implementations for MySQL and PostgreSQL. Batch loading applies MySQL row updates in very large groups to Vertica using CSV files and Vertica COPY commands. The following diagram shows direct replication using a single pipeline to apply transactions from a Tungsten master replication to Vertica. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFxfR1azcIoYqZY4RSMbdkutsuKKlZ8-qiA2KgTiz2H6qQb6JtAr6LeJUmJlE_T6m7EwMLuupeYw28n73mJmCkGZKk0XRaJZhB-KReA45jaueFfTX1rUR8sRE_NX2aBBpQInAe7ee_Vlc/s1600/set-based-mysql-to-vertica.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFxfR1azcIoYqZY4RSMbdkutsuKKlZ8-qiA2KgTiz2H6qQb6JtAr6LeJUmJlE_T6m7EwMLuupeYw28n73mJmCkGZKk0XRaJZhB-KReA45jaueFfTX1rUR8sRE_NX2aBBpQInAe7ee_Vlc/s400/set-based-mysql-to-vertica.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
Tungsten replication operates more or less normally up to the point where we apply to Vertica. This is the job of a new applier class called <a href="http://code.google.com/p/tungsten-replicator/source/browse/trunk/replicator/src/java/com/continuent/tungsten/replicator/applier/batch/SimpleBatchApplier.java">SimpleBatchApplier</a>. It implements the CSV loading as follows. <br />
<br />
First, as new transactions arrive Tungsten writes them to CSV files named after the Vertica tables to which they apply. For instance, say we have updates for a table simple_tab in schema test with the following format (slightly truncated from the vsql \d output):<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> Schema | Table | Column | Type | Size | </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">--------+------------+-----------------+--------------+------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> test | simple_tab | id | int | 8 | </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> test | simple_tab | </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">f_data | varchar(100) | 100</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> | </span><br />
<br />
The updates go into file test.simple_tab. Here is an example of the data in the CSV file.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">"64087","I","17","Some data to be inserted","1"</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">"64088","I","18","Some more data to be inserted","2"</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">"64088","D","0",null,"3"</span><br />
<div>
<br /></div>
The CSV file includes a Tungsten seqno (global transaction ID), an operation code (I for insert, D for delete), and the primary key. For inserts, we have additional columns containing data. Deletes just contain nulls for those columns. The last column is a row number, which allows us to order the rows when they are loaded into Vertica.<br />
<br />
Tungsten keeps writing transactions until it reaches the block commit maximum (for example 25,000 transactions). It then closes each CSV file and loads the contents into a staging table that has the base name plus a prefix, here "stage_xxx_." The staging table format mimics the CSV file columns. For example, the previous example might have a staging table like the following:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">Schema | Table | Column | Type | Size | </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">--------+----------------------+-----------------+--------------+------+</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> test | stage_xxx_simple_tab | tungsten_seqno | int | 8 |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> | stage_xxx_simple_tab | tungsten_opcode | char(1) | 1 |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> | stage_xxx_simple_tab | id | int | 8 |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> | stage_xxx_simple_tab | f_data | varchar(100) | 100 |</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> | stage_xxx_simple_tab | tungsten_row_id | int | 8 |</span><br />
<br />
Finally, Tungsten applies the deletes and inserts to table test.simple_tab by executing SQL commands like the following:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">DELETE FROM test.simple_tab WHERE id IN</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> (SELECT id FROM </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.stage_xxx_simple_tab</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> WHERE tungsten_opcode = 'D');</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">INSERT INTO </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.simple_tab(id, f_data)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> SELECT id, f_data </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> FROM </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.stage_xxx_simple_tab AS stage_a</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> WHERE tungsten_opcode='I' AND tungsten_row_id IN</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">(</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">SELECT MAX(tungsten_row_id) </span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> FROM </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">test</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.stage_xxx_simple_tab GROUP BY id);</span><br />
<div>
<br /></div>
Simple right? The SQL commands are actually generated from templates that specify the SQL to execute when connecting to Vertica, to load a CSV file into the staging table, and to merge changes from the staging table to the base (i.e., real) table. You can find the template files in directory tungsten-replicator/samples/scripts/batch. The template file format is documented <a href="http://code.google.com/p/tungsten-replicator/wiki/Replicator_Batch_Loading#Connect,_Copy,_and_Load_Scripts">here</a>.<br />
<br />
Tungsten MySQL to Vertica replication is currently in field testing. The performance on the MySQL side is excellent, as you would expect with asynchronous replication. On the Vertica side we find that batch loading operates far faster than using JDBC interfaces. Tungsten has a block commit feature that allows you to commit very large numbers of transactions at once. Tests show that Tungsten easily commits around 20,000 transactions per block using CSV files. <br />
<br />
We added a <a href="http://code.google.com/p/tungsten-replicator/source/browse/trunk/replicator/src/java/com/continuent/tungsten/replicator/applier/batch/VerticaStreamBatchApplier.java">specialized batch loader class</a> to perform CSV uploads to Vertica from other hosts, which further reduces the load on Vertica servers. (It still needs <a href="http://code.google.com/p/tungsten-replicator/issues/detail?id=338">a small fix</a> to work with Vertica 5 JDBC but works with Vertica 4.) Taking together the new Vertica replication features look as if they will be very successful for implementing real-time analytics. Reading the binlog on MySQL minimizes master overhead and fetches exactly the rows that have changed within seconds of being committed. Batch loading on Vertica takes advantage of parallel load, again reducing overhead in the reporting tier. <br />
<br />
<b>A New Replication Paradigm: Set-Based Apply</b><br />
<br />
Batch loading is significant for reasons other than conveniently moving data between MySQL and Vertica. Batch loading is also the beginning of a new model for replication. I would like to expand on this briefly as it will likely be a theme in future work on Tungsten.<br />
<br />
Up until this time, Tungsten Replicator has followed the principle of rigorously applying transactions to replicas in serial order without any deviations whatsoever. If you INSERT and then UPDATE a row, it always works because Tungsten applies them to the slave in the same order. This consistency is one of the reasons for the success of Tungsten overall, as serialization short-cuts usually end up hitting weird corner cases and are also hard to test. However, the serialized apply model is horribly inefficient on data warehouses, because single SQL statements execute very slowly. <br />
<br />
The SQL-based procedure for updating replicas that we saw in the previous section is based on a model that I call <i>set-based apply</i>. It works by treating the changes in a group of transactions as an <i>ordered set</i> (actually a relation) consisting of insert and delete operations. The algorithm is easiest to explain with an example. The following diagram shows how three row operation on table t in the MySQL binlog morph to four changes, of which we actually apply only the last two. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8SFmWy7QZtw3wxoOAIIIez4oJ64o0bzaMO3iEd1oaaV86lp96tmCVD3kG3nT7exwz4d-DV0fW9TWwWWwXiCtIQnh3EyRIgOSvBPQ6TjJ6Mx22OeH4SQ7CiJ42ZMtWz1lChs7QCn_6TTo/s1600/set-based-apply.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="73" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8SFmWy7QZtw3wxoOAIIIez4oJ64o0bzaMO3iEd1oaaV86lp96tmCVD3kG3nT7exwz4d-DV0fW9TWwWWwXiCtIQnh3EyRIgOSvBPQ6TjJ6Mx22OeH4SQ7CiJ42ZMtWz1lChs7QCn_6TTo/s400/set-based-apply.jpg" width="400" /></a></div>
<br />
Set-based apply merges the ordered change set to the base table using the following rules:<br />
<ol>
<li>Delete any rows from the base table where there is change set DELETE for the primary key <i>and</i> the first operation on that key is not an INSERT. This deletes any rows that previously existed. </li>
<li>Apply the last INSERT on each key <i>provided</i> it is not followed by a DELETE. This inserts any row that was not later deleted. </li>
</ol>
This is a form of logical reduction using a combination of staging tables and CSV loading as described in the previous section. The rules are implemented as SQL queries. Taken together these two rules apply changes in a way that is identical to applying them straight from the binlog. Using SQL is not the most efficient approach but is relatively simple to implement and easy for users to understand. <br />
<br />
Set-based apply offers interesting capabilities because sets, particularly relations, have powerful mathematical properties. We can use set theory to reason about and develop optimized handling to solve problems like conflict resolution in multi-master systems. I will get back to this topic in a future post. <br />
<br />
Meanwhile, there are obvious ways to speed up the apply process for data warehouses by performing more of the set reduction in Tungsten and less in SQL. We can also take advantage of existing Tungsten parallelization capabilities. I predict that this will offer the same sort of efficiency gains for data warehouse loading as <a href="http://scale-out-blog.blogspot.com/2011/10/benchmarking-tungsten-parallel.html">Tungsten parallel apply provides for I/O-bound MySQL slaves</a>. Log-based replication is simply a very good way of handling real-time loading and there are lots of ways to optimize it provided we follow a sound processing model. <br />
<br />
<b>Conclusion</b><br />
<br />
This first article on enabling real-time analytics explained how Tungsten loads data in real-time from Vertica to MySQL. The focus has been allowing users to serve up reports quickly from MySQL-based data, but Tungsten replication obviously applies to many other problems involving data warehouses. <br />
<br />
In the next article I will turn from dry theory to practical details. We will walk through the details of configuring MySQL to Vertica replication, so that you can try setting up real-time data loading yourself.<br />
<br />
P.S. If optimized batch loading seems like something you can help solve, Continuent is hiring. This is just one of a number of cutting-edge problems we are working on.Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com9tag:blogger.com,1999:blog-768233104244702633.post-54179884335136028322012-04-30T00:57:00.000-07:002012-04-30T07:09:24.499-07:00If You *Must* Deploy Multi-Master Replication, Read This FirstAn increasing number of organizations run applications that depend on MySQL multi-master replication between remote sites. I have worked on several such implementations recently. This article summarizes the lessons from those experiences that seem most useful when deploying multi-master on existing as well as new applications. <br />
<br />
Let's start by defining terms. <i>Multi-master replication</i> means that applications update the same tables on different masters, and the changes replicate automatically between those masters. <i>Remote sites</i> mean that the masters are separated by a wide area network (WAN), which implies high average network latency of 100ms or more. WAN network latency is also characterized by a <a href="http://en.wikipedia.org/wiki/Long_Tail">long tail</a>, ranging from seconds due to congestion to hours or even days if a ship runs over the wrong undersea cable. <br />
<br />
With the definitions in mind we can proceed to the lessons. The list is not exhaustive but includes a few insights that may not be obvious if you are new to multi-master topologies. Also, I have omitted issues like monitoring replication, using InnoDB to make slaves crash-safe, or provisioning new nodes. If you use master/slave replication, you are likely familiar with these topics already. <br />
<br />
<b>1. Use the Right Replication Technology and Configure It Properly</b><br />
<br />
The best overall tool for MySQL multi-master replication between sites is <a href="http://code.google.com/p/tungsten-replicator/">Tungsten</a>. The main reason for this assertion is that Tungsten uses a flexible, asynchronous, point-to-point, master/slave replication model that handles a wide variety of topologies such as star replication or all-to-all. Even so, you have to configure Tungsten properly. The following topology is currently my favorite:<br />
<ul>
<li><u>All-to-all topology</u>. Each master replicates directly to every other master. This handles prolonged network outages or replication failures well, because one or more masters can drop out without breaking replication between the remaining masters or requiring reconfiguration. When the broken master(s) return, replication just resumes on all sides. All-to-all does not work well if you have a large number of masters. </li>
<li><u>Updates are not logged on slaves</u>. This keeps master binlogs simple, which is helpful for debugging, and eliminates the possibility of loops. It also requires some extra configuration if the masters have their own slaves, as would be the case in a <a href="http://www.continuent.com/solutions/overview">Tungsten Enterprise cluster</a>. </li>
</ul>
There are many ways to set up multi-master replication replication, and the right choice varies according to the number of masters, whether you have local clustering, or other considerations. <a href="http://datacharmer.blogspot.com/">Giuseppe Maxia</a> has described many topologies, for example <a href="http://datacharmer.blogspot.com/2011/11/replication-multiple-masters-stars.html">here</a>, and the Tungsten Cookbook <a href="http://code.google.com/p/tungsten-replicator/wiki/TRCMultiMasterInstallation#Multi-Master_Installation">has even more details</a>. <br />
<br />
One approach you should approach with special caution is <a href="http://onlamp.com/onlamp/2006/04/20/advanced-mysql-replication.html">MySQL circular replication</a>. In topologies of three or more nodes, circular replication results in broken systems if one of the masters fails. Also, you should be wary of any kind of synchronous multi-master replication across sites that are separated by more than 50 kilometers (i.e. 1-2ms latency). Synchronous replication makes a siren-like promise of consistency but the price you pay is slow performance under normal conditions and broken replication when WAN links go down. <br />
<br />
<b>2. Use Row-Based Replication to Avoid Data Drift</b><br />
<br />
Replication depends on deterministic updates--a transaction that changes 10 rows on the original master should change exactly the same rows when it executes against a replica. Unfortunately many SQL statements that are deterministic in master/slave replication are non-deterministic in multi-master topologies. Consider the following example, which gives a 10% raise to employees in department #35. <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> UPDATE emp SET salary = salary * 1.1 WHERE dep_id = 35;</span><br />
<br />
If all masters add employees, then the number of employees who actually get the raise will vary depending on whether such additions have replicated to all masters. Your servers will very likely become inconsistent with statement replication. The fix is to enable row-based replication using <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">binlog-format=row</span> in my.cnf. Row replication transfers the exact row updates from each master to the others and eliminates ambiguity. <br />
<br />
<b>3. Prevent Key Collisions on INSERTs</b><br />
<br />
For applications that use auto-increment keys, MySQL offers a useful trick to ensure that such keys do not collide between masters using the auto-increment-increment and auto-increment-offset parameters in my.cnf. The following example ensures that auto-increment keys start at 1 and increment by 4 to give values like 1, 5, 9, etc. on this server.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">server-id=1</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">auto-increment-offset = 1</span><br />
<div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">auto-increment-increment = 4</span></div>
</div>
<div>
<br /></div>
<div>
This works so long as your applications use auto-increment keys faithfully. However, any table that either does not have a primary key or where the key is not an auto-increment field is suspect. You need to hunt them down and ensure the application generates a proper key that does not collide across masters, for example using UUIDs or by putting the server ID into the key. Here is a query on the MySQL information schema to help locate tables that do not have an auto-increment primary key. </div>
<div>
<br /></div>
<div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">SELECT t.table_schema, t.table_name </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> FROM information_schema.tables t </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> WHERE NOT EXISTS </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> (SELECT * FROM information_schema.columns c</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> WHERE t.table_schema = c.table_schema </span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> AND t.table_name = c.table_name</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> AND c.column_key = 'PRI'</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> AND c.extra = 'auto_increment')</span></div>
<div>
<b><b><br /></b></b></div>
<div>
<b><b>4. Beware of Semantic Conflicts in Applications</b></b></div>
</div>
<br />
Neither Tungsten nor MySQL native replication can resolve conflicts, though we are starting to design this capability for Tungsten. You need to avoid them in your applications. Here are a few tips as you go about this. <br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
First, avoid obvious conflicts. These include inserting data with the same keys on different masters (described above), updating rows in two places at once, or deleting rows that are updated elsewhere. Any of these can cause errors that will break replication or cause your masters to become out of sync. The good news is that many of these problems are not hard to detect and eliminate using properly formatted transactions. The bad news is that these are the easy conflicts. There are others that are much harder to address. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
For example, accounting systems need to generate unbroken sequences of numbers for invoices. A common approach is to use a table that holds the next invoice number and increment it in the same transaction that creates a new invoice. Another accounting example is reports that need to read the value of accounts consistently, for example at monthly close. Neither example works off-the-shelf in a multi-master system with asynchronous replication, as they both require some form of synchronization to ensure global consistency across masters. These and other such cases may force substantial application changes. Some applications simply do not work with multi-master topologies for this reason. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b><br /></b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b>5. Remove Triggers or Make Them Harmless</b></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Triggers are a bane of replication. They conflict with row replication if they run by accident on the slave. They can also create strange conflicts due to weird behavior/bugs (like <a href="http://bugs.mysql.com/bug.php?id=64089">this</a>) or other problems like needing definer accounts present. MySQL native replication turns <a href="http://dev.mysql.com/doc/refman/5.5/en/replication-features-triggers.html">triggers off on slaves when using row replication</a>, which is a very nice feature that prevents a lot of problems. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
Tungsten on the other hand cannot suppress slave-side triggers. You must instead alter each trigger to add an IF statement that prevents the trigger from running on the slave. The technique is <a href="http://code.google.com/p/tungsten-replicator/wiki/TRCAdministration#Tungsten_Limitations">described in the Tungsten Cookbook</a>. It is actually quite flexible and has some advantages for cleaning up data because you can also suppress trigger execution on the master. </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
You should regard all triggers with suspicion when moving to multi-master. If you cannot eliminate triggers, at least find them, look at them carefully to ensure they do not generate conflicts, and test them very thoroughly before deployment. Here's a query to help you hunt them down: </div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">SELECT trigger_schema, trigger_name </span></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> FROM information_schema.triggers;</span></div>
<div>
<br /></div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<b></b></div>
<b>6. Have a Plan for Sorting Out Mixed Up Data</b><br />
<br />
Master/slave replication has its discontents, but at least sorting out messed up replicas is simple: re-provision from another slave or the master. No so with multi-master topologies--you can easily get into a situation where all masters have transactions you need to preserve and the only way to sort things out is to track down differences and update masters directly. Here are some thoughts on how to do this.<br />
<ol>
<li><u>Ensure you have tools to detect inconsistencies</u>. Tungsten has built-in consistency checking with the 'trepctl check' command. You can also use the <a href="http://www.percona.com/doc/percona-toolkit/2.1/pt-table-checksum.html">Percona Toolkit pt-table-checksum</a> to find differences. Be forewarned that neither of these works especially well on large tables and may give false results if more than one master is active when you run them. </li>
<li><u>Consider relaxing foreign key constraints</u>. I love foreign keys because they keep data in sync. However, they can also create problems for fixing messed up data, because the constraints may break replication or make it difficult to go table-by-table when synchronizing across masters. There is an argument for being a little more relaxed in multi-master settings. </li>
<li><u>Switch masters off if possible</u>. Fixing problems is a lot easier if you can quiesce applications on all but one master. </li>
<li><u>Know how to fix data</u>. Being handy with SQL is very helpful for fixing up problems. I find <a href="http://dev.mysql.com/doc/refman/5.5/en/select-into.html">SELECT INTO OUTFILE</a> and <a href="http://dev.mysql.com/doc/refman/5.5/en/load-data.html">LOAD DATA INFILE</a> quite handy for moving changes between masters. Don't forget SET SESSION LOG_FILE_BIN=0 to keep changes from being logged and breaking replication elsewhere. There are also various synchronization tools like <a href="http://www.percona.com/doc/percona-toolkit/2.1/pt-table-sync.html">pt-table-sync</a>, but I do not know enough about them to make recommendations. </li>
</ol>
At this point it's probably worth mentioning commercial support. Unless you are a replication guru, it is very comforting to have somebody to call when you are dealing with messed up masters. Even better, expert advice early on can help you avoid problems in the first place. <br />
<br />
(Disclaimer: <a href="http://www.continuent.com/">My company</a> sells support for Tungsten so I'm not unbiased. That said, commercial outfits really earn their keep on problems like this.)<br />
<br />
<b>7. Test Everything</b><br />
<br />
Cutting corners on testing for multi-master can really hurt. This article has described a lot of things to look for, so put together a test plan and check for them. Here are a few tips on procedure:<br />
<ol>
<li>Set up a realistic pre-prod test with production data snapshots. </li>
<li>Have a way to reset your test environment quickly from a single master, so you can get back to a consistent state to restart testing. </li>
<li>Run tests on all masters, not just one. You never know if things are properly configured everywhere until you try. </li>
<li>Check data consistency after tests. Quiesce your applications and run a consistency check to compare tables across masters. </li>
</ol>
It is tempting to take shortcuts or slack off, so you'll need to find ways to improve your motivation. If it helps, picture yourself explaining to the people you work for why your DBMS servers have conflicting data with broken replication, and the problem is getting worse because you cannot take applications offline to fix things. It is a lot easier to ask for more time to test. An even better approach is to hire great QA people and give them time to do the job right.<br />
<br />
<b>Summary</b><br />
<br />
Before moving to a multi-master replication topology you should ask yourself whether the trouble is justified. You can get many of the benefits of multi-master with <a href="http://scale-out-blog.blogspot.com/2011/08/system-of-record-approach-to-multi.html">system-of-record architectures</a> with a lot less heartburn. That said, an increasing number of applications do require full multi-master across multiple sites. If you operate one of them, I hope this article is helpful in getting you deployed or improving what you already have.<br />
<br />
Tungsten does a pretty good job of multi-master replication already, but I am optimistic we can make it much better. There is a wealth of obvious features around conflict resolution, data repair, and up-front detection of problems that will make life better for Tungsten users and reduce our support load. Plus I believe we can make it easier for developers to write applications that run on multi-master DBMS topologies. You will see more about how we do this in future articles on this blog.Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com6tag:blogger.com,1999:blog-768233104244702633.post-63454463346205276662012-04-22T19:46:00.001-07:002012-04-22T19:46:56.164-07:00Replication Is Bad for MySQL Temp TablesExperienced MySQL DBAs know that temp tables cause major problems for MySQL replication. It turns out the converse is also true: replication can cause major problems for temporary tables. <br />
<br />
In a recent customer engagement we enabled <a href="http://code.google.com/p/tungsten-replicator/">Tungsten Replicator</a> with a MySQL application that originally ran on a server that did not use replication. QA promptly discovered reports that previously ran in 10 seconds were now running in as many minutes. It turned out that the reports used temp tables to assemble data, and these were being written into the master binlog. This created bloated binlogs and extremely slow reports. We fixed the problem by enabling row replication (i.e., binlog-format=row in my.cnf). <br />
<br />
A common DBA response to temp table problems is to try to eliminate them completely, as suggested in the excellent <a href="http://www.amazon.com/High-Performance-MySQL-Optimization-Replication/dp/1449314287">High Performance MySQL, 3rd Edition</a>. (See p. 502.) Elimination is a good philosophy when applications use temp tables to generate updates. However, it does not work for reporting. Temp tables allow you to stage data for complex reports across a series of transactions, then pull the final results into a report writer like <a href="http://jasperforge.org/projects/jasperreports">JasperReports</a>. This modular approach is easy to implement and maintain afterwards. Eliminating temp tables in such cases can create an unmaintainable mess. <br />
<br />
The real solution with report temp tables is to keep them out of the master binlog. Here is a list of common ways to do so. Let me know if you know others.<br />
<br />
<b>* Turn off binlog updates.</b> Issue 'SET SESSION SQL_LOG_BIN=0' when generating reports. The downside is that it requires SUPER privilege to set. Also, if you make a code mistake and update normal tables with this setting, your changes will not be replicated.<br />
<br />
<b>* Use a non-replicated database. </b>Configure the master my.cnf with binlog-ignore-db as follows to ignore any update (including on temp tables) that is issued when database 'scratch' is the default database: <br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">binlog_ignore_db = scratch</span><br />
<div>
<br /></div>
<div>
This approach does not require special privileges. However coding errors or connection pool misconfigurations are obvious liabilities. Your application must either connect to the scratch database or issue an explicit use command. Otherwise, temp table operations <i>will</i> be logged, as in the following example:</div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">use not_scratch;</span></div>
<div>
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">create temporary table scratch.report1_temp(name varchar(256), entry_time date, exit_time date);</span></div>
<div>
<br /></div>
<div>
<b>* Use a slave with the binlog disabled. </b>Remove th<span class="Apple-style-span" style="font-family: inherit;">e log-bin o</span>ption from my.cnf. This works well if you have extra reporting slaves that are caught up. However, it may not work if the reports must be fully up-to-date or you need the ability to promote the slave quickly to a master, in which case the binlog must be enabled. </div>
<div>
<br /></div>
<div>
<b>* Use row replication. </b> You can set row replication at the session level using 'SET SESSION binlog_format=row', which requires SUPER privilege, or overall by setting binlog-format in my.cnf. In this case CREATE TEMPORARY TABLE and updates on temp tables do not appear in the binlog at all. The downside of enabling row replication fully is that it can lead to bloated logs and <a href="http://scale-out-blog.blogspot.com/2012/02/agony-of-big-transactions-in-mysql.html">blocked servers if you have very large transactions</a>. SQL operations like DELETE that affect multiple rows are stored far more compactly in statement replication. Also, reloading mysqldump files can be very slow in row replication compared to statement replication, which can handle block inserts generated by the --extended-insert option. <br />
<div>
<br /></div>
<div>
The proper solution to keep replication from hurting your use of temp tables will vary depending on your application as well as the way you run your site. For my money, though, this is a good example of where row replication really helps and deserves a closer look. </div>
<div>
<br /></div>
<div>
MySQL could use some feature improvements in the area of temp tables and replication. I find it surprising that <a href="http://dev.mysql.com/doc/refman/5.1/en/binary-log-mixed.html">mixed mode replication does not fully suppress temp table binlog updates</a>. Only row replication does so. Second, it would be great to have a CREATE TABLE option to suppress logging particular tables to the binlog. This would allow applications to make the logging decision at schema design time. Finally, global options to suppress binlogging of specific table types, such as temp tables and MEMORY tables would be useful. Perhaps we will see some of these in future MySQL releases. </div>
<br />
<table></table>
</div>Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com15tag:blogger.com,1999:blog-768233104244702633.post-52824681883300177892012-04-14T14:04:00.000-07:002012-04-18T17:14:39.902-07:00Oracle Missed at MySQL User Conference...Not!The MySQL UC this past week was the best in years. Percona did an outstanding job of organizing the main <a href="http://www.percona.com/live/mysql-conference-2012/">Percona Live event</a> that ran Tuesday through Thursday. About 1000 people attended, which is up from the 800 or so at the O'Reilly-run conference in 2011. There were also excellent follow-on events on Friday for <a href="http://www.skysql.com/events/mysql-solutions-day/schedule">MariaDB/SkySQL</a>, <a href="http://www.drizzle.org/content/drizzle-day-fri-13-apr-2012-santa-clara">Drizzle</a>, and <a href="http://sphinxsearch.com/conference2012/schedule.html">Sphinx</a>. <br />
<div>
<br /></div>
<div>
What made this conference different was the renewed energy around MySQL and the number of companies using it. </div>
<div>
<ol>
<li>Big web properties like Facebook, Twitter, Google, and Craigslist continue to anchor the MySQL community and drive innovation from others through a combination of funding, encouragement, and patches. </li>
<li>Many new companies we have not heard from before like <a href="http://www.percona.com/live/mysql-conference-2012/sessions/scaling-pinterest">Pinterest</a>, <a href="http://www.percona.com/live/mysql-conference-2012/sessions/building-high-volume-reporting-system-amazon-using-mysql-tungsten-and-vertica">BigDoor,</a> <a href="http://www.percona.com/live/mysql-conference-2012/sessions/one-many-story-sharding-box">Box.net</a>, and <a href="http://www.skysql.com/events/mysql-solutions-day/schedule">Constant Contact</a> talked about their experience building major new applications on MySQL. </li>
<li>The vendor exhibition hall at Percona Live was hopping. Every vendor I spoke to had a great show and plans to return next year. There is great innovation around MySQL from many creative companies. I'm very proud my company, <a href="http://www.continuent.com/">Continuent</a>, is a part of this. </li>
<li><span class="Apple-style-span" style="font-style: normal;">The demand for MySQL expertise was completely out of hand. So many talks e</span>nded with "...and we are hiring" that it became something of a joke. The message board was likewise packed with help wanted ads. </li>
</ol>
</div>
<div>
When Oracle acquired Sun Microsystems a couple of years ago, it triggered a lot of uncertainty about the future of MySQL. This concern turns out to be unfounded. Oracle does excellent engineering work, especially on InnoDB, but had no involvement either official or unofficial at the conference. This was actually a good thing. </div>
<div>
<br /></div>
<div>
By not participating, Oracle helped demonstrate that MySQL is no longer dependent on any single vendor and has taken on a real life of its own driven by the people who use it. MySQL fans owe Oracle a vote of thanks for not attending this year. Next year I hope they will be back to join the fun. <br />
<br />
<i>p.s., It has come to my attention since writing this article that 800 may not be correct attendance for the O'Reilly 2011 conference. The 1000 figure is from Percona. Speaking as an attendee they seemed about the same size. Please feel free to comment if you have accurate numbers. </i></div>Robert Hodgeshttp://www.blogger.com/profile/05379726998057344092noreply@blogger.com4