Mobile Internet Access in Germany for Open Source Road Warriors

Reliable Internet access is a long-standing problem for road warriors visiting foreign countries.  Open source developers in particular have problems reconciling travel with addiction to high-bandwidth network access from laptop computers.  Wi-Fi hotspots are scarce, costly, often slow, and in some cases complicated by inconvenient local laws like Italy's Pisanu Decree.  International mobile network access plans are ridiculously expensive or like DROAM have download limits that make them useless for serious programming.

The best solution in many cases is to look for a local pre-paid mobile access plan in each country you visit.   Mobile networks are widely available and fast in developed regions, and there are cheap plans that limit the amount you pay while providing solid connectivity.  On a recent trip to Germany I found a great solution for local Internet access from FONIC, which offers a prepaid data plan using their FONIC Surf-Stick.  The Surf-Stick is a USB modem that plugs into a USB port on your laptop and looks like the following. 

After you buy the Surf Stick, you can add money to it using credits purchased in local stores.  It's a relatively simple solution provided you understand a little about networking and can work your way through a bit of German.

The rest of this article is a description of how to use FONIC in Germany, as well as a review of the performance and a couple of downsides.  I do not have any connection with FONIC other than using their products.  You may have different experiences or find something better.  If so, please write an article about it.

Getting Started

I bought got my Surf Stick for 49 Euro at Saturn, a big German consumer electronics chain.  You get the modem itself plus a SIM card and one day of free surfing.  You can get also the stick from other FONIC partners or order it off the FONIC website.  Surprisingly it does not appear to be available in places like Frankfurt or Munich Airport.

The first step once you have the modem is to initialize the SIM on FONIC's site by registering yourself as a new user.  To do this you'll need to have your FONIC phone number and activation code ("Freischaltungsnummer"), which are written on the side of the envelope that contains the SIM and that you should try to avoid losing.  You'll also need a German address that you enter as part of the sign-up.  A hotel address is fine.  It takes a couple of hours after registration to complete provisioning.  

Once provisioned, connecting is easy.  Put the SIM in the surf stick and plug the stick in a USB port on your computer.  On Mac OS X, you'll see an application called "Mobile Partner."  Here are the steps to activate a connection: 

1. Click the Mobile Partner app.  
2. Enter your PIN (also on the SIM envelope) and press Return. 
3. The application will stall for a few seconds while it looks for the modem.  Once complete it will show a screen entitled "Verbinden" (connect).   
4. Click Verbinden.  A couple of pop-ups will appear and you are online. 
5. When you want to drop the connection click "Trennen" (disconnect).  

You will now need to load some money into your account.  There's no way to top up online with a credit card--you either have to register a German bank account or purchase credits.  For foreigners the only practical thing is therefore to buy a credit, which is called an "Aufladebon."  You can get them at stores like Lidl, Rossman, and Jet as well as Esso gas stations.  Just ask for the following in German:  FONIC Aufladebon für 20 Euro, which is a 20 Euro credit and gives you 8 days of surfing.   I recommend buying this at the same store where you get your modem, which means the start-up cost is about 70 Euros.  

The Aufladebon itself a bit disappointing:  a printed receipt with a 12 digit number and some instructions in German.  Start your Mobile Partner app, connect to the Internet, and click on "Guthaben verwalten" (manage credit).  Select "Guthabenkarte aufladen" (load credit), enter your 12 digit number, and press enter.   Your credit is now topped up for a while.  You can check the level using the "Guthabenabfrage" (check credit) button.  

Performance

I used the FONIC modem for about a week.  It functioned well in every location that had at all reasonable connectivity.  FONIC handles transitions across cells really well--provided there's connectivity it seems to use the fastest available protocol, such as HSPA, and switches seamlessly down to lower capacity protocols like Edge if that is all that is available.  You can use it in a car or train without experiencing application problems--I used it on the Autobahn between Hamburg and Lübeck and it worked surprisingly well.   (Just be clear, somebody else was driving at the time.) 

With good connectivity, the rated speed is 7.2Mbps down and 5.76 upload, which is comparable to average household connections.  PDFs and similar files downloaded at 150K/sec.  Upload was similarly snappy.  

Applications like ssh and even remote debugging/profiling worked very well.  In my case this included running Yourkit Java profiler connecting into an application running on servers elsewhere in Germany.  I also hosted GotoMeeting web conferences.  The latency seems higher than DSL--typical latencies run about 100 to 200ms, which is about the same latency you get when connecting from the US West Coast to sites in Europe. 

Daily transfer limits are very competitive.  The current limit is 500Mb per day at full speed access, though in practice it seemed to cut off around 600Mb for me.  This is better than offerings from the big guys like T-Mobile and Vodafone.  FONIC implements the limit quite elegantly.  First you get a nice SMS that you are near the limit, followed by another that you are over the limit.  However, after that you do not get cut off.  The connection just switches to 64kbps until the next day.  This is still faster than the Wi-Fi connectivity in many hotel rooms.

If you don't want to wait for the friendly SMS message about limits, you can check your limits easily on using the Mobile Partner app.

Speaking of the app, Mobile Partner seems really solid.  There were no bugs, crashes or weird hangs the way you get with some of the US providers like Verizon's prepaid Internet app. I only ran it on Mac OS X but I would guess it is just as good on Windows.  

Caveats

In general FONIC is outstanding.   Life is not perfect, so here are a few of the limitations I found.   

1. German.  FONIC is 100% German, including the website and the app.  You'll need to know some German or make friends fast.  Also, the data plan apparently only works in Germany.  
2. Phone connectivity.  If your cell phone does not work, FONIC won't either.  The nice thing is that you can have connectivity drop out for a bit while crossing cells in a train or car without losing TCP/IP sessions.  
3. Occasional routing problems. On one evening in one particular location the FONIC modem did not work at all well, reporting ping latencies of 50+ seconds or dropping out entirely.  It was not apparent whether this was due to poor cell coverage or another issue.  It worked fine everywhere else.

There appear to be some competitors to FONIC but I have no idea how well they work.  One of the advantages of FONIC is that they have many partners in Germany that sell both modems as well as the credits.  That would also be a consideration in evaluating competing products.

Summary

Some day it may be possible to get decent mobile data plans that work for international travel.  Given the complexity of telecom regulation and the motivations of local carriers to prevent competition, you probably don't want to hold your breath until they arrive.

In the meantime, FONIC looks like a good Internet access solution for anybody traveling to Germany for more than a few days who needs constant, high-capacity access.  I was stuck in a couple of locations that had no network access, so it was a life-saver in those cases.  However, FONIC is also cheaper than hotel Wi-Fi and has better performance than many Wi-Fi hotspots.   If you move about or are uncertain about the quality of the Internet connectivity where you are staying, I highly recommend it.

If you know of similar plans for other countries, please share your experiences.  I would love to find something like FONIC for Italy, Spain, and the US.  

Introducing Tungsten On-Disk Queues for Parallel Replication

Tungsten Replicator has offered shard-based parallel replication to slaves since late 2010.  The initial implementation uses in-memory queues.  Working purely in memory keeps latency low and throughput high.   On the other hand, working in memory consumes valuable RAM.  It also forces us to buffer all in-flight transactions and therefore greatly limits the span of time permissible between the slowest and fastest shard.

Hence our newest improvement:  on-disk parallel queues.  In this article I will cover how parallel replication works in general, how on-disk queues help with parallel replication, and then show how to set up from the latest builds.

First, let's review the basic mechanics.  Parallel replication is really "parallel apply," which means taking a stream of serialized transactions, splitting them into separate streams, and applying them in parallel to a slave DBMS.  In the Tungsten pipeline architecture, we implement this kind of flow using a combination of stages and stores.  One stage reads transactions from the persistent transaction history log (aka THL) to a "parallel store."  The parallel store splits the stream into a set of queues.  The next stage extracts from those queues and applies to the slave.   It looks like the following picture:

From a conceptual point of view the incoming thl-to-q task thread performs an indexing function.  It  guides construction of the queues read by task threads in the q-to-dbms stage.  Within this framework there are many ways to feed events into the parallel queues.    In the case of on-disk queues there are two obvious design options.  

1. Read data out of the THL and split them into separate transaction logs per parallel queue.  This is very similar to the in-memory approach, except that the queues are now on disk (or SSD or whatever storage you pick).  It can be implemented without adding any extra threads to the parallel store. 
2. Leave all data in THL.  Add a cursor for each parallel queue that scans the THL and picks only the transactions that belong in that parallel queue.   This requires extra threads to do the scans, hence is more complex to implement.  

Both approaches achieve the primary goal, which is to keep the transactions in storage until we actually need them and thereby minimize memory usage.  This in turn solves a major problem, namely that individual shards can now be many thousands or even millions of transactions apart in the serial history.  Beyond that, it is not completely obvious which approach is better.

For example, option 1 isolates the reads to individual files.  This minimizes overall I/O at the cost of making it more random, since reads and writes are spread over many files.  Option 2 avoids extra writes and keeps I/O sequential, but introduces a bunch of threads doing the equivalent of table scans across the same patch of storage.  Up to a point we can assume that pages are coming out of the OS page cache rather than storage but this assumption will not hold for all operating environments and workloads.  The only way to prove the trade-offs is to implement and test.  (We may end up implementing both.)

After some discussion internally at Continuent as well as with Harrison Fisk from Facebook, we picked option 2 for the initial implementation.  Here is a diagram that shows how it works.

Here is a quick tour of the implementation in Java class THLParallelQueue.  This class maintains an in-memory blocking queue for each channel.  Each queue has a corresponding read thread that scans the THL and places matching events into the queue.  The THLParallelQueue class synchronizes read threads and handles issues like serialization and clean shutdown.  Some memory is therefore consumed, for queues, but they are quite small and amount to far less than keeping all transactions in memory.

So much for the theoretical description. If you would like to test on-disk queues yourself, you can get started in three steps.

1. Download the latest nightly build of Tungsten Replicator to /tmp. 
2. Untar and cd into the resulting release directory.  
3. Install using the new tungsten-installer as described by Guiseppe Maxia.  

Here is an example of set-up commands.  My test system uses logos1 as the master and logos2 as the slave.  The MySQL DBMS build is Percona Server 5.1.54.  Tungsten is installed in /opt/tungsten.

# Download and unpack build.
cd /tmp
wget --no-check-certificate https://s3.amazonaws.com/files.continuent.com/builds/nightly/tungsten-2.0-snapshots/tungsten-replicator-2.0.4-154.tar.gz
# Untar.
tar -xvzf tungsten-replicator-2.0.4-154.tar.gz
cd tungsten-replicator-2.0.4-154
# Set up and start replicators. 
export TUNGSTEN_HOME=/opt/tungsten
/tmp/tungsten-replicator-2.0.4-154/tools/tungsten-installer \
  --master-slave  \
  --master-host=logos1  \
  --datasource-user=tungsten  \
  --datasource-password=secret  \
  --service-name=percona \
  --home-directory=${TUNGSTEN_HOME} \
  --cluster-hosts=logos1,logos2 \
  --relay-directory=${TUNGSTEN_HOME}/relay-logs \
  --datasource-log-directory=/usr/local/percona-5.1.54/data \
  --thl-directory=${TUNGSTEN_HOME}/thl-logs \
  --channels=10 \
  --svc-parallelization-type=disk \
  --start-and-report                                  

Note the bold options to select disk queues--"memory" is the other option--and the number of channels.  You can confirm you have the right queue installed by running the following command against any slave replicator.  You should see the storage class THLParallelQueue in the status output.

$ trepctl -host logos2 status -name stores
Processing status command (stores)...
NAME                VALUE
----                -----
doChecksum        : false
logDir            : /opt/rhodges4/thl-logs/percona
logFileRetention  : 7d
logFileSize       : 100000000
maximumStoredSeqNo: 0
minimumStoredSeqNo: 0
name              : thl
storeClass        : com.continuent.tungsten.replicator.thl.THL
NAME                VALUE
----                -----
criticalPartition : -1
discardCount      : 0
eventCount        : 1
headSeqno         : 0
maxSize           : 10
name              : parallel-queue
queues            : 10
serializationCount: 0
serialized        : false
stopRequested     : false
store.0           : THLParallelReadTask task_id=0 thread_name=store-thl-0 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.1           : THLParallelReadTask task_id=1 thread_name=store-thl-1 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.2           : THLParallelReadTask task_id=2 thread_name=store-thl-2 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.3           : THLParallelReadTask task_id=3 thread_name=store-thl-3 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.4           : THLParallelReadTask task_id=4 thread_name=store-thl-4 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.5           : THLParallelReadTask task_id=5 thread_name=store-thl-5 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.6           : THLParallelReadTask task_id=6 thread_name=store-thl-6 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.7           : THLParallelReadTask task_id=7 thread_name=store-thl-7 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.8           : THLParallelReadTask task_id=8 thread_name=store-thl-8 hi_seqno=0 lo_seqno=0 read=1 discarded=1 events=0
store.9           : THLParallelReadTask task_id=9 thread_name=store-thl-9 hi_seqno=0 lo_seqno=0 read=1 discarded=0 events=0
storeClass        : com.continuent.tungsten.replicator.thl.THLParallelQueue
syncEnabled       : true
syncInterval      : 2000
Finished status command (stores)...

Now for some fine print.  On-disk queues are implemented but are still undergoing QA.  There are bugs.  The most important problem is performance--the latency is a lot higher than expected on some of our systems, which I suspect is due to an as-yet undiagnosed bug.  If you try them out now you can expect to hit a few problems.  On the other hand, we take any and all feedback quite seriously, so this is your chance provide input and help guide the final implementation.  Please log issues on the Tungsten Replicator issue tracker or bring up questions on the tungsten-discuss mailing list.

Finally, if you would like to learn more about the parallel queue implementation, check out the design documentation on our wiki as well as the source code.  They are both pretty readable.