High Scalability

“Data is everywhere, never be at a single location. Not scalable, not maintainable.” –Alex Szalay

While Galileo played life and death doctrinal games over the mysteries revealed by the telescope, another revolution went unnoticed, the microscope gave up mystery after mystery and nobody yet understood how subversive would be what it revealed. For the first time these new tools of perceptual augmentation allowed humans to peek behind the veil of appearance. A new new eye driving human invention and discovery for hundreds of years.

Data is another material that hides, revealing itself only when we look at different scales and investigate its underlying patterns. If the universe is truly made of information, then we are looking into truly primal stuff. A new eye is needed for Data and an ambitious project called Data-scope aims to be the lens.

A detailed paper on the Data-Scope tells more about what it is:

The Data-Scope is a new scientific instrument, capable of ‘observing’ immense volumes of data from various scientific domains such as astronomy, fluid mechanics, and bioinformatics. The system will have over 6PB of storage, about 500GBytes per sec aggregate sequential IO, about 20M IOPS, and about 130TFlops. The Data-Scope is not a traditional multi-user computing cluster, but a new kind of instrument, that enables people to do science with datasets ranging between 100TB and 1000TB.  There  is a vacuum today in data-intensive scientific computations, similar to the one that lead to the development of the BeoWulf cluster: an inexpensive yet efficient template for data intensive computing in academic environments based on commodity components. The proposed Data-Scope aims to fill this gap.

A very accessible interview by Nicole Hemsoth with Dr. Alexander Szalay, Data-Scope team lead, is available at The New Era of Computing: An Interview with "Dr. Data". Roberto Zicari also has a good interview with Dr. Szalay in Objects in Space vs. Friends in Facebook.

The paper is filled with lots of very specific recommendations on their hardware choices and architecture, so please read the paper for the deeper details. Many BigData operations have the same IO/scale/storage/processing issues Data-Scope is solving, so it’s well worth a look. Here are some of the highlights:

 

One of the benefits of web applications is that they are generally transported via TCP, which is a connection-oriented protocol designed to assure delivery. TCP has a variety of native mechanisms through which delivery issues can be addressed – from window sizes to selective acks to idle time specification to ramp up parameters. All these technical knobs and buttons serve as a way for operators and administrators to tweak the protocol, often at run time, to ensure the exchange of requests and responses upon which web applications rely. This is unlike UDP, which is more of a “fire and forget” protocol in which the server doesn’t really care if you receive the data or not.

Who's Hiring?

• Anybody interested in helping manage a 100+ Linux server deployment? Next Big Sound is a an analytics company for the music industry and is looking someone to help them scale.
• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• aiCache creates a better user experience by increasing the speed scale and stability of your web-site.
• ElasticHosts award winning cloud server hosting launches across North America. Adding data centers in Los Angeles and Toronto. Free trial. Just visit our website.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...

There are so many architectural ideas swirling in the bit wind these days. Two of the biggest battles are cloud vs. bare metal and RAM vs. disk vs. SSD. 37signals has published two solid articles that are counter hype cycle in their message:

Technologists who grew up when RAM cost $1,000 per megabyte can have a hard time dealing with the luxury of RAM being virtually free.

The progress of technology is throwing an ever greater number of optimizations into the “premature evil” bucket never to be seen again.

37signals made quite a stir with their money shot of the 864GB of RAM they bought for a mere $12K as part of their caching layer for Basecamp. That's a lot of memory for not a lot of money. There's nothing like actually seeing it in the flesh to bring the point home. Does that make Memory Based Architectures a little more appealing?

37signals then followed up with another provocative article: Three years later, Mr. Moore is still letting us punt on database sharding. The gist is scaling up is working for them. RAM is getting cheaper and FusionIO is getting faster, so they've been able to avoid architecture complexifications like sharding. Does that make SSD based architectures a little more appealing?

StackExchange is in much the same position, with a different stack, but with sympatico core ideas and comparable results. The learning: In your transaction oriented features, if you aren't Googleish in your requirements, then scale-up using bare metal, RAM, and SSD may be the way to go. The tug you feel towards the cloud and horizontal scaling may just be a strong consensus wind a blowin'.

Some of the key takeways are: 

If you’ve got the time, we’ve got the HighScalability:

• 9nm : IBM's carbon nanotube transistor that outperforms silicon; YouTube: 4 Billion Views/Day; 864GB RAM: 37signals Memcache, $12K
• Quotable Quotes:
  • Chad Dickerson: You can only get growth by feeding opportunities.
  • @launchany: It amazes me how many NoSQL database vendors spend more time detailing their scalability and no time detailing the data model and design
  • Google: Let's make TCP faster.
  • WhatsApp: we are now able to easily push our systems to over 2 million tcp connections!
  • Sidney Dekker: In a complex system…doing the same thing twice will not predictably or necessarily lead to the same results.
  • @Rasmusfjord: Just heard about an Umbraco site running on Azure that handles 20.000 requests /*second*
• Herb Sutter with an epic post, Welcome to the Jungle, touching on a lot of themes we've explored on HighScalability, only in a dramatically more competent way. What's after the current era of multi-core CPUs has played out? Mainstream computers from desktops to ‘smartphones’ are being permanently transformed into heterogeneous supercomputer clusters. Henceforth, a single compute-intensive application will need to harness different kinds of cores, in immense numbers, to get its job done. Different parts of even the same application naturally want to run on different kinds of cores. Applications will need to be at least massively parallel, and ideally able to use non-local cores and heterogeneous cores. Programming languages and systems will increasingly be forced to deal with heterogeneous distributed parallelism. Perhaps our most difficult mental adjustment, however, will be to learn to think of the cloud as part of the mainstream machine – to view all these local and non-local cores as being equally part of the target machine that executes our application, where the network is just another bus that connects us to more cores. If you haven’t done so already, now is the time to take a hard look at the design of your applications, determine what existing features – or, better still, what potential and currently-unimaginable demanding new features – are CPU-sensitive now or are likely to become so soon, and identify how those places could benefit from local and distributed parallelism. Now is also the time for you and your team to grok the requirements, pitfalls, styles, and idioms of hetero-parallel (e.g., GPGPU) and cloud programming.

There's so much more the Internet has to say on Scalability. Click below to be in on all the secrets...

MoreSQL is a cheeky idea Alex Tatiyants invented in NoSQL No More: Let’s double down with MoreSQL advocating that the cure for NoSQL is not less SQL, but even more SQL. Use SQL everywhere, for everything. This is of course creates yet another NewSQL. Hopefully we've exhausted all tasteful variants of the SQL motif. 

While the post is ironic (I think), Google may be into MoreSQL in a big way, as described in a well written and exceptionally detailed paper: Tenzing - A SQL Implementation On The MapReduce Framework. If you are looking for the secrets of how to get back to the good old days of joins and aggregate operators using a SQL syntax, while enjoying the scalability of NoSQL, this paper is a must read.

Abstract:

Tenzing is a query engine built on top of MapReduce for ad hoc analysis of Google data. Tenzing supports a mostly complete SQL implementation (with several extensions) combined with several key characteristics such as heterogeneity, high performance, scalability, reliability, metadata awareness, low latency, support for columnar storage and structured data, and easy extensibility. Tenzing is currently used internally at Google by 1000+ employees and serves 10000+ queries per day over 1.5 petabytes of compressed data. In this paper, we describe the architecture and implementation of Tenzing, and present benchmarks of typical analytical queries.

This is a guest post by Siddharth Anand, a senior member of LinkedIn's Distributed Data Systems team. 

Preamble Ramble

If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”.

In 2007, Amazon published a paper on Dynamo. The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud.

It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing.

Huh?

Facebook Timeline is audacious in scope. It wants to compile a complete scrollable version of your life story from photos, locations, videos, status updates, and everything you do. That could be many decades of data (hopefully) that must stored and made quickly available at any point in time. A huge technical challenge, even for Facebook, which we know are experts in handing big data. And they built it all in 6 months.

Facebook's Ryan Mack shares quite a bit of Timeline's own implementation story in his excellent article: Building Timeline: Scaling up to hold your life story. 

Five big takeaways from the article are:

If you’ve got the time, we’ve got the HighScalability:

• Google+: 90 million users; Internet 2011: 2.1 billion Internet users, 1 trillion YouTube views, 5.9 billion mobile subscriptions; Fusion-io: One Billion IOPS; 12 atoms: size of IBM's new memory bit; 32 Million: Stack monthly visitors; Gmail: 350 Million Users; TimTebow: 1.5 million Tweets
• Quotable Quotes:
  • Similarity: There is no canonical schema anymore. Instead you should ask: What high-volume queries will I need to serve with my data? Then work backwards from there.
  • @kvirjee: Dis/Agree? -- "there is no problem but scalability, and architecture is its solution"
  • @robpegoraro : Eternal vigilance can be crowdsourced.
• Didn't Bill Gates say once that 48 bits would always be enough for an ID? Well, Oracle ran out of bits: Fundamental Oracle flaw revealed. 64 bits, that's the ticket, ipv6 went 128 bits. 
• The day Kodak died: We developed the world's first consumer digital camera but we could not get approval to launch or sell it because of fear of the effects on the film market... a huge opportunity missed.

To read much more the Internet has to say on Scalability, please click below...

You program in a dynamic language, that runs on a JVM, that runs on a OS designed 40 years ago for a completely different purpose, that runs on virtualized hardware. Does this make sense? We've talked about this idea before in Machine VM + Cloud API - Rewriting The Cloud From Scratch, where the vision is to treat cloud virtual hardware as a compiler target, and converting high-level language source code directly into kernels that run on it.

As new technologies evolve the friction created by our old tool chains and architecture models becomes ever more obvious. Take, for example, what a team at UCSD is releasing: a phase-change memory prototype  - a solid state storage device that provides performance thousands of times faster than a conventional hard drive and up to seven times faster than current state-of-the-art solid-state drives (SSDs). However, PCM has access latencies several times slower than DRAM.

This technology has obvious mind blowing implications, but an interesting not so obvious implication is what it says about our current standard datacenter stack. Gary Athens has written an excellent article, Revamping storage performance, spelling it all out in more detail:

How do you scale an inbox that has multiple highly volatile feeds? That's a problem faced by social networks like Tumblr, Facebook, and Twitter. Follow a few hundred event sources and it's hard to scalably order an inbox so that you see a correct view as event sources continually publish new events.

This can be considered like a view materialization problem in a database. In a database a view is a virtual table defined by a query that can be accessed like a table. Materialization refers to when the data behind the view is created. If a view is a join on several tables and that join is performed when the view is accessed, then performance will be slow. If the view is precomputed access to the view will be fast, but more resources are used, especially considering that the view may never be accessed.

Your wall/inbox/stream is a view on all the people/things you follow. If you never look at your inbox then materializing the view in your inbox is a waste of resources, yet you'll be mad if displaying your inbox takes forever because all your event streams must be read, sorted, and filtered. 

What's a smart way of handling the materialization problem? That's what is addressed in a very good paper on the subject, Feeding Frenzy: Selectively Materializing Users’ Event Feeds, from researchers at Yahoo!, who found:

The best policy is to decide whether to push or pull events on a per producer/consumer basis. This technique minimizes system cost both for workloads with a high query rate and those with a high event rate. It also exposes a knob, the push threshold, that we can tune to reduce latency in return for higher system cost.

I learned about this paper from Tumblr's Blake Matheny, in an interview with him for a forthcoming post. This is broadly how they handle the inbox problem at Tumblr. More details later.

Abstract from the paper:

Who's Hiring?

• Anybody interested in helping manage a 100+ Linux server deployment? Next Big Sound is a an analytics company for the music industry and is looking someone to help them scale.
• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 
• Callfire, one of the largest cloud telephony platforms on the web, is hiring a Sr. Software Engineer. You can learn more here.
• Spokeo is hiring backend & frontend developers, and system administrators to revolutionize the people search industry. Please visit here for more information.

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• ElasticHosts award winning cloud server hosting launches across North America. Adding data centers in Los Angeles and Toronto. Free trial. Just visit our website.
• SingleHop is IaaS. Flexible + On-Demand Public Hosting and Computing Instances. Design and deploy your Public Cloud Instance in minutes. Sign-up now and the first month is $1.
• 1&1 Internet. Flexible, dynamic high performance cloud servers for professionals. 3 Months Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• ScaleOut StateServer® Delivers Map/Reduce Analysis and Scalable Application Performance. Gain competitive advantage with rapid access to business intelligence. Download a free evaluation trial today.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...

With a name like HighScalability... it has to be good:

• Facebook: 1 Billion Users?; Internet Archive: 500,000 users/day, 6PB of data, 150 billion pages, 1000 queries a second; 6,180: The number of patents granted to IBM in 2011; 676: The number of patents granted to Apple in 2011; Live TV is Dead; Kickstarter: 10,000 successfully funded projects; $82bn: Apple's cash hoard; 100 Billion Planets: Our home sweet galaxy; Creative: 100-core system-on-a-chip; 15 million: Lines of code in the Linux kernel; According to Twitter: Justin Bieber > President Obama.
• Quotable quotes:
  • @florind : I just realized that the Santa story is a classical scalability myth.
  • @juokaz : doesn't always use dating sites, but when he does, he finds out about them on High Scalability http://bit.ly/xYfBmq. True story
  • @niclashulting : The Yahoo! homepage is updated 45,000 times every five minutes." A content strategy is vital.
• Google’s Data Center Engineer Shares Secrets of ‘Warehouse’ Computing. Cade Metz interviews Google's guru of the datacenter, Luiz André Barroso: delivering good Internet service requires designing the software and hardware of entire datacenter to work together as one computer; split application pieces across an array of computers; modesty is key, select modest machines with modest processors and spread applications as thin as possible; wimpy cores won't process work fast enough to be useful, that's too thin.

To read much more the Internet has to say on Scalability, please click below...

The Peregrine falcon is a bird of prey, famous for its high speed diving attacks, feeding primarily on much slower Hadoops. Wait, sorry, it is Kevin Burton of Spinn3r's new Peregrine project--a new FAST modern map reduce framework optimized for iterative and pipelined map reduce jobs--that feeds on Hadoops.

If you don't know Kevin, he does a lot of excellent technical work that he's kind enough to share it on his blog. Only he hasn't been blogging much lately, he's been heads down working on Peregrine. Now that Peregrine has been released, here's a short email interview with Kevin on why you might want to take up falconry, the ancient sport of MapReduce.

What does Spinn3r do that Peregrine is important to you? For the rest of the interview please click below...

Jeremiah Peschka with a great a set of Notes on Scalability, just in case you do reach your wildest expectations of success:

1. Build it to Break. Plan for the fact that everything you make is going to break. Design in layers that are independent and redundant.
2. Everything is a Feature. Your application is a set of features created by a series of conscious choices made by considering trade-offs. 
3. Scale Out, Not Up. Purchasing more hardware is easier than coding and managing horizontal resources. 
4. Buy More Storage. Large numbers of smaller, faster drives have more IOPS than fewer, larger drives.
5. You’re Going to Do It Wrong. Be prepared to iterate on your ideas. You will make mistakes.  Be prepared to re-write code and to quickly move on to the next idea.

Please read the original article for a much more expansive treatment. 

Seldom do we get to hear stories of the bumps and bruses earned by a popular website during its formative years. Ross Snyder, a Sr. Software Engineer at Etsy, changes that with an engaging talk he gave at Surge 2011: Scaling Etsy: What Went Wrong, What Went Right.

Ross gives a detailed and honest account of how Etsy went from a raw startup in 2005, to a startup struggling with their success in 2007, to the mean, handmade, super scaling, ops driven machine they’ve become in 2011.

There’s lots to learn from this illuminating story of transformation:

Origin Story

OMG, it's 2012:

• Harry Bombarda Twilight; 200 Million: Chinese online shoppers; Quantum 150 qubit computer: all the power of today's supercomputers;  Sperm: two aspirins worth could repopulate the world; 1 Billion: the number of iOS and Android apps downloaded in a week; Watson: 250 Servers, 2,880 cores, 10 racks, 16 Terabytes RAM, 80 Teraflops; Reddit: 2 Billion Pageviews
• Quotable Quotes:
  • Robert Martin : The hallmark of a really good architecture is that it allows major decisions to be deferred. 
  • Building Memory-efficient Java Applications: Practices and Challenges : More abstractions = less awareness of costs.
  • Ian Muir : When we do something that Microsoft did not anticipate, it's nothing but pain.
  • @kekline : Want to know a secret - NoSQL's rapid growth is really about NoNormalization
  • Jeremy Zawodny : The fact that I can look back on code I wrote a few years ago and identify ways that I’d do it better is good. It means I’m still learning. But the fact that I can successfully resist the urge to change the code is even better.
  • John Boyd : people first, ideas second, hardware last.
• So cool: Glowing bacteria biopixels: The sensor displays of the future. Bacteria talk to each other using quorum sensing, which means they talk using molecules. This doesn't scale to millions of bacteria. The solution: a microfluidic chips were designed to harness the localized trigger and broadcast it to the plethora of shared colonies existing on the chip. Each of the bacteria cells on the microfluidic chip is called a “biopixel." The future of sensing technology is going to be in living sensors.

The Internet has a lot more to Say on Scalability, please read the whispers below...

In the "should I or shouldn't I" debate around deploying SSD, it always helps to have real-world data. Fiesta! with a live-blog summary of a presentation by Kenny Gorman on Shutterfly on MongoDB Performance Tuning.

What if you still need more performance after doing all of this tuning? One option is to use SSDs. Shutterfly uses Facebook’s flashcache: kernel module to cache data on SSD. Designed for MySQL/InnoDB. SSD in front of a disk, but exposed as a single mount point. This only makes sense when you have lots of physical I/O. Shutterfly saw a speedup of 500% w/ flashcache. A benefit is that you can delay sharding: less complexity.

The whole series of posts has a lot of great information and is worth a longer look, especially if you are considering using MongoDB. 

How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve, we get a little insight as to their method for the madness, nothing really detailed, but still interesting.

Problem Setup

Who's Hiring?

• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 
• Callfire, one of the largest cloud telephony platforms on the web, is hiring a Sr. Software Engineer. You can learn more here.
• Spokeo is hiring backend & frontend developers, and system administrators to revolutionize the people search industry. Please visit here for more information.

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• SingleHop is IaaS. Flexible + On-Demand Public Hosting and Computing Instances. Design and deploy your Public Cloud Instance in minutes. Sign-up now and the first month is $1.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• ScaleOut StateServer® Delivers Map/Reduce Analysis and Scalable Application Performance. Gain competitive advantage with rapid access to business intelligence. Download a free evaluation trial today.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...