High Scalability

Scaling Hotjar's Architecture: 9 Lessons Learned

High Scalability - Wed, 2016-06-29 15:56

Hotjar offers free website analytics so they have a challenging mission: handle hundreds of millions of requests per day from mostly free users. Marc von Brockdorff, Co-Founder & Director of Engineering at Hotjar, summarized the lessons they've learned in: 9 Lessons Learned Scaling Hotjar's Tech Architecture To Handle 21,875,000 Requests Per Hour.

In response to the criticism their architecture looks like a hot mess, Erik Näslund, Chief Architect at Hotjar, gives the highlights of their architecture:

  • We use nginx + lua for the really hot code paths where python doesn't quite cut it. No language is perfect and you might have to break out of your comfort zone and use something different every now and then.
  • Redis, Memcached, Postgres, Elasticsearch and S3 are all suitable for different kinds of data storage and we eventually needed them all to be able to query and store data in a cost effective way. We didn't start out using 5 different data-stores though...it's something that we "grew into".
  • Each application server is a (majestic) monolith. Micro-services are one way of architecting things, monoliths are another - I'm still waiting to be convinced that one way is superior to the other when it comes to a smaller team of developers.

What have they learned?

Categories: High Scalability

How Facebook Live Streams to 800,000 Simultaneous Viewers

High Scalability - Mon, 2016-06-27 15:56

Fewer companies know how to build world spanning distributed services than there are countries with nuclear weapons. Facebook is one of those companies and Facebook Live, Facebook’s new live video streaming product, is one one of those services.

Facebook CEO Mark Zuckerberg

The big decision we made was to shift a lot of our video efforts to focus on Live, because it is this emerging new format; not the kind of videos that have been online for the past five or ten years...We’re entering this new golden age of video. I wouldn’t be surprised if you fast-forward five years and most of the content that people see on Facebook and are sharing on a day-to-day basis is video.

If you are in the advertising business what could better than a supply of advertising ready content that is never ending, always expanding, and freely generated? It’s the same economics Google exploited when it started slapping ads on an exponentially growing web.

An example of Facebook’s streaming prowess is a 45 minute video of two people exploding a watermelon with rubber bands. It reached a peak of over 800,000 simultaneous viewers who also racked up over 300,000 comments. That’s the kind of viral scale you can generate with a social network of 1.5 billion users.

As a comparison The 2015 Super Bowl was watched by 114 million viewers with an average 2.36 million on the live stream. On Twitch there was a peak of 840,000 viewers at E3 2015. The September 16th Republican debate peaked at 921,000 simultaneous live streams.

So Facebook is right up there with the state of the art. Keep in mind Facebook would have a large number of other streams going on at the same time as well.

A Wired article quotes Chris Cox, Facebook’s chief product officer, who said Facebook:

  • Has more than a hundred people working on Live. (it started with ~12 and now there are more than 150 engineers on the project)

  • Needs to be able to serve up millions of simultaneous streams without crashing.

  • Need to be able to support millions of simultaneous viewers on a stream, as well as seamless streams across different devices and service providers around the world.

Cox said that “It turns out it’s a really hard infrastructure problem.”

Wouldn't it be interesting if we had some details about how that infrastructure problem was solved? Woe is we. But wait, we do!

 Federico Larumbe from Facebook’s Traffic Team, which works on the caching software powering Facebook’s CDN and the Global Load Balancing system, gave an excellent talk: Scaling Facebook Live, where he shares some details about how Live works.

Here’s my gloss on the talk. It’s impressive.

Origin Story
Categories: High Scalability

Stuff The Internet Says On Scalability For June 24th, 2016

High Scalability - Fri, 2016-06-24 16:03

Hey, it's HighScalability time:


A complete and accurate demonstration of the internals of a software system.

 

If you like this sort of Stuff then please support me on Patreon.
  • 79: podcasts for developers; 100 million: daily voice calls made on WhatsApp; 2,000; cars Tesla builds each week; 2078 lbs: weight it takes to burst an exercise ball; 500 million: Instagram users; > 100M: hours watched per day on Netflix; 400 PPM: Antarctica’s CO2 Level; 2.5 PB: New Relic SSD storage; 

  • Quotable Quotes:
    • Alan Kay: The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
    • @jaykreps: Actually, yes: distributed systems are hard, but getting 100+ engineers to work productively on one app is harder.
    • @adrianco: All in 2016: Serverless Architecture: AWS Lambda, Codeless Architecture: Mendix, Architectureless Architecture: SaaS
    • @AhmetAlpBalkan: "That's MS SQL Server running on Ubuntu on Docker Swarm on Docker Datacenter on @Azure" @markrussinovich #dockercon
    • Erik Darling: EVERYTHING’S EASY WHEN IT’S WORKING
    • @blueben: Bold claim by @brianl: Most of the best tech of the last 10 years has come out of work at Google. #VelocityConf
    • Joe: there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. Because it really never works like that.
    • Eran Hammer: The web is the present and it’s a f*cking mess. Deal with it.
    • @etherealmind: If you believe in DevOps/NetOps you have to believe that leaving Europe is a difficult but better course of action. Small, fast & fluid
    • Sanfilippo: Redis is currently not good for data problems where write safety is very important. One of the main design sacrifices Redis makes in order to provide many good things is data retention. It has best effort consistency and it has a configurable level of write safety, but it’s optimized for use cases where most of the time, you have your data, but in cases of large incidents you can lose a little bit of it. 
    • David Smith: The best time you are ever going to have to make a new app is when there's a new iOS update. Go through the diffs. Go through the What's New? Find something that couldn't be possible before and make an app around that. 
    • @DanielEricLee: There was a timezone bug so I wrote a test and then the test failed because the CI runs in a different timezone and then I became a farmer
    • @jasongorman: Reminds me of someone I know who led a dev team who built something that won an award. None of team invited to awards bash. Only snr mgmt.
    • David Robinson: My advice to graduate students: create public artifacts
    • @cdixon: Because distributed clusters of commodity machines are more powerful.
    • @wmf: 128-port TORs have outrun the compute capacity of most racks, so putting two mini-TORs in 1U is a great idea.
    • msravi: I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibility. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.
    • Mobile Breakfast: What is clear that we will get rid of the classic circuit-switched technology and move to all IP networks fairly soon in the US.
    • Douglas Rushkoff: I think they all come down to how you can optimize your business or the economy for flow over growth; for the circulation of money rather than the extraction of money.
    • Alan Hastings~ [Species] that go into synchrony may be more subject to extinction because a single driver can trigger a collapse
    • @TechCrunch: More cars than phones were connected to cell service in Q1 http://tcrn.ch/28MLtmt  by @kristenhg
    • @docker: "Nobody cares about #containers, it's the application that matters!" - @solomonstre, CTO @Docker #DockerCon
    • @cmeik: The most commercially successful NoSQL database: Lotus Notes.
    • Brittany Fleit: behavior-based push results in open rates 800 percent higher than immediate blasts. Personalizing a message based on individual actions garners much more engagement.

  • Dilbert, of course, nails it on AI.

  • How will hackers be stopped from using Return-oriented programming (ROP) to execute privilege escalation attacks? ROP was created when "clever hackers realized that they could successively invoke snippets of code at the end of existing subroutines as 'proxies' to accomplish the work they needed done." Randomizing memory locations didn't stop them so Intel created new hardware magic called Control-flow Enforcement Technology. Intel added a new "ENDBRANCH" instruction and created a new "shadow stack". It turns out the immense power and beauty of the stack in von neumann architectures is also a great weakness. The story of life. Steve Gibson with an inspiring deep dive on CET in Security Now 565

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability

The Technology Behind Apple Photos and the Future of Deep Learning and Privacy

High Scalability - Mon, 2016-06-20 15:56

There’s a war between two visions of how the ubiquitous AI assisted future will be rendered: on the cloud or on the device. And as with any great drama it helps the story along if we have two archetypal antagonists. On the cloud side we have Google. On the device side we have Apple. Who will win? Both? Neither? Or do we all win?

If you would have asked me a week ago I would have said the cloud would win. Definitely. If you read an article like Jeff Dean On Large-Scale Deep Learning At Google you can’t help but be amazed at what Google is accomplishing. Impressive. Wide ranging. Smart. Systematic. Dominant.

Apple has been largely absent from the trend of sprinkling deep learning fairy dust on their products. This should not be all that surprising. Apple moves at their own pace. Apple doesn’t reach for early adopters, they release a technology when it’s a win for the mass consumer market.

There’s an idea because Apple is so secretive they might have hidden away vast deep learning chops we don’t even know about yet. We, of course, have no way of knowing.

What may prove more true is that Apple is going about deep learning in a different way: differential privacy + powerful on device processors + offline training with downloadable models + a commitment to really really not knowing anything personal about you + the deep learning equivalent of perfect forward secrecy.

Photos vs Photos
Categories: High Scalability

Stuff The Internet Says On Scalability For June 17th, 2016

High Scalability - Fri, 2016-06-17 15:56

Hey, it's HighScalability time:


You've seen the Netflix Death Star microservices map. Here's a map of microbes conversing on your skin.

 

If you like this sort of Stuff then please support me on Patreon.
  • 4281: # of unread articles in my HackerNews feed; 23%: of all corporate cash is held by Microsoft, Apple, Google; 400 million: number of new servers needed by 2020; ~25,740TB: storage Backblaze adds per month; 3 bits: IBM stores per memory cell; 488 million: faked comments by China per year; 90%: revenue Spotify makes fron 30% of users; 780 million: miles of Tesla driving data; 4 days: median time to binge watch a season on Netlix; $33: cost of Nike Air Max; $50 billion: amount Apple has paid out to app developers; $270 million: amount Line makes from selling stickers; 4,600: # of trees Apple will plant aorund the Spaceship; 200 million: Google photos users; $1.8 billion: Series F round for Snapchat; 3x: capacity of the roadway with driverless cars; 138%: growth in Alibaba's cloud; 

  • Quotable Quotes:
    • @swardley: By this time next year, AMZN should be comfortably worth more than IBM + HPQ + HPE + CISCO + VMW + ORCL + NetApp + EMC combined.
    • @pcalcado: The #serverless revolution, or as we call it in my hometown: "JavaScript folks finally find out about CGI"
    • @kellan: Those patents you filed at Y! that would "only ever be used defensively"? Up for sale
    • @mipsytipsy: oooo, the term FaaS (Functions as a Service) is WAY better than #serverless.  Accurate, doesn't irritate me at all!
    • Ian Eyberg:  We need a new paradigm. We really need to be deploying software not the systems the software lives on. We need to be configuring software not the system that they are on. This is a new way of thinking.
    • @amernetflix: 2 Million #PacketsPerSecond on a #aws public #cloudinstance. 
    • @antirez: /me hopes that because at Redis Conf there were already folks telling me “don’t bother too much with Docker, we are moving away”.
    • @swardley: Asked "Do I not like Docker?" - I like Docker but I view containers as an important but for most, ultimately invisible subsystem ...
    • @jeffjarvis: Before AMP, 51% of WaPost users returned in 7 days; after AMP that's up to 63% says David Merrell at #io16
    • @HackerNewsOnion: How LinkedIn Scaled To Billions Of Unread Messages
    • @nzben: 90% Domain Driven Design 10% Switch Statements
    • @igrigorik: 0.3s latency improvement → £8M revenue gain for thetrainline(.com): http://bit.ly/1NiFzdP  - great case study.
    • @thegrugq: A monk asked Satoshi: “Why do you not sign a challenge msg?” Satoshi answered: “signing proves nothing!” The monk was englightened
    • #bitkoans
    • @johnrobb: Stanford:  Apple could manufacture its products in the US with robotic assembly at same price as China now
    • @Khanoisseur: Amazon changes prices 2.5 million times a day. Wal-Mart and Best Buy change prices 50,000 times over entire month.
    • @Khanoisseur: Amazon pushes code live every 12 seconds and can test a feature on 5000 users by turning it on for just 45 seconds
    • southpolesteve: I have one hard example I can share. We had a node service that was running on ec2 and cost ~$2500/mo. Moved the code directly over to lambda. Now ~$400/mo.
    • Keith Chen~ The behavioural economist at UCLA said users are willing to accept surge pricing increases as high as 9.9 times the normal price of a ride if their smartphone's battery is close to dying.
    • @Bill_Gross: With its always on cellular connection, every 10 hours Tesla gets a MILLION miles of additional data
    • @timallenwagner: Nice example of converting batch system to real-time serverless analytics!
    • @danielbryantuk: "In one study 35% of catastrophic failures were caused by what I call 'dev laziness' " @caitie #qconnewyork
    • @dechampsgu~ "Improving Anything but The Bottleneck is Close to Meaningless" @ziobrando  #dddx
    • @heinrichhartman: "We [#SQLite] don't compete against Oracle, we compete against fopen(3)" Richard Hipp on @changelog https://changelog.com/201/  (47:00)
    • @Pinboard: Any complex website is an interdependent ballet of dozens of mutually supporting services. You can’t reduce it to a word like “up” or “down”
    • @kevinmarks: “The web is already decentralized,” Mr. Berners-Lee said. “The problem is the dominance of one search engine…
    • CHARITY.WTF: I’ve seen what happens when application developers think they don’t have to care about the skills associated with operations engineering.  When they forget that no matter how pretty the abstractions are, you’re still dealing with dusty old concepts like “persistent state” and “queries” and “unavailability” and so forth, or when they literally just think they can throw money at a service to make it go faster because that’s totally how services work.
    • @beaucronin: A dollar-an-hour EC2 instance today would have been at or near the top of Top500 in the early/mid 90s
    • @cloud_opinion: Is this the real ops? / Is this just devops? / Caught in a deploy / No escape from techdebt / Open your eyes / Look up to the cloud and see #serverles
    • lostcolony: In context, I'm pretty sure it was just saying "Unlike computation, bandwidth, and memory size, we haven't seen much improvement in latency, and even if we focused on it, we have a very clear limit we can't get past".
    • People have said a lot more Stuff. Read the full article to see what've they said.

  • You know all those elaborate A/B testing and new feature testing systems built into websites so companies can gather data and learn more about how their product works in the real world? It's not just for websites. Tesla Tests Self-Driving Functions with Secret Updates to Its Customers’ Cars. It appears by deploying actual cars with real drivers Tesla has the Big Data advantage when it comes to creating self-driving cars. Though like much of the modern world it's damn spooky: Tesla can pull down data from the sensors inside its customers’ vehicles to see how people are driving and the road and traffic conditions they experience. It uses that data to test the effectiveness of new self-driving features. The company even secretly tests new autonomous software by remotely installing it on customer vehicles so it can react to real road and traffic conditions, without controlling the vehicle.

  • Can you be a Libertarian and use the government as an amplification attack on an enemy? The Stunning and Expected End of Gawker.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability

The Image Optimization Technology that Serves Millions of Requests Per Day

High Scalability - Wed, 2016-06-15 15:56

This article will touch upon how Kraken.io built and scaled an image optimization platform which serves millions of requests per day, with the goal of maintaining high performance at all times while keeping costs as low as possible. We present our infrastructure as it is in its current state at the time of writing, and touch upon some of the interesting things we learned in order to get it here.

Let’s make an image optimizer

You want to start saving money on your CDN bills and generally speed up your websites by pushing less bytes over the wire to your user’s browser. Chances are that over 60% of your traffic are images alone.

Using ImageMagick (you did read ImageTragick, right?) you can slash down the quality of a JPEG file with a simple command:

$ convert -quality 70 original.jpg optimized.jpg

$ ls -la

-rw-r--r--  1 matylla  staff  5897 May 16 14:24 original.jpg

-rw-r--r--  1 matylla  staff  2995 May 16 14:25 optimized.jpg

Congratulations. You’ve just brought down the size of that JPEG by ~50% by butchering it’s quality. The image now looks like Minecraft. It can’t look like that - it sells your products and services. Ideally, images on the Web should have outstanding quality and carry no unnecessary bloat in the form of excessively high quality or EXIF metadata.

You now open your favourite image-editing software and start playing with Q levels while saving a JPEG for the Web. It turns out that this particular image you test looks great at Q76. You start saving all your JPEGs with quality set to 76. But hold on a second… Some images look terrible even with Q80 while some would look just fine even at Q60.

Ok. You decide to automate it somehow - who wants to manually test the quality of millions of images you have the “privilege” of maintaining. So you create a script that generates dozens of copies of an input image at different Q levels. Now you need a metric that will tell you which Q level is perfect for a particular image. MSE? SSIM? MS-SSIM? PSNR? You’re so desperate that you even start calculating and comparing perceptual hashes of different versions of your input image.

Some metrics perform better than others. Some work well for specific types of images. Some are blazingly fast while the others take a long time to complete. You can get away by reducing the number of loops in which you process each image but then chances are that you miss your perfect Q level and the image will either be heavier than it could be or quality degradation will be too high.

And what about product images against white backgrounds? You really want to reduce ringing/haloing artifacts around the subject. What about custom chroma-subsampling settings on per-image basis? That red dress against white background looks all washed-out now. You’ve learned that stripping EXIF metadata will bring the file size down a bit but you’ve also removed Orientation tag and now your images are all rotated incorrectly.

And that’s only the JPEG format. For your PNGs probably you’d want to re-compress your 7-Zip or Deflate compressed images with something more cutting-edge like Google’s Zopfli. You spin up your script and watch the fan on your CPU start to melt...

Categories: High Scalability

Sponsored Post: Gusto, Awake Networks, Spotify, Telenor Digital, Kinsta, Aerospike, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

High Scalability - Tue, 2016-06-14 16:10

Who's Hiring?
  • IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.

  • Awake Networks is an early stage network security and analytics startup that processes, analyzes, and stores billions of events at network speed. We help security teams respond to intrusions with super-human  efficiency and provide macroscopic and microscopic insight into the networks they defend. We're looking for folks that are excited about building systems that handle scale in a constrained environment. We have many open-ended problems to solve around stream-processing, distributed systems, machine learning, query processing, data modeling, and much more! Please check out our jobs page to learn more.

  • Site Reliability Engineer Manager. We at Spotify are looking for an engineering leader (Chapter Lead) to manage the NYC part of the Site Reliability Engineering team. This team works with the infrastructure that powers the music service used by millions of users, built by hundreds of engineers. We create tools, develop infrastructure, and teach good practices to help Spotify engineers move faster. As a Chapter Lead your primary responsibility is to the people on your team: ensuring that the members are growing as engineers, doing valuable work, performing well, and generally having a great time at Spotify. Read more and apply here

  • Site Reliability Engineer. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. Read more and apply here

  • Backend Engineer. We at Spotify are looking for senior backend engineers to join our team of talented engineers that share a common interest in distributed backend systems, their scalability and continued development.  You will build the backend systems that power our application, scale highly distributed systems, and continuously improve our engineering practices. Read more and apply here

  • Security Engineer. The security team at Spotify is a distributed team supporting autonomous development teams with a focus on raising security awareness, sharing responsibility, and building tools. We aim to constantly improve the security posture for our fast-paced, rapidly-changing environment in a manner that will keep up with our scale. We’re knowledgeable in many domains of security and are willing to teach (and learn) from anyone at the company. Read more and apply here

  • Data Architect. You will be a key figure in a rapidly growing team, where the role will highly depend on you. You must have extensive experience in Cloud Computing and AWS and deeply understand databases and/or Information Architecture (PostgreSQL, Cassandra, MongoDB, Redis, etc.). And if you also know your way in the Hadoop ecosystem (including Spark and HDFS), Kafka, Cassandra and other big data technologies, this will be more than enough. You have an understanding of how to structure the data sources and data feeds of the Data Insights big data solution, plan for integration and maintenance of the data as well as have an eye on the logical design and on how the data flows through the different stages. Please apply here at Telenor Digital.

  • Data Engineers. You know Java, and possibly Clojure or Scala, are effective in a Linux terminal (shell scripting, configuration files, etc.), have experience with some SQL database, preferably PostgreSQL, have experience with Apache Kafka, Apache Spark, Elasticsearch. You enjoy automating things and building systems. Machine learning experience is considered a plus, and Continuous Integration + delivery is important to you, and writing tests a given. You are humble and passionate; you like to listen and can understand the viewpoints of others and strive to be a good dialog partner, but you can focus on delivery once a direction is decided. Please apply here at Telenor Digital.

  • Software Engineers, Analytics. You've got strong front-end developer skills: HTML, CSS, and Javascript, with knowledge of D3.js or other charting libraries - Clojurescript is a plus; have worked with various programming languages, like Java, Clojure, or Python; have experience with SQL (PostgreSQL). You have experience with Cloud Computing, especially with AWS, a deep foundation in computer science; data structures, algorithms and programming languages, as well as networking and concurrency; exposure to architectural patterns of a large, high-scale web applications; experience with shell scripting, configuration files, etc. and enjoy automating things and building systems. Please apply here at Telenor Digital.

  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • NoSQL Databases & Docker Containers: From Development to Deployment. What is Docker and why is it important to Developers, Admins and DevOps when they are using a NoSQL database? Find out in this on-demand webinar by Alvin Richards, VP of Product at Aerospike, the enterprise-grade NoSQL database. The video includes a demo showcasing the core Docker components (Machine, Engine, Swarm and Compose) and integration with Aerospike. See how much simpler Docker can make building and deploying multi-node, Aerospike-based applications!  

  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.
Cool Products and Services
  • Kinsta provides high speed, automatically scalable managed WordPress hosting services for businesses large and small. All servers run on Google Cloud and all individual sites are completely compartmentalized using the latest LXD technology. All sites include powerful SSH access and tools like Git and WP-CLI are available out-of-the-box.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: High Scalability

Sponsored Post: Spotify, Telenor Digital, Kinsta, Aerospike, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

High Scalability - Thu, 2016-05-12 16:06

Who's Hiring?
  • Site Reliability Engineer Manager. We at Spotify are looking for an engineering leader (Chapter Lead) to manage the NYC part of the Site Reliability Engineering team. This team works with the infrastructure that powers the music service used by millions of users, built by hundreds of engineers. We create tools, develop infrastructure, and teach good practices to help Spotify engineers move faster. As a Chapter Lead your primary responsibility is to the people on your team: ensuring that the members are growing as engineers, doing valuable work, performing well, and generally having a great time at Spotify. Read more and apply here

  • Site Reliability Engineer. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. Read more and apply here

  • Backend Engineer. We at Spotify are looking for senior backend engineers to join our team of talented engineers that share a common interest in distributed backend systems, their scalability and continued development.  You will build the backend systems that power our application, scale highly distributed systems, and continuously improve our engineering practices. Read more and apply here

  • Security Engineer. The security team at Spotify is a distributed team supporting autonomous development teams with a focus on raising security awareness, sharing responsibility, and building tools. We aim to constantly improve the security posture for our fast-paced, rapidly-changing environment in a manner that will keep up with our scale. We’re knowledgeable in many domains of security and are willing to teach (and learn) from anyone at the company. Read more and apply here

  • Data Architect. You will be a key figure in a rapidly growing team, where the role will highly depend on you. You must have extensive experience in Cloud Computing and AWS and deeply understand databases and/or Information Architecture (PostgreSQL, Cassandra, MongoDB, Redis, etc.). And if you also know your way in the Hadoop ecosystem (including Spark and HDFS), Kafka, Cassandra and other big data technologies, this will be more than enough. You have an understanding of how to structure the data sources and data feeds of the Data Insights big data solution, plan for integration and maintenance of the data as well as have an eye on the logical design and on how the data flows through the different stages. Please apply here at Telenor Digital.

  • Data Engineers. You know Java, and possibly Clojure or Scala, are effective in a Linux terminal (shell scripting, configuration files, etc.), have experience with some SQL database, preferably PostgreSQL, have experience with Apache Kafka, Apache Spark, Elasticsearch. You enjoy automating things and building systems. Machine learning experience is considered a plus, and Continuous Integration + delivery is important to you, and writing tests a given. You are humble and passionate; you like to listen and can understand the viewpoints of others and strive to be a good dialog partner, but you can focus on delivery once a direction is decided. Please apply here at Telenor Digital.

  • Software Engineers, Analytics. You've got strong front-end developer skills: HTML, CSS, and Javascript, with knowledge of D3.js or other charting libraries - Clojurescript is a plus; have worked with various programming languages, like Java, Clojure, or Python; have experience with SQL (PostgreSQL). You have experience with Cloud Computing, especially with AWS, a deep foundation in computer science; data structures, algorithms and programming languages, as well as networking and concurrency; exposure to architectural patterns of a large, high-scale web applications; experience with shell scripting, configuration files, etc. and enjoy automating things and building systems. Please apply here at Telenor Digital.

  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.

  • NoSQL Databases & Docker Containers: From Development to Deployment. What is Docker and why is it important to Developers, Admins and DevOps when they are using a NoSQL database? Find out in this on-demand webinar by Alvin Richards, VP of Product at Aerospike, the enterprise-grade NoSQL database. The video includes a demo showcasing the core Docker components (Machine, Engine, Swarm and Compose) and integration with Aerospike. See how much simpler Docker can make building and deploying multi-node, Aerospike-based applications! 
Cool Products and Services
  • Kinsta provides high speed, automatically scalable managed WordPress hosting services for businesses large and small. All servers run on Google Cloud and all individual sites are completely compartmentalized using the latest LXD technology. All sites include powerful SSH access and tools like Git and WP-CLI are available out-of-the-box.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: High Scalability

Performance and Scaling in Enterprise Systems

High Scalability - Wed, 2016-05-11 15:56

This is a guest post from Vlad Mihalcea the author of the High-Performance Java Persistence book, on the notion of performance and scalability in enterprise systems.

An enterprise application needs to store and retrieve as much data and as fast as possible. In application performance management, the two most important metrics are response time and throughput.

The lower the response time, the more responsive an application becomes. Response time is, therefore, the measure of performance. Scaling is about maintaining low response times while increasing system load, so throughput is the measure of scalability.

Response time and throughput
Categories: High Scalability

Stuff The Internet Says On Scalability For May 6th, 2016

High Scalability - Fri, 2016-05-06 15:56

Hey, it's HighScalability time:


Who wants in on the over? We are not alone if the probability a habitable zone planet develops a technological species is larger than 10-24.

 

If you like this sort of Stuff then please support me on Patreon.
  • 100,000+: bare metal servers run by Twitter; 10 billion: Snapchat videos delivered daily; $2.57 billion: AWS fourth quarter revenues; 40 light years: potentially habitable planets; 1700: seed banks around the world; 560x: throughput after SSD optimization; 12: data science algorithms; $2.8 billion: new value of Pivotal’s Cloud Foundry;  

  • Quotable Quotes:
    • @skap5: Pied Piper's product is its stock and anything that makes its price go up! #SiliconValley
    • Seth Godin: It pays to have big dreams but low overhead. 
    • Craig Venter~ Our knowledge of the genome hasn't changed a lot since 2003, but it's about to start changing rapidly. One of the key things for understanding the genome is to get very large numbers of genomes so we can understand out of the 6.2 billion or so letters of genetic code the less than 3% that we have different amongst the entire human population. We need very large data sets to understand the differences and significances. That's where the cost and speed of sequencing has had such an immediate impact. 
    • @elonmusk: Rocket reentry is a lot faster and hotter than last time, so odds of making it are maybe even, but we should learn a lot either way
    • @EconBizFin: The space race was once between capitalism and communism. Now it's individual capitalists
    • Grit: substitute nuance for novelty. Rather than constantly moving on to a new thrill try to find another dimension of the thing you are already doing to make it more thrilling. 
    • David Rosenthal: Overall the message is: Storage Will Be Much Less Free Than It Used To Be
    • StorageMojo: The losers are the systems that make customers pay for features they no longer need. Winners will successfully blend ease of use with performance and availability – at a competitive price.
    • Tim Harford: These distractions were actually grists to their creative mill. They were able to think outside the box because their box was full of holes.
    • Juan Enriquez~ Plastics was the wrong advice to Dustin Hoffman in The Graduate. The word should have been silicon. In 2015 the word is lifecode, the various means we have to program life.
    • Benjamin Treynor Sloss: If you've ever wondered about how run reliable services, this beautifully written intro from the SRE book is the best 5-minute guide on the topic.
    • cs702: Without AWS, Amazon would have reported losses! 
    • Kode Vicious: A single cache miss is more expensive than many instructions.
    • @aminggs: “your database… unlikely to provide serializability, your multi-core processor… unlikely to provide linearizability” 
    • @mrogati: A decade in academia taught me a bunch of sophisticated algorithms; a decade in industry taught me when not to use them.
    • @mjpt777: Hardware tries so hard to make software fast; software tries so hard to make hardware slow.
    • @jyarow: Echo sold 3 million units. Gets stories that it’s next great business for Amazon. Apple Watch sold 12 million units, gets panned as a flop.
    • @balajis: 5/ At that time, the highest truth comes not from faith in god or trust in the state, but from the ability to check the math of the network.
    • Benedict Evans: The smartphone install base does have a lot of room to grow, but that's a function of replacement at close to existing volumes, and even that will be largely done in a few more years. Hence: smartphone sales growth is slowing down. 
    • @giupan: Colocated teams where Devs are sitting together with Product and UX outperform distributed teams. Don't split up skills @cagan #craftconf
    • Mathias Bynens: To me, this stuff is extremely interesting on a technical level. It’s also a little scary, however, to realize that malicious actors can use these techniques to invade your privacy while you’re browsing the web, without you ever knowing.
    • Le Corbusier: yes, the Parthenon is perhaps the most beautiful instance, the perfect example of a particular standard of architecture. The Parthenon may have achieved the platonic ideal of the standard of architecture we’ve previously established. But there are many possible standards to acknowledge, each dependent on need and use, and standards are established by experiment.
    • PaulHoule: Atom chips have always been crippled to keep them from cannibalizing more expensive chips. Skylake is a fine tablet chip, in fact, that's really what Skylake is good for. They are probably producing them in high enough numbers now that they can give up on Atom
    • Chau Tu: CyArk wants to preserve our world’s important cultural heritage sites before they turn to dust...with new imaging technologies to steadily build a digital archive of the past, for the future. 
    • Neill Turner: Over time i think OpenStack will be a niche product for large corporates that don’t want to use public clouds. For everyone else they with be doing hybrid IT – that is extending their existing IT infrastructure into the Public Cloud. When they see what is left to run to run outside public clouds then they can see where to take that portion of the workloads.

  • What if going to Mars is how we fix our economy? Trump: Before going to Mars, America needs to fix its economy. A problem can't be solved at the same level it was created. Someone smart said that once. Isn't expanding the economy in to space the only way we'll be able to generate the constant growth a modern economy so desperately devowers? Walls don't lift boats.

  • Is it dystopian to hire real meat people to train your AI? Interesting question posed by John Robb in a tweet: "they were there not to work, but to serve as training modules for Facebook’s algorithm" Journos at Fbook

  • Peter Bailis offers in a heartfelt visionary article four pieces of advice to get the database community out of its identity crisis: 1) Kill the reference architecture and rethink our conception of “database.” 2) Solve new, emerging, real problems outside traditional relational database systems. 3) Use data-intensive tools, both the tools that you’re building and the tools that others have built. 4) Do bold, weird, and hard projects and actually follow through. Examples in action: Peter Alvaro’s work on Molly and Lineage-Driven Fault Injection; Chris Ré’s work on DeepDive; A recent project I wish the database community had done is TensorFlow at Google. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability

Gone Fishin'

High Scalability - Mon, 2016-05-02 15:56

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. See you on down the road...

Categories: High Scalability

Stuff The Internet Says On Scalability For April 29th, 2016

High Scalability - Fri, 2016-04-29 15:56

Hey, it's HighScalability time:


The Universe in one image (Pablo Budassi). Imagine an ancient being leaning over, desperately scrying to figure out what they have wrought.

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • 50 minutes: Facebook daily average use; 1.65 billion: Facebook Monthly active users; 25PB: size of Internet archive; 7 years: speedup of encryption adoption from the Snowden revelations; 10 million: strands of DNA Microsoft is buying to store data; 300TB: open data from CERN; 2PB: data from PanSTARRS' imaging survey; 100 billion: words translated by Google per day; 204 million: Weather Channel views in March on Facebook; 

  • Quotable Quotes:
    • @antevens: -> Describe your perfect date. ......<- YYYY-MM-DD HH:MM:SS.XXXXXX
    • @ValaAfshar: 1995: top 15 Internet companies worth $17 billion. 2015: top 15 Internet companies worth $2.4 trillion.
    • @BenedictEvans: The move to mobile took away Facebook's monopoly of social, but gave it much greater scale, engagement & revenue potential.
    • Sundar Pichai: We will move from mobile first to an AI first world.
    • Chris Sacca~ We [Google] literally could feel a scale that had never been felt before on the planet. We had a globe where you could visualize in searches in real-time. A dot would indicate every single search on the planet. In the middle of the night there would be a search in the Gobi desert.
    • @stack72: Just had a recruiter contact me about a role with "microservices on a servers architecture” - twice I’ve seen that now in 2 days #TheFuture?
    • Jason Waxman [Intel]: We see that the world is moving to scale computing in data centers. Our projection is between 70 and 80 percent of the compute, network, and storage will be going into what we call scale data centers by 2025.
    • @BenedictEvans: In 2009 only half of Facebook's MAUs were on it every day. Mobile has taken that to 2/3, at much greater scale.
    • Dan Rayburn: Amazon and Google Enticing Customers With Cheap Storage, But Beware Of Egress Charges
    • @manumarchal: CERN LHC computing challenge is more than 400k CPUs + 300PB of data. It's is also global distribution. #dotScale
    • @bridgetkromhout: Decouple and segregate systems requiring different trust levels for faster iteration. @adrianco #craftconf
    • @dkalintsev: GE on stage at AWS Summit: “50% TCO saving compared to best what we could do in-house”
    • @etherealmind: How messed up was GE management to let their costs get this out of control ?
    • @Ellen_Friedman: #dotscale Oliver Keeble CERN - superb: computing is key. Collisions are transient; data is persisted at huge scale
    • @stratecheryAggregation Theory leads to monopoly; expect more antitrust cases, but only in Europe
    • @kelseyhightower: Moving to microservices won't save you. Borrowing money in smaller chunks doesn't change the fact that you're broke.
    • @jrauser: 1/ Inspired by this HN comment …, I offer a story about software rewrites and Bezos as a technical leader.
    • aytekin: This is a story that has happened over and over again. When you rewrite software, you lose all those hundreds of tiny things which were added for really good reasons. Don't do it blindly.
    • @BWJones: The F-35 program, which at $1.5 T would fund the entire NIH biomedical research portfolio for 41 years.
    • @balinski: "Centralization is a disease" #dotScale #scalability #cloudcomputing
    • Tony Bain: So despite the noise surrounding NoSQL, in a head to head comparison of volume of use, NoSQL use seems so very small.  At a guess, I would predict that for every NoSQL database in existence there would be at least 1000 relational databases.  Probably more.  You would be forgiven for thinking NoSQL use was almost insignificant. 
    • @jaksprats: NVM is gonna put big data on a single machine, very interesting for non-BulkSynchronousParallel GraphDBs like Neo4j
    • @frontofstore: US department stores' sales per sq ft down 26% in last ten years - many closures forecast, anchors killing malls.
    • There are even more Quotable Quotes in the full article. See you there.

  • If you thought HyperCard was a trip you were correct. Bill Atkinson in a fascinating two part Triangulation interview (12) shared that HyperCard was inspired by a LSD trip. It's a far ranging interview that covers Steve Jobs, why the movies about Jobs sucked, Apple's early days, the web's HyperCard inspiration, photography, spirituality, color theory, philosophy, learning, and lots more.

  • In case you were wondering (I certainly was): Pied Piper compression (Silicon Valley HBO). This is the Pied Piper code shown on Silicon Valley HBO Season 3 Episode 1. Worth a deca-unicorn or two.

  • Design Details has a fun podcast with Facebookers talking about Facebook bots and the Facebook design process in general. 124: Dazzle (feat. Jeremy Goldberg). Are bots useful? (yes, but not a convincing argument). Do we have to be nice to bots? (to a point because you never know who if you are talking to a person). Bots aren't all automated, they can be a combination of automated and human interactions. Bots should use strategies to help convince people they are talking with another human, like playing with typing indicator delays to simulate typing. Same for simulating reading. Animation and delays should speed up over time. Regressive design, the idea that over time parts of the UI remove themselves as users use the application more. Fight for designs you believe in. Understand, identify, execute. Truly understand what you are doing at a deep level. Identify the things you can be the most impactful on. Facebook measures you on impact. Lots of talk about design crits and pillars and pillar centered design crits. We often think of ourselves as problem solvers, our job isn't so much problem solving as communicating proposed solutions to problems. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability

How Walmart Canada’s responsive redesign boosted conversions by 20%: a case Study

High Scalability - Thu, 2016-04-28 15:56

With conversion optimization on the rise, it is a great idea to look into case studies which help you learn and adopt them to your personal needs positively. To find the material on conversion optimization you need from case studies, here are some few pointers:

  • Find out which case studies reflect your situation currently in business and your future aspirations.
  • Find out why a certain aspect of a case study worked and how to adopt it to specifically address your website’s needs.
  • Ensure you keep the references of your case study in order to go back to them when you need them.
Find out which case studies reflect your situation currently in business and your future aspirations. Find out why a certain aspect of a case study worked and how to adopt it to specifically address your website’s needs. Ensure you keep the references of your case study in order to go back to them when you need them.

Find out which case studies reflect your situation currently in business and your future aspirations.Find out why a certain aspect of a case study worked and how to adopt it to specifically address your website’s needs.Ensure you keep the references of your case study in order to go back to them when you need them.

Receptive web design is time consuming and requires monetary resources to implement. It took Walmart Canada almost a year of work to refine the site and make it fully responsive. However, their results show that it provided a Return on Investment in months owing to the increased revenue of mobile gadgets.

Research

Categories: High Scalability

The Platform Advantage of Amazon, Facebook, and Google

High Scalability - Wed, 2016-04-27 15:56

Where’s the mag­ic? [Amazon] The databas­ing and stream­ing and sync­ing in­fras­truc­ture we build on is pret­ty slick, but that’s not the se­cret. The man­age­ment tools are nifty, too; but that’s not it ei­ther. It’s the trib­al knowl­edge: How to build Cloud in­fras­truc­ture that works in a fal­li­ble, messy, un­sta­ble world.

Tim Bray, Senior Principal Engineer at Amazon, in Cloud Eventing

Ben Thompson makes the case in Apple's Organizational Crossroads and in a recent episode of Exponent that Apple has a services problem. With the reaching of peak iPhone Apple naturally wants to turn to services as a way to expand revenues. The problem is Apple has a mixed history of delivering services at scale and Ben suggests that the strength of Apple, its functional organization, is a weakness when it comes to making services. The same skill set you need to create great devices is not the same skill set you need to create great services. He suggests: “Apple’s services need to be separated from the devices that are core to the company, and the managers of those services need to be held accountable via dollars and cents.”

If Apple has this problem they are not the only ones. Only a few companies seemed to have cross the chasm of learning how to deliver a stream of new features at a worldwide scale: Amazon, Facebook, and Google. And of these Amazon is the clear winner.

This is the Amazon Web Services console, it shows the amazing number of services Amazon produces, and it doesn’t even include whole new platforms like the Echo:

Categories: High Scalability

Sponsored Post: Aerospike, TrueSight Pulse, Redis Labs, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

High Scalability - Tue, 2016-04-26 15:56

Who's Hiring?
  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.

  • How can your business stand out from the crowd? Bringing to market an innovative differentiation – without too many technical challenges – can be the key. The most forward-looking organizations are coding business logic using the fastest, most agile and scalable technologies available today. In a webinar on May 11 entitled “Exposing Differentiation: A New Era of Scalable Infrastructure Arrives”, Data Scientist Dez Blanchfield and Chief Analyst Dr. Robin Bloor will explain how a nexus of innovations has transformed what’s possible. They’ll be briefed by Brian Bulkowski, CTO and Co-Founder of Aerospike (the high-performance NoSQL database), who will discuss how leading companies are changing their infrastructure to meet the new demands of customized digital experiences, fraud prevention, risk analysis, and other application and data uses. Sign up here to reserve your seat!
Cool Products and Services
  • TrueSight Pulse is SaaS IT performance monitoring with one-second resolution, visualization and alerting. Monitor on-prem, cloud, VMs and containers with custom dashboards and alert on any metric. Start your free trial with no code or credit card.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: High Scalability

The Joy of Deploying Apache Storm on Docker Swarm

High Scalability - Mon, 2016-04-25 15:56

This is a guest repost from Baqend Tech on deploying and redeploying an Apache Storm cluster on top of Docker Swarm instead of deploying on VMs. It's an interesting topic because of the experience Wolfram Wingerath called it "a real joy", which is not a phrase you hear often in tech. Curious, I asked what made using containers such a good experience over using VMs? Here's his reply:

Being pretty new to Docker and Docker Swarm, I'm sure there are many good and bad sides I am not aware of, yet. From my point of view, however, the thing that makes deployment (and operation in general) on top of Docker way more fun than on VMs or even on bare metal is that Docker abstracts from heterogeneity and many issues. Once you have Docker running, you can start something like a MongoDB or a Redis server with a single-line statement. If you have a Docker Swarm cluster, you can do the same, but Docker takes care of distributing the thing you just started to some server in your cluster. Docker even takes care of downloading the correct image in case you don't have it on your machine right now. You also don't have to fight as much with connectivity issues, because every machine can reach every other machine as long as they are in the same Docker network. As demonstrated in the tutorial, this even goes for distributed setups, as long as you have an _overlay_ network.

 

When I wrote the lines you were quoting in your email, I had a situation in the back of my head that had occurred a few months back when I had to set up and operate an Apache Storm cluster with 16+ nodes. There were several issues such as my inexperience with AWS (coming from OpenStack) and strange connectivity problems relating to Netty (used by Storm) and AWS hostname resolution that had not occurred in my OpenStack setup and eventually cost us several days and several hundred bucks to fix. I really think that you can shield from problems like that by using Docker, simply because your environment remains the same: Docker.

On to the tutorial...
Categories: High Scalability

Stuff The Internet Says On Scalability For April 22nd, 2016

High Scalability - Fri, 2016-04-22 15:56

Hey, it's HighScalability time:


A perfect 10. Really stuck that landing. Nadia Comaneci approves.

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • $1B: Supercell’s Clash Royale projected annual haul; 3x: Messenger and WhatsApp send more messages than SMS; 20%: of big companies pay zero corporate taxes; Tens of TB's RAM: Netflix's Container Runtime; 1 Million: People use Facebook over Tor; $10.0 billion: Microsoft raining money in the cloud; 

  • Quotable Quotes:
    • @nehanarkhede: @LinkedIn's use of @apachekafka:1.4 trillion msg/day, 1400 brokers. Powers database replication, change capture etc
    • @kenkeiter~ Full-duplex on a *single antenna* -- this is huge.  (single chip, too -- that's the other huge part, obviously) 
    • John Langford: In the next few years, I expect machine learning to solve no important world issues.
    • Dan Rayburn: By My Estimate, Apple’s Internal CDN Now Delivers 75% Of Their Own Content
    • @BenedictEvans: If Google sees the device as dumb glass, Apple sees the cloud as dumb pipes & dumb storage. Both views could lead to weakness
    • @JordanRinke: We need less hackathons, more apprenticeships. Less bootcamps, more classes. Less rockstars, more mentors. Develop people instead of product
    • @alicegoldfuss: Nagios screaming / Data center ablaze? No / Cable was unplugged
    • Mark Bates: As I was working on the software part time, I was keen to minimise the [cognitive] scope required when making changes. A RoR monolith was the best choice in this case.
    • Google: Our tests have shown that AMP documents load an average of four times faster and use 10 times less data than the equivalent non-amp’ed result.
    • @stevesi: In earning's call @sundarpichai says going “from mobile-first to AI-first world" emphasizing AI and machine learning across services.
    • Rex Sorgatz: Unfortunately, the entire thesis of my story is that having the history of recorded music in your pocket dictates that you will develop tastes outside “the usual.”
    • Newzoo: Clash Royale has rocketed to such quick success because of its strong core gameplay elements combined with some serious pressure to spend real money to keep up with your friends
    • vgt:  I'm going to plug Google Cloud's Preemptible VMs as a simpler alternative to Spot Instances: - Preemptible VMs are sold at a fixed 70% off discount, removing pricing volatility entirely
    • @mfdii~ "Cloud Native" is code words for "rewrite the entire f*cking app"
    • There are so many great Quotable Quotes this week they wouldn't all fit in the summary. Please see the full post to read them all.

  • Imperfection as a strategy. Why a Chip That’s Bad at Math Can Help Computers Tackle Harder Problems: In a simulated test using software that tracks objects such as cars in video, Singular’s approach [computer chips are hardwired to be incapable of performing mathematical calculations correctly] was  capable of processing frames almost 100 times faster than a conventional processor restricted to doing correct math—while using less than 2 percent as much power.

  • You have to fight magic with magic, super-villains with super-heroes, and algorithms with algorithms. How I Investigated Uber Surge Pricing in D.C. Also, Investigating the algorithms that govern our lives.

  • Mitchell Hashimoto in The Cloudcast #246  on some cloud trends. Seeing a lot of interest in non-Amazon clouds right now. A lot of interest in Azure is coming from more boring successful companies, not hot Silicon Valley startups.  This is not a clean market segmentation, but there are three flavors of cloud: Google Compute for the green field startup crowd, Amazon for enterprise, and Azure for super-enterprise. One enterprise attractor for Azure is Azure Stack, an on-premises solution. Mitchell is seeing a broad adoption of the cloud across industries you may not expect to be using the cloud. Also seeing a transition to a multi-cloud strategy to create pricing leverage. The idea seems to be to rehearse and plan to move to another cloud, though they may not actually do it, but when pricing negotiations come up there's a lot of leverage saying you can move to a completely different platform. The cloud is not so much a pay as you go model for this use case, it's more about trying to lock-in long term cost savings. International companies are interested in price, but also what features are available in different regions and when they become available.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability

How Twitter Handles 3,000 Images Per Second

High Scalability - Wed, 2016-04-20 15:56

Today Twitter is creating and persisting 3,000 (200 GB) images per second. Even better, in 2015 Twitter was able to save $6 million due to improved media storage policies.

It was not always so. Twitter in 2012 was primarily text based. A Hogwarts without all the cool moving pictures hanging on the wall. It’s now 2016 and Twitter has moved into to a media rich future. Twitter has made the transition through the development of a new Media Platform capable of supporting photos with previews, multi-photos, gifs, vines, and inline video.

Henna Kermani, a Software Development Engineer at Twitter, tells the story of the Media Platform in an interesting talk she gave at Mobile @Scale London: 3,000 images per second. The talk focuses primarily on the image pipeline, but she says most of the details also apply to the other forms of media as well.

Some of the most interesting lessons from the talk:

  • Doing the simplest thing that can possibly work can really screw you. The simple method of uploading a tweet with an image as an all or nothing operation was a form of lock-in. It didn’t scale well, especially on poor networks, which made it difficult for Twitter to add new features.

  • Decouple. By decoupling media upload from tweeting Twitter was able independently optimize each pathway and gain a lot of operational flexibility. 

  • Move handles not blobs. Don’t move big chunks of data through your system. It eats bandwidth and causes performance problems for every service that has to touch the data. Instead, store the data and refer to it with a handle.

  • Moving to segmented resumable uploads resulted in big decreases in media upload failure rates.

  • Experiment and research. Twitter found through research that a 20 day TTL (time to live) on image variants (thumbnails, small, large, etc) was a sweet spot, a good balance between storage and computation. Images had a low probability of being accessed after 20 days so they could be deleted, which saves nearly 4TB of data storage per day, almost halves the number of compute servers needed, and saves millions of dollars a year.

  • On demand. Old image variants could be deleted because they could be recreated on the fly rather than precomputed. Performing services on demand increases flexibility, it lets you be lot smarter about how tasks are performed, and gives a central point of control.

  • Progressive JPEG is a real winner as a standard image format. It has great frontend and backend support and performs very well on slower networks.

Lots of good things happened on Twitter’s journey to a media rich future, let’s learn how they did it...

The Old Way - Twitter in 2012
Categories: High Scalability

Hadoop and Salesforce Integration: the Ultimate Successful Database Merger

High Scalability - Mon, 2016-04-18 15:56

How we can transfer salesforce data to hadoop? It is big challenge to everyday users. What are different features of data transfer tools.

Categories: High Scalability

Stuff The Internet Says On Scalability For April 15th, 2016

High Scalability - Fri, 2016-04-15 15:56

Hey, it's HighScalability time:


What happens when Beyoncé meets eCommerce? Ring the alarm.

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • $14 billion: one day of purchases on Alibaba; 47 megawatts: Microsoft's new data center space for its MegaCloud; 50%: do not speak English on Facebook; 70-80%: of all Intel servers shipped will be deployed in large scale datacenters by 2025; 1024 TB: of storage for 3D imagery currently in Google Earth; $7: WeChat average revenue per user; 1 trillion: new trees; 

  • Quotable Quotes:
    • @PicardTips: Picard management tip: Know your audience. Display strength to Klingons, logic to Vulcans, and opportunity to Ferengi.
    • Mark Burgess: Microservices cannot be a panacea. What we see clearly from cities is that they can be semantically valuable, but they can be economically expensive, scaling with superlinear cost. 
    • ethanpil: I'm crying. Remember when messaging was built on open platforms and standards like XMPP and IRC? The golden year(s?) when Google Talk worked with AIM and anyone could choose whatever client they preferred?
    • @acmurthy: @raghurwi from @Microsoft talking about scaling Hadoop YARN to 100K+ clusters. Yes, 100,000 
    • @ryanbigg: Took a Rails view rendering time from ~300ms to 50ms. Rewrote it in Elixir: it’s now 6-7ms. #MyElixirStatus
    • Dmitriy Samovskiy: In the past, our [Operations] primary purpose in life was to build and babysit production. Today operations teams focus on scale.
    • @Agandrau: Sir Tim Berners-Lee thinks that if we can predict what the internet will look like in 20 years, than we are not creative enough. #www2016
    • @EconBizFin: Apple and Tesla are today’s most talked-about companies, and the most vertically integrated
    • Kevin Fishner: Nomad was able to schedule one million containers across 5,000 hosts in Google Cloud in under five minutes.
    • David Rosenthal: The Web we have is a huge success disaster. Whatever replaces it will be at least as big a success disaster. Lets not have the causes of the disaster be things we knew about all along.
    • Kurt Marko: The days of homogeneous server farms with racks and racks of largely identical systems are over.
    • Jonathan Eisen: This is humbling, we know virtually nothing right now about the biology of most of the tree of life.
    • @adrianco: Google has a global network IP model (more convenient), AWS regional (more resilient). Choices...
    • @jason_kint: Stupid scary stats in this. Ad tech responsible for 70% of server calls and 50% of your mobile data plan.
    • apy: I found myself agreeing with many of Pike’s statements but then not understanding how he wound up at Go. 
    • @TomBirtwhistle: The head of Apple Music claims YouTube accounts for 40% of music consumption yet only 4% of online industry revenue 
    • @0x604: Immutable Laws of Software: Anyone slower than you is over-engineering, anyone faster is introducing technical debt
    • surrealvortex: I'm currently using flame graphs at work. If your application hasn't been profiled recently, you'll usually get lots of improvement for very little effort. Some 15 minutes of work improved CPU usage of my team's biggest fleet by ~40%. Considering we scaled up to 1500 c3.4xlarge hosts at peak in NA alone on that fleet, those 15 minutes kinda made my month :)
    • @cleverdevil: Did you know that Virtual Machines spin up in the opposite direction in the southern hemisphere? Little known fact.
    • ksec: Yes, and I think Intel is not certain to win, just much more likely. The Power9 is here is targeting 2H 2017 release. Which is actually up against Intel Skylake/Kabylake Xeon Purley Platform in similar timeframe.
    • @jon_moore: Platforms make promises; constraints are the contracts that allow platforms to do their jobs. #oreillysacon
    • @CBILIEN: Scaling data platforms:compute and storage have to be scaled independently #HS16Dublin

  • A morning reverie. Chopped for programmers. Call it Crashed. You have three rounds with four competitors. Each round is an hour. The competitors must create a certain kind of program, say a game, or a productivity tool, anything really, using a basket of three selected technologies, say Google Cloud, wit.ai, and Twilio. Plus the programmer can choose to use any other technologies from the pantry that is the Internet. The program can take any form the programmer chooses. It could be a web app, iOS or Android app, an Alexa skill, a Slack bot, anything, it's up to the creativity of the programmer. The program is judged by an esteemed panel based on creativity, quality, and how well the basket technologies are highlighted. When a programmer loses a round they have been Crashed. The winner becomes the Crashed Champion. Sound fun?

  • Jeff Dean when talking about deep learning at Google makes it clear a big part of their secret sauce is being able to train neural nets at scale using their bespoke distributed infrastructure. Now Google has released Tensor Flow with distributed computing support. It's not clear if this is the same infrastructure Google uses internally, but it seems to work: using the distributed trainer, we trained the Inception network to 78% accuracy in less than 65 hours using 100 GPUs. Also, the tensorflow playground is a cool way to visualize what's going on inside.

  • Christopher Meiklejohn with an interesting history of the Remote Procedure Call. It started way back in 1974: RFC 674, “Procedure Call Protocol Documents, Version 2”. RFC 674 attempts to define a general way to share resources across all 70 nodes of the Internet

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: High Scalability
Syndicate content