How it works: YouPorn

YouPorn is one of the most visited porn site on the web. I applied for a job as developer in 2009 because of a post who talk about its technological stack.

I heard again about its infrastructure because, about an year ago, @ErikPickupYP spoke about the great switchover at CooFoo and @antirez tweet some details regarding datastore. The development team rewrote the entire site using Redis as primary database.

Original stack was based on Perl and Catalyst and powered the site from 2006 to 2011. After acquisition they rewrote the site using a well designed LAMP stack.

The chosen framework is Symfony2 (which uses Doctrine as ORM) running over nginx with PHP-FPM helped by Varnish (speed up requests, manage cache and check servers status) and HAProxy (load balance and health check of servers). Syslog-ng handle logs. They maintain two pools of servers: a write pool with a fail-over to backup-Master and a read pool will servers except the master.

Datastore is the most interesting part. Initially they used MySQL but more than 200 million of pageviews and 300K query per second are too much to be handled using only MySQL. First try was to add ActiveMQ to enqueue writes but a separate Java infrastructure is too expensive to be maintained. Finally they add Redis in front of MySQL and use it as main datastore.

Now all reads come from Redis. MySQL is used to allow the building new sorted sets as requirements change and it’s highly normalized because it’s not used directly for the site. After the switchover additional Redis nodes were added, not because Redis was overworked, but because the network cards couldn’t keep up with Redis :D

Lists are stored in a sorted set and MySQL is used as source to rebuild them when needed. Pipelining allows Redis to be faster and Append-only-file (AOF) is an efficient strategy to easily backup data.

In the end YouPorn uses a LAMP stack “on-steroids” which smartly uses Redis and other modern middlewares.

Sources:

Tweet about this on Twitter40Share on LinkedIn0Share on Facebook54Share on Google+4Email this to someone
  • http://blog.abahgat.com/ Alessandro Bahgat

    Nice article, I’d be curious to know how they interview candidates :)

    • http://www.facebook.com/people/Nick-Selby/505238768 Nick Selby

      “You want this job?” “Yes, I’d really like this job.” “Well, how MUCH do you want this job?” “I’ll do ANYTHING for this job…” “Anything???””

    • VladasDirzys

      Not sure how, but I think I know where…
      http://www.lbpost.com/images/news/backroom-casting-couch.jpg

  • http://gionn.net/ Gionn

    Hey, the first thing to know about Symfony is how to write its name, or maybe it’s just spell checking ;)

    • http://twitter.com/zenkay Andrea Mostosi

      My fault, just corrected ;D
      Thanks for reporting

  • Pingback: 简记 YouPorn 的技术架构 » 了解并拥有