joestump.net

I start all of my talks at conferences on architecture and scaling with describing the distinct differences between scaling and performance. I define scaling in one word: specialization. I, somewhat jokingly, retort to the question of what performance is with “Who cares?”.

The reason for not caring (much) about actual performance is that whether or not you use single quotes, double quotes, objects, functions, Python, Ruby, PHP, foreach, etc. has nothing to do with whether or not your application and site will scale.

Scaling is entirely about IO. Fundamentally, it’s about whether your data is being stored in a manner that makes retrieving it at the rate of today’s high traffic websites possible. In other words, Ruby isn’t the reason you can’t store 250,000,000 records in MySQL and do range scans. It’s because MySQL (and most RDBMS’s) suck.

I’ve been playing with Django a lot lately for a side project. Despite being a pragmatic coder to a fault, I’ve decided to truly learn another language and I felt Django would ease the shock a bit. So far I love it. Django’s patterns make a lot of sense and I’m loving the goodies a true OO language like Python gives me. Since starting down this path I’ve been getting two questions over and over:

1. Does this mean Digg is going to be using Django?
2. Why didn’t you choose Ruby on Rails?

Digg will not be using Django or any other framework anytime soon. We deal with traffic that most Django developers will probably never see. Our stack receives billions of requests a month. That kind of traffic, as I stated earlier, requires specialization. Django is the exact opposite of this. It’s a generic web framework made to answer the majority of web programmer’s basic needs. The majority of web programmers don’t deal with the problems we deal with. I’m sure if you ripped out a lot of what makes Django so great (e.g. the models, admin, etc.) Django would be fine (e.g. If we used it only for mapping requests to views and templates), but then it wouldn’t really be Django would it?

I chose Python and Django over Ruby and Ruby on Rails for a number of reasons. First and foremost is that we use Python here at Digg. Learning Python will only enhance my ability to perform my duties at Digg. I, personally, dislike the Perlisms in Ruby. Additionally, Ruby seems to skew towards implicitness in the language, while Python skews toward explicitness. I like explicitness.

To sum things up, I have nothing technically against Ruby as a language, I love frameworks for regular development work and Perl’s syntax kills small children.

Choose what you love and be happy coders.

I’m working on the next generation data access layer for Digg right now, which is basically a REST layer built on top of a partitioned and multihomed database setup. The general idea is that we’ll send GET, POST, PUT and DELETE requests to URI’s on our services layer to access and manipulate data. PHP makes accessing GET and POST easy via $_GET and $_POST. DELETE isn’t an issue since what we’re deleting is just the entity defined by the URI (e.g. Sending DELETE to /2.0/User/1234.xml will delete User 1234).

After a few days work I can create, fetch and delete entities from this setup. Today I started working on implementing the PUT method. I always knew PHP wasn’t exactly top notch when it came to PUT support, but I had no idea how annoying it would be to find a simple solution for parsing PUT information. After some digging around this is what I’ve figured out.

$put = array();
parse_str(file_get_contents('php://input'), $put);

That should parse everything into a native PHP array, including arguments like foo[bar]=1&foo[baz]=2. If anyone knows of a more native way of doing this please let me know.

It’s 6PM on a Thursday night and I’m about ready to head over to the Open Web Awards presented by Mashables.com to celebrate Digg’s wins with a few of my fellow Diggers. The only downer is that we don’t have more Diggers to share the fruits of our labor with. It reminded me that I should probably tell all 10 people who read this site that we’re looking for talented people to work with us in our San Francisco office (in Potrero Hill). Below is a little insight into what you’d be doing if you worked at Digg.

• Program for the 36^th largest site on the intertubes according to Compete.com. Digg.com does 20,000,000+ unique visits a month. That’s a lot of zeros!
• We use the LAMP stack (Debian GNU/Linux, Apache, MySQL and PHP with some Python thrown around from time to time) and expect you to be proficient with that.
• Learn from some of the brightest minds in the PHP, design and operations communities.
• Play with Memcache, Gearman, Mogile, PEAR, etc. in a high volume environment.
• Create and contribute to open source projects while you work on high traffic and scalability problems at Digg.
• 20 days of PTO a year and access to all sorts of great benefits (medical, dental, vision, etc.).

I love working at Digg. It’s fun, fast pace and I work with some of the brightest minds in the industry. If you have any questions or interest please email jobs@digg.com.

Thanks to Southwest delaying my 1.5 hour flight to Portland by 2 hours I’ve finally gotten time to package up the latest release of Framework. I jumped ahead to 0.4.0 from the 0.2.x series due to the fact that 0.4.0 isn’t really 100% backwards compatible and because I felt it was a major enough release to skip ahead a number. Lots of bug fixes, features, etc.

The biggest enhancements are that I switched to a pure exception model for error handling. I even went so far to switch the default PEAR_Error handler to throw Framework_Exception’s. Also, as asked by a ton of people I’ve created the ability to change your DB abstraction layers or not use one at all.

Like many other interesting projects the Digg iPhone project started with a conversation over a few beers and a challenge: code it in 48 hours and Kevin would give me a free iPhone. Being the unabashed Apple fanboy I am and, also, being a self-respecting coder I set out to create Digg for the iPhone.  After spending a short time white boarding the application, Daniel mocked up the design and I set off to code it.

Technically speaking, it’s no revolutionary application and I didn’t spend the entire 48 hours working on the application. I did, of course, find time to go see Transformers (awesome) over the weekend. The JavaScript was borrowed from Joe Hewitt and adapted a bit using jQuery. The application iteself is based on our API using the Services_Digg PEAR package I maintain. I’ve been talking with the jQuery team about some limitation in the animations and plan on packaging up a more robust iPhone JavaScript library based on jQuery sometime in the near future (hopefully).

Yesterday Kevin announced the iPhone application and Daniel Burka, our designer at Digg, has covered the details about designing for the iPhone. So far the response to the application has been positive. It was fun and I’m glad everyone is enjoying it. I know I did while riding the bus to work today.

And to answer everyone’s questions. I got an 8GB version last night, the keyboard is interesting, it’s breathtakingly gorgeous and I’ll write YAiPR (Yet Another iPhone Review) soon.

UPDATE: As you might have noticed, we’re having some technical issues rolling out the new comments. Please bare with us as we work out the kinks.

UPDATE: We reworked a few things and the comments are now live again.

Today Digg launched it’s redesign of the comment system, which was programmed by yours truly. Daniel has written up a detailed overview of the design decisions so if you’re interested in the design aspects you’ll definitely want to check that out.

There were a few fairly complex technical changes to the comment systems, which I’ll outline and then go into a little detail about.

• We’ve been talking about moving towards a services oriented architecture on and off since I started in February. Steve had coded the API and the decision was made that the new comments system would use Digg’s public API.
• Comment threads would be loaded dynamically using AJAX and JSON.
• All commenting, editing, etc. would happen via AJAX.

The API ended up making the PHP code behind the scenes relatively painless. Each page loads using two small calls; one to figure out how many comments are on a story and another to fetch the first 50 or so comments (we use a fuzzy limit so it’ll just load all of them if there’s, say, 55 comments on a story). Both our local proxy and the comments code use the Services_Digg package we released via PEAR when the API launched. In fact, the entire permalink page is built using the API now, which is pretty neat.

By far the most complex portion of the comments system was how dynamic it was going to be. Threads would be zipping in and out, we’d be creating 90% of the HTML dynamically in the DOM from JSON, posting and editing over AJAX, etc. It was during design that Micah and I also plotted to remove script.aculo.us and replace it with the smaller jQuery library. The entire comment system is, in fact, a series of jQuery plugins.

Probably the coolest, technically speaking, portion of the new comments is the manner in which most of the page is created. No longer do we create static HTML in PHP and send you a huge HTML page. Instead we give you the basics and, via AJAX/JSON, we make requests to the API and dynamically create the DOM using the FlyDOM jQuery plugin. The FlyDOM JSON templates are a stroke of genius if you’re looking at loading JSON dynamically into the DOM. The advantage of this is that initial page loads are much snappier and you can load the threads you wish to read on demand.

I really picked up the whole dynamically created DOM ball and ran with if. If you notice, on the initial page load there aren’t any forms anywhere in the DOM. Those, also, are created dynamically on request. An interesting side effect to this is that there’s about 4x as much JavaScript code on the new comments than there is PHP.

The major technical and design changes of comments should lead to faster load times, less bandwidth being eaten up and, hopefully, a better user experience. I hope you enjoy them and, as always, welcome comments and input.

As Kevin would say, “Digg on!”

As some of you know I’ve been working on a little side project and I’d like to officially announce that correlate.us is open for business. Whatever that means.

I’ve been kicking this idea around for quite some time. The basic idea is that you authenticate your various Web 2.0 accounts and then correlate.us goes out and aggregates that data, groups it by tags and makes it generally more browseable for everyone involved.

The best part about it is that you can add friends. Once you’ve added a few friends you can view your friends’ activity online in an aggregated view as well. Don’t forget to add correlate.us to your facebook profile.

A recent project I’ve been working on required 4:3 thumbnails of all images no matter the original image’s orientation. This required me to do a little math and cropping before actually making the thumbnail. I use the package Image_Transform to figure out which orientation the image is and then crop it appropriately.


$o = Image_Transform::factory('IM');
if (PEAR::isError($o)) {
    return $o;
}

$result = $o->load($ds1);
if (PEAR::isError($result)) {
    return $result;
}

$w = $o->getImageWidth();
$h = $o->getImageHeight();

if ($h > $w) {
    $newWidth = $w;
    $newHeight = ($newWidth * .75);
    $newY = ($h * .15);
    $newX = 0;
} else {
    $newWidth = ($w / 2);
    $newHeight = ($newWidth * .75);
    $newY = ($h / 2) - ($newHeight / 2);
    $newX = ($x / 2) + ($newWidth / 2);
}

$o->crop($newWidth, $newHeight, $newX, $newY);
$o->save($ds2);

The first check checks to see if the image is in landscape or portrait (in portrait the height will be greater than the width). With portrait’s I use the width, multiple that by 0.75 to get my 4:3 ratio and finish by going 15% down from the top of the portrait (assuming that you’re focusing your portrait in the upper portion of the image). With landscape images I simply go outwards from the center of the image (again, assuming you’re focusing your landscape in the center of the image).

I’ve been programming PHP now for, oh, almost a decade and I’ve been a contributor to PEAR for about four years now. It took me this long to finally get to the point that I think writing unit tests, in some form, is a good idea. I jumped on the OOP bandwagon about 6 years ago and drank the MVC koolaid about 4 years ago. As of a few weeks ago, I’ve jumped on the phpt bandwagon. I’m not going to cover the basics, because they’re well covered in a few different places around the web, but I am going to cover some of the nuances I’ve learned in the last few weeks.First off, you can run phpt files using the PEAR command line utility. I’ve been creating a tests directory and putting all of my phpt files in there. I normally name them 000-class-method.phpt and then increment the sequence number at the front for each test by 5 so I can do a decent job of predicting how tests run. Once you have a few tests you can run your tests in two different manners from the command line:

1. pear run-tests
2. pear run-tests 000-class-method.phpt

When you run your tests you’ll get results of which tests passed, which failed, if any of them were skipped and the reason why they were skipped, and files for debugging. If 000-class-method.phpt fails, there are a few files that will be of interest to you:

• 000-class-method.php is the actual PHP file ran, which is found in the --FILE-- portion of your phpt file.
• 000-class-method.outis what was actually output by the PHP file ran from the --FILE-- portion of your phpt file.
• 000-class-method.exp is what the test expected the output of 000-class-method.php to be, which is found in the --EXPECT-- portion of your phpt file.

You pretty much have everything you need to debug why a test failed. I normally diff the exp and out files and then debug the test (or my code, which is usually the case) using the php file.

Another great thing about phpt files is that you can use a cgi PHP binary to debug actual GET requests (phpt supports spoofing $_GET, $_POST and $_COOKIE using the cgi binary). This lets you create tests like the following:

--TEST--
Test a GET request
--GET--
foo=bar&baz=foo
--FILE--
<?php
echo $_GET['bar'] . "\n";
echo $_GET['baz'] . "\n";
?>
--EXPECT--
bar
foo


This test should pass without incident. The only catch is that you need to specify your cgi binary path using the --cgi argument when you run the tests (e.g. pear run-tests --cgi=/path/to/php-cgi).

Overall, I’m pretty happy with my newfound testing experiences. My only complaint is that phpt files don’t support spoofing of $_SESSION natively. A small complaint to be sure.

So if you’re interested in seeing what I’ve been doing for Digg lately, you can now go to the site, log in and send invites to your friends. You’ve actually always been able to do this, but now you’re able to keep track of how many friends you’ve invited, those friend invites will be announced to the world on the homepage and you’ll automagically be added to the invitee’s friends list.

Once you’ve invited a friend, the friend needs to register and digg at least three stories before they count as a referral. At that point a little note is posted to the frontpage announcing your invite and your friend referral count in your user profile goes up. I, of course, invite all of you to join.

I’ve also been working on other magical doings, but I can’t talk about those quite yet. I hope to talk more about that work after it goes live as well.