|
a jaundiced eye: dark fiber
for tuesday, september 16, 1997. The Sporting News
It's time, once again, for a bit of sport. I like to call this particular sport "Teach the Bastards a Lesson."
This is a sport with a fabled history among the rank-and-file of Geek Nation. Remember, for example, the story about the BBS operator who set
his modem to auto-dial Jerry Falwell's fund-rasing 800 line, filling the line with 30 seconds of silence each time, at an estimated cost of several
thousands of dollars to the hapless, humorless Chistian soldier?
We're not talking about peaceful petitioning here, either. We're not talking about the more legitimized forms of computer resistance that have
become so popular of late. Filling the mailboxes of Senators Hatch, Helms and Thurmond with NC-17 rated e-mails in protest of the CDA? Reasonably
effective, sure, and lots of fun besides, but still, such actions are well within the bounds of traditional political dissent. Let's face it, folks
-- that's the cyber-equivalent of sitting in a circle in the lobby of the Dean's office and singing "Kumbaya." No, what we're talking about here is
unabashed bit-wrasslin' terrorism, straight from the pages of 2600 and alt.hackers.malicious: denial-of-service attacks, wicked spoofing, spam and anti-spam bombs,
digital maledicta of all kinds. Because that's the real sport, you see: pulling a really dirty trick on somebody who desperately deserves it.
Of course, "Teach the Bastards a Lesson" is ultimately a spectator sport for the vast majority of us, for a number of reasons. First, only a select
few have the technical wherewithal to play this game effectively; and of those, only a small percentage actually have the stomach for it -- it does
entail bending the law in the vast majority of cases, after all. Besides, in many instances, it's less than completely satisfying anyway, because the
victims often aren't bright enough to understand the nature of the attack.
Wouldn't it be great if we could all play "Teach the Bastards a Lesson," though? Wouldn't it be great if it were completely legal? And wouldn't it
be great if the bastards were a really, really big bunch of bastards? And wouldn't it be sweetest of all if the bastards brought it entirely upon
themselves?
Maybe by now you've figured out my motives. Yes, Virginia, there is a really big bastard. His name, and this should come as no surprise to those
of you who've been paying attention for the past fifteen years, is Bill Gates.
Let's shift gears, shall we?
By now, some of you may have had the opportunity to take IE4 for a spin. Most of
you probably haven't. If you're anything like me, you're too busy building the web to participate in the Endless Beta Hype Machine. Of
course, I have my tried-and-true methods for keeping up with developments in the web world, and by now I've gleaned a lot about IE4 from the trade
rags I keep in my bathroom. Much of the data out there is fairly useless; it's hard to get a feel for a product until you've really played with it
for a while, and most technical reviewers are too busy trying to sound knowledgeable to make the really salient points that need to be made.
Having said that, I would like to offer up my personal and heartfelt thanks to Lincoln Stein.
Anyone familiar with his work knows that he is a veritable Web Knight, and in his article in the October 1997 edition of
Web Techniques, Sir Lincoln points out a gaping hole in the breastplate of the Dragon from Redmond. And all I can say is, boys, git yer lances: we're a-going jousting!
It's an article about IE4 and Robots. You should go find it and read it. Today. Now. This minute. I mean it! Go! Now!
Okay, maybe not right now; read on. And since I don't want to lose you completely in the meantime, here's the Reader's Digest Condensed Version of
Sir Lincoln's tract, suitably paraphrased to satisfy Jaundiced Eye's strict adherence to all laws of copyright.
IE4 has a "subscribe" feature. If you like a page on a site, you can "subscribe" to it, meaning that you can tell IE4 to fetch it as often as
you'd like. This "subscribe" feature also allows the retrieval of multiple pages, and allows you to set the page depth as well. This, of course,
means that the IE4 "subscribe" feature is, in point of fact, a robot. Unfortunately, Microsoft disagrees with this assessment, and for that
reason, Microsoft has chosen not to bind this "subscribe" feature to the robot rules that have helped to make the web a more civilized place. And
since the IE4 "subscribe" requests are indistinguishable from a regular IE4 user's requests, your site -- you know, the one that has an Oracle back-end
two levels deep that will crank out a million records if tackled by an unruly robot? -- has no way of telling this rude-boy to piss off. It's
worse than that, even, but I'll leave the details to Sir Lincoln.
So, to recap, briefly: IE4 = idiot site-killer robot.
Microsoft's reaction to this? Sir Lincoln quotes Lewis Geer of the IE4 Development team: "robot rules were meant for servers, not human clients."
I see. Well, thanks for clearing that up.
This comment is disheartening not only because of its obvious ridiculousness, but also because it comes from someone who should clearly
know better. It is the job of the development manager to know a product inside-and-out. A product is not simply the sum of its features; a product
is also the environment in which it operates. This is especially true of an application that functions in a client-server model, and let's face it,
we're not dealing with your office's NT domain here. If a site is even moderately successful in driving traffic, it's not unreasonable to expect a
thousand visitors in a day. Spread out over the course of an entire day, that's a simple load to bear. But IE4's defaults will lead hundreds of
thousands of users to subscribe to sites at exactly the same time -- 12 midnight. Think about it: can your site handle five hundred users at the
same exact instant, all pulling at a depth of five pages?
The nastiest irony of all here is that we, as webmasters, might be penalized for our successes. If we provide compelling content, and if
we turn it over regularly enough to encourage users to visit often, then we make our sites prime candidates for subscription. This might
be a great thing. It might be a nightmare. We won't know until our log files fill with 502s and we get buried in e-mails from irritated users,
wondering if we've packed up our servers in moth balls.
It's clear that Microsoft hasn't quite thought the whole idea through. Rather than confront the issues head-on, however, the IE4 development team
chooses to ignore the potential for disaster. Geer further implies, with his throwaway comment, that the reasonable and prudent webmasters who take
the precaution of robot-proofing their web sites don't know as much about the internet as his Microsoft engineers do. Which brings us, at last, to
the real issue here: Microsoft's arrogance. They know how the web works, and you don't, so quit whining about a problem that isn't going to
materialize, ok?
Back to the premise: you want to play "Teach the Bastards a Lesson?" Here's your golden opportunity. Microsoft, by releasing a product that
threatens to cripple perfectly well-designed web sites through an irresponsibly ill-conceived "subscribe" feature, makes an ideal target.
Hmmmm...wheels turning...www.microsoft.com...retrieve every hour...99 levels deep...lots and
lots of sluggish Active Server pages...lots and lots of ActiveX controls...saved straight into a trash folder...Rosebud...oh, am I saying
these things out loud? I must have been sleepwalking. Oh dear, how reckless of me.
You know what to do. They're asking for it.
Let's stick it to 'em.
Editor's note: the problem with MSIE4 ingoring robots.txt seems to have been opened as a bug by Microsoft in MSIE build 507, and closed in build 718. The latest versions of MSIE do not display the behavior described by the author. For the record, it appears as though MSIE PR1 falls between the cracks, being version 543. Interesting questions remain, though, especially regarding the issue of default times for subscription. Even if the authors/maintainers of a site do carefully observe the robots exclusion standard, there still remain issues of whether any given site can withstand the mass subscription of millions to those pages considered "safe" by the authors. So take note, site maintainers, and check your logs periodically to see if your participation in the "web->TV" craze (perhaps unwilling) becomes a detriment to your Web service in general.
© 1997-2001
Steven Champeon. All rights reserved. |