on security research

I’ve been pondering URL rewriting for the past couple of days – trying to come up with some way a client of a web site can first: determine if URL rewriting is occurring on a given web server, and second: in cases where it is used, determine what the rewrite rules are.
As I have been thinking about this, it occurred to me that, despite the proliferation of security research whitepapers and blog posts, there is a scarcity of ‘this is the process I went through to do this research’ information out there.

There are mountains of articles and documents, with dizzying arrays of statistics and metrics (often intermingled with a fair amount of marketing fluff), and yet most of the whitepapers, and certainly the various conference presentations, simply don’t talk about the process – preferring instead to present the end results.
As security professionals, we gather together at a multitude of conferences where we do a wonderful job displaying all of this shiny data and showing off new marvelous tricks to each other with varying degrees of self-indulgence. Yet most of how we came to have such cool stuff is left out of the picture entirely.

I understand why that is, of course. Simply put, the process is boring! It’s full of failure, and repeatedly throwing things at a wall and observing what happens. Nobody wants to sit in a small room with a couple hundred hackers listening to someone drone on for an hour about how “this didn’t work…and neither did this”, I get that. Added to that is the fact that, in some cases, the research is being done for a corporate (or government) entity. In such a situation, the process may be withheld not from a lack of desire to share on the researcher’s part, but because they are not permitted to do so by the organization for which the work was done.

Despite these reasons, in my opinion it is a disservice to ourselves, to the profession, and to others whom may be interested in performing their own research, when we all we do is deliver an end product in glossy PDF or a shiny PowerPoint presentation. That is simply not research, it’s promotion. Research, in an academic sense, implies documenting the entire process: both success and failure. This is not what I find when I look at the typical infosec industry output.

Accordingly, I’ve decided that I will share how I go about this particular project, and not just release some PDF or tool as a result of it. I’ll post my process here, any notes and thoughts, as well as any code I come up with. (Well, links to code anyway. I’ll probably keep the code itself in github).

One of the reasons I’m doing this is that I expect to fail. =)

As I’ve considered how one can detect URL rewriting, and as I’ve started investigating the details of how it works, my initial thought is that detecting it simply won’t be possible.

If that’s correct, I think it’s important that I present what I tried, along with the fact that ultimately it didn’t work. That’s vital information, in that it prevents someone else from wasting cycles repeating a process that’s already been done.

As well, understanding why something failed may lead to discovering a way to succeed.

OK… this rant being done now, my next post will start the process of documenting my research into detecting URL rewriting.