==Phrack Inc.==

              Volume 0x0b, Issue 0x39, Phile #0x0a of 0x12

|=-------------=[ Against the System: Rise of the Robots ]=--------------=|
|=-----------------------------------------------------------------------=|
|=-=[ (C)Copyright 2001 by Michal Zalewski <[email protected]> ]=-=|


-- [1] Introduction -------------------------------------------------------

 "[...] big difference between the web and traditional well controlled
  collections is that there is virtually no control over what people can
  put on the web. Couple this flexibility to publish anything with the
  enormous influence of search engines to route traffic and companies
  which deliberately manipulating search engines for profit become a
  serious problem."

                     -- Sergey Brin, Lawrence Page (see references, [A])

 Consider a remote exploit that is able to compromise a remote system
 without sending any attack code to his victim. Consider an exploit
 which simply creates local file to compromise thousands of computers,
 and which does not involve any local resources in the attack. Welcome to
 the world of zero-effort exploit techniques. Welcome to the world of
 automation, welcome to the world of anonymous, dramatically difficult
 to stop attacks resulting from increasing Internet complexity.

 Zero-effort exploits create their 'wishlist', and leave it somewhere
 in cyberspace - can be even its home host, in the place where others
 can find it. Others - Internet workers (see references, [D]) - hundreds
 of never sleeping, endlessly browsing information crawlers, intelligent
 agents, search engines... They come to pick this information, and -
 unknowingly - to attack victims. You can stop one of them, but can't
 stop them all. You can find out what their orders are, but you can't
 guess what these orders will be tomorrow, hidden somewhere in the abyss
 of not yet explored cyberspace.

 Your private army, close at hand, picking orders you left for them
 on their way. You exploit them without having to compromise them. They
 do what they are designed for, and they do their best to accomplish it.
 Welcome to the new reality, where our A.I. machines can rise against us.

 Consider a worm. Consider a worm which does nothing. It is carried and
 injected by others - but not by infecting them. This worm creates a
 wishlist - wishlist of, for example, 10,000 random addresses. And waits.
 Intelligent agents pick this list, with their united forces they try to
 attack all of them. Imagine they are not lucky, with 0.1% success ratio.
 Ten new hosts infected. On every of them, the worm does extactly the
 same - and agents come back, to infect 100 hosts. The story goes - or
 crawls, if you prefer.

 Agents work virtually invisibly, people get used to their presence
 everywhere. And crawlers just slowly go ahead, in never-ending loop.
 They work systematically, they do not choke with excessive data - they
 crawl, there's no "boom" effect. Week after week after week, they try
 new hosts, carefully, not overloading network uplinks, not generating
 suspected traffic, recurrent exploration never ends. Can you notice
 they carry a worm? Possibly...

-- [2] An example ---------------------------------------------------------

 When this idea came to my mind, I tried to use the simpliest test, just
 to see if I am right. I targeted, if that's the right word, general-purpose
 web indexing crawlers. I created very short HTML document and put it
 somewhere. And waited few weeks. And then they come. Altavista, Lycos
 and dozens of others. They found new links and picked them
 enthusiastically, then disappeared for days.

 bigip1-snat.sv.av.com:
   GET /indexme.html HTTP/1.0

 sjc-fe5-1.sjc.lycos.com:
   GET /indexme.html HTTP/1.0

 [...]

 They came back later, to see what I gave them to parse.

   http://somehost/cgi-bin/script.pl?p1=../../../../attack
   http://somehost/cgi-bin/script.pl?p1=;attack
   http://somehost/cgi-bin/script.pl?p1=|attack
   http://somehost/cgi-bin/script.pl?p1=`attack`
   http://somehost/cgi-bin/script.pl?p1=$(attack)
   http://somehost:54321/attack?`id`
   http://somehost/AAAAAAAAAAAAAAAAAAAAA...


 Our bots followed them exploiting hypotetical vulnerabilities,
 compromising remote servers:

 sjc-fe6-1.sjc.lycos.com:
   GET /cgi-bin/script.pl?p1=;attack HTTP/1.0

 212.135.14.10:
   GET /cgi-bin/script.pl?p1=$(attack) HTTP/1.0

 bigip1-snat.sv.av.com:
   GET /cgi-bin/script.pl?p1=../../../../attack HTTP/1.0

 [...]

 (BigIP is one of famous "I observe you" load balancers from F5Labs)
 Bots happily connected to non-http ports I prepared for them:

 GET /attack?`id` HTTP/1.0
 Host: somehost
 Pragma: no-cache
 Accept: text/*
 User-Agent: Scooter/1.0
 From: [email protected]

 GET /attack?`id` HTTP/1.0
 User-agent: Lycos_Spider_(T-Rex)
 From: [email protected]
 Accept: */*
 Connection: close
 Host: somehost:54321

 GET /attack?`id` HTTP/1.0
 Host: somehost:54321
 From: [email protected]
 Accept: */*
 User-Agent: FAST-WebCrawler/2.2.6 ([email protected]; [...])
 Connection: close

 [...]

 But not only publicly available crawlbot engines can be targeted.
 Crawlbots from alexa.com, ecn.purdue.edu, visual.com, poly.edu,
 inria.fr, powerinter.net, xyleme.com, and even more unidentified
 crawl engines found this page and enjoyed it. Some robots didn't
 pick all URLs. For example, some crawlers do not index scripts
 at all, others won't use non-standard ports. But majority of
 the most powerful bots will do - and even if not, trivial filtering
 is not the answer. Many IIS vulnerabilities and so on can be triggered
 without invoking any scripts.

 What if this server list was randomly generated, 10,000 IPs or 10,000
 .com domains? What is script.pl is replaced with invocations of
 three, four, five or ten most popular IIS vulnerabilities or
 buggy Unix scripts? What if one out of 2,000 is actually exploited?

 What if somehost:54321 points to vulnerable service which can
 be exploited with partially user-dependent contents of HTTP
 requests (I consider majority of fool-proof services that do not
 drop connections after first invalid command vulnerable)? What if...

 There is an army of robots, different species, different functions,
 different levels of intelligence. And these robots will do whatever
 you tell them to do. It is scary.

-- [3] Social considerations ----------------------------------------------

 Who is guilty if webcrawler compromises your system? The most obvious
 answer is: the author of original webpage crawler visited. But webpage
 authors are hard to trace, and web crawler indexing cycle takes
 weeks. It is hard to determine when specific page was put on the net
 - they can be delivered in so many ways, processed by other robots
 earlier; there is no tracking mechanism we can find in SMTP protocol and
 many others. Moreover, many crawlers don't remember where they "learned"
 new  URLs. Additional problems are caused by indexing flags, like "noindex"
 without "nofollow" option. In many cases, author's identity and attack
 origin wouldn't be determined, while compromises would take place.

 And, finally, what if having particular link followed by bots wasn't
 what the author meant? Consider "educational" papers, etc - bots won't
 read the disclaimer and big fat warning "DO NOT TRY THESE LINKS"...

 By analogy to other cases, e.g. Napster forced to filter their contents
 (or shutdown their services) because of copyrighted information exchanged
 by their users, causing losses, it is reasonable to expect that
 intelligent bot developers would be forced to implement specific filters,
 or to pay enormous compensations to victims suffering because of bot
 abuse.

 On the other hand, it seems almost impossible to successfully filter
 contents to elliminate malicious code, if you consider the number and
 wide variety of known vulnerabilities. Not to mention targeted attacks
 (see references, [B], for more information on proprietary solutions and
 their insecuritities). So the problem persists. Additional issue is that
 not all crawler bots are under U.S. jurisdiction, which makes whole
 problem more complicated (in many countries, U.S. approach is found at
 least controversial).

-- [4] Defense ------------------------------------------------------------

 As discussed above, webcrawler itself has very limited defense and
 avoidance possibilities, due to wide variety of web-based
 vulnerabilities. One of more reasonable defense ideas is to use
 secure and up-to-date software, but - obviously - this concept is
 extremely unpopular for some reasons - www.google.com, with
 unique documents filter enabled, returns 62,100 matches for "cgi
 vulnerability" query (see also references, [D]).

 Another line of defense from bots is using /robots.txt standard
 robot exclusion mechanism (see references, [C], for specifications).
 The price you have to pay is partial or complete exclusion of your
 site from search engines, which, in most cases, is undesired. Also,
 some robots are broken, and do not respect /robots.txt when following
 a direct link to new website.

-- [5] References ---------------------------------------------------------

 [A] "The Anatomy of a Large-Scale Hypertextual Web Search Engine"
     Googlebot concept, Sergey Brin, Lawrence Page, Stanford University
     URL: http://www7.scu.edu.au/programme/fullpapers/1921/com1921.htm

 [B] Proprietary web solutions security, Michal Zalewski
     URL: http://lcamtuf.coredump.cx/milpap.txt

 [C] "A Standard for Robot Exclusion", Martijn Koster
     URL: http://info.webcrawler.com/mak/projects/robots/norobots.html

 [D] "The Web Robots Database"
     URL: http://www.robotstxt.org/wc/active.html
     URL: http://www.robotstxt.org/wc/active/html/type.html

 [E] "Web Security FAQ", Lincoln D. Stein
     URL: http://www.w3.org/Security/Faq/www-security-faq.html

|=[ EOF ]=---------------------------------------------------------------=|

You are viewing proxied material from gopher.black. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.