Commits

Fred T-H committed 00969c3

Adding ibrowse to the source tree

ibrowse will likely be the client used by chut when handling
authentication and permissions (asking for them on other servers).

Might also be useful for a few tests and benchmarks of the mochiweb
server, but we're not there yet.

Comments (0)

Files changed (25)

+%% Chut source
 {'src/*', [debug_info,
            {i, "src"},
            {i, "include"},
            {outdir, "ebin"}]}.
+
+%% ibrowse source
+{'lib/ibrowse/src/*', [debug_info,
+                       {i, "lib/ibrowse/src"},
+                       {outdir, "ebin"}]}. 

lib/ibrowse/.gitignore

+*.beam
+

lib/ibrowse/BSD_LICENSE

+Copyright (c) 2010, Chandrashekhar Mullaparthi
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+    * Neither the name of the T-Mobile nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/ibrowse/LICENSE

+ibrowse - a HTTP client written in erlang
+Copyright (C) 2005-2010 Chandrashekhar Mullaparthi <chandrashekhar dot mullaparthi at gmail dot com>
+
+This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+

lib/ibrowse/Makefile

+include vsn.mk
+
+all:
+	(cd src ; make)
+
+clean:
+	(cd src ; make clean)
+
+install: all
+	mkdir -p $(DESTDIR)/lib/ibrowse-$(IBROWSE_VSN)/
+	cp -r ebin $(DESTDIR)/lib/ibrowse-$(IBROWSE_VSN)/

lib/ibrowse/README

+ibrowse is a HTTP client. The following are a list of features.
+        - RFC2616 compliant (AFAIK) 
+        - supports GET, POST, OPTIONS, HEAD, PUT, DELETE, TRACE, 
+          MKCOL, PROPFIND, PROPPATCH, LOCK, UNLOCK, MOVE and COPY
+        - Understands HTTP/0.9, HTTP/1.0 and HTTP/1.1
+        - Understands chunked encoding
+        - Can generate requests using Chunked Transfer-Encoding
+        - Pools of connections to each webserver
+        - Pipelining support
+        - Download to file
+        - Asynchronous requests. Responses are streamed to a process
+        - Basic authentication
+        - Supports proxy authentication
+        - Can talk to Secure webservers using SSL
+        - any other features in the code not listed here :)
+
+ibrowse is available under two different licenses. LGPL or the BSD license.
+
+Comments to : Chandrashekhar.Mullaparthi@gmail.com
+
+Version : 1.6.2
+
+Latest version : git://github.com/cmullaparthi/ibrowse.git
+
+CONTRIBUTIONS & CHANGE HISTORY
+==============================
+17-07-2010 - * Merged change made by Filipe David Manana to use the base64
+               module for encoding/decoding. 
+
+11-06-2010 - * Removed use of deprecated concat_binary. Patch supplied by
+               Steve Vinoski
+
+10-06-2010 - * Fixed bug in https requests not going via the proxy
+
+12-05-2010 - * Added support for the CONNECT method to tunnel HTTPS through
+               a proxy. When a https URL is requested through a proxy, 
+               ibrowse will automatically use the CONNECT method to first
+               setup a tunnel through the proxy. Once this succeeds, the 
+               actual request is dispatched. Successfully tested with the 
+               new SSL implementation in R13B-03
+             * Added SSL support for direct connections. 
+               See ibrowse:spawn_worker_process/1 and 
+               ibrowse:spawn_link_worker_process/1
+             * Added option to return raw status line and raw unparsed headers
+
+23-04-2010 - * Fixes to URL parsing by Karol Skocik
+
+08-11-2009 - * Added option headers_as_is
+
+04-10-2009 - * Patch from Kostis Sagonas to cleanup some code and suppress
+               dialyzer warnings
+
+24-09-2009 - * When a filename was supplied with the 'save_response_to_file' 
+               option, the option was being ignored. Bug report from 
+               Adam Kocoloski
+
+05-09-2009 - * Introduced option to allow caller to set socket options.
+
+29-07-2009 - * The ETS table created for load balancing of requests was not
+               being deleted which led to the node not being able to create
+               any more ETS tables if queries were made to many number of 
+               webservers. ibrowse now deletes the ETS table it creates once the
+               last connection to a webserver is dropped. 
+               Reported by Seth Falcon.
+             * Spurious data being returned at end of body in certain cases of
+               chunked encoded responses from the server. 
+               Reported by Chris Newcombe.
+
+03-07-2009 - Added option {stream_to, {Pid, once}} which allows the caller
+             to control when it wants to receive more data. If this option
+             is used, the call ibrowse:stream_next(Req_id) should be used
+             to get more data.
+           - Patch submitted by Steve Vinoski to remove compiler warnings
+             about the use of obsolete guards
+
+29-06-2009 - * Fixed following issues reported by Oscar Hellstr�m
+               - Use {active, once} instead of {active, true}
+               - Fix 'dodgy' timeout handling
+               - Use binaries internally instead of lists to reduce memory
+                 consumption on 64 bit platforms. The default response format
+                 is still 'list' to maintain backwards compatibility. Use the
+                 option {response_format, binary} to get responses as binaries.
+             * Fixed chunking bug (reported by Adam Kocoloski)
+             * Added new option {inactivity_timeout, Milliseconds} to timeout
+               requests if no data is received on the link for the specified
+               interval. Useful when responses are large and links are flaky.
+             * Added ibrowse:all_trace_off/0 to turn off all tracing
+             * Change to the way responses to asynchronous requests are
+               returned. The following messages have been removed.
+                * {ibrowse_async_response, Req_id, {chunk_start, Chunk_size}}
+                 * {ibrowse_async_response, Req_id, chunk_end}
+             * Fixed Makefiles as part of Debian packaging 
+               (thanks to Thomas Lindgren)
+             * Moved repository from Sourceforge to Github
+
+11-06-2009 - * Added option to control size of streamed chunks. Also added 
+               option for the client to receive responses in binary format.
+
+21-05-2008 - * Fixed bug in reading some options from the ibrowse.conf file.
+               Reported by Erik Reitsma on the erlyaws mailing list
+             * Fixed bug when cleaning up closing connections
+
+27-03-2008 - * Major rewrite of the load balancing feature. Additional module,
+               ibrowse_lb.erl, introduced to achieve this.
+             * Can now get a handle to a connection process which is not part of
+               the load balancing pool. Useful when an application is making
+               requests to a webserver which are time consuming (such as 
+               uploading a large file). Such requests can be put on a separate
+               connection, and all other smaller/quicker requests can use the
+               load balancing pool. See ibrowse:spawn_worker_process/2 and
+               ibrowse:spawn_link_worker_process/2
+             * Ram Krishnan sent a patch to enable a client to send a lot of
+               data in a request by providing a fun which is invoked by the
+               connection handling process. This fun can fetch the data from 
+               any where. This is useful when trying to upload a large file
+               to a webserver.
+             * Use the TCP_NODELAY option on every socket by default
+             * Rudimentary support for load testing of ibrowse. Undocumented, 
+               but see ibrowse_test:load_test/3. Use the source, Luke!
+             * New function ibrowse:show_dest_status/2 to view state of
+               connections/pipelines to a web server
+
+20-02-2008 - Ram Krishnan sent another patch for another hidden bug in the
+             save_response_to_file feature.
+
+07-02-2008 - Ram Krishnan (kriyative _at_ gmail dot com) sent a simple patch to
+             enable specifying the filename in the save_response_to_file option.
+             When testing the patch, I realised that my original implementation
+             of this feature was quite flaky and a lot of corner cases were 
+             not covered.  Fixed all of them. Thanks Ram!
+
+17-10-2007 - Matthew Reilly (matthew dot reilly _at_ sipphone dot com) 
+             sent a bug report and a fix. If the chunk trailer spans two TCP 
+             packets, then ibrowse fails to recognise that the chunked transfer
+             has ended.
+
+29-08-2007 - Bug report by Peter Kristensen(ptx _at_ daimi dot au dot dk). 
+             ibrowse crashes when the webserver returns just the Status line 
+             and nothing else.
+
+28-06-2007 - Added host_header option to enable connection to secure sites
+             via stunnel
+
+20-04-2007 - Geoff Cant sent a patch to remove URL encoding for digits in
+             ibrowse_lib:url_encode/1.
+             ibrowse had a dependency on the inets application because the 
+             ibrowse_http_client.erl invoked httpd_util:encode_base64/1. This
+             dependency is now removed and the encode_base64/1 has been 
+             implemented in ibrowse_lib.erl
+
+06-03-2007 -  Eric Merritt sent a patch to support WebDAV requests.
+
+12-01-2007 -  Derek Upham sent in a bug fix. The reset_state function was not
+              behaving correctly when the transfer encoding was not chunked.
+
+13-11-2006 -  Youn�s Hafri reported a bug where ibrowse was not returning the 
+                temporary filename when the server was closing the connection 
+                after sending the data (as in HTTP/1.0).
+              Released ibrowse under the BSD license
+
+12-10-2006 -  Chris Newcombe reported bug in dealing with requests where no 
+              body is expected in the response. The first request would succeed
+              and the next request would hang.
+
+24-May-2006 - Sean Hinde reported a bug. Async responses with pipelining was 
+              returning the wrong result.
+
+08-Dec-2005 - Richard Cameron (camster@citeulike.org). Patch to ibrowse to
+              prevent port number being included in the Host header when port
+              80 is intended.
+
+22-Nov-2005 - Added ability to generate requests using the Chunked 
+              Transfer-Encoding.
+
+08-May-2005 - Youn�s Hafri made a CRUX LINUX port of ibrowse.
+              http://yhafri.club.fr/crux/index.html
+
+Here are some usage examples. Enjoy!
+
+5> ibrowse:start().
+{ok,<0.94.0>}
+
+%% A simple GET
+6> ibrowse:send_req("http://intranet/messenger/", [], get).
+{ok,"200",
+    [{"Server","Microsoft-IIS/5.0"},
+     {"Content-Location","http://intranet/messenger/index.html"},
+     {"Date","Fri, 17 Dec 2004 15:16:19 GMT"},
+     {"Content-Type","text/html"},
+     {"Accept-Ranges","bytes"},
+     {"Last-Modified","Fri, 17 Dec 2004 08:38:21 GMT"},
+     {"Etag","\"aa7c9dc313e4c41:d77\""},
+     {"Content-Length","953"}],
+    "<html>\r\n\r\n<head>\r\n<title>Messenger</title>\r\n<meta name=\"GENERATOR\" content=\"Microsoft FrontPage 5.0\">\r\n<meta name=\"ProgId\" content=\"FrontPage.Editor.Document\">\r\n<meta name=\"description\" content=\"Messenger Home Page\">\r\n</head>\r\n\r\n<frameset border=\"0\" frameborder=\"0\" rows=\"60,*\">\r\n  <frame src=\"/messenger/images/topnav.html\" name=\"mFrameTopNav\" scrolling=\"NO\" target=\"mFrameMain\">\r\n  <frameset cols=\"18%,*\">\r\n    <frameset rows=\"*,120\">\r\n      <frame src=\"index-toc.html\" name=\"mFrameTOC\" target=\"mFrameMain\" scrolling=\"auto\"  noresize=\"true\">\r\n      <frame src=\"/shared/search/namesearch.html\" name=\"mFrameNameSearch\" scrolling=\"NO\" target=\"mFrameMain\">\r\n    </frameset>\r\n    <frame src=\"home/16-12-04-xmascardsmms.htm\" name=\"mFrameMain\" scrolling=\"auto\" target=\"mFrameMain\" id=\"mFrameMain\">\r\n  </frameset>\r\n  <noframes>\r\n  <body>\r\n\r\n  <p><i>This site requires a browser that can view frames.</i></p>\r\n\r\n  </body>\r\n  </noframes>\r\n</frameset>\r\n\r\n</html>"}
+
+%% =============================================================================
+%% A GET using a proxy
+7> ibrowse:send_req("http://www.google.com/", [], get, [], 
+                 [{proxy_user, "XXXXX"},
+                  {proxy_password, "XXXXX"},
+                  {proxy_host, "proxy"},
+                  {proxy_port, 8080}], 1000).
+{ok,"302",
+    [{"Date","Fri, 17 Dec 2004 15:22:56 GMT"},
+     {"Content-Length","217"},
+     {"Content-Type","text/html"},
+     {"Set-Cookie",
+      "PREF=ID=f58155c797f96096:CR=1:TM=1103296999:LM=1103296999:S=FiWdtAqQvhQ0TvHq; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com"},
+     {"Server","GWS/2.1"},
+     {"Location",
+      "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1103296999:S%3Do8bEY2FIHwdyGenS&prev=/"},
+     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],
+    "<HTML><HEAD><TITLE>302 Moved</TITLE></HEAD><BODY>\n<H1>302 Moved</H1>\nThe document has moved\n<A HREF=\"http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1103296999:S%3Do8bEY2FIHwdyGenS&amp;prev=/\">here</A>.\r\n</BODY></HTML>\r\n"}
+
+%% =============================================================================
+%% A GET response saved to file. A temporary file is created and the
+%% filename returned. The response will only be saved to file is the
+%% status code is in the 200 range. The directory to download to can
+%% be set using the application env var 'download_dir' - the default
+%% is the current working directory.
+8> ibrowse:send_req("http://www.erlang.se/", [], get, [],
+                 [{proxy_user, "XXXXX"},
+                  {proxy_password, "XXXXX"},
+                  {proxy_host, "proxy"},
+                  {proxy_port, 8080},
+                  {save_response_to_file, true}], 1000).
+{error,req_timedout}
+
+%% =============================================================================
+9> ibrowse:send_req("http://www.erlang.se/", [], get, [],
+                 [{proxy_user, "XXXXX"},
+                  {proxy_password, "XXXXX"},
+                  {proxy_host, "proxy"},
+                  {proxy_port, 8080},
+                  {save_response_to_file, true}], 5000).
+{ok,"200",
+    [{"Transfer-Encoding","chunked"},
+     {"Date","Fri, 17 Dec 2004 15:24:36 GMT"},
+     {"Content-Type","text/html"},
+     {"Server","Apache/1.3.9 (Unix)"},
+     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],
+    {file,"/Users/chandru/code/ibrowse/src/ibrowse_tmp_file_1103297041125854"}}
+
+%% =============================================================================
+%% Setting size of connection pool and pipeline size. This sets the
+%% number of maximum connections to this server to 10 and the pipeline
+%% size to 1. Connections are setup a required.
+11> ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10},
+                                             {max_pipeline_size, 1}]).
+ok
+
+%% =============================================================================
+%% Example using the HEAD method
+56> ibrowse:send_req("http://www.erlang.org", [], head).
+{ok,"200",
+    [{"Date","Mon, 28 Feb 2005 04:40:53 GMT"},
+     {"Server","Apache/1.3.9 (Unix)"},
+     {"Last-Modified","Thu, 10 Feb 2005 09:31:23 GMT"},
+     {"Etag","\"8d71d-1efa-420b29eb\""},
+     {"Accept-ranges","bytes"},
+     {"Content-Length","7930"},
+     {"Content-Type","text/html"}],
+    []}
+
+%% =============================================================================
+%% Example using the OPTIONS method
+62> ibrowse:send_req("http://www.sun.com", [], options).   
+{ok,"200",
+    [{"Server","Sun Java System Web Server 6.1"},
+     {"Date","Mon, 28 Feb 2005 04:44:39 GMT"},
+     {"Content-Length","0"},
+     {"P3p",
+      "policyref=\"http://www.sun.com/p3p/Sun_P3P_Policy.xml\", CP=\"CAO DSP COR CUR ADMa DEVa TAIa PSAa PSDa CONi TELi OUR  SAMi PUBi IND PHY ONL PUR COM NAV INT DEM CNT STA POL PRE GOV\""},
+     {"Set-Cookie",
+      "SUN_ID=X.X.X.X:169191109565879; EXPIRES=Wednesday, 31-Dec-2025 23:59:59 GMT; DOMAIN=.sun.com; PATH=/"},
+     {"Allow",
+      "HEAD, GET, PUT, POST, DELETE, TRACE, OPTIONS, MOVE, INDEX, MKDIR, RMDIR"}],
+    []}
+
+%% =============================================================================
+%% Example of using Asynchronous requests
+18> ibrowse:send_req("http://www.google.com", [], get, [], 
+                     [{proxy_user, "XXXXX"}, 
+                      {proxy_password, "XXXXX"}, 
+                      {proxy_host, "proxy"}, 
+                      {proxy_port, 8080}, 
+                      {stream_to, self()}]).
+{ibrowse_req_id,{1115,327256,389608}}
+19> flush().
+Shell got {ibrowse_async_headers,{1115,327256,389608},
+           "302",
+           [{"Date","Thu, 05 May 2005 21:06:41 GMT"},
+            {"Content-Length","217"},
+            {"Content-Type","text/html"},
+            {"Set-Cookie",
+             "PREF=ID=b601f16bfa32f071:CR=1:TM=1115327201:LM=1115327201:S=OX5hSB525AMjUUu7; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com"},
+            {"Server","GWS/2.1"},
+            {"Location",
+             "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1115327201:S%3DDS9pDJ4IHcAuZ_AS&prev=/"},
+            {"Via",
+             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}
+Shell got {ibrowse_async_response,{1115,327256,389608},
+           "<HTML><HEAD><TITLE>302 Moved</TITLE></HEAD><BODY>\n<H1>302 Moved</H1>\nThe document has moved\n<A HREF=\"http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1115327201:S%3DDS9pDJ4IHcAuZ_AS&amp;prev=/\">here</A>.\r\n</BODY></HTML>\r\n"}
+Shell got {ibrowse_async_response_end,{1115,327256,389608}}
+ok
+
+%% =============================================================================
+%% Another example of using async requests
+24> ibrowse:send_req("http://yaws.hyber.org/simple_ex2.yaws", [], get, [], 
+                     [{proxy_user, "XXXXX"}, 
+                      {proxy_password, "XXXXX"}, 
+                      {proxy_host, "proxy"}, 
+                      {proxy_port, 8080}, 
+                      {stream_to, self()}]).
+{ibrowse_req_id,{1115,327430,512314}}
+25> flush().
+Shell got {ibrowse_async_headers,{1115,327430,512314},
+           "200", 
+           [{"Date","Thu, 05 May 2005 20:58:08 GMT"},
+            {"Content-Length","64"},
+            {"Content-Type","text/html;charset="},
+            {"Server",
+             "Yaws/1.54 Yet Another Web Server"},
+            {"Via",
+             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}
+Shell got {ibrowse_async_response,{1115,327430,512314},
+           "<html>\n\n\n<h1> Yesssssss </h1>\n\n<h2> Hello again </h2>\n\n\n</html>\n"}
+Shell got {ibrowse_async_response_end,{1115,327430,512314}}
+
+%% =============================================================================
+%% Example of request which fails when using the async option. Here
+%% the {ibrowse_req_id, ReqId} is not returned. Instead the error code is
+%% returned.  
+68> ibrowse:send_req("http://www.earlyriser.org", [], get, [], [{stream_to, self()}]).
+{error,conn_failed}
+
+%% Example of request using both Proxy-Authorization and authorization by the final webserver.
+17> ibrowse:send_req("http://www.erlang.se/lic_area/protected/patches/erl_756_otp_beam.README", 
+                     [], get, [], 
+                     [{proxy_user, "XXXXX"}, 
+                      {proxy_password, "XXXXX"}, 
+                      {proxy_host, "proxy"}, 
+                      {proxy_port, 8080}, 
+                      {basic_auth, {"XXXXX", "XXXXXX"}}]).
+{ok,"200",
+    [{"Accept-Ranges","bytes"},
+     {"Date","Thu, 05 May 2005 21:02:09 GMT"},
+     {"Content-Length","2088"},
+     {"Content-Type","text/plain"},
+     {"Server","Apache/1.3.9 (Unix)"},
+     {"Last-Modified","Tue, 03 May 2005 15:08:18 GMT"},
+     {"ETag","\"1384c8-828-427793e2\""},
+     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],
+    "Patch Id:\t\terl_756_otp_beam\nLabel:\t\t\tinets patch\nDate:\t\t\t2005-05-03\nTrouble Report Id:\tOTP-5513, OTP-5514, OTP-5516, OTP-5517, OTP-5521, OTP-5537\nSeq num:\t\tseq9806\nSystem:\t\t\totp\nRelease:\t\tR10B\nOperating System:\tall\nArchitecture:\t\tall\nErlang machine:\t\tBEAM\nApplication:\t\tinets-4.4\nFiles:\t\t\tall\n\nDescription:\n\n    OTP-5513  The server did not handle HTTP-0.9 messages with an implicit\n\t      version.\n\n    OTP-5514  An internal server timeout killed the request handling\n\t      process without sending a message back to the client. As this\n\t      timeout only affects a single request it has been set to\n\t      infinity (if the main server process dies the request\n\t      handling process will also die and the client will receive an\n\t      error). This might make a client that does not use a timeout\n\t      hang for a longer period of time, but that is an expected\n\t      behavior!\n\n    OTP-5516  That a third party closes the http servers accept socket is\n\t      recoverable for inets, hence intes will only produce an info\n\t      report as there was no error in inets but measures where\n\t      taken to avoid failure due to errors elsewhere.\n\n    OTP-5517  The HTTP client proxy settings where ignored. Bug introduced\n\t      in inets-4.3.\n\n    OTP-5521  Inets only sent the \"WWW-Authenticate\" header at the first\n\t      attempt to get a page, if the user supplied the wrong\n\t      user/password combination the header was not sent again. This\n\t      forces the user to kill the browser entirely after a failed\n\t      login attempt, before the user may try to login again. Inets\n\t      now always send the authentication header.\n\n    OTP-5537  A major rewrite of big parts of the HTTP server code was\n\t      performed. There where many things that did not work\n\t      satisfactory. Cgi script handling can never have worked\n\t      properly and the cases when it did sort of work, a big\n\t      unnecessary delay was enforced. Headers where not always\n\t      treated as expected and HTTP version handling did not work,\n\t      all responses where sent as version HTTP/1.1 no matter what.\n\n\n"}
+
+%% =============================================================================
+%% Example of a TRACE request. Very interesting! yaws.hyber.org didn't
+%% support this. Nor did www.google.com. But good old BBC supports
+%% this.
+35> 37> ibrowse:send_req("http://www.bbc.co.uk/", [], trace, [], 
+                         [{proxy_user, "XXXXX"},
+                          {proxy_password, "XXXXX"},
+                          {proxy_host, "proxy"},
+                          {proxy_port, 8080}]).         
+{ok,"200",
+    [{"Transfer-Encoding","chunked"},
+     {"Date","Thu, 05 May 2005 21:40:27 GMT"},
+     {"Content-Type","message/http"},
+     {"Server","Apache/2.0.51 (Unix)"},
+     {"Set-Cookie",
+      "BBC-UID=7452e72a29424c5b0b232c7131c7d9395d209b7170e8604072e0fcb3630467300; expires=Mon, 04-May-09 21:40:27 GMT; path=/; domain=bbc.co.uk;"},
+     {"Set-Cookie",
+      "BBC-UID=7452e72a29424c5b0b232c7131c7d9395d209b7170e8604072e0fcb3630467300; expires=Mon, 04-May-09 21:40:27 GMT; path=/; domain=bbc.co.uk;"},
+     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],
+    "TRACE / HTTP/1.1\r\nHost: www.bbc.co.uk\r\nConnection: keep-alive\r\nX-Forwarded-For: 172.24.28.29\r\nVia: 1.1 hatproxy01 (NetCache NetApp/5.6.2)\r\nCookie: BBC-UID=7452e72a29424c5b0b232c7131c7d9395d209b7170e8604072e0fcb3630467300\r\n\r\n"}

lib/ibrowse/c_src/build_darwin

+cc -o ../priv/ibrowse_drv.so -I ~/R9C-0/usr/include/ -bundle -flat_namespace -undefined suppress -fno-common ibrowse_drv.c

lib/ibrowse/c_src/ibrowse_drv.c

+/* Created 07/March/2004 Chandrashekhar Mullaparthi
+
+   $Id: ibrowse_drv.c,v 1.1 2005/05/05 22:28:28 chandrusf Exp $
+   
+   Erlang Linked in driver to URL encode a set of data
+*/
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "erl_driver.h"
+
+static ErlDrvData ibrowse_drv_start(ErlDrvPort port, char* buff);
+static void ibrowse_drv_stop(ErlDrvData handle);
+static void ibrowse_drv_command(ErlDrvData handle, char *buff, int bufflen);
+static void ibrowse_drv_finish(void);
+static int ibrowse_drv_control(ErlDrvData handle, unsigned int command, 
+                       char* buf, int count, char** res, int res_size);
+
+/* The driver entry */
+static ErlDrvEntry ibrowse_driver_entry = {
+    NULL,                          /* init, N/A */
+    ibrowse_drv_start,             /* start, called when port is opened */
+    ibrowse_drv_stop,              /* stop, called when port is closed */
+    NULL,                          /* output, called when erlang has sent */
+    NULL,                          /* ready_input, called when input descriptor 
+				      ready */
+    NULL,                          /* ready_output, called when output 
+				      descriptor ready */
+    "ibrowse_drv",                 /* char *driver_name, the argument 
+				      to open_port */
+    NULL,                          /* finish, called when unloaded */
+    NULL,                          /* void * that is not used (BC) */
+    ibrowse_drv_control,           /* control, port_control callback */
+    NULL,                          /* timeout, called on timeouts */
+    NULL,                          /* outputv, vector output interface */
+    NULL,
+    NULL,
+    NULL,                          /* call, synchronous call to driver */
+    NULL
+};
+
+typedef struct ibrowse_drv_data {
+  unsigned int count;
+  void *alloc_ptr;
+} State;
+
+static State *ibrowse_drv_data;
+
+DRIVER_INIT(ibrowse_drv)
+{
+  ibrowse_drv_data = NULL;
+  return &ibrowse_driver_entry;
+}
+
+static ErlDrvData ibrowse_drv_start(ErlDrvPort port, char *buff)
+{
+  State *state;
+
+  state = driver_alloc(sizeof(State));
+  state->count = 0;
+  state->alloc_ptr = NULL;
+
+  ibrowse_drv_data = state;
+  return ((ErlDrvData) state);
+}
+
+void ibrowse_drv_stop(ErlDrvData desc)
+{
+  return;
+}
+
+static int ibrowse_drv_control(ErlDrvData handle, unsigned int command,
+			       char *buf, int bufflen, char **rbuf, int rlen)
+{
+  State* state = (State *) handle;
+  unsigned int j = 0, i = 0;
+  unsigned int temp = 0, rlen_1 = 0;
+  char* replybuf;
+
+  fprintf(stderr, "alloc_ptr -> %d\n", state->alloc_ptr);
+/*   if(state->alloc_ptr != NULL) */
+/*     { */
+/*       driver_free(state->alloc_ptr); */
+/*     } */
+
+  /* Calculate encoded length. If same as bufflen, it means there is
+     no encoding to do. Do return an empty list */
+  rlen_1 = calc_encoded_length(buf, bufflen);
+  if(rlen_1 == bufflen)
+    {
+      *rbuf = NULL;
+      state->alloc_ptr = NULL;
+      return 0;
+    }
+  *rbuf = driver_alloc(rlen_1);
+  state->alloc_ptr = *rbuf;
+  fprintf(stderr, "*rbuf -> %d\n", *rbuf);
+  replybuf = *rbuf;
+
+  for(i=0;i<bufflen;i++)
+    {
+      temp = buf[i];
+      if( 'a' <= temp && temp <= 'z'
+	  || 'A' <= temp && temp <= 'Z'
+	  || '0' <= temp && temp <= '9'
+	  || temp == '-' || temp == '_' || temp == '.' )
+	{
+	  replybuf[j++] = temp;
+	  /*	  printf("j -> %d\n", j); */
+	}
+      else
+	{
+	  replybuf[j++] = 37;
+	  /* printf("temp -> %d\n", temp);
+	     printf("d2h(temp >> 4) -> %d\n", d2h(temp >> 4));
+	     printf("d2h(temp & 15) -> %d\n", d2h(temp & 15)); */
+	  replybuf[j++] = d2h(temp >> 4);
+	  replybuf[j++] = d2h(temp & 15);
+	  /*	  printf("j -> %d\n", j); */
+	}
+    }
+  return rlen_1;
+}
+
+/* Calculates the length of the resulting buffer if a string is URL encoded */
+int calc_encoded_length(char* buf, int bufflen)
+{
+  unsigned int count=0, i=0, temp=0;
+
+  for(i=0;i<bufflen;i++)
+    {
+      temp = buf[i];
+      if( 'a' <= temp && temp <= 'z'
+	  || 'A' <= temp && temp <= 'Z'
+	  || '0' <= temp && temp <= '9'
+	  || temp == '-' || temp == '_' || temp == '.' )
+	{
+	  count++;
+	}
+      else
+	{
+	  count = count+3;
+	}
+    }
+  return count;
+}
+
+/* Converts an integer in the range 1-15 into it's ascii value
+   1 -> 49 (ascii value of 1)
+   10 -> 97 (ascii value of a)
+*/
+int d2h(unsigned int i)
+{
+  if( i < 10 )
+    {
+      return i + 48;
+    }
+  else
+    {
+      return i + 97 - 10;
+    }
+}

lib/ibrowse/doc/ibrowse.html

+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>Module ibrowse</title>
+<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
+</head>
+<body bgcolor="white">
+<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<hr>
+
+<h1>Module ibrowse</h1>
+<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The ibrowse application implements an HTTP 1.1 client.
+<p>Copyright � 2005-2010 Chandrashekhar Mullaparthi</p>
+
+<p><b>Version:</b> 1.6.0</p>
+<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
+<p><b>Authors:</b> Chandrashekhar Mullaparthi (<a href="mailto:chandrashekhar dot mullaparthi at gmail dot com"><tt>chandrashekhar dot mullaparthi at gmail dot com</tt></a>).</p>
+
+<h2><a name="description">Description</a></h2><p>The ibrowse application implements an HTTP 1.1 client. This  
+module implements the API of the HTTP client. There is one named  
+process called 'ibrowse' which assists in load balancing and maintaining configuration. There is one load balancing process per unique webserver. There is  
+one process to handle one TCP connection to a webserver  
+(implemented in the module ibrowse_http_client). Multiple connections to a  
+webserver are setup based on the settings for each webserver. The  
+ibrowse process also determines which connection to pipeline a  
+certain request on.  The functions to call are send_req/3,  
+send_req/4, send_req/5, send_req/6.</p>
+ 
+  <p>Here are a few sample invocations.</p>
+ 
+  <code>
+  ibrowse:send_req("http://intranet/messenger/", [], get).
+  <br><br>
+ 
+  ibrowse:send_req("http://www.google.com/", [], get, [],
+                [{proxy_user, "XXXXX"},
+                 {proxy_password, "XXXXX"},
+                 {proxy_host, "proxy"},
+                 {proxy_port, 8080}], 1000).
+  <br><br>
+ 
+ ibrowse:send_req("http://www.erlang.org/download/otp_src_R10B-3.tar.gz", [], get, [],
+                [{proxy_user, "XXXXX"},
+                 {proxy_password, "XXXXX"},
+                 {proxy_host, "proxy"},
+                 {proxy_port, 8080},
+                 {save_response_to_file, true}], 1000).
+  <br><br>
+ 
+  ibrowse:send_req("http://www.erlang.org", [], head).
+ 
+  <br><br>
+  ibrowse:send_req("http://www.sun.com", [], options).
+ 
+  <br><br>
+  ibrowse:send_req("http://www.bbc.co.uk", [], trace).
+ 
+  <br><br>
+  ibrowse:send_req("http://www.google.com", [], get, [],
+                    [{stream_to, self()}]).
+  </code>
+ 
+<h2><a name="index">Function Index</a></h2>
+<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#all_trace_off-0">all_trace_off/0</a></td><td>Turn Off ALL tracing.</td></tr>
+<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
+<tr><td valign="top"><a href="#get_config_value-1">get_config_value/1</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#get_config_value-2">get_config_value/2</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
+<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
+<tr><td valign="top"><a href="#rescan_config-0">rescan_config/0</a></td><td>Clear current configuration for ibrowse and load from the file
+  ibrowse.conf in the IBROWSE_EBIN/../priv directory.</td></tr>
+<tr><td valign="top"><a href="#rescan_config-1">rescan_config/1</a></td><td></td></tr>
+<tr><td valign="top"><a href="#send_req-3">send_req/3</a></td><td>This is the basic function to send a HTTP request.</td></tr>
+<tr><td valign="top"><a href="#send_req-4">send_req/4</a></td><td>Same as send_req/3.</td></tr>
+<tr><td valign="top"><a href="#send_req-5">send_req/5</a></td><td>Same as send_req/4.</td></tr>
+<tr><td valign="top"><a href="#send_req-6">send_req/6</a></td><td>Same as send_req/5.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-4">send_req_direct/4</a></td><td>Same as send_req/3 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-5">send_req_direct/5</a></td><td>Same as send_req/4 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-6">send_req_direct/6</a></td><td>Same as send_req/5 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-7">send_req_direct/7</a></td><td>Same as send_req/6 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#set_dest-3">set_dest/3</a></td><td>Deprecated.</td></tr>
+<tr><td valign="top"><a href="#set_max_pipeline_size-3">set_max_pipeline_size/3</a></td><td>Set the maximum pipeline size for each connection to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#set_max_sessions-3">set_max_sessions/3</a></td><td>Set the maximum number of connections allowed to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#show_dest_status-0">show_dest_status/0</a></td><td></td></tr>
+<tr><td valign="top"><a href="#show_dest_status-2">show_dest_status/2</a></td><td>Shows some internal information about load balancing to a
+  specified Host:Port.</td></tr>
+<tr><td valign="top"><a href="#spawn_link_worker_process-1">spawn_link_worker_process/1</a></td><td>Same as spawn_worker_process/1 except the the calling process
+  is linked to the worker process which is spawned.</td></tr>
+<tr><td valign="top"><a href="#spawn_link_worker_process-2">spawn_link_worker_process/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#spawn_worker_process-1">spawn_worker_process/1</a></td><td>Creates a HTTP client process to the specified Host:Port which
+  is not part of the load balancing pool.</td></tr>
+<tr><td valign="top"><a href="#spawn_worker_process-2">spawn_worker_process/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#start-0">start/0</a></td><td>Starts the ibrowse process without linking.</td></tr>
+<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td>Starts the ibrowse process linked to the calling process.</td></tr>
+<tr><td valign="top"><a href="#stop-0">stop/0</a></td><td>Stop the ibrowse process.</td></tr>
+<tr><td valign="top"><a href="#stop_worker_process-1">stop_worker_process/1</a></td><td>Terminate a worker process spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#stream_next-1">stream_next/1</a></td><td>Tell ibrowse to stream the next chunk of data to the
+  caller.</td></tr>
+<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#trace_off-0">trace_off/0</a></td><td>Turn tracing off for the ibrowse process.</td></tr>
+<tr><td valign="top"><a href="#trace_off-2">trace_off/2</a></td><td>Turn tracing OFF for all connections to the specified HTTP
+  server.</td></tr>
+<tr><td valign="top"><a href="#trace_on-0">trace_on/0</a></td><td>Turn tracing on for the ibrowse process.</td></tr>
+<tr><td valign="top"><a href="#trace_on-2">trace_on/2</a></td><td>Turn tracing on for all connections to the specified HTTP
+  server.</td></tr>
+</table>
+
+<h2><a name="functions">Function Details</a></h2>
+
+<h3 class="function"><a name="all_trace_off-0">all_trace_off/0</a></h3>
+<div class="spec">
+<p><tt>all_trace_off() -&gt; ok</tt></p>
+</div><p>Turn Off ALL tracing</p>
+
+<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
+<div class="spec">
+<p><tt>code_change(OldVsn, State, Extra) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="get_config_value-1">get_config_value/1</a></h3>
+<div class="spec">
+<p><tt>get_config_value(Key) -&gt; any()</tt></p>
+</div><p>Internal export</p>
+
+<h3 class="function"><a name="get_config_value-2">get_config_value/2</a></h3>
+<div class="spec">
+<p><tt>get_config_value(Key, DefVal) -&gt; any()</tt></p>
+</div><p>Internal export</p>
+
+<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
+<div class="spec">
+<p><tt>handle_call(Request, From, State) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
+<div class="spec">
+<p><tt>handle_cast(Msg, State) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
+<div class="spec">
+<p><tt>handle_info(Info, State) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="init-1">init/1</a></h3>
+<div class="spec">
+<p><tt>init(X1) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="rescan_config-0">rescan_config/0</a></h3>
+<div class="spec">
+<p><tt>rescan_config() -&gt; any()</tt></p>
+</div><p>Clear current configuration for ibrowse and load from the file
+  ibrowse.conf in the IBROWSE_EBIN/../priv directory. Current
+  configuration is cleared only if the ibrowse.conf file is readable
+  using file:consult/1</p>
+
+<h3 class="function"><a name="rescan_config-1">rescan_config/1</a></h3>
+<div class="spec">
+<p><tt>rescan_config(File) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="send_req-3">send_req/3</a></h3>
+<div class="spec">
+<p><tt>send_req(Url::string(), Headers::<a href="#type-headerList">headerList()</a>, Method::<a href="#type-method">method()</a>) -&gt; <a href="#type-response">response()</a></tt>
+<ul class="definitions"><li><tt><a name="type-headerList">headerList()</a> = [{<a href="#type-header">header()</a>, <a href="#type-value">value()</a>}]</tt></li>
+<li><tt><a name="type-header">header()</a> = atom() | string()</tt></li>
+<li><tt><a name="type-value">value()</a> = term()</tt></li>
+<li><tt><a name="type-method">method()</a> = get | post | head | options | put | delete | trace | mkcol | propfind | proppatch | lock | unlock | move | copy</tt></li>
+<li><tt>Status = string()</tt></li>
+<li><tt>ResponseHeaders = [<a href="#type-respHeader">respHeader()</a>]</tt></li>
+<li><tt><a name="type-respHeader">respHeader()</a> = {<a href="#type-headerName">headerName()</a>, <a href="#type-headerValue">headerValue()</a>}</tt></li>
+<li><tt><a name="type-headerName">headerName()</a> = string()</tt></li>
+<li><tt><a name="type-headerValue">headerValue()</a> = string()</tt></li>
+<li><tt><a name="type-response">response()</a> = {ok, Status, ResponseHeaders, ResponseBody} | {ibrowse_req_id, <a href="#type-req_id">req_id()</a>} | {error, Reason}</tt></li>
+<li><tt><a name="type-req_id">req_id()</a> = term()</tt></li>
+<li><tt>ResponseBody = string() | {file, Filename}</tt></li>
+<li><tt>Reason = term()</tt></li>
+</ul></p>
+</div><p>This is the basic function to send a HTTP request.
+  The Status return value indicates the HTTP status code returned by the webserver</p>
+
+<h3 class="function"><a name="send_req-4">send_req/4</a></h3>
+<div class="spec">
+<p><tt>send_req(Url, Headers, Method::<a href="#type-method">method()</a>, Body::<a href="#type-body">body()</a>) -&gt; <a href="#type-response">response()</a></tt>
+<ul class="definitions"><li><tt><a name="type-body">body()</a> = [] | string() | binary() | <a href="#type-fun_arity_0">fun_arity_0()</a> | {<a href="#type-fun_arity_1">fun_arity_1()</a>, <a href="#type-initial_state">initial_state()</a>}</tt></li>
+<li><tt><a name="type-initial_state">initial_state()</a> = term()</tt></li>
+</ul></p>
+</div><p>Same as send_req/3.
+  If a list is specified for the body it has to be a flat list. The body can also be a fun/0 or a fun/1. <br>
+  If fun/0, the connection handling process will repeatdely call the fun until it returns an error or eof. <pre>Fun() = {ok, Data} | eof</pre><br>
+  If fun/1, the connection handling process will repeatedly call the fun with the supplied state until it returns an error or eof. <pre>Fun(State) = {ok, Data} | {ok, Data, NewState} | eof</pre></p>
+
+<h3 class="function"><a name="send_req-5">send_req/5</a></h3>
+<div class="spec">
+<p><tt>send_req(Url::string(), Headers::<a href="#type-headerList">headerList()</a>, Method::<a href="#type-method">method()</a>, Body::<a href="#type-body">body()</a>, Options::<a href="#type-optionList">optionList()</a>) -&gt; <a href="#type-response">response()</a></tt>
+<ul class="definitions"><li><tt><a name="type-optionList">optionList()</a> = [<a href="#type-option">option()</a>]</tt></li>
+<li><tt><a name="type-option">option()</a> = {max_sessions, integer()} | {response_format, <a href="#type-response_format">response_format()</a>} | {stream_chunk_size, integer()} | {max_pipeline_size, integer()} | {trace, <a href="#type-boolean">boolean()</a>} | {is_ssl, <a href="#type-boolean">boolean()</a>} | {ssl_options, [SSLOpt]} | {pool_name, atom()} | {proxy_host, string()} | {proxy_port, integer()} | {proxy_user, string()} | {proxy_password, string()} | {use_absolute_uri, <a href="#type-boolean">boolean()</a>} | {basic_auth, {<a href="#type-username">username()</a>, <a href="#type-password">password()</a>}} | {cookie, string()} | {content_length, integer()} | {content_type, string()} | {save_response_to_file, <a href="#type-srtf">srtf()</a>} | {stream_to, <a href="#type-stream_to">stream_to()</a>} | {http_vsn, {MajorVsn, MinorVsn}} | {host_header, string()} | {inactivity_timeout, integer()} | {connect_timeout, integer()} | {socket_options, Sock_opts} | {transfer_encoding, {chunked, ChunkSize}} | {headers_as_is, <a href="#type-boolean">boolean()</a>} | {give_raw_headers, <a href="#type-boolean">boolean()</a>}</tt></li>
+<li><tt><a name="type-stream_to">stream_to()</a> = <a href="#type-process">process()</a> | {<a href="#type-process">process()</a>, once}</tt></li>
+<li><tt><a name="type-process">process()</a> = pid() | atom()</tt></li>
+<li><tt><a name="type-username">username()</a> = string()</tt></li>
+<li><tt><a name="type-password">password()</a> = string()</tt></li>
+<li><tt>SSLOpt = term()</tt></li>
+<li><tt>Sock_opts = [Sock_opt]</tt></li>
+<li><tt>Sock_opt = term()</tt></li>
+<li><tt>ChunkSize = integer()</tt></li>
+<li><tt><a name="type-srtf">srtf()</a> = <a href="#type-boolean">boolean()</a> | <a href="#type-filename">filename()</a></tt></li>
+<li><tt><a name="type-filename">filename()</a> = string()</tt></li>
+<li><tt><a name="type-response_format">response_format()</a> = list | binary</tt></li>
+</ul></p>
+</div><p>Same as send_req/4.
+  For a description of SSL Options, look in the <a href="http://www.erlang.org/doc/apps/ssl/index.html">ssl</a> manpage. If the
+  HTTP Version to use is not specified, the default is 1.1.
+  <br>
+  <ul>
+  <li>The <code>host_header</code> option is useful in the case where ibrowse is
+  connecting to a component such as <a href="http://www.stunnel.org">stunnel</a> which then sets up a
+  secure connection to a webserver. In this case, the URL supplied to
+  ibrowse must have the stunnel host/port details, but that won't
+  make sense to the destination webserver. This option can then be
+  used to specify what should go in the <code>Host</code> header in
+  the request.</li>
+  <li>The <code>stream_to</code> option can be used to have the HTTP
+  response streamed to a process as messages as data arrives on the
+  socket. If the calling process wishes to control the rate at which
+  data is received from the server, the option <code>{stream_to,
+  {process(), once}}</code> can be specified. The calling process
+  will have to invoke <code>ibrowse:stream_next(Request_id)</code> to
+  receive the next packet.</li>
+ 
+  <li>When both the options <code>save_response_to_file</code> and <code>stream_to</code>
+  are specified, the former takes precedence.</li>
+ 
+  <li>For the <code>save_response_to_file</code> option, the response body is saved to
+  file only if the status code is in the 200-299 range. If not, the response body is returned
+  as a string.</li>
+  <li>Whenever an error occurs in the processing of a request, ibrowse will return as much
+  information as it has, such as HTTP Status Code and HTTP Headers. When this happens, the response
+  is of the form <code>{error, {Reason, {stat_code, StatusCode}, HTTP_headers}}</code></li>
+ 
+  <li>The <code>inactivity_timeout</code> option is useful when
+  dealing with large response bodies and/or slow links. In these
+  cases, it might be hard to estimate how long a request will take to
+  complete. In such cases, the client might want to timeout if no
+  data has been received on the link for a certain time interval.</li>
+ 
+  <li>
+  The <code>connect_timeout</code> option is to specify how long the
+  client process should wait for connection establishment. This is
+  useful in scenarios where connections to servers are usually setup
+  very fast, but responses might take much longer compared to
+  connection setup. In such cases, it is better for the calling
+  process to timeout faster if there is a problem (DNS lookup
+  delays/failures, network routing issues, etc). The total timeout
+  value specified for the request will enforced. To illustrate using
+  an example:
+  <code>
+  ibrowse:send_req("http://www.example.com/cgi-bin/request", [], get, [], [{connect_timeout, 100}], 1000).
+  </code>
+  In the above invocation, if the connection isn't established within
+  100 milliseconds, the request will fail with
+  <code>{error, conn_failed}</code>.<br>
+  If connection setup succeeds, the total time allowed for the
+  request to complete will be 1000 milliseconds minus the time taken
+  for connection setup.
+  </li>
+ 
+  <li> The <code>socket_options</code> option can be used to set
+  specific options on the socket. The <code>{active, true | false | once}</code>
+  and <code>{packet_type, Packet_type}</code> will be filtered out by ibrowse.  </li>
+ 
+  <li> The <code>headers_as_is</code> option is to enable the caller
+  to send headers exactly as specified in the request without ibrowse
+  adding some of its own. Required for some picky servers apparently.  </li>
+ 
+  <li>The <code>give_raw_headers</code> option is to enable the
+  caller to get access to the raw status line and raw unparsed
+  headers. Not quite sure why someone would want this, but one of my
+  users asked for it, so here it is. </li>
+  </ul>
+ </p>
+
+<h3 class="function"><a name="send_req-6">send_req/6</a></h3>
+<div class="spec">
+<p><tt>send_req(Url, Headers::<a href="#type-headerList">headerList()</a>, Method::<a href="#type-method">method()</a>, Body::<a href="#type-body">body()</a>, Options::<a href="#type-optionList">optionList()</a>, Timeout) -&gt; <a href="#type-response">response()</a></tt>
+<ul class="definitions"><li><tt>Timeout = integer() | infinity</tt></li>
+</ul></p>
+</div><p>Same as send_req/5.
+  All timeout values are in milliseconds.</p>
+
+<h3 class="function"><a name="send_req_direct-4">send_req_direct/4</a></h3>
+<div class="spec">
+<p><tt>send_req_direct(Conn_pid, Url, Headers, Method) -&gt; any()</tt></p>
+</div><p>Same as send_req/3 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-5">send_req_direct/5</a></h3>
+<div class="spec">
+<p><tt>send_req_direct(Conn_pid, Url, Headers, Method, Body) -&gt; any()</tt></p>
+</div><p>Same as send_req/4 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-6">send_req_direct/6</a></h3>
+<div class="spec">
+<p><tt>send_req_direct(Conn_pid, Url, Headers, Method, Body, Options) -&gt; any()</tt></p>
+</div><p>Same as send_req/5 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-7">send_req_direct/7</a></h3>
+<div class="spec">
+<p><tt>send_req_direct(Conn_pid, Url, Headers, Method, Body, Options, Timeout) -&gt; any()</tt></p>
+</div><p>Same as send_req/6 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="set_dest-3">set_dest/3</a></h3>
+<div class="spec">
+<p><tt>set_dest(Host, Port, T) -&gt; any()</tt></p>
+</div><p>Deprecated. Use set_max_sessions/3 and set_max_pipeline_size/3
+  for achieving the same effect.</p>
+
+<h3 class="function"><a name="set_max_pipeline_size-3">set_max_pipeline_size/3</a></h3>
+<div class="spec">
+<p><tt>set_max_pipeline_size(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum pipeline size for each connection to a specific Host:Port.</p>
+
+<h3 class="function"><a name="set_max_sessions-3">set_max_sessions/3</a></h3>
+<div class="spec">
+<p><tt>set_max_sessions(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum number of connections allowed to a specific Host:Port.</p>
+
+<h3 class="function"><a name="show_dest_status-0">show_dest_status/0</a></h3>
+<div class="spec">
+<p><tt>show_dest_status() -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="show_dest_status-2">show_dest_status/2</a></h3>
+<div class="spec">
+<p><tt>show_dest_status(Host, Port) -&gt; any()</tt></p>
+</div><p>Shows some internal information about load balancing to a
+  specified Host:Port. Info about workers spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2 is not
+  included.</p>
+
+<h3 class="function"><a name="spawn_link_worker_process-1">spawn_link_worker_process/1</a></h3>
+<div class="spec">
+<p><tt>spawn_link_worker_process(Url::string()) -&gt; {ok, pid()}</tt></p>
+</div><p>Same as spawn_worker_process/1 except the the calling process
+  is linked to the worker process which is spawned.</p>
+
+<h3 class="function"><a name="spawn_link_worker_process-2">spawn_link_worker_process/2</a></h3>
+<div class="spec">
+<p><tt>spawn_link_worker_process(Host::string(), Port::integer()) -&gt; {ok, pid()}</tt></p>
+</div>
+
+<h3 class="function"><a name="spawn_worker_process-1">spawn_worker_process/1</a></h3>
+<div class="spec">
+<p><tt>spawn_worker_process(Url::string()) -&gt; {ok, pid()}</tt></p>
+</div><p>Creates a HTTP client process to the specified Host:Port which
+  is not part of the load balancing pool. This is useful in cases
+  where some requests to a webserver might take a long time whereas
+  some might take a very short time. To avoid getting these quick
+  requests stuck in the pipeline behind time consuming requests, use
+  this function to get a handle to a connection process. <br>
+  <b>Note:</b> Calling this function only creates a worker process. No connection
+  is setup. The connection attempt is made only when the first
+  request is sent via any of the send_req_direct/4,5,6,7 functions.<br>
+  <b>Note:</b> It is the responsibility of the calling process to control
+  pipeline size on such connections.
+ </p>
+
+<h3 class="function"><a name="spawn_worker_process-2">spawn_worker_process/2</a></h3>
+<div class="spec">
+<p><tt>spawn_worker_process(Host::string(), Port::integer()) -&gt; {ok, pid()}</tt></p>
+</div>
+
+<h3 class="function"><a name="start-0">start/0</a></h3>
+<div class="spec">
+<p><tt>start() -&gt; any()</tt></p>
+</div><p>Starts the ibrowse process without linking. Useful when testing using the shell</p>
+
+<h3 class="function"><a name="start_link-0">start_link/0</a></h3>
+<div class="spec">
+<p><tt>start_link() -&gt; {ok, pid()}</tt></p>
+</div><p>Starts the ibrowse process linked to the calling process. Usually invoked by the supervisor ibrowse_sup</p>
+
+<h3 class="function"><a name="stop-0">stop/0</a></h3>
+<div class="spec">
+<p><tt>stop() -&gt; any()</tt></p>
+</div><p>Stop the ibrowse process. Useful when testing using the shell.</p>
+
+<h3 class="function"><a name="stop_worker_process-1">stop_worker_process/1</a></h3>
+<div class="spec">
+<p><tt>stop_worker_process(Conn_pid::pid()) -&gt; ok</tt></p>
+</div><p>Terminate a worker process spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2. Requests in
+  progress will get the error response <pre>{error, closing_on_request}</pre></p>
+
+<h3 class="function"><a name="stream_next-1">stream_next/1</a></h3>
+<div class="spec">
+<p><tt>stream_next(Req_id::<a href="#type-req_id">req_id()</a>) -&gt; ok | {error, unknown_req_id}</tt></p>
+</div><p>Tell ibrowse to stream the next chunk of data to the
+  caller. Should be used in conjunction with the
+  <code>stream_to</code> option</p>
+
+<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
+<div class="spec">
+<p><tt>terminate(Reason, State) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="trace_off-0">trace_off/0</a></h3>
+<div class="spec">
+<p><tt>trace_off() -&gt; any()</tt></p>
+</div><p>Turn tracing off for the ibrowse process</p>
+
+<h3 class="function"><a name="trace_off-2">trace_off/2</a></h3>
+<div class="spec">
+<p><tt>trace_off(Host, Port) -&gt; ok</tt></p>
+</div><p>Turn tracing OFF for all connections to the specified HTTP
+  server.</p>
+
+<h3 class="function"><a name="trace_on-0">trace_on/0</a></h3>
+<div class="spec">
+<p><tt>trace_on() -&gt; any()</tt></p>
+</div><p>Turn tracing on for the ibrowse process</p>
+
+<h3 class="function"><a name="trace_on-2">trace_on/2</a></h3>
+<div class="spec">
+<p><tt>trace_on(Host, Port) -&gt; ok</tt>
+<ul class="definitions"><li><tt>Host = string()</tt></li>
+<li><tt>Port = integer()</tt></li>
+</ul></p>
+</div><p>Turn tracing on for all connections to the specified HTTP
+  server. Host is whatever is specified as the domain name in the URL</p>
+<hr>
+
+<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<p><i>Generated by EDoc, May 17 2010, 23:21:42.</i></p>
+</body>
+</html>

lib/ibrowse/doc/ibrowse_lib.html

+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>Module ibrowse_lib</title>
+<link rel="stylesheet" type="text/css" href="stylesheet.css">
+</head>
+<body bgcolor="white">
+
+<h1>Module ibrowse_lib</h1>
+Module with a few useful functions.
+<ul><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
+
+<h2><a name="description">Description</a></h2>Module with a few useful functions
+<h2><a name="index">Function Index</a></h2>
+<table width="100%" border="1"><tr><td valign="top"><a href="#dec2hex-2">dec2hex/2</a></td><td>dec2hex taken from gtk.erl in std dist
+  M = integer() -- number of hex digits required
+  N = integer() -- the number to represent as hex.</td></tr>
+<tr><td valign="top"><a href="#decode_base64-1">decode_base64/1</a></td><td>Implements the base64 decoding algorithm.</td></tr>
+<tr><td valign="top"><a href="#decode_rfc822_date-1">decode_rfc822_date/1</a></td><td/></tr>
+<tr><td valign="top"><a href="#drv_ue-1">drv_ue/1</a></td><td/></tr>
+<tr><td valign="top"><a href="#drv_ue-2">drv_ue/2</a></td><td/></tr>
+<tr><td valign="top"><a href="#encode_base64-1">encode_base64/1</a></td><td>Implements the base64 encoding algorithm.</td></tr>
+<tr><td valign="top"><a href="#status_code-1">status_code/1</a></td><td>Given a status code, returns an atom describing the status code.</td></tr>
+<tr><td valign="top"><a href="#url_encode-1">url_encode/1</a></td><td>URL-encodes a string based on RFC 1738.</td></tr>
+</table>
+
+<h2><a name="functions">Function Details</a></h2>
+
+<h3><a name="dec2hex-2">dec2hex/2</a></h3>
+<p><tt>dec2hex(M::integer(), N::integer()) -&gt; string()</tt></p>
+<p>dec2hex taken from gtk.erl in std dist
+  M = integer() -- number of hex digits required
+  N = integer() -- the number to represent as hex</p>
+
+<h3><a name="decode_base64-1">decode_base64/1</a></h3>
+<p><tt>decode_base64(List::In) -&gt; Out | <a href="#type-exit">exit({error, invalid_input})</a></tt>
+<ul><li><tt>In = string() | binary()</tt></li><li><tt>Out = string() | binary()</tt></li></ul></p>
+<p>Implements the base64 decoding algorithm. The output data type matches in the input data type.</p>
+
+<h3><a name="decode_rfc822_date-1">decode_rfc822_date/1</a></h3>
+<tt>decode_rfc822_date(String) -&gt; term()
+</tt>
+
+<h3><a name="drv_ue-1">drv_ue/1</a></h3>
+<tt>drv_ue(Str) -&gt; term()
+</tt>
+
+<h3><a name="drv_ue-2">drv_ue/2</a></h3>
+<tt>drv_ue(Str, Port) -&gt; term()
+</tt>
+
+<h3><a name="encode_base64-1">encode_base64/1</a></h3>
+<p><tt>encode_base64(List::In) -&gt; Out</tt>
+<ul><li><tt>In = string() | binary()</tt></li><li><tt>Out = string() | binary()</tt></li></ul></p>
+<p>Implements the base64 encoding algorithm. The output data type matches in the input data type.</p>
+
+<h3><a name="status_code-1">status_code/1</a></h3>
+<p><tt>status_code(StatusCode::<a href="#type-status_code">status_code()</a>) -&gt; StatusDescription</tt>
+<ul><li><tt><a name="type-status_code">status_code()</a> = string() | integer()</tt></li><li><tt>StatusDescription = atom()</tt></li></ul></p>
+<p>Given a status code, returns an atom describing the status code.</p>
+
+<h3><a name="url_encode-1">url_encode/1</a></h3>
+<p><tt>url_encode(Str) -&gt; UrlEncodedStr</tt>
+<ul><li><tt>Str = string()</tt></li><li><tt>UrlEncodedStr = string()</tt></li></ul></p>
+<p>URL-encodes a string based on RFC 1738. Returns a flat list.</p>
+</body>
+</html>

lib/ibrowse/doc/short-desc

+A powerful HTTP/1.1 client written in erlang

lib/ibrowse/ebin/ibrowse.app

+{application, ibrowse,
+        [{description, "HTTP client application"},
+         {vsn, "1.6.2"},
+         {modules, [ ibrowse, 
+		     ibrowse_http_client, 
+		     ibrowse_app, 
+		     ibrowse_sup, 
+		     ibrowse_lib,
+		     ibrowse_lb ]},
+         {registered, []},
+         {applications, [kernel,stdlib,sasl]},
+	 {env, []},
+	 {mod, {ibrowse_app, []}}]}.

lib/ibrowse/priv/ibrowse.conf

+%% Configuration file for specifying settings for HTTP servers which this
+%% client will connect to.
+%% The format of each entry is (one per line)
+%% {dest, Hostname, Portnumber, MaxSessions, MaxPipelineSize, Options}.
+%%
+%% where Hostname = string()
+%%       Portnumber = integer()
+%%       MaxSessions = integer()
+%%       MaxPipelineSize = integer()
+%%       Options = [{Tag, Val} | ...]
+%%       Tag = term()
+%%       Value = term()
+%% e.g.
+%% {dest, "covig02", 8000, 10, 10, [{is_ssl, true}, {ssl_options, [option()]}]}.
+%% If SSL is to be used, both the options, is_ssl and ssl_options MUST be specified
+%% where option() is all options supported in the ssl module
+
+{{options, "www.google.co.uk", 80}, [{proxy_host, "proxy"}, {proxy_port, 8080}, {proxy_user, "cmullaparthi"}, {proxy_password, "20nov99"}]}.

lib/ibrowse/src/Emakefile.src

+'../src/ibrowse'.
+'../src/ibrowse_http_client'.
+'../src/ibrowse_app'.
+'../src/ibrowse_sup'.
+'../src/ibrowse_lib'.
+'../src/ibrowse_lb'.
+'../src/ibrowse_test'.

lib/ibrowse/src/Makefile

+include ../vsn.mk
+
+ERL_FILES = ibrowse.erl \
+		ibrowse_http_client.erl \
+		ibrowse_app.erl \
+		ibrowse_sup.erl \
+		ibrowse_lib.erl \
+		ibrowse_lb.erl \
+		ibrowse_test.erl
+
+
+INCLUDE_DIRS = -I./ 
+
+ERLC ?= erlc
+ERLC_EMULATOR ?= erl -boot start_clean
+COMPILER_OPTIONS = -W +warn_unused_vars +nowarn_shadow_vars +warn_unused_import
+
+.SUFFIXES: .erl .beam $(SUFFIXES)
+EBIN = ../ebin
+
+all: $(ERL_FILES:%.erl=$(EBIN)/%.beam) ../ebin/ibrowse.app
+
+$(EBIN)/%.beam: %.erl
+	${ERLC} $(COMPILER_OPTIONS) $(INCLUDE_DIRS) -o ../ebin $<
+
+$(EBIN)/%.app:	%.app.src ../vsn.mk Makefile
+	sed -e s^%IBROWSE_VSN%^$(IBROWSE_VSN)^ \
+		 $< > $@
+
+clean:
+	rm -f $(EBIN)/*.beam $(EBIN)/*.app
+

lib/ibrowse/src/ibrowse.app.src

+{application, ibrowse,
+        [{description, "HTTP client application"},
+         {vsn, "%IBROWSE_VSN%"},
+         {modules, [ ibrowse, 
+		     ibrowse_http_client, 
+		     ibrowse_app, 
+		     ibrowse_sup, 
+		     ibrowse_lib,
+		     ibrowse_lb ]},
+         {registered, []},
+         {applications, [kernel,stdlib,sasl]},
+	 {env, []},
+	 {mod, {ibrowse_app, []}}]}.

lib/ibrowse/src/ibrowse.erl

+%%%-------------------------------------------------------------------
+%%% File    : ibrowse.erl
+%%% Author  : Chandrashekhar Mullaparthi <chandrashekhar.mullaparthi@t-mobile.co.uk>
+%%% Description : Load balancer process for HTTP client connections.
+%%%
+%%% Created : 11 Oct 2003 by Chandrashekhar Mullaparthi <chandrashekhar.mullaparthi@t-mobile.co.uk>
+%%%-------------------------------------------------------------------
+%% @author Chandrashekhar Mullaparthi <chandrashekhar dot mullaparthi at gmail dot com>
+%% @copyright 2005-2010 Chandrashekhar Mullaparthi
+%% @version 1.6.0
+%% @doc The ibrowse application implements an HTTP 1.1 client. This
+%% module implements the API of the HTTP client. There is one named
+%% process called 'ibrowse' which assists in load balancing and maintaining configuration. There is one load balancing process per unique webserver. There is
+%% one process to handle one TCP connection to a webserver
+%% (implemented in the module ibrowse_http_client). Multiple connections to a
+%% webserver are setup based on the settings for each webserver. The
+%% ibrowse process also determines which connection to pipeline a
+%% certain request on.  The functions to call are send_req/3,
+%% send_req/4, send_req/5, send_req/6.
+%%
+%% <p>Here are a few sample invocations.</p>
+%%
+%% <code>
+%% ibrowse:send_req("http://intranet/messenger/", [], get). 
+%% <br/><br/>
+%% 
+%% ibrowse:send_req("http://www.google.com/", [], get, [], 
+%%               [{proxy_user, "XXXXX"},
+%%                {proxy_password, "XXXXX"},
+%%                {proxy_host, "proxy"},
+%%                {proxy_port, 8080}], 1000). 
+%% <br/><br/>
+%%
+%%ibrowse:send_req("http://www.erlang.org/download/otp_src_R10B-3.tar.gz", [], get, [],
+%%               [{proxy_user, "XXXXX"},
+%%                {proxy_password, "XXXXX"},
+%%                {proxy_host, "proxy"},
+%%                {proxy_port, 8080},
+%%                {save_response_to_file, true}], 1000).
+%% <br/><br/>
+%%
+%% ibrowse:send_req("http://www.erlang.org", [], head).
+%%
+%% <br/><br/>
+%% ibrowse:send_req("http://www.sun.com", [], options).
+%%
+%% <br/><br/>
+%% ibrowse:send_req("http://www.bbc.co.uk", [], trace).
+%%
+%% <br/><br/>
+%% ibrowse:send_req("http://www.google.com", [], get, [], 
+%%                   [{stream_to, self()}]).
+%% </code>
+%%
+
+-module(ibrowse).
+-behaviour(gen_server).
+%%--------------------------------------------------------------------
+%% Include files
+%%--------------------------------------------------------------------
+
+%%--------------------------------------------------------------------
+%% External exports
+-export([start_link/0, start/0, stop/0]).
+
+%% gen_server callbacks
+-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
+         terminate/2, code_change/3]).
+
+%% API interface
+-export([
+         rescan_config/0,
+         rescan_config/1,
+         get_config_value/1,
+         get_config_value/2,
+         spawn_worker_process/1,
+         spawn_worker_process/2,
+         spawn_link_worker_process/1,
+         spawn_link_worker_process/2,
+         stop_worker_process/1,
+         send_req/3,
+         send_req/4,
+         send_req/5,
+         send_req/6,
+         send_req_direct/4,
+         send_req_direct/5,
+         send_req_direct/6,
+         send_req_direct/7,
+         stream_next/1,
+         set_max_sessions/3,
+         set_max_pipeline_size/3,
+         set_dest/3,
+         trace_on/0,
+         trace_off/0,
+         trace_on/2,
+         trace_off/2,
+         all_trace_off/0,
+         show_dest_status/0,
+         show_dest_status/2
+        ]).
+
+-ifdef(debug).
+-compile(export_all).
+-endif.
+
+-import(ibrowse_lib, [
+                      parse_url/1,
+                      get_value/3,
+                      do_trace/2
+                     ]).
+                      
+-record(state, {trace = false}).
+
+-include("ibrowse.hrl").
+-include_lib("stdlib/include/ms_transform.hrl").
+
+-define(DEF_MAX_SESSIONS,10).
+-define(DEF_MAX_PIPELINE_SIZE,10).
+
+%%====================================================================
+%% External functions
+%%====================================================================
+%%--------------------------------------------------------------------
+%% Function: start_link/0
+%% Description: Starts the server
+%%--------------------------------------------------------------------
+%% @doc Starts the ibrowse process linked to the calling process. Usually invoked by the supervisor ibrowse_sup
+%% @spec start_link() -> {ok, pid()}
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+%% @doc Starts the ibrowse process without linking. Useful when testing using the shell
+start() ->
+    gen_server:start({local, ?MODULE}, ?MODULE, [], [{debug, []}]).
+
+%% @doc Stop the ibrowse process. Useful when testing using the shell.
+stop() ->
+    catch gen_server:call(ibrowse, stop).
+
+%% @doc This is the basic function to send a HTTP request.
+%% The Status return value indicates the HTTP status code returned by the webserver
+%% @spec send_req(Url::string(), Headers::headerList(), Method::method()) -> response()
+%% headerList() = [{header(), value()}]
+%% header() = atom() | string()
+%% value() = term()
+%% method() = get | post | head | options | put | delete | trace | mkcol | propfind | proppatch | lock | unlock | move | copy
+%% Status = string()
+%% ResponseHeaders = [respHeader()]
+%% respHeader() = {headerName(), headerValue()}
+%% headerName() = string()
+%% headerValue() = string()
+%% response() = {ok, Status, ResponseHeaders, ResponseBody} | {ibrowse_req_id, req_id() } | {error, Reason}
+%% req_id() = term()
+%% ResponseBody = string() | {file, Filename}
+%% Reason = term()
+send_req(Url, Headers, Method) ->
+    send_req(Url, Headers, Method, [], []).
+
+%% @doc Same as send_req/3. 
+%% If a list is specified for the body it has to be a flat list. The body can also be a fun/0 or a fun/1. <br/>
+%% If fun/0, the connection handling process will repeatdely call the fun until it returns an error or eof. <pre>Fun() = {ok, Data} | eof</pre><br/>
+%% If fun/1, the connection handling process will repeatedly call the fun with the supplied state until it returns an error or eof. <pre>Fun(State) = {ok, Data} | {ok, Data, NewState} | eof</pre>
+%% @spec send_req(Url, Headers, Method::method(), Body::body()) -> response()
+%% body() = [] | string() | binary() | fun_arity_0() | {fun_arity_1(), initial_state()}
+%% initial_state() = term()
+send_req(Url, Headers, Method, Body) ->
+    send_req(Url, Headers, Method, Body, []).
+
+%% @doc Same as send_req/4. 
+%% For a description of SSL Options, look in the <a href="http://www.erlang.org/doc/apps/ssl/index.html">ssl</a> manpage. If the
+%% HTTP Version to use is not specified, the default is 1.1.
+%% <br/>
+%% <ul>
+%% <li>The <code>host_header</code> option is useful in the case where ibrowse is
+%% connecting to a component such as <a
+%% href="http://www.stunnel.org">stunnel</a> which then sets up a
+%% secure connection to a webserver. In this case, the URL supplied to
+%% ibrowse must have the stunnel host/port details, but that won't
+%% make sense to the destination webserver. This option can then be
+%% used to specify what should go in the <code>Host</code> header in
+%% the request.</li>
+%% <li>The <code>stream_to</code> option can be used to have the HTTP
+%% response streamed to a process as messages as data arrives on the
+%% socket. If the calling process wishes to control the rate at which
+%% data is received from the server, the option <code>{stream_to,
+%% {process(), once}}</code> can be specified. The calling process
+%% will have to invoke <code>ibrowse:stream_next(Request_id)</code> to
+%% receive the next packet.</li>
+%%
+%% <li>When both the options <code>save_response_to_file</code> and <code>stream_to</code> 
+%% are specified, the former takes precedence.</li>
+%%
+%% <li>For the <code>save_response_to_file</code> option, the response body is saved to
+%% file only if the status code is in the 200-299 range. If not, the response body is returned
+%% as a string.</li>
+%% <li>Whenever an error occurs in the processing of a request, ibrowse will return as much
+%% information as it has, such as HTTP Status Code and HTTP Headers. When this happens, the response
+%% is of the form <code>{error, {Reason, {stat_code, StatusCode}, HTTP_headers}}</code></li>
+%%
+%% <li>The <code>inactivity_timeout</code> option is useful when
+%% dealing with large response bodies and/or slow links. In these
+%% cases, it might be hard to estimate how long a request will take to
+%% complete. In such cases, the client might want to timeout if no
+%% data has been received on the link for a certain time interval.</li>
+%%
+%% <li>
+%% The <code>connect_timeout</code> option is to specify how long the
+%% client process should wait for connection establishment. This is
+%% useful in scenarios where connections to servers are usually setup
+%% very fast, but responses might take much longer compared to
+%% connection setup. In such cases, it is better for the calling
+%% process to timeout faster if there is a problem (DNS lookup
+%% delays/failures, network routing issues, etc). The total timeout
+%% value specified for the request will enforced. To illustrate using
+%% an example:
+%% <code>
+%% ibrowse:send_req("http://www.example.com/cgi-bin/request", [], get, [], [{connect_timeout, 100}], 1000).
+%% </code>
+%% In the above invocation, if the connection isn't established within
+%% 100 milliseconds, the request will fail with 
+%% <code>{error, conn_failed}</code>.<br/>
+%% If connection setup succeeds, the total time allowed for the
+%% request to complete will be 1000 milliseconds minus the time taken
+%% for connection setup.
+%% </li>
+%% 
+%% <li> The <code>socket_options</code> option can be used to set
+%% specific options on the socket. The <code>{active, true | false | once}</code> 
+%% and <code>{packet_type, Packet_type}</code> will be filtered out by ibrowse.  </li>
+%%
+%% <li> The <code>headers_as_is</code> option is to enable the caller
+%% to send headers exactly as specified in the request without ibrowse
+%% adding some of its own. Required for some picky servers apparently.  </li>
+%%
+%% <li>The <code>give_raw_headers</code> option is to enable the
+%% caller to get access to the raw status line and raw unparsed
+%% headers. Not quite sure why someone would want this, but one of my
+%% users asked for it, so here it is. </li>
+%% </ul>
+%%
+%% @spec send_req(Url::string(), Headers::headerList(), Method::method(), Body::body(), Options::optionList()) -> response()
+%% optionList() = [option()]
+%% option() = {max_sessions, integer()}        |
+%%          {response_format,response_format()}|
+%%          {stream_chunk_size, integer()}     |
+%%          {max_pipeline_size, integer()}     |
+%%          {trace, boolean()}                 | 
+%%          {is_ssl, boolean()}                |
+%%          {ssl_options, [SSLOpt]}            |
+%%          {pool_name, atom()}                |
+%%          {proxy_host, string()}             |
+%%          {proxy_port, integer()}            |
+%%          {proxy_user, string()}             |
+%%          {proxy_password, string()}         |
+%%          {use_absolute_uri, boolean()}      |
+%%          {basic_auth, {username(), password()}} |
+%%          {cookie, string()}                 |
+%%          {content_length, integer()}        |
+%%          {content_type, string()}           |
+%%          {save_response_to_file, srtf()}    |
+%%          {stream_to, stream_to()}           |
+%%          {http_vsn, {MajorVsn, MinorVsn}}   |
+%%          {host_header, string()}            |
+%%          {inactivity_timeout, integer()}    |
+%%          {connect_timeout, integer()}       |
+%%          {socket_options, Sock_opts}        |
+%%          {transfer_encoding, {chunked, ChunkSize}} | 
+%%          {headers_as_is, boolean()}         |
+%%          {give_raw_headers, boolean()}
+%%
+%% stream_to() = process() | {process(), once}
+%% process() = pid() | atom()
+%% username() = string()
+%% password() = string()
+%% SSLOpt = term()
+%% Sock_opts = [Sock_opt]
+%% Sock_opt = term()
+%% ChunkSize = integer()
+%% srtf() = boolean() | filename()
+%% filename() = string()
+%% response_format() = list | binary
+send_req(Url, Headers, Method, Body, Options) ->
+    send_req(Url, Headers, Method, Body, Options, 30000).
+
+%% @doc Same as send_req/5. 
+%% All timeout values are in milliseconds.
+%% @spec send_req(Url, Headers::headerList(), Method::method(), Body::body(), Options::optionList(), Timeout) -> response()
+%% Timeout = integer() | infinity
+send_req(Url, Headers, Method, Body, Options, Timeout) ->
+    case catch parse_url(Url) of
+        #url{host = Host,
+             port = Port,
+             protocol = Protocol} = Parsed_url ->
+            Lb_pid = case ets:lookup(ibrowse_lb, {Host, Port}) of
+                         [] ->
+                             get_lb_pid(Parsed_url);
+                         [#lb_pid{pid = Lb_pid_1}] ->
+                             Lb_pid_1
+                     end,
+            Max_sessions = get_max_sessions(Host, Port, Options),
+            Max_pipeline_size = get_max_pipeline_size(Host, Port, Options),
+            Options_1 = merge_options(Host, Port, Options),
+            {SSLOptions, IsSSL} =
+                case (Protocol == https) orelse
+                     get_value(is_ssl, Options_1, false) of
+                    false -> {[], false};
+                    true -> {get_value(ssl_options, Options_1, []), true}
+                end,
+            case ibrowse_lb:spawn_connection(Lb_pid, Parsed_url,
+                                             Max_sessions, 
+                                             Max_pipeline_size,
+                                             {SSLOptions, IsSSL}) of
+                {ok, Conn_Pid} ->
+                    do_send_req(Conn_Pid, Parsed_url, Headers,
+                                Method, Body, Options_1, Timeout);
+                Err ->
+                    Err
+            end;
+        Err ->
+            {error, {url_parsing_failed, Err}}
+    end.
+
+merge_options(Host, Port, Options) ->
+    Config_options = get_config_value({options, Host, Port}, []),
+    lists:foldl(
+      fun({Key, Val}, Acc) ->
+                        case lists:keysearch(Key, 1, Options) of
+                            false ->
+                                [{Key, Val} | Acc];
+                            _ ->
+                                Acc
+                        end
+      end, Options, Config_options).
+
+get_lb_pid(Url) ->
+    gen_server:call(?MODULE, {get_lb_pid, Url}).
+
+get_max_sessions(Host, Port, Options) ->
+    get_value(max_sessions, Options,
+              get_config_value({max_sessions, Host, Port}, ?DEF_MAX_SESSIONS)).
+
+get_max_pipeline_size(Host, Port, Options) ->
+    get_value(max_pipeline_size, Options,
+              get_config_value({max_pipeline_size, Host, Port}, ?DEF_MAX_PIPELINE_SIZE)).
+
+%% @doc Deprecated. Use set_max_sessions/3 and set_max_pipeline_size/3
+%% for achieving the same effect.
+set_dest(Host, Port, [{max_sessions, Max} | T]) ->
+    set_max_sessions(Host, Port, Max),
+    set_dest(Host, Port, T);
+set_dest(Host, Port, [{max_pipeline_size, Max} | T]) ->
+    set_max_pipeline_size(Host, Port, Max),
+    set_dest(Host, Port, T);
+set_dest(Host, Port, [{trace, Bool} | T]) when Bool == true; Bool == false ->
+    ibrowse ! {trace, true, Host, Port},
+    set_dest(Host, Port, T);
+set_dest(_Host, _Port, [H | _]) ->
+    exit({invalid_option, H});
+set_dest(_, _, []) ->
+    ok.
+    
+%% @doc Set the maximum number of connections allowed to a specific Host:Port.
+%% @spec set_max_sessions(Host::string(), Port::integer(), Max::integer()) -> ok
+set_max_sessions(Host, Port, Max) when is_integer(Max), Max > 0 ->
+    gen_server:call(?MODULE, {set_config_value, {max_sessions, Host, Port}, Max}).
+
+%% @doc Set the maximum pipeline size for each connection to a specific Host:Port.
+%% @spec set_max_pipeline_size(Host::string(), Port::integer(), Max::integer()) -> ok
+set_max_pipeline_size(Host, Port, Max) when is_integer(Max), Max > 0 ->
+    gen_server:call(?MODULE, {set_config_value, {max_pipeline_size, Host, Port}, Max}).
+
+do_send_req(Conn_Pid, Parsed_url, Headers, Method, Body, Options, Timeout) ->
+    case catch ibrowse_http_client:send_req(Conn_Pid, Parsed_url,
+                                            Headers, Method, ensure_bin(Body),
+                                            Options, Timeout) of
+        {'EXIT', {timeout, _}} ->
+            {error, req_timedout};
+        {'EXIT', Reason} ->
+            {error, {'EXIT', Reason}};
+        {ok, St_code, Headers, Body} = Ret when is_binary(Body) ->
+            case get_value(response_format, Options, list) of
+                list ->
+                    {ok, St_code, Headers, binary_to_list(Body)};
+                binary ->
+                    Ret
+            end;
+        Ret ->
+            Ret
+    end.
+
+ensure_bin(L) when is_list(L)                     -> list_to_binary(L);
+ensure_bin(B) when is_binary(B)                   -> B;
+ensure_bin(Fun) when is_function(Fun)             -> Fun;
+ensure_bin({Fun}) when is_function(Fun)           -> Fun;
+ensure_bin({Fun, _} = Body) when is_function(Fun) -> Body.
+
+%% @doc Creates a HTTP client process to the specified Host:Port which
+%% is not part of the load balancing pool. This is useful in cases
+%% where some requests to a webserver might take a long time whereas
+%% some might take a very short time. To avoid getting these quick
+%% requests stuck in the pipeline behind time consuming requests, use
+%% this function to get a handle to a connection process. <br/>
+%% <b>Note:</b> Calling this function only creates a worker process. No connection
+%% is setup. The connection attempt is made only when the first
+%% request is sent via any of the send_req_direct/4,5,6,7 functions.<br/>
+%% <b>Note:</b> It is the responsibility of the calling process to control
+%% pipeline size on such connections.
+%%
+%% @spec spawn_worker_process(Url::string()) -> {ok, pid()}
+spawn_worker_process(Url) ->
+    ibrowse_http_client:start(Url).
+
+%% @spec spawn_worker_process(Host::string(), Port::integer()) -> {ok, pid()}
+spawn_worker_process(Host, Port) ->
+    ibrowse_http_client:start({Host, Port}).
+
+%% @doc Same as spawn_worker_process/1 except the the calling process
+%% is linked to the worker process which is spawned.
+%% @spec spawn_link_worker_process(Url::string()) -> {ok, pid()}
+spawn_link_worker_process(Url) ->
+    ibrowse_http_client:start_link(Url).
+
+%% @spec spawn_link_worker_process(Host::string(), Port::integer()) -> {ok, pid()}
+spawn_link_worker_process(Host, Port) ->
+    ibrowse_http_client:start_link({Host, Port}).
+
+%% @doc Terminate a worker process spawned using
+%% spawn_worker_process/2 or spawn_link_worker_process/2. Requests in
+%% progress will get the error response <pre>{error, closing_on_request}</pre>
+%% @spec stop_worker_process(Conn_pid::pid()) -> ok
+stop_worker_process(Conn_pid) ->
+    ibrowse_http_client:stop(Conn_pid).
+
+%% @doc Same as send_req/3 except that the first argument is the PID
+%% returned by spawn_worker_process/2 or spawn_link_worker_process/2
+send_req_direct(Conn_pid, Url, Headers, Method) ->
+    send_req_direct(Conn_pid, Url, Headers, Method, [], []).
+
+%% @doc Same as send_req/4 except that the first argument is the PID
+%% returned by spawn_worker_process/2 or spawn_link_worker_process/2
+send_req_direct(Conn_pid, Url, Headers, Method, Body) ->
+    send_req_direct(Conn_pid, Url, Headers, Method, Body, []).
+
+%% @doc Same as send_req/5 except that the first argument is the PID
+%% returned by spawn_worker_process/2 or spawn_link_worker_process/2
+send_req_direct(Conn_pid, Url, Headers, Method, Body, Options) ->
+    send_req_direct(Conn_pid, Url, Headers, Method, Body, Options, 30000).
+
+%% @doc Same as send_req/6 except that the first argument is the PID
+%% returned by spawn_worker_process/2 or spawn_link_worker_process/2
+send_req_direct(Conn_pid, Url, Headers, Method, Body, Options, Timeout) ->
+    case catch parse_url(Url) of
+        #url{host = Host,
+             port = Port} = Parsed_url ->
+            Options_1 = merge_options(Host, Port, Options),
+            case do_send_req(Conn_pid, Parsed_url, Headers, Method, Body, Options_1, Timeout) of
+                {error, {'EXIT', {noproc, _}}} ->
+                    {error, worker_is_dead};
+                Ret ->
+                    Ret
+            end;
+        Err ->
+            {error, {url_parsing_failed, Err}}
+    end.
+
+%% @doc Tell ibrowse to stream the next chunk of data to the
+%% caller. Should be used in conjunction with the
+%% <code>stream_to</code> option
+%% @spec stream_next(Req_id :: req_id()) -> ok | {error, unknown_req_id}
+stream_next(Req_id) ->    
+    case ets:lookup(ibrowse_stream, {req_id_pid, Req_id}) of
+        [] ->
+            {error, unknown_req_id};
+        [{_, Pid}] ->
+            catch Pid ! {stream_next, Req_id},
+            ok
+    end.
+
+%% @doc Turn tracing on for the ibrowse process
+trace_on() ->
+    ibrowse ! {trace, true}.
+%% @doc Turn tracing off for the ibrowse process
+trace_off() ->
+    ibrowse ! {trace, false}.
+
+%% @doc Turn tracing on for all connections to the specified HTTP
+%% server. Host is whatever is specified as the domain name in the URL
+%% @spec trace_on(Host, Port) -> ok
+%% Host = string() 
+%% Port = integer()
+trace_on(Host, Port) ->
+    ibrowse ! {trace, true, Host, Port},
+    ok.
+
+%% @doc Turn tracing OFF for all connections to the specified HTTP
+%% server.
+%% @spec trace_off(Host, Port) -> ok
+trace_off(Host, Port) ->
+    ibrowse ! {trace, false, Host, Port},
+    ok.
+
+%% @doc Turn Off ALL tracing
+%% @spec all_trace_off() -> ok
+all_trace_off() ->
+    ibrowse ! all_trace_off,
+    ok.
+
+show_dest_status() ->
+    Dests = lists:filter(fun({lb_pid, {Host, Port}, _}) when is_list(Host),
+                                                             is_integer(Port) ->
+                                 true;
+                            (_) ->
+                                 false
+                         end, ets:tab2list(ibrowse_lb)),
+    All_ets = ets:all(),
+    io:format("~-40.40s | ~-5.5s | ~-10.10s | ~s~n",
+              ["Server:port", "ETS", "Num conns", "LB Pid"]),
+    io:format("~80.80.=s~n", [""]),
+    lists:foreach(fun({lb_pid, {Host, Port}, Lb_pid}) ->
+                          case lists:dropwhile(
+                                 fun(Tid) ->
+                                         ets:info(Tid, owner) /= Lb_pid
+                                 end, All_ets) of
+                              [] ->
+                                  io:format("~40.40s | ~-5.5s | ~-5.5s | ~s~n",
+                                            [Host ++ ":" ++ integer_to_list(Port),
+                                             "",
+                                             "",
+                                             io_lib:format("~p", [Lb_pid])]
+                                           );
+                              [Tid | _] ->
+                                  catch (
+                                    begin
+                                        Size = ets:info(Tid, size),
+                                        io:format("~40.40s | ~-5.5s | ~-5.5s | ~s~n",
+                                                  [Host ++ ":" ++ integer_to_list(Port),
+                                                   io_lib:format("~p", [Tid]),
+                                                   integer_to_list(Size),
+                                                   io_lib:format("~p", [Lb_pid])]
+                                                 )
+                                    end
+                                   )
+                                  end
+                  end, Dests).
+                                          
+%% @doc Shows some internal information about load balancing to a
+%% specified Host:Port. Info about workers spawned using
+%% spawn_worker_process/2 or spawn_link_worker_process/2 is not
+%% included.
+show_dest_status(Host, Port) ->
+    case ets:lookup(ibrowse_lb, {Host, Port}) of
+        [] ->
+            no_active_processes;
+        [#lb_pid{pid = Lb_pid}] ->
+            io:format("Load Balancer Pid     : ~p~n", [Lb_pid]),
+            io:format("LB process msg q size : ~p~n", [(catch process_info(Lb_pid, message_queue_len))]),
+            case lists:dropwhile(
+                   fun(Tid) ->
+                           ets:info(Tid, owner) /= Lb_pid
+                   end, ets:all()) of
+                [] ->
+                    io:format("Couldn't locate ETS table for ~p~n", [Lb_pid]);
+                [Tid | _] ->
+                    First = ets:first(Tid),
+                    Last = ets:last(Tid),
+                    Size = ets:info(Tid, size),
+                    io:format("LB ETS table id       : ~p~n", [Tid]),
+                    io:format("Num Connections       : ~p~n", [Size]),
+                    case Size of
+                        0 ->
+                            ok;
+                        _ ->
+                            {First_p_sz, _} = First,
+                            {Last_p_sz, _} = Last,
+                            io:format("Smallest pipeline     : ~1000.p~n", [First_p_sz]),
+                            io:format("Largest pipeline      : ~1000.p~n", [Last_p_sz])
+                    end
+            end
+    end.
+
+%% @doc Clear current configuration for ibrowse and load from the file
+%% ibrowse.conf in the IBROWSE_EBIN/../priv directory. Current
+%% configuration is cleared only if the ibrowse.conf file is readable
+%% using file:consult/1
+rescan_config() ->
+    gen_server:call(?MODULE, rescan_config).
+
+%% Clear current configuration for ibrowse and load from the specified
+%% file. Current configuration is cleared only if the specified
+%% file is readable using file:consult/1
+rescan_config(File) when is_list(File) ->
+    gen_server:call(?MODULE, {rescan_config, File}).
+
+%%====================================================================
+%% Server functions
+%%====================================================================
+
+%%--------------------------------------------------------------------
+%% Function: init/1
+%% Description: Initiates the server
+%% Returns: {ok, State}          |
+%%          {ok, State, Timeout} |
+%%          ignore               |
+%%          {stop, Reason}
+%%--------------------------------------------------------------------
+init(_) ->
+    process_flag(trap_exit, true),
+    State = #state{},
+    put(my_trace_flag, State#state.trace),
+    put(ibrowse_trace_token, "ibrowse"),
+    ets:new(ibrowse_lb, [named_table, public, {keypos, 2}]),
+    ets:new(ibrowse_conf, [named_table, protected, {keypos, 2}]),
+    ets:new(ibrowse_stream, [named_table, public]),
+    import_config(),
+    {ok, #state{}}.
+
+import_config() ->
+    case code:priv_dir(ibrowse) of
+        {error, _} = Err ->
+            Err;
+        PrivDir ->
+            Filename = filename:join(PrivDir, "ibrowse.conf"),
+            import_config(Filename)
+    end.
+
+import_config(Filename) ->
+    case file:consult(Filename) of
+        {ok, Terms} ->
+            ets:delete_all_objects(ibrowse_conf),
+            Fun = fun({dest, Host, Port, MaxSess, MaxPipe, Options}) 
+                     when is_list(Host), is_integer(Port),
+                          is_integer(MaxSess), MaxSess > 0,
+                          is_integer(MaxPipe), MaxPipe > 0, is_list(Options) ->
+                          I = [{{max_sessions, Host, Port}, MaxSess},
+                               {{max_pipeline_size, Host, Port}, MaxPipe},
+                               {{options, Host, Port}, Options}],
+                          lists:foreach(
+                            fun({X, Y}) ->
+                                    ets:insert(ibrowse_conf,
+                                               #ibrowse_conf{key = X, 
+                                                             value = Y})
+                            end, I);
+                     ({K, V}) ->
+                          ets:insert(ibrowse_conf,
+                                     #ibrowse_conf{key = K,
+                                                   value = V});
+                     (X) ->
+                          io:format("Skipping unrecognised term: ~p~n", [X])
+                  end,
+            lists:foreach(Fun, Terms);
+        Err ->
+            Err
+    end.
+
+%% @doc Internal export
+get_config_value(Key) ->
+    [#ibrowse_conf{value = V}] = ets:lookup(ibrowse_conf, Key),
+    V.
+
+%% @doc Internal export
+get_config_value(Key, DefVal) ->
+    case ets:lookup(ibrowse_conf, Key) of
+        [] ->
+            DefVal;
+        [#ibrowse_conf{value = V}] ->
+            V
+    end.
+
+set_config_value(Key, Val) ->
+    ets:insert(ibrowse_conf, #ibrowse_conf{key = Key, value = Val}).
+%%--------------------------------------------------------------------
+%% Function: handle_call/3
+%% Description: Handling call messages
+%% Returns: {reply, Reply, State}          |
+%%          {reply, Reply, State, Timeout} |
+%%          {noreply, State}               |
+%%          {noreply, State, Timeout}      |
+%%          {stop, Reason, Reply, State}   | (terminate/2 is called)
+%%          {stop, Reason, State}            (terminate/2 is called)
+%%--------------------------------------------------------------------
+handle_call({get_lb_pid, #url{host = Host, port = Port} = Url}, _From, State) ->
+    Pid = do_get_connection(Url, ets:lookup(ibrowse_lb, {Host, Port})),
+    {reply, Pid, State};
+
+handle_call(stop, _From, State) ->
+    do_trace("IBROWSE shutting down~n", []),
+    {stop, normal, ok, State};
+
+handle_call({set_config_value, Key, Val}, _From, State) ->
+    set_config_value(Key, Val),
+    {reply, ok, State};
+
+handle_call(rescan_config, _From, State) ->
+    Ret = (catch import_config()),
+    {reply, Ret, State};
+
+handle_call({rescan_config, File}, _From, State) ->
+    Ret = (catch import_config(File)),
+    {reply, Ret, State};
+
+handle_call(Request, _From, State) ->
+    Reply = {unknown_request, Request},
+    {reply, Reply, State}.
+
+%%--------------------------------------------------------------------
+%% Function: handle_cast/2
+%% Description: Handling cast messages
+%% Returns: {noreply, State}          |
+%%          {noreply, State, Timeout} |
+%%          {stop, Reason, State}            (terminate/2 is called)
+%%--------------------------------------------------------------------
+
+handle_cast(_Msg, State) ->
+    {noreply, State}.
+
+%%--------------------------------------------------------------------
+%% Function: handle_info/2
+%% Description: Handling all non call/cast messages
+%% Returns: {noreply, State}          |
+%%          {noreply, State, Timeout} |
+%%          {stop, Reason, State}            (terminate/2 is called)
+%%--------------------------------------------------------------------
+handle_info(all_trace_off, State) ->
+    Mspec = [{{ibrowse_conf,{trace,'$1','$2'},true},[],[{{'$1','$2'}}]}],
+    Trace_on_dests = ets:select(ibrowse_conf, Mspec),
+    Fun = fun(#lb_pid{host_port = {H, P}, pid = Pid}, _) ->
+                  case lists:member({H, P}, Trace_on_dests) of
+                      false ->
+                          ok;
+                      true ->
+                          catch Pid ! {trace, false}
+                  end;
+             (_, Acc) ->
+                  Acc
+          end,
+    ets:foldl(Fun, undefined, ibrowse_lb),
+    ets:select_delete(ibrowse_conf, [{{ibrowse_conf,{trace,'$1','$2'},true},[],['true']}]),
+    {noreply, State};
+                                  
+handle_info({trace, Bool}, State) ->
+    put(my_trace_flag, Bool),
+    {noreply, State};
+
+handle_info({trace, Bool, Host, Port}, State) ->
+    Fun = fun(#lb_pid{host_port = {H, P}, pid = Pid}, _)
+             when H == Host,
+                  P == Port ->
+                  catch Pid ! {trace, Bool};
+             (_, Acc) ->
+                  Acc
+          end,
+    ets:foldl(Fun, undefined, ibrowse_lb),
+    ets:insert(ibrowse_conf, #ibrowse_conf{key = {trace, Host, Port},
+                                           value = Bool}),
+    {noreply, State};
+                     
+handle_info(_Info, State) ->
+    {noreply, State}.
+
+%%--------------------------------------------------------------------
+%% Function: terminate/2
+%% Description: Shutdown the server
+%% Returns: any (ignored by gen_server)
+%%--------------------------------------------------------------------
+terminate(_Reason, _State) ->
+    ok.
+
+%%--------------------------------------------------------------------
+%% Func: code_change/3
+%% Purpose: Convert process state when code is changed
+%% Returns: {ok, NewState}
+%%--------------------------------------------------------------------
+code_change(_OldVsn, State, _Extra) ->
+    {ok, State}.
+
+%%--------------------------------------------------------------------
+%%% Internal functions
+%%--------------------------------------------------------------------
+do_get_connection(#url{host = Host, port = Port}, []) ->
+    {ok, Pid} = ibrowse_lb:start_link([Host, Port]),
+    ets:insert(ibrowse_lb, #lb_pid{host_port = {Host, Port}, pid = Pid}),
+    Pid;
+do_get_connection(_Url, [#lb_pid{pid = Pid}]) ->
+    Pid.

lib/ibrowse/src/ibrowse.hrl

+-ifndef(IBROWSE_HRL).
+-define(IBROWSE_HRL, "ibrowse.hrl").
+
+-record(url, {abspath, host, port, username, password, path, protocol}).
+
+-record(lb_pid, {host_port, pid}).
+
+-record(client_conn, {key, cur_pipeline_size = 0, reqs_served = 0}).
+
+-record(ibrowse_conf, {key, value}).
+
+-endif.

lib/ibrowse/src/ibrowse_app.erl

+%%%-------------------------------------------------------------------
+%%% File    : ibrowse_app.erl
+%%% Author  : Chandrashekhar Mullaparthi <chandrashekhar.mullaparthi@t-mobile.co.uk>
+%%% Description : 
+%%%
+%%% Created : 15 Oct 2003 by Chandrashekhar Mullaparthi <chandrashekhar.mullaparthi@t-mobile.co.uk>
+%%%-------------------------------------------------------------------
+-module(ibrowse_app).
+
+-behaviour(application).
+%%--------------------------------------------------------------------
+%% Include files
+%%--------------------------------------------------------------------
+
+%%--------------------------------------------------------------------
+%% External exports
+%%--------------------------------------------------------------------
+-export([
+	 start/2,
+	 stop/1
+        ]).
+
+%%--------------------------------------------------------------------
+%% Internal exports
+%%--------------------------------------------------------------------
+-export([
+        ]).
+
+%%--------------------------------------------------------------------
+%% Macros
+%%--------------------------------------------------------------------
+
+%%--------------------------------------------------------------------
+%% Records
+%%--------------------------------------------------------------------
+
+%%====================================================================
+%% External functions
+%%====================================================================
+%%--------------------------------------------------------------------
+%% Func: start/2
+%% Returns: {ok, Pid}        |
+%%          {ok, Pid, State} |
+%%          {error, Reason}   
+%%--------------------------------------------------------------------
+start(_Type, _StartArgs)