diff options
Diffstat (limited to 'netwerk/docs/http_server_for_testing.rst')
| -rw-r--r-- | netwerk/docs/http_server_for_testing.rst | 482 |
1 files changed, 482 insertions, 0 deletions
diff --git a/netwerk/docs/http_server_for_testing.rst b/netwerk/docs/http_server_for_testing.rst new file mode 100644 index 0000000000..dbf6ce7520 --- /dev/null +++ b/netwerk/docs/http_server_for_testing.rst @@ -0,0 +1,482 @@ +HTTP server for unit tests +========================== + +This page describes the JavaScript implementation of an +HTTP server located in ``netwerk/test/httpserver/``. + +Server functionality +~~~~~~~~~~~~~~~~~~~~ + +Here are some of the things you can do with the server: + +- map a directory of files onto an HTTP path on the server, for an + arbitrary number of such directories (including nested directories) +- define custom error handlers for HTTP error codes +- serve a given file for requests for a specific path, optionally with + custom headers and status +- define custom "CGI" handlers for specific paths using a + JavaScript-based API to create the response (headers and actual + content) +- run multiple servers at once on different ports (8080, 8081, 8082, + and so on.) + +This functionality should be more than enough for you to use it with any +test which requires HTTP-provided behavior. + +Where you can use it +~~~~~~~~~~~~~~~~~~~~ + +The server is written primarily for use from ``xpcshell``-based +tests, and it can be used as an inline script or as an XPCOM component. The +Mochitest framework also uses it to serve its tests, and +`reftests <https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt>`__ +can optionally use it when their behavior is dependent upon specific +HTTP header values. + +Ways you might use it +~~~~~~~~~~~~~~~~~~~~~ + +- application update testing +- cross-"server" security tests +- cross-domain security tests, in combination with the right proxy + settings (for example, using `Proxy + AutoConfig <https://en.wikipedia.org/wiki/Proxy_auto-config>`__) +- tests where the behavior is dependent on the values of HTTP headers + (for example, Content-Type) +- anything which requires use of files not stored locally +- open-id : the users could provide their own open id server (they only + need it when they're using their browser) +- micro-blogging : users could host their own micro blog based on + standards like RSS/Atom +- rest APIs : web application could interact with REST or SOAP APIs for + many purposes like : file/data storage, social sharing and so on +- download testing + +Using the server +~~~~~~~~~~~~~~~~ + +The best and first place you should look for documentation is +``netwerk/test/httpserver/nsIHttpServer.idl``. It's extremely +comprehensive and detailed, and it should be enough to figure out how to +make the server do what you want. I also suggest taking a look at the +less-comprehensive server +`README <https://searchfox.org/mozilla-central/source/netwerk/test/httpserver/README>`__, +although the IDL should usually be sufficient. + +Running the server +^^^^^^^^^^^^^^^^^^ + +From test suites, the server should be importable as a testing-only JS +module: + +.. code:: javascript + + ChromeUtils.import("resource://testing-common/httpd.js"); + +Once you've done that, you can create a new server as follows: + +.. code:: javascript + + let server = new HttpServer(); // Or nsHttpServer() if you don't use ChromeUtils.import. + + server.registerDirectory("/", nsILocalFileForBasePath); + + server.start(-1); // uses a random available port, allows us to run tests concurrently + const SERVER_PORT = server.identity.primaryPort; // you can use this further on + + // and when the tests are done, most likely from a callback... + server.stop(function() { /* continue execution here */ }); + +You can also pass in a numeric port argument to the ``start()`` method, +but we strongly suggest you don't do it. Using a dynamic port allow us +to run your test in parallel with other tests which reduces wait times +and makes everybody happy. If you really have to use a hardcoded port, +you will have to annotate your test in the xpcshell manifest file with +``run-sequentially = REASON``. +However, this should only be used as the last possible option. + +.. note:: + + Note: You **must** make sure to stop the server (the last line above) + before your test completes. Failure to do so will result in the + "XPConnect is being called on a scope without a Components property" + assertion, which will cause your test to fail in debug builds, and + you'll make people running tests grumbly because you've broken the + tests. + +Debugging errors +^^^^^^^^^^^^^^^^ + +The server's default error pages don't give much information, partly +because the error-dispatch mechanism doesn't currently accommodate doing +so and partly because exposing errors in a real server could make it +easier to exploit them. If you don't know why the server is acting a +particular way, edit +`httpd.js <https://searchfox.org/mozilla-central/source/netwerk/test/httpserver/httpd.js>`__ +and change the value of ``DEBUG`` to ``true``. This will cause the +server to print information about the processing of requests (and errors +encountered doing so) to the console, and it's usually not difficult to +determine why problems exist from that output. ``DEBUG`` is ``false`` by +default because the information printed with it set to ``true`` +unnecessarily obscures tinderbox output. + +Header modification for files +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The server supports modifying the headers of the files (not request +handlers) it serves. To modify the headers for a file, create a sibling +file with the first file's name followed by ``^headers^``. Here's an +example of how such a file might look: + +.. code:: + + HTTP 404 I want a cool HTTP description! + Content-Type: text/plain + +The status line is optional; all other lines specify HTTP headers in the +standard HTTP format. Any line ending style is accepted, and the file +may optionally end with a single newline character, to play nice with +Unix text tools like ``diff`` and ``hg``. + +Hidden files +^^^^^^^^^^^^ + +Any file which ends with a single ``^`` is inaccessible when querying +the web server; if you try to access such a file you'll get a +``404 File Not Found`` page instead. If for some reason you need to +serve a file ending with a ``^``, just tack another ``^`` onto the end +of the file name and the file will then become available at the +single-``^`` location. + +At the moment this feature is basically a way to smuggle header +modification for files into the file system without making those files +accessible to clients; it remains to be seen whether and how hidden-file +capabilities will otherwise be used. + +SJS: server-side scripts +^^^^^^^^^^^^^^^^^^^^^^^^ + +Support for server-side scripts is provided through the SJS mechanism. +Essentially an SJS is a file with a particular extension, chosen by the +creator of the server, which contains a function with the name +``handleRequest`` which is called to determine the response the server +will generate. That function acts exactly like the ``handle`` function +on the ``nsIHttpRequestHandler`` interface. First, tell the server what +extension you're using: + +.. code:: javascript + + const SJS_EXTENSION = "cgi"; + server.registerContentType(SJS_EXTENSION, "sjs"); + +Now just create an SJS with the extension ``cgi`` and write whatever you +want. For example: + +.. code:: javascript + + function handleRequest(request, response) + { + response.setStatusLine(request.httpVersion, 200, "OK"); + response.write("Hello world! This request was dynamically " + + "generated at " + new Date().toUTCString()); + } + +Further examples may be found `in the Mozilla source +tree <https://searchfox.org/mozilla-central/search?q=&path=.sjs>`__ +in existing tests. The request object is an instance of +``nsIHttpRequest`` and the response is a ``nsIHttpResponse``. +Please refer to the `IDL +documentation <https://searchfox.org/mozilla-central/source/netwerk/test/httpserver/nsIHttpServer.idl>` +for more details. + +Storing information across requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +HTTP is basically a stateless protocol, and the httpd.js server API is +for the most part similarly stateless. If you're using the server +through the XPCOM interface you can simply store whatever state you want +in enclosing environments or global variables. However, if you're using +it through an SJS your request is processed in a near-empty environment +every time processing occurs. To support stateful SJS behavior, the +following functions have been added to the global scope in which a SJS +handler executes, providing a simple key-value state storage mechanism: + +.. code:: + + /* + * v : T means v is of type T + * function A() : T means A() has type T + */ + + function getState(key : string) : string + function setState(key : string, value : string) + function getSharedState(key : string) : string + function setSharedState(key : string, value : string) + function getObjectState(key : string, callback : function(value : object) : void) // SJS API, XPCOM differs, see below + function setObjectState(key : string, value : object) + +A key is a string with arbitrary contents. The corresponding value is +also a string, for the non-object-saving functions. For the +object-saving functions, it is (wait for it) an object, or also +``null``. Initially all keys are associated with the empty string or +with ``null``, depending on whether the function accesses string- or +object-valued storage. A stored value persists across requests and +across server shutdowns and restarts. The state methods are available +both in SJS and, for convenience when working with the server both via +XPCOM and via SJS, XPCOM through the ``nsIHttpServer`` interface. The +variants are designed to support different needs. + +.. warning:: + + **Warning:** Be careful using state: you, the user, are responsible + for synchronizing all uses of state through any of the available + methods. (This includes the methods that act only on per-path state: + you might still run into trouble there if your request handler + generates responses asynchronously. Further, any code with access to + the server XPCOM component could modify it between requests even if + you only ever used or modified that state while generating + synchronous responses.) JavaScript's run-to-completion behavior will + save you in simple cases, but with anything moderately complex you + are playing with fire, and if you do it wrong you will get burned. + +``getState`` and ``setState`` +''''''''''''''''''''''''''''' + +``getState`` and ``setState`` are designed for the case where a single +request handler needs to store information from a first request of it +for use in processing a second request of it — say, for example, if you +wanted to implement a request handler implementing a counter: + +.. code:: javascript + + /** + * Generates a response whose body is "0", "1", "2", and so on. each time a + * request is made. (Note that browser caching might make it appear + * to not quite have that behavior; a Cache-Control header would fix + * that issue if desired.) + */ + function handleRequest(request, response) + { + var counter = +getState("counter"); // convert to number; +"" === 0 + response.write("" + counter); + setState("counter", "" + ++counter); + } + +The useful feature of these two methods is that this state doesn't bleed +outside the single path at which it resides. For example, if the above +SJS were at ``/counter``, the value returned by ``getState("counter")`` +at some other path would be completely distinct from the counter +implemented above. This makes it much simpler to write stateful handlers +without state accidentally bleeding between unrelated handlers. + +.. note:: + + **Note:** State saved by this method is specific to the HTTP path, + excluding query string and hash reference. ``/counter``, + ``/counter?foo``, and ``/counter?bar#baz`` all share the same state + for the purposes of these methods. (Indeed, non-shared state would be + significantly less useful if it changed when the query string + changed!) + +.. note:: + + **Note:** The predefined ``__LOCATION__`` state + contains the native path of the SJS file itself. You can pass the + result directly to the ``nsILocalFile.initWithPath()``. Example: + ``thisSJSfile.initWithPath(getState('__LOCATION__'));`` + +``getSharedState`` and ``setSharedState`` +''''''''''''''''''''''''''''''''''''''''' + +``getSharedState`` and ``setSharedState`` make up the functionality +intentionally not supported by ``getState`` and set\ ``State``: state +that exists between different paths. If you used the above handler at +the paths ``/sharedCounters/1`` and ``/sharedCounters/2`` (changing the +state-calls to use shared state, of course), the first load of either +handler would return "0", a second load of either handler would return +"1", a third load either handler would return "2", and so on. This more +powerful functionality allows you to write cooperative handlers that +expose and manipulate a piece of shared state. Be careful! One test can +screw up another test pretty easily if it's not careful what it does +with this functionality. + +``getObjectState`` and ``setObjectState`` +''''''''''''''''''''''''''''''''''''''''' + +``getObjectState`` and ``setObjectState`` support the remaining +functionality not provided by the above methods: storing non-string +values (object values or ``null``). These two methods are the same as +``getSharedState`` and ``setSharedState``\ in that state is visible +across paths; ``setObjectState`` in one handler will expose that value +in another handler that uses ``getObjectState`` with the same key. (This +choice was intentional, because object values already expose mutable +state that you have to be careful about using.) This functionality is +particularly useful for cooperative request handlers where one request +*suspends* another, and that second request must then be *resumed* at a +later time by a third request. Without object-valued storage you'd need +to resort to polling on a string value using either of the previous +state APIs; with this, however, you can make precise callbacks exactly +when a particular event occurs. + +``getObjectState`` in an SJS differs in one important way from +``getObjectState`` accessed via XPCOM. In XPCOM the method takes a +single string argument and returns the object or ``null`` directly. In +SJS, however, the process to return the value is slightly different: + +.. code:: javascript + + function handleRequest(request, response) + { + var key = request.hasHeader("key") + ? request.getHeader("key") + : "unspecified"; + var obj = null; + getObjectState(key, function(objval) + { + // This function is called synchronously with the object value + // associated with key. + obj = objval; + }); + response.write("Keyed object " + + (obj && Object.prototype.hasOwnProperty.call(obj, "doStuff") + ? "has " + : "does not have ") + + "a doStuff method."); + } + +This idiosyncratic API is a restriction imposed by how sandboxes +currently work: external functions added to the sandbox can't return +object values when called within the sandbox. However, such functions +can accept and call callback functions, so we simply use a callback +function here to return the object value associated with the key. + +Advanced dynamic response creation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default behavior of request handlers is to fully construct the +response, return, and only then send the generated data. For certain use +cases, however, this is infeasible. For example, a handler which wanted +to return an extremely large amount of data (say, over 4GB on a 32-bit +system) might run out of memory doing so. Alternatively, precise control +over the timing of data transmission might be required so that, for +example, one request is received, "paused" while another request is +received and completes, and then finished. httpd.js solves this problem +by defining a ``processAsync()`` method which indicates to the server +that the response will be written and finished by the handler. Here's an +example of an SJS file which writes some data, waits five seconds, and +then writes some more data and finishes the response: + +.. code:: javascript + + var timer = null; + + function handleRequest(request, response) + { + response.processAsync(); + response.setHeader("Content-Type", "text/plain", false); + response.write("hello..."); + + timer = Cc["@mozilla.org/timer;1"].createInstance(Ci.nsITimer); + timer.initWithCallback(function() + { + response.write("world!"); + response.finish(); + }, 5 * 1000 /* milliseconds */, Ci.nsITimer.TYPE_ONE_SHOT); + } + +The basic flow is simple: call ``processAsync`` to mark the response as +being sent asynchronously, write data to the response body as desired, +and when complete call ``finish()``. At the moment if you drop such a +response on the floor, nothing will ever terminate the connection, and +the server cannot be stopped (the stop API is asynchronous and +callback-based); in the future a default connection timeout will likely +apply, but for now, "don't do that". + +Full documentation for ``processAsync()`` and its interactions with +other methods may, as always, be found in +``netwerk/test/httpserver/nsIHttpServer.idl``. + +Manual, arbitrary response creation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The standard mode of response creation is fully synchronous and is +guaranteed to produce syntactically correct responses (excluding +headers, which for the most part may be set to arbitrary values). +Asynchronous processing enables the introduction of response handling +coordinated with external events, but again, for the most part only +syntactically correct responses may be generated. The third method of +processing removes the correct-syntax property by allowing a response to +contain completely arbitrary data through the ``seizePower()`` method. +After this method is called, any data subsequently written to the +response is written directly to the network as the response, skipping +headers and making no attempt whatsoever to ensure any formatting of the +transmitted data. As with asynchronous processing, the response is +generated asynchronously and must be finished manually for the +connection to be closed. (Again, nothing will terminate the connection +for a response dropped on the floor, so again, "don't do that".) This +mode of processing is useful for testing particular data formats that +are either not HTTP or which do not match the precise, canonical +representation that httpd.js generates. Here's an example of an SJS file +which writes an apparent HTTP response whose status text contains a null +byte (not allowed by HTTP/1.1, and attempting to set such status text +through httpd.js would throw an exception) and which has a header that +spans multiple lines (httpd.js responses otherwise generate only +single-line headers): + +.. code:: javascript + + function handleRequest(request, response) + { + response.seizePower(); + response.write("HTTP/1.1 200 OK Null byte \u0000 makes this response malformed\r\n" + + "X-Underpants-Gnomes-Strategy:\r\n" + + " Phase 1: Collect underpants.\r\n" + + " Phase 2: ...\r\n" + + " Phase 3: Profit!\r\n" + + "\r\n" + + "FAIL"); + response.finish(); + } + +While the asynchronous mode is capable of producing certain forms of +invalid responses (through setting a bogus Content-Length header prior +to the start of body transmission, among others), it must not be used in +this manner. No effort will be made to preserve such implementation +quirks (indeed, some are even likely to be removed over time): if you +want to send malformed data, use ``seizePower()`` instead. + +Full documentation for ``seizePower()`` and its interactions with other +methods may, as always, be found in +``netwerk/test/httpserver/nsIHttpServer.idl``. + +Example uses of the server +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Shorter examples (for tests which only do one test): + +- ``netwerk/test/unit/test_bug331825.js`` +- ``netwerk/test/unit/test_httpcancel.js`` +- ``netwerk/test/unit/test_cookie_header.js`` + +Longer tests (where you'd need to do multiple async server requests): + +- ``netwerk/test/httpserver/test/test_setstatusline.js`` +- ``netwerk/test/unit/test_content_sniffer.js`` +- ``netwerk/test/unit/test_authentication.js`` +- ``netwerk/test/unit/test_event_sink.js`` +- ``netwerk/test/httpserver/test/`` + +Examples of modifying HTTP headers in files may be found at +``netwerk/test/httpserver/test/data/cern_meta/``. + +Future directions +~~~~~~~~~~~~~~~~~ + +The server, while very functional, is not yet complete. There are a +number of things to fix and features to add, among them support for +pipelining, support for incrementally-received requests (rather than +buffering the entire body before invoking a request handler), and better +conformance to the MUSTs and SHOULDs of HTTP/1.1. If you have +suggestions for functionality or find bugs, file them in +`Testing-httpd.js <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=General>`__ +. |