path: root/modules/http/README.rst

.. _mod-http:

HTTP/2 services
---------------

This is a module that does the heavy lifting to provide an HTTP/2 enabled
server that provides endpoint for other modules
in order to enable them to export restful APIs and websocket streams.
One example is statistics module that can stream live metrics on the website,
or publish metrics on request for Prometheus scraper.

The server allows other modules to either use default endpoint that provides
built-in webpage, restful APIs and websocket streams, or create new endpoints.

By default the server provides plain HTTP and TLS on the same port. See below
if you want to use only one of these.

.. warning:: This module provides access to various API endpoints
             and must not be directly exposed to untrusted parties.
             Use `reverse-proxy`_ like Apache_ or Nginx_ if you need to
             authenticate API clients.

Example configuration
^^^^^^^^^^^^^^^^^^^^^

By default, the web interface starts HTTPS/2 on port 8053 using an ephemeral
certificate that is valid for 90 days and is automatically renewed. It is of
course self-signed. Why not use something like
`Let's Encrypt <https://letsencrypt.org>`_?

.. code-block:: lua

	-- Load HTTP module with defaults
	modules = {
		http = {
			host = 'localhost', -- Default: 'localhost'
			port = 8053,        -- Default: 8053
			geoip = 'GeoLite2-City.mmdb' -- Optional, see
			-- e.g. https://dev.maxmind.com/geoip/geoip2/geolite2/
			-- and install mmdblua library
			endpoints = {},
		}
	}

Now you can reach the web services and APIs, done!

.. code-block:: bash

	$ curl -k https://localhost:8053
	$ curl -k https://localhost:8053/stats


Configuring TLS
^^^^^^^^^^^^^^^
You can disable unecrypted HTTP and enforce HTTPS by passing
``tls = true`` option.

.. code-block:: lua

	http = {
		tls = true,
	}

If you want to provide your own certificate and key, you're welcome to do so:

.. code-block:: lua

	http = {
		cert = 'mycert.crt',
		key  = 'mykey.key',
	}

The format of both certificate and key is expected to be PEM, e.g. equivalent to
the outputs of following:

.. code-block:: bash

	openssl ecparam -genkey -name prime256v1 -out mykey.key
	openssl req -new -key mykey.key -out csr.pem
	openssl req -x509 -days 90 -key mykey.key -in csr.pem -out mycert.crt

It is also possible to disable HTTPS altogether by passing ``tls = false`` option.
Plain HTTP gets handy if you want to use `reverse-proxy`_ like Apache_ or Nginx_
for authentication to API etc.
(Unencrypted HTTP could be fine for localhost tests as, for example,
Safari doesn't allow WebSockets over HTTPS with a self-signed certificate.
Major drawback is that current browsers won't do HTTP/2 over insecure connection.)


Built-in services
^^^^^^^^^^^^^^^^^

The HTTP module has several built-in services to use.

.. csv-table::
 :header: "Endpoint", "Service", "Description"

 "``/stats``", "Statistics/metrics", "Exported metrics in JSON."
 "``/metrics``", "Prometheus metrics", "Exported metrics for Prometheus_"
 "``/trace/:name/:type``", "Tracking", "Trace resolution of the query and return the verbose logs."

Enabling Prometheus metrics endpoint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The module exposes ``/metrics`` endpoint that serves internal metrics in Prometheus_ text format.
You can use it out of the box:

.. code-block:: bash

	$ curl -k https://localhost:8053/metrics | tail
	# TYPE latency histogram
	latency_bucket{le=10} 2.000000
	latency_bucket{le=50} 2.000000
	latency_bucket{le=100} 2.000000
	latency_bucket{le=250} 2.000000
	latency_bucket{le=500} 2.000000
	latency_bucket{le=1000} 2.000000
	latency_bucket{le=1500} 2.000000
	latency_bucket{le=+Inf} 2.000000
	latency_count 2.000000
	latency_sum 11.000000

You can namespace the metrics in configuration, using `http.prometheus.namespace` attribute:

.. code-block:: lua

	http = {
		host = 'localhost',
	}

	-- Set Prometheus namespace
	http.prometheus.namespace = 'resolver_'

You can also add custom metrics or rewrite existing metrics before they are returned to Prometheus client.

.. code-block:: lua

	http = {
		host = 'localhost',
	}

	-- Add an arbitrary metric to Prometheus
	http.prometheus.finalize = function (metrics)
		table.insert(metrics, 'build_info{version="1.2.3"} 1')
	end

Tracing requests
^^^^^^^^^^^^^^^^

With the ``/trace`` endpoint you can trace various aspects of the request execution.
The basic mode allows you to resolve a query and trace verbose logs (and messages received):

.. code-block:: bash

   $ curl https://localhost:8053/trace/e.root-servers.net
   [ 8138] [iter] 'e.root-servers.net.' type 'A' created outbound query, parent id 0
   [ 8138] [ rc ] => rank: 020, lowest 020, e.root-servers.net. A
   [ 8138] [ rc ] => satisfied from cache
   [ 8138] [iter] <= answer received:
   ;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 8138
   ;; Flags: qr aa  QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 0

   ;; QUESTION SECTION
   e.root-servers.net.		A

   ;; ANSWER SECTION
   e.root-servers.net. 	3556353	A	192.203.230.10

   [ 8138] [iter] <= rcode: NOERROR
   [ 8138] [resl] finished: 4, queries: 1, mempool: 81952 B

How to expose services over HTTP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The module provides a table ``endpoints`` of already existing endpoints, it is free for reading and
writing. It contains tables describing a triplet - ``{mime, on_serve, on_websocket}``.
In order to register a new service, simply add it to the table:

.. code-block:: lua

	local on_health = {'application/json',
	function (h, stream)
		-- API call, return a JSON table
		return {state = 'up', uptime = 0}
	end,
	function (h, ws)
		-- Stream current status every second
		local ok = true
		while ok do
			local push = tojson('up')
			ok = ws:send(tojson({'up'}))
			require('cqueues').sleep(1)
		end
		-- Finalize the WebSocket
		ws:close()
	end}
	-- Load module
	modules = {
		http = {
			endpoints = { ['/health'] = on_health }
		}
	}

Then you can query the API endpoint, or tail the WebSocket using curl.

.. code-block:: bash

	$ curl -k https://localhost:8053/health
	{"state":"up","uptime":0}
	$ curl -k -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: localhost:8053/health"  -H "Sec-Websocket-Key: nope" -H "Sec-Websocket-Version: 13" https://localhost:8053/health
	HTTP/1.1 101 Switching Protocols
	upgrade: websocket
	sec-websocket-accept: eg18mwU7CDRGUF1Q+EJwPM335eM=
	connection: upgrade

	?["up"]?["up"]?["up"]

Since the stream handlers are effectively coroutines, you are free to keep state and yield using cqueues.
This is especially useful for WebSockets, as you can stream content in a simple loop instead of
chains of callbacks.

Last thing you can publish from modules are *"snippets"*. Snippets are plain pieces of HTML code that are rendered at the end of the built-in webpage. The snippets can be extended with JS code to talk to already
exported restful APIs and subscribe to WebSockets.

.. code-block:: lua

	http.snippets['/health'] = {'Health service', '<p>UP!</p>'}

How to expose RESTful services
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A RESTful service is likely to respond differently to different type of methods and requests,
there are three things that you can do in a service handler to send back results.
First is to just send whatever you want to send back, it has to respect MIME type that the service
declared in the endpoint definition. The response code would then be ``200 OK``, any non-string
responses will be packed to JSON. Alternatively, you can respond with a number corresponding to
the HTTP response code or send headers and body yourself.

.. code-block:: lua

	-- Our upvalue
	local value = 42

	-- Expose the service
	local service = {'application/json',
	function (h, stream)
		-- Get request method and deal with it properly
		local m = h:get(':method')
		local path = h:get(':path')
		log('[service] method %s path %s', m, path)
		-- Return table, response code will be '200 OK'
		if m == 'GET' then
			return {key = path, value = value}
		-- Save body, perform check and either respond with 505 or 200 OK
		elseif m == 'POST' then
			local data = stream:get_body_as_string()
			if not tonumber(data) then
				return 500, 'Not a good request'
			end
			value = tonumber(data)
		-- Unsupported method, return 405 Method not allowed
		else
			return 405, 'Cannot do that'
		end
	end}
	-- Load the module
	modules = {
		http = {
			endpoints = { ['/service'] = service }
		}
	}

In some cases you might need to send back your own headers instead of default provided by HTTP handler,
you can do this, but then you have to return ``false`` to notify handler that it shouldn't try to generate
a response.

.. code-block:: lua

	local headers = require('http.headers')
	function (h, stream)
		-- Send back headers
		local hsend = headers.new()
		hsend:append(':status', '200')
		hsend:append('content-type', 'binary/octet-stream')
		assert(stream:write_headers(hsend, false))
		-- Send back data
		local data = 'binary-data'
		assert(stream:write_chunk(data, true))
		-- Disable default handler action
		return false
	end

How to expose more interfaces
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Services exposed in the previous part share the same external interface. This means that it's either accessible to the outside world or internally, but not one or another. This is not always desired, i.e. you might want to offer DNS/HTTPS to everyone, but allow application firewall configuration only on localhost. ``http`` module allows you to create additional interfaces with custom endpoints for this purpose.

.. code-block:: lua

	http.add_interface {
		endpoints = {
			['/conf'] = {
				'application/json', function (h, stream)
					return 'configuration API\n'
				end
			},
		},
		-- Same options as the config() method
		host = 'localhost',
		port = '8054',
	}

This way you can have different internal-facing and external-facing services at the same time.

Dependencies
^^^^^^^^^^^^

* `lua-http <https://github.com/daurnimator/lua-http>`_ (>= 0.1) available in LuaRocks

    If you're installing via Homebrew on OS X, you need OpenSSL too.

    .. code-block:: bash

       $ brew update
       $ brew install openssl
       $ brew link openssl --force # Override system OpenSSL

    Any other system can install from LuaRocks directly:

    .. code-block:: bash

       $ luarocks install http

* `mmdblua <https://github.com/daurnimator/mmdblua>`_ available in LuaRocks

    .. code-block:: bash

       $ luarocks install --server=https://luarocks.org/dev mmdblua
       $ curl -O https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
       $ gzip -d GeoLite2-City.mmdb.gz

.. _Prometheus: https://prometheus.io
.. _reverse-proxy: https://en.wikipedia.org/wiki/Reverse_proxy
.. _Apache: https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html
.. _Nginx: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/