diff options
Diffstat (limited to 'www/vfs.html')
| -rw-r--r-- | www/vfs.html | 493 |
1 files changed, 493 insertions, 0 deletions
diff --git a/www/vfs.html b/www/vfs.html new file mode 100644 index 0000000..c35951d --- /dev/null +++ b/www/vfs.html @@ -0,0 +1,493 @@ +<!DOCTYPE html> +<html><head> +<meta name="viewport" content="width=device-width, initial-scale=1.0"> +<meta http-equiv="content-type" content="text/html; charset=UTF-8"> +<link href="sqlite.css" rel="stylesheet"> +<title>The SQLite OS Interface or "VFS"</title> +<!-- path= --> +</head> +<body> +<div class=nosearch> +<a href="index.html"> +<img class="logo" src="images/sqlite370_banner.gif" alt="SQLite" border="0"> +</a> +<div><!-- IE hack to prevent disappearing logo --></div> +<div class="tagline desktoponly"> +Small. Fast. Reliable.<br>Choose any three. +</div> +<div class="menu mainmenu"> +<ul> +<li><a href="index.html">Home</a> +<li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a> +<li class='wideonly'><a href='about.html'>About</a> +<li class='desktoponly'><a href="docs.html">Documentation</a> +<li class='desktoponly'><a href="download.html">Download</a> +<li class='wideonly'><a href='copyright.html'>License</a> +<li class='desktoponly'><a href="support.html">Support</a> +<li class='desktoponly'><a href="prosupport.html">Purchase</a> +<li class='search' id='search_menubutton'> +<a href="javascript:void(0)" onclick='toggle_search()'>Search</a> +</ul> +</div> +<div class="menu submenu" id="submenu"> +<ul> +<li><a href='about.html'>About</a> +<li><a href='docs.html'>Documentation</a> +<li><a href='download.html'>Download</a> +<li><a href='support.html'>Support</a> +<li><a href='prosupport.html'>Purchase</a> +</ul> +</div> +<div class="searchmenu" id="searchmenu"> +<form method="GET" action="search"> +<select name="s" id="searchtype"> +<option value="d">Search Documentation</option> +<option value="c">Search Changelog</option> +</select> +<input type="text" name="q" id="searchbox" value=""> +<input type="submit" value="Go"> +</form> +</div> +</div> +<script> +function toggle_div(nm) { +var w = document.getElementById(nm); +if( w.style.display=="block" ){ +w.style.display = "none"; +}else{ +w.style.display = "block"; +} +} +function toggle_search() { +var w = document.getElementById("searchmenu"); +if( w.style.display=="block" ){ +w.style.display = "none"; +} else { +w.style.display = "block"; +setTimeout(function(){ +document.getElementById("searchbox").focus() +}, 30); +} +} +function div_off(nm){document.getElementById(nm).style.display="none";} +window.onbeforeunload = function(e){div_off("submenu");} +/* Disable the Search feature if we are not operating from CGI, since */ +/* Search is accomplished using CGI and will not work without it. */ +if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){ +document.getElementById("search_menubutton").style.display = "none"; +} +/* Used by the Hide/Show button beside syntax diagrams, to toggle the */ +function hideorshow(btn,obj){ +var x = document.getElementById(obj); +var b = document.getElementById(btn); +if( x.style.display!='none' ){ +x.style.display = 'none'; +b.innerHTML='show'; +}else{ +x.style.display = ''; +b.innerHTML='hide'; +} +return false; +} +var antiRobot = 0; +function antiRobotGo(){ +if( antiRobot!=3 ) return; +antiRobot = 7; +var j = document.getElementById("mtimelink"); +if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href"); +} +function antiRobotDefense(){ +document.body.onmousedown=function(){ +antiRobot |= 2; +antiRobotGo(); +document.body.onmousedown=null; +} +document.body.onmousemove=function(){ +antiRobot |= 2; +antiRobotGo(); +document.body.onmousemove=null; +} +setTimeout(function(){ +antiRobot |= 1; +antiRobotGo(); +}, 100) +antiRobotGo(); +} +antiRobotDefense(); +</script> +<div class=fancy> +<div class=nosearch> +<div class="fancy_title"> +The SQLite OS Interface or "VFS" +</div> +<div class="fancy_toc"> +<a onclick="toggle_toc()"> +<span class="fancy_toc_mark" id="toc_mk">►</span> +Table Of Contents +</a> +<div id="toc_sub"><div class="fancy-toc1"><a href="#introduction">1. Introduction</a></div> +<div class="fancy-toc1"><a href="#the_vfs_in_relation_to_the_rest_of_sqlite">2. The VFS In Relation To The Rest Of SQLite</a></div> +<div class="fancy-toc1"><a href="#multiple_vfses">3. Multiple VFSes</a></div> +<div class="fancy-toc2"><a href="#standard_unix_vfses">3.1. Standard Unix VFSes</a></div> +<div class="fancy-toc2"><a href="#standard_windows_vfses">3.2. Standard Windows VFSes</a></div> +<div class="fancy-toc2"><a href="#specifying_which_vfs_to_use">3.3. Specifying Which VFS To Use</a></div> +<div class="fancy-toc2"><a href="#vfs_shims">3.4. VFS Shims</a></div> +<div class="fancy-toc2"><a href="#other_example_vfses">3.5. Other Example VFSes</a></div> +<div class="fancy-toc1"><a href="#vfs_implementations">4. VFS Implementations</a></div> +</div> +</div> +<script> +function toggle_toc(){ +var sub = document.getElementById("toc_sub") +var mk = document.getElementById("toc_mk") +if( sub.style.display!="block" ){ +sub.style.display = "block"; +mk.innerHTML = "▼"; +} else { +sub.style.display = "none"; +mk.innerHTML = "►"; +} +} +</script> +</div> + + + + + +<h1 id="introduction"><span>1. </span>Introduction</h1> + +<p> +This article describes the SQLite OS portability layer or "VFS" - the +module at the bottom of the SQLite implementation stack +that provides portability across operating systems. +</p> + +<h1 id="the_vfs_in_relation_to_the_rest_of_sqlite"><span>2. </span>The VFS In Relation To The Rest Of SQLite</h1> + +<div> +<img src="images/vfs1.gif" align="right" hspace="10"> +</div> + +<p> +The internal organization of the SQLite library can be viewed as the +stack of modules shown to the right. +The Tokenizer, Parser, and Code Generator components are used to +process SQL statements and convert them into executable programs +in a virtual machine language or byte code. +Roughly speaking, these top three layers implement +<a href="c3ref/prepare.html">sqlite3_prepare_v2()</a>. The byte code generated by the top three +layers is a <a href="c3ref/stmt.html">prepared statement</a>. +The Virtual Machine module is responsible for running the SQL statement +byte code. The B-Tree module organizes a database file into multiple +key/value stores with ordered keys and logarithmic performance. +The Pager module is responsible for loading pages of the database +file into memory, for implementing and controlling transactions, and +for creating and maintaining the journal files that prevent database +corruption following a crash or power failure. +The OS Interface is a thin abstraction that provides a common set of +routines for adapting SQLite to run on different operating systems. +Roughly speaking, the bottom four layers implement +<a href="c3ref/step.html">sqlite3_step()</a>. +</p> + +<p> +This article is about the bottom layer. +</p> + +<p>The OS Interface - also called the "VFS" - is what makes SQLite +portable across operating systems. Whenever any of the other modules +in SQLite needs to communicate with the operating +system, they invoke methods in the VFS. The VFS then invokes the +operating-specific code needed to satisfy the request. +Hence, porting SQLite to a new +operating system is simply a matter of writing a new OS interface layer +or "VFS".</p> + +<h1 id="multiple_vfses"><span>3. </span>Multiple VFSes</h1> + +<p> +The standard SQLite source tree contains built-in VFSes for unix +and windows. Alternative VFSes can be +added at start-time or run-time using the +<a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface. +</p> + +<p> +Multiple VFSes can be registered at the same time. +Each VFS has a unique names. +Separate <a href="c3ref/sqlite3.html">database connections</a> within the same process can be using +different VFSes at the same time. For that matter, if a single +database connection has multiple database files open using +the <a href="lang_attach.html">ATTACH</a> command, then each attached database might be using a +different VFS. +</p> + +<h2 id="standard_unix_vfses"><span>3.1. </span>Standard Unix VFSes</h2> + +<p> +Unix builds come with multiple built-in VFSes. The default VFS +for unix is called "unix" and is used in most applications. +Other VFSes that might be found in unix (depending on compile-time +options) include: +</p> + +<ol> +<li><p><b>unix-dotfile</b> - uses dot-file locking rather than + POSIX advisory locks. +</p></li><li><p><b>unix-excl</b> - obtains and holds an exclusive lock on + database files, preventing other processes from accessing the + database. Also keeps the <a href="walformat.html#shm">wal-index</a> in heap rather than in + shared memory. +</p></li><li><p><b>unix-none</b> - all file locking operations are no-ops. +</p></li><li><p><b>unix-namedsem</b> - uses named semaphores for file locking. + VXWorks only. +</p></li></ol> + +<p> +The various unix VFSes differ only in the way they handle file locking - +they share most of their implementation in common with one another and +are all located in the same SQLite source file: +<a href="https://www.sqlite.org/src/doc/trunk/src/os_unix.c">os_unix.c</a>. +Note that except for "unix" and "unix-excl", the various unix VFSes all +use incompatible locking implementations. If two processes are accessing +the same SQLite database using different unix VFSes, they may +not see each others locks and may end up interfering with one another, +resulting in database corruption. The "unix-none" VFS in particular +does no locking at all and will easily result in database corruption if +used by two or more database connections at the same time. +Programmers are encouraged to use only "unix" or "unix-excl" unless +there is a compelling reason to do otherwise. +</p> + +<h2 id="standard_windows_vfses"><span>3.2. </span>Standard Windows VFSes</h2> + + +<p> +Windows builds also come with multiple built-in VFSes. The default +Windows VFS is called "win32" and is used in most applications. +Other VFSes that might be found on windows builds include: +</p> + +<ol> +<li><p><b>win32-longpath</b> - like "win32" except that pathnames can + be up to 65534 bytes in length, whereas pathnames max out at + 1040 bytes in "win32". +</p></li><li><p><b>win32-none</b> - all file locking operations are no-ops. +</p></li><li><p><b>win32-longpath-none</b> - combination of "win32-longpath" + and "win32-none" - long pathnames are supported and all lock + operations are no-ops. +</p></li></ol> + +<p>As with unix, most of the code for the various Windows VFSes is shared. + +</p><h2 id="specifying_which_vfs_to_use"><span>3.3. </span>Specifying Which VFS To Use</h2> + +<p> +There is always one VFS which is the default VFS. On unix systems, +the "unix" VFS comes up as the default and on windows it is "win32". +If no other actions are taken, new database connections will make use +of the default VFS. +</p> + +<p> +The default VFS can be changed by registering or re-registering the +VFS using the <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface with a second parameter +of 1. Hence, if a (unix) process wants to always use the "unix-nolock" VFS +in place of "unix", the following code would work: +</p> + +<blockquote><pre> +sqlite3_vfs_register(sqlite3_vfs_find("unix-nolock"), 1); +</pre></blockquote> + +<p> +An alternate VFS can also be specified as the 4th parameter to the +<a href="c3ref/open.html">sqlite3_open_v2()</a> function. For example: +</p> + +<blockquote><pre> +int rc = sqlite3_open_v2("demo.db", &db, SQLITE_OPEN_READWRITE, "unix-nolock"); +</pre></blockquote> + +<p> +Finally, if <a href="uri.html">URI filenames</a> have been enabled, then the alternative +VFS can be specified using the "vfs=" parameter on the URI. This technique +works with <a href="c3ref/open.html">sqlite3_open()</a>, <a href="c3ref/open.html">sqlite3_open16()</a>, <a href="c3ref/open.html">sqlite3_open_v2()</a>, and +when a new database is <a href="lang_attach.html">ATTACH</a>-ed to an existing database connection. +For example: +</p> + +<blockquote><pre> +ATTACH 'file:demo2.db?vfs=unix-none' AS demo2; +</pre></blockquote> + +<p> +The VFS specified by a URI has the highest priority. After that comes +a VFS specified as the fourth argument to <a href="c3ref/open.html">sqlite3_open_v2()</a>. The +default VFS is used if no VFS is specified otherwise. +</p> + +<a name="shim"></a> + +<h2 id="vfs_shims"><span>3.4. </span>VFS Shims</h2> + +<p> +From the point of view of the uppers layers of the SQLite stack, each +open database file uses exactly one VFS. +But in practice, a particular VFS might +just be a thin wrapper around another VFS that does the real work. +We call a wrapper VFS a "shim". +</p> + +<p> +A simple example of a shim is the "vfstrace" VFS. This is a VFS +(implemented in the +<a href="https://www.sqlite.org/src/doc/trunk/src/test_vfstrace.c">test_vfstrace.c</a> +source file) that writes a message associated with each VFS method call +into a log file, then passes control off to another VFS to do the actual +work. +</p> + +<h2 id="other_example_vfses"><span>3.5. </span>Other Example VFSes</h2> + +<p> +The following are other VFS implementations available in the public +SQLite source tree: +</p> + +<ul> +<li><p> +<a href="https://www.sqlite.org/src/file/ext/misc/appendvfs.c">appendvfs.c</a> - +This VFS allows an SQLite database to be appended to the end of some +other file. This can be used, for example, to append an SQLite database +onto the end of an executable such that, when run, it can easily +locate the appended database. The <a href="cli.html">command-line shell</a> will use this +VFS if launched with the --append option, and its .archive command +will use it given the --append flag. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_demovfs.c">test_demovfs.c</a> - +This file implements a very simple VFS named "demo" that uses POSIX +functions such as +open(), read(), write(), fsync(), close(), fsync(), sleep(), time(), +and so forth. This VFS only works on unix systems. But it is not +intended as a replacement for the standard "unix" VFS used by default +on unix platforms. The "demo" VFS is deliberately kept very simple +so that it can be used as a learning aid or as template for building +other VFSes or for porting SQLite to new operating systems. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_quota.c">test_quota.c</a> - +This file implements a shim called "quota" that enforces cumulative +file size limits on a collection of database files. An auxiliary +interface is used to define "quota groups". A quota group is a +set of files (database files, journals, and temporary files) whose +names all match a <a href="lang_expr.html#glob">GLOB</a> pattern. The sum of the sizes of all files +in each quota group is tracked, and if that sum exceeds a threshold +defined for the quota group, a callback function is invoked. That +callback can either increase the threshold or cause the operation +that would have exceeded the quota to fail with an +<a href="rescode.html#full">SQLITE_FULL</a> error. One of the uses of this shim is used to enforce +resource limits on application databases in Firefox. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_multiplex.c">test_multiplex.c</a> - +This file implements a shim that allows database files to exceed the +maximum file size of the underlying filesystem. This shim presents +an interface to the upper six layers of SQLite that makes it look like +very large files are being used, when in reality each such large file +is split up into many smaller files on the underlying system. +This shim has been used, for example, to allow databases to grow +larger than 2 gibibytes on FAT16 filesystems. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_onefile.c">test_onefile.c</a> - +This file implements a demonstration VFS named "fs" that shows how SQLite +can be used on an embedded device that lacks a filesystem. Content is +written directly to the underlying media. A VFS derived from this +demonstration code could be used by a gadget with a limited amount of +flash memory to make SQLite behave as the filesystem for the flash memory +on the device. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_journal.c">test_journal.c</a> - +This file implements a shim used during SQLite testing that verifies that +the database and rollback journal are written in the correct order and +are "synced" at appropriate times in order to guarantee that the database +can recover from a power lose are hard reset at any time. The shim +checks several invariants on the operation of databases and rollback +journals and raises exceptions if any of those invariants are violated. +These invariants, in turn, assure that the database is always recoverable. +Running a large suite of test cases using this shim provides added +assurance that SQLite databases will not be damaged by unexpected +power failures or device resets. + +</p></li><li><p> +<a href="https://www.sqlite.org/src/doc/trunk/src/test_vfs.c">test_vfs.c</a> - +This file implements a shim that can be used to simulate filesystem faults. +This shim is used during testing to verify that SQLite responses sanely +to hardware malfunctions or to other error conditions such as running out +of filesystem space that are difficult to test on a real system. +</p></li></ul> + +<p> +There are other VFS implementations both in the core SQLite source code +library and in available extensions. The list above is not meant to be +exhaustive but merely representative of the kinds of features that can +be realized using the VFS interface. +</p> + +<h1 id="vfs_implementations"><span>4. </span>VFS Implementations</h1> + +<p> +A new VFS is implemented by subclassing three objects: +</p> + +<ul> +<li><a href="c3ref/vfs.html">sqlite3_vfs</a> +</li><li><a href="c3ref/io_methods.html">sqlite3_io_methods</a> +</li><li><a href="c3ref/file.html">sqlite3_file</a> +</li></ul> + +<p> +An <a href="c3ref/vfs.html">sqlite3_vfs</a> object defines the name of the VFS and the core +methods that implement the interface to the operating system, such +as checking for existence of files, deleting files, creating files +and opening and for reading and/or writing, converting filenames +into their canonical form. The <a href="c3ref/vfs.html">sqlite3_vfs</a> object also contains +methods for obtaining randomness from the operating system, for +suspending a process (sleeping) and for finding the current date and +time. +</p> + +<p> +The <a href="c3ref/file.html">sqlite3_file</a> object represents an open file. +The xOpen method of <a href="c3ref/vfs.html">sqlite3_vfs</a> constructs an <a href="c3ref/file.html">sqlite3_file</a> +object when the file is opened. The <a href="c3ref/file.html">sqlite3_file</a> keeps track +of the state of the file while it is opened. +</p> + +<p> +The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object holds the methods used to interact +with an open file. Each <a href="c3ref/file.html">sqlite3_file</a> contains a pointer to +an <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object that is appropriate for the file it +represents. The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object contains methods to do +things such as read and write from the file, to truncate the file, +to flush any changes to persistent storage, to find the size of the +file, to lock and unlock the file, and to close file and destroy +the <a href="c3ref/file.html">sqlite3_file</a> object. +</p> + +<p> +Writing the code for a new VFS involves constructing a subclass for +the <a href="c3ref/vfs.html">sqlite3_vfs</a> object and then registering that VFS object using +a call to <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a>. The VFS implementation also +provides subclasses for <a href="c3ref/file.html">sqlite3_file</a> and <a href="c3ref/io_methods.html">sqlite3_io_methods</a> but +those objects are not registered directly with SQLite. Instead, the +<a href="c3ref/file.html">sqlite3_file</a> object is returned from the xOpen method of +<a href="c3ref/vfs.html">sqlite3_vfs</a> and the <a href="c3ref/file.html">sqlite3_file</a> object points to an instance +of the <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object. +</p> + + |