# page_content

c2: SQLite Is Transactional
c5: transactional.html
c4:
SQLite is Transactional

A transactional database is one in which all changes and queries
appear to be
Atomic, Consistent, Isolated, and Durable
(ACID).
SQLite implements 
serializable
transactions that are atomic, consistent, isolated, and durable,
even if the transaction is interrupted by a program crash, an
operating system crash, or a power failure to the computer.



We here restate and amplify the previous sentence for emphasis:
All changes within a single transaction in SQLite either occur
completely or not at all, even if the act of writing the change
out to the disk is interrupted by

a program crash,
an operating system crash, or
a power failure.




The claim of the previous paragraph is extensively checked in the
SQLite regression test suite using a special test harness that 
simulates the effects on a database file of operating system crashes 
and power failures.



Additional information

This page last modified on  2021-02-12 00:39:31 UTC


c2: Datatypes In SQLite
c5: datatype3.html
c4:
1. Datatypes In SQLite
Most SQL database engines (every SQL database engine other than SQLite,
as far as we know) uses static, rigid typing.  With static typing, the datatype
of a value is determined by its container - the particular column in
which the value is stored.

SQLite uses a more general dynamic type system.  In SQLite, the datatype
of a value is associated with the value itself, not with its container.
The dynamic type system of SQLite is backwards
compatible with the more common static type systems of other database engines
in the sense that SQL statements that work on statically typed databases
work the same way in SQLite.  However, the dynamic typing in SQLite allows
it to do things which are not possible in traditional rigidly typed
databases.  Flexible typing is a feature of SQLite, not a bug.

Update:
As of version 3.37.0 (2021-11-27), SQLite provides STRICT tables
that do rigid type enforcement, for developers who prefer that kind of thing.



2. Storage Classes and Datatypes

Each value stored in an SQLite database (or manipulated by the
database engine) has one of the following storage classes:

  NULL.
  The value is a NULL value.

  INTEGER. The value is a signed integer, stored in 0, 1,
  2, 3, 4, 6, or 8 bytes depending on the magnitude of the value.

  REAL. The value is a floating point value, stored as
  an 8-byte IEEE floating point number.

  TEXT. The value is a text string, stored using the
  database encoding (UTF-8, UTF-16BE or UTF-16LE).

  BLOB. The value is a blob of data, stored exactly as
  it was input.


A storage class is more general than a datatype.
The INTEGER storage class, for example, includes 7 different integer
datatypes of different lengths.
This makes a difference on disk. 
But as soon as INTEGER values are read off of disk and into memory for
processing, they are converted to the most general datatype
(8-byte signed integer).
And so for the most part, "storage class" is indistinguishable from 
"datatype" and the two terms can be used interchangeably.

Any column in an SQLite version 3 database,
except an INTEGER PRIMARY KEY column, may be used to store a value 
of any storage class.

All values in SQL statements, whether they are literals embedded in SQL
statement text or parameters bound to 
precompiled SQL statements
have an implicit storage class.
Under circumstances described below, the
database engine may convert values between numeric storage classes
(INTEGER and REAL) and TEXT during query execution. 




2.1. Boolean Datatype

SQLite does not have a separate Boolean storage class.
Instead, Boolean values are stored as integers 0 (false) and 1 (true).

SQLite recognizes the keywords "TRUE" and "FALSE",
as of version 3.23.0 (2018-04-02) but those keywords are
really just alternative spellings for the integer literals 1 and 0
respectively.



2.2. Date and Time Datatype

SQLite does not have a storage class set aside for storing
dates and/or times.
Instead, the built-in Date And Time Functions of SQLite are capable of 
storing dates and times as TEXT, REAL, or INTEGER values:


TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since
noon in Greenwich on November 24, 4714 B.C. according to the
proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since
1970-01-01 00:00:00 UTC.


Applications can choose to store dates and times in any of these
formats and freely convert between formats using the built-in
date and time functions.




3. Type Affinity


SQL database engines that use rigid typing will usually try to
automatically convert values to the appropriate datatype.  Consider this:


CREATE TABLE t1(a INT, b VARCHAR(10));
INSERT INTO t1(a,b) VALUES('123',456);



Rigidly-typed database will convert the string '123' into an
integer 123 and the integer 456 into a string '456' prior to
doing the insert.


In order to maximize compatibility between SQLite and other database
engines, and so that the example above will work on SQLite as it does
on other SQL database engines,
SQLite supports the concept of "type affinity" on columns.
The type affinity of a column is the recommended type for data stored
in that column.  The important idea here is that the type is recommended, not
required.  Any column can still store any type of data.
It is just that some columns, given the choice, will prefer to use
one storage class over another.  The preferred storage class for
a column is called its "affinity".


Each column in an SQLite 3 database is assigned one of the
following type affinities:

	TEXT
	NUMERIC
	INTEGER
        REAL
	BLOB


(Historical note:  The "BLOB" type affinity used to be called "NONE".
But that term was easy to confuse with "no affinity" and so it was
renamed.)

A column with TEXT affinity stores all data using storage classes
NULL, TEXT or BLOB. If numerical data is inserted into a column with
TEXT affinity it is converted into text form before being stored.

A column with NUMERIC affinity may contain values using all five
storage classes.  When text data is inserted into a NUMERIC column, the
storage class of the text is converted to INTEGER or REAL (in order of
preference) if the text is a well-formed integer or real literal, respectively.
If the TEXT value is a well-formed integer literal that is too large
to fit in a 64-bit signed integer, it is converted to REAL.
For conversions between TEXT and REAL storage classes, only the first
15 significant decimal digits of the number are preserved.
If the TEXT value is not a well-formed integer or real literal,
then the value is stored as TEXT.
For the purposes of this paragraph, hexadecimal integer
literals are not considered well-formed and are stored as TEXT.
(This is done for historical compatibility with versions of SQLite
prior to version 3.8.6 2014-08-15 where hexadecimal integer
literals were first introduced into SQLite.)
If a floating point value that can be represented exactly as an integer
is inserted into a column with NUMERIC affinity, the value is
converted into an integer.
No attempt is made to convert NULL or BLOB values.

A string might look like a floating-point literal with
a decimal point and/or exponent notation but as long as
the value can be expressed as an integer, the NUMERIC affinity will convert
it into an integer. Hence, the string '3.0e+5' is stored in a
column with NUMERIC affinity as the integer 300000, not as the floating
point value 300000.0.

A column that uses INTEGER affinity behaves the same as a column
with NUMERIC affinity.  The difference between INTEGER and NUMERIC affinity
is only evident in a CAST expression:  The expression
"CAST(4.0 AS INT)" returns an integer 4, whereas
"CAST(4.0 AS NUMERIC)" leaves the value as a floating-point 4.0.

A column with REAL affinity behaves like a column with NUMERIC
affinity except that it forces integer values into floating point
representation.  (As an internal optimization, small floating point
values with no fractional component and stored in columns with REAL
affinity are written to disk as integers in order to take up less 
space and are automatically converted back into floating point as
the value is read out.
This optimization is completely invisible at the SQL level and can only
be detected by examining the raw bits of the database file.)

A column with affinity BLOB does not prefer one storage class over
another and no attempt is made to coerce data from one storage class into
another.



3.1. Determination Of Column Affinity

For tables not declared as STRICT,
the affinity of a column is determined by the declared type
of the column, according to the following rules in the order shown:


  If the declared type contains the string "INT" then it
  is assigned INTEGER affinity.

  If the declared type of the column contains any of the strings
  "CHAR", "CLOB", or "TEXT" then that
  column has TEXT affinity.  Notice that the type VARCHAR contains the
  string "CHAR" and is thus assigned TEXT affinity.

  If the declared type for a column
  contains the string "BLOB" or if
  no type is specified then the column has affinity BLOB.

  If the declared type for a column
  contains any of the strings "REAL", "FLOA",
  or "DOUB" then the column has REAL affinity.

  Otherwise, the affinity is NUMERIC.


Note that the order of the rules for determining column affinity
is important.  A column whose declared type is "CHARINT" will match
both rules 1 and 2 but the first rule takes precedence and so the 
column affinity will be INTEGER.

3.1.1. Affinity Name Examples

The following table shows how many common datatype names from
more traditional SQL implementations are converted into affinities by the five rules of the
previous section.  This table shows only a small subset of the
datatype names that SQLite will accept.  Note that numeric arguments
in parentheses that following the type name (ex: "VARCHAR(255)") are
ignored by SQLite - SQLite does not impose any length restrictions
(other than the large global SQLITE_MAX_LENGTH limit) on the length of
strings, BLOBs or numeric values.

 

Example Typenames From TheCREATE TABLE Statement
        or CAST Expression
    Resulting Affinity
    Rule Used To Determine Affinity


  INT
  INTEGER
  TINYINT
  SMALLINT
  MEDIUMINT
  BIGINT
  UNSIGNED BIG INT
  INT2
  INT8
INTEGER
1


  CHARACTER(20)
  VARCHAR(255)
  VARYING CHARACTER(255)
  NCHAR(55)
  NATIVE CHARACTER(70)
  NVARCHAR(100)
  TEXT
  CLOB
TEXT
2


  BLOB
  no datatype specified
BLOB
3


  REAL
  DOUBLE
  DOUBLE PRECISION
  FLOAT
REAL
4


  NUMERIC
  DECIMAL(10,5)
  BOOLEAN
  DATE
  DATETIME
NUMERIC
5



Note that a declared type of "FLOATING POINT" would give INTEGER
affinity, not REAL affinity, due to the "INT" at the end of "POINT".
And the declared type of "STRING" has an affinity of NUMERIC, not TEXT.



3.2. Affinity Of Expressions

Every table column has a type affinity (one of BLOB, TEXT, INTEGER,
REAL, or NUMERIC) but expressions do not necessarily have an affinity.

Expression affinity is determined by the following rules:


  The right-hand operand of an IN or NOT IN
  operator has no affinity if the operand is a list, or has the same
  affinity as the affinity of the result set expression if the operand
  is a SELECT.

  When an expression is a simple reference to a column of a
  real table (not a VIEW or subquery) then the expression
  has the same affinity as the table column.
  
  Parentheses around the column name are ignored.  Hence if
  X and Y.Z are column names, then (X) and (Y.Z) are also considered
  column names and have the affinity of the corresponding columns.
  
  Any operators applied to column names, including the no-op 
   unary "+" operator, convert the column name into an expression which
   always has no affinity.  Hence even if X and Y.Z are column names, the
   expressions +X and +Y.Z are not column names and have no affinity.
  

  An expression of the form "CAST(expr AS type)"
  has an affinity that is the same as a column with a declared
  type of "type".

  A COLLATE operator has the same affinity as its left-hand side operand.

  Otherwise, an expression has no affinity.




3.3. Column Affinity For Views And Subqueries

The "columns" of a VIEW or FROM-clause subquery are really
the expressions
in the result set of the SELECT statement that implements the VIEW
or subquery.  Thus, the affinity for columns of a VIEW or subquery
are determined by the expression affinity rules above.
Consider an example:


CREATE TABLE t1(a INT, b TEXT, c REAL);
CREATE VIEW v1(x,y,z) AS SELECT b, a+c, 42 FROM t1 WHERE b!=11;


The affinity of the v1.x column will be the same as the affinity
of t1.b (TEXT), since v1.x maps directly into t1.b.  But
columns v1.y and v1.z both have no affinity, since those columns
map into expression a+c and 42, and expressions always have no
affinity.



3.3.1. Column Affinity For Compound Views

When the SELECT statement that implements a VIEW or 
FROM-clause subquery is a compound SELECT then the affinity of 
each column of the VIEW or subquery will
be the affinity of the corresponding result column for
one of the individual SELECT statements that make up 
the compound.  
However, it is indeterminate which of the SELECT statements will 
be used to determine affinity.
Different constituent SELECT statements might be used to determine 
affinity at different times during query evaluation.  The choice
might vary across different versions of SQLite.  The choice might
change between one query and the next in the same version of SQLite.
The choice might be different at different times within the same
query.  Hence, you can never be sure what affinity will be used
for columns of a compound SELECT that have different affinities in
the constituent subqueries.

Best practice is to avoid mixing affinities in a compound SELECT
if you care about the datatype of the result.  Mixing affinities in
a compound SELECT can lead to surprising and unintuitive results.
See, for example, forum post 02d7be94d7.

3.4. Column Affinity Behavior Example

The following SQL demonstrates how SQLite uses column affinity
to do type conversions when values are inserted into a table.



CREATE TABLE t1(
    t  TEXT,     -- text affinity by rule 2
    nu NUMERIC,  -- numeric affinity by rule 5
    i  INTEGER,  -- integer affinity by rule 1
    r  REAL,     -- real affinity by rule 4
    no BLOB      -- no affinity by rule 3
);

-- Values stored as TEXT, INTEGER, INTEGER, REAL, TEXT.
INSERT INTO t1 VALUES('500.0', '500.0', '500.0', '500.0', '500.0');
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|text

-- Values stored as TEXT, INTEGER, INTEGER, REAL, REAL.
DELETE FROM t1;
INSERT INTO t1 VALUES(500.0, 500.0, 500.0, 500.0, 500.0);
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|real

-- Values stored as TEXT, INTEGER, INTEGER, REAL, INTEGER.
DELETE FROM t1;
INSERT INTO t1 VALUES(500, 500, 500, 500, 500);
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
text|integer|integer|real|integer

-- BLOBs are always stored as BLOBs regardless of column affinity.
DELETE FROM t1;
INSERT INTO t1 VALUES(x'0500', x'0500', x'0500', x'0500', x'0500');
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
blob|blob|blob|blob|blob

-- NULLs are also unaffected by affinity
DELETE FROM t1;
INSERT INTO t1 VALUES(NULL,NULL,NULL,NULL,NULL);
SELECT typeof(t), typeof(nu), typeof(i), typeof(r), typeof(no) FROM t1;
null|null|null|null|null





4. Comparison Expressions

SQLite version 3 has the usual set of SQL comparison operators
including "=", "==", "&lt;", "&lt;=", "&gt;", "&gt;=", "!=", "",
"IN", "NOT IN", "BETWEEN", "IS", and "IS NOT", .



4.1. Sort Order

The results of a comparison depend on the storage classes of the
operands, according to the following rules:

  A value with storage class NULL is considered less than any
  other value (including another value with storage class NULL).

  An INTEGER or REAL value is less than any TEXT or BLOB value.
  When an INTEGER or REAL is compared to another INTEGER or REAL, a
  numerical comparison is performed.

  A TEXT value is less than a BLOB value.  When two TEXT values
  are compared an appropriate collating sequence is used to determine 
  the result.  

  When two BLOB values are compared, the result is
  determined using memcmp().




4.2. Type Conversions Prior To Comparison

SQLite may attempt to convert values between the storage classes
INTEGER, REAL, and/or TEXT before performing a comparison.
Whether or not any conversions are attempted before the comparison takes
place depends on the type affinity of the operands.

To "apply affinity" means to convert an operand to a particular storage
class if and only if the conversion does not lose essential information.
Numeric values can always be converted into TEXT.  TEXT values 
can be converted into numeric values if the text content is a well-formed 
integer or real literal, but not a hexadecimal integer literal.
BLOB values are converted into TEXT values by simply interpreting
the binary BLOB content as a text string in the current database
encoding.

Affinity is applied to operands of a comparison operator prior to
the comparison according to the following rules in the order shown:


If one operand has INTEGER, REAL or NUMERIC affinity
and the other operand has TEXT or BLOB or no affinity
then NUMERIC affinity is applied to other operand.

If one operand has TEXT affinity and the other has no affinity,
then TEXT affinity is applied to the other operand.

Otherwise, no affinity is applied and both operands are compared
as is.


The expression "a BETWEEN b AND c" is treated as two separate
binary comparisons "a &gt;= b AND a &lt;= c", even if that means
different affinities are applied to 'a' in each of the comparisons.
Datatype conversions in comparisons of the
form "x IN (SELECT y ...)" are handled as if
the comparison were really "x=y".
The expression "a IN (x, y, z, ...)" is equivalent to "a = +x OR
a = +y OR a = +z OR ...".  
In other words, the values to the right of the IN operator (the "x", "y",
and "z" values in this example) are considered to have no affinity, 
even if they happen to be column values or CAST expressions.  


4.3. Comparison Example



CREATE TABLE t1(
    a TEXT,      -- text affinity
    b NUMERIC,   -- numeric affinity
    c BLOB,      -- no affinity
    d            -- no affinity
);

-- Values will be stored as TEXT, INTEGER, TEXT, and INTEGER respectively
INSERT INTO t1 VALUES('500', '500', '500', 500);
SELECT typeof(a), typeof(b), typeof(c), typeof(d) FROM t1;
text|integer|text|integer

-- Because column "a" has text affinity, numeric values on the
-- right-hand side of the comparisons are converted to text before
-- the comparison occurs.
SELECT a &lt; 40,   a &lt; 60,   a &lt; 600 FROM t1;
0|1|1

-- Text affinity is applied to the right-hand operands but since
-- they are already TEXT this is a no-op; no conversions occur.
SELECT a &lt; '40', a &lt; '60', a &lt; '600' FROM t1;
0|1|1

-- Column "b" has numeric affinity and so numeric affinity is applied
-- to the operands on the right.  Since the operands are already numeric,
-- the application of affinity is a no-op; no conversions occur.  All
-- values are compared numerically.
SELECT b &lt; 40,   b &lt; 60,   b &lt; 600 FROM t1;
0|0|1

-- Numeric affinity is applied to operands on the right, converting them
-- from text to integers.  Then a numeric comparison occurs.
SELECT b &lt; '40', b &lt; '60', b &lt; '600' FROM t1;
0|0|1

-- No affinity conversions occur.  Right-hand side values all have
-- storage class INTEGER which are always less than the TEXT values
-- on the left.
SELECT c &lt; 40,   c &lt; 60,   c &lt; 600 FROM t1;
0|0|0

-- No affinity conversions occur.  Values are compared as TEXT.
SELECT c &lt; '40', c &lt; '60', c &lt; '600' FROM t1;
0|1|1

-- No affinity conversions occur.  Right-hand side values all have
-- storage class INTEGER which compare numerically with the INTEGER
-- values on the left.
SELECT d &lt; 40,   d &lt; 60,   d &lt; 600 FROM t1;
0|0|1

-- No affinity conversions occur.  INTEGER values on the left are
-- always less than TEXT values on the right.
SELECT d &lt; '40', d &lt; '60', d &lt; '600' FROM t1;
1|1|1



All of the results in the example are the same if the comparisons are
commuted - if expressions of the form "a&lt;40" are rewritten
as "40&gt;a".

5. Operators

Mathematical operators (+, -, *, /, %, &lt;&lt;, &gt;&gt;,
&amp;, and |) interpret both operands as if they were numbers.
STRING or BLOB operands automatically convert into REAL or INTEGER values.
If the STRING or BLOB looks like a real number (if it has a decimal point
or an exponent) or if the value is outside the range that can be represented
as a 64-bit signed integer, then it converts to REAL.  Otherwise the operand
converts to INTEGER.
The implied type conversion of mathematical operands is slightly different
from CAST to NUMERIC in that string and BLOB values that
look like real numbers but have no fractional part are kept as REAL
instead of being converted into INTEGER as they would be for CAST to NUMERIC.
The conversion from STRING or BLOB into REAL or INTEGER is performed
even if it is lossy and irreversible.
Some mathematical operators (%, &lt;&lt;, &gt;&gt;, &amp;, and |) expect
INTEGER operands.  For those operators, REAL operands are converted into INTEGER
in the same way as a CAST to INTEGER.
The &lt;&lt;, &gt;&gt;, &amp;, and | operators always return an INTEGER (or NULL)
result, but the % operator returns either INTEGER or REAL (or NULL)
depending on the type of its operands.
A NULL operand on a mathematical operator yields a NULL result.
An operand on a mathematical operator that does not look in any way
numeric and is not NULL is converted to 0 or 0.0.
Division by zero gives a result of NULL.


6. Sorting, Grouping and Compound SELECTs

When query results are sorted by an ORDER BY clause, values with storage
class NULL come first, followed by INTEGER and REAL values
interspersed in numeric order, followed by TEXT values in collating
sequence order, and finally BLOB values in memcmp() order.  No storage
class conversions occur before the sort.

When grouping values with the GROUP BY clause values with
different storage classes are considered distinct, except for INTEGER
and REAL values which are considered equal if they are numerically
equal. No affinities are applied to any values as the result of a
GROUP by clause.

The compound SELECT operators UNION,
INTERSECT and EXCEPT perform implicit comparisons between values.
No affinity is applied to comparison operands for the implicit
comparisons associated with UNION, INTERSECT, or EXCEPT - the values
are compared as is.



7. Collating Sequences

When SQLite compares two strings, it uses a collating sequence or
collating function (two terms for the same thing) to determine which
string is greater or if the two strings are equal.
SQLite has three built-in collating functions:  BINARY, NOCASE, and 
RTRIM.


BINARY - Compares string data using memcmp(), regardless
                   of text encoding.
NOCASE - Similar to binary, except that it uses
     sqlite3_strnicmp() for the comparison.  Hence the 26 upper case
     characters of ASCII are folded to their lower case equivalents before
     the comparison is performed.  Note that only ASCII characters
     are case folded.  SQLite does not attempt to do full
     UTF case folding due to the size of the tables required.
     Also note that any U+0000 characters in the string are considered
     string terminators for comparison purposes.

RTRIM - The same as binary, except that trailing space
     characters are ignored.


An application can register additional collating functions using
the sqlite3_create_collation() interface.

Collating functions only matter when comparing string values.
Numeric values are always compared numerically, and BLOBs are always
compared byte-by-byte using memcmp().

7.1. Assigning Collating Sequences from SQL


Every column of every
table has an associated collating function.  If no collating function
is explicitly defined, then the collating function defaults to BINARY.
The COLLATE clause of the column definition is used
to define alternative collating functions for a column.






The rules for determining which collating function to use for a
binary comparison operator (=, &lt;, &gt;, &lt;=, &gt;=, !=, IS, and
IS NOT) are as follows:


If either operand has an explicit collating function assignment
using the postfix COLLATE operator, then the explicit collating function
is used for comparison, with precedence to the collating function of the
left operand.

If either operand is a column, then the collating function of
that column is used with precedence to the left operand.
For the purposes of the previous sentence, a column name
preceded by one or more unary "+" operators and/or CAST operators
is still considered a column name.


Otherwise, the BINARY collating function is used for comparison.




An operand of a comparison is considered to have an explicit
collating function assignment (rule 1 above) 
if any subexpression of the operand uses
the postfix COLLATE operator.  Thus, if a COLLATE operator is used
anywhere in a comparison expression, the collating function defined
by that operator is used for string comparison regardless of what 
table columns might be a part of that expression.  If two or more
COLLATE operator subexpressions appear anywhere in a comparison, the 
left most explicit collating function is used regardless of how deeply the
COLLATE operators are nested in the expression and regardless of
how the expression is parenthesized.



The expression "x BETWEEN y and z" is logically
equivalent to two comparisons "x &gt;= y AND x &lt;= z" and works with
respect to collating functions as if it were two separate comparisons.
The expression "x IN (SELECT y ...)" is handled in the same way as the
expression "x = y" for the purposes of determining the collating sequence.
The collating sequence used for expressions of the form 
"x IN (y, z, ...)" is the collating sequence of x.
If an explicit collating sequence is required on an IN operator
it should be applied to the left operand, like this: 
"x COLLATE nocase IN (y,z, ...)".
  


Terms of the ORDER BY clause that is part of a SELECT
statement may be assigned a collating sequence using the 
COLLATE operator, in which case the specified collating function is
used for sorting.
Otherwise, if the expression sorted by an ORDER BY clause is
a column, then the collating sequence of the column is used to
determine sort order. If the expression is not a column and has no
COLLATE clause, then the BINARY collating sequence is used.
  

7.2. Collation Sequence Examples

The examples below identify the collating sequences that would be used to
determine the results of text comparisons that may be performed by various
SQL statements. Note that a text comparison may not be required, and no
collating sequence used, in the case of numeric, blob or NULL values.



CREATE TABLE t1(
    x INTEGER PRIMARY KEY,
    a,                 /* collating sequence BINARY */
    b COLLATE BINARY,  /* collating sequence BINARY */
    c COLLATE RTRIM,   /* collating sequence RTRIM  */
    d COLLATE NOCASE   /* collating sequence NOCASE */
);
                   /* x   a     b     c       d */
INSERT INTO t1 VALUES(1,'abc','abc', 'abc  ','abc');
INSERT INTO t1 VALUES(2,'abc','abc', 'abc',  'ABC');
INSERT INTO t1 VALUES(3,'abc','abc', 'abc ', 'Abc');
INSERT INTO t1 VALUES(4,'abc','abc ','ABC',  'abc');
 
/* Text comparison a=b is performed using the BINARY collating sequence. */
SELECT x FROM t1 WHERE a = b ORDER BY x;
--result 1 2 3

/* Text comparison a=b is performed using the RTRIM collating sequence. */
SELECT x FROM t1 WHERE a = b COLLATE RTRIM ORDER BY x;
--result 1 2 3 4

/* Text comparison d=a is performed using the NOCASE collating sequence. */
SELECT x FROM t1 WHERE d = a ORDER BY x;
--result 1 2 3 4

/* Text comparison a=d is performed using the BINARY collating sequence. */
SELECT x FROM t1 WHERE a = d ORDER BY x;
--result 1 4

/* Text comparison 'abc'=c is performed using the RTRIM collating sequence. */
SELECT x FROM t1 WHERE 'abc' = c ORDER BY x;
--result 1 2 3

/* Text comparison c='abc' is performed using the RTRIM collating sequence. */
SELECT x FROM t1 WHERE c = 'abc' ORDER BY x;
--result 1 2 3

/* Grouping is performed using the NOCASE collating sequence (Values
** 'abc', 'ABC', and 'Abc' are placed in the same group). */
SELECT count(*) FROM t1 GROUP BY d ORDER BY 1;
--result 4

/* Grouping is performed using the BINARY collating sequence.  'abc' and
** 'ABC' and 'Abc' form different groups */
SELECT count(*) FROM t1 GROUP BY (d || '') ORDER BY 1;
--result 1 1 2

/* Sorting or column c is performed using the RTRIM collating sequence. */
SELECT x FROM t1 ORDER BY c, x;
--result 4 1 2 3

/* Sorting of (c||'') is performed using the BINARY collating sequence. */
SELECT x FROM t1 ORDER BY (c||''), x;
--result 4 2 3 1

/* Sorting of column c is performed using the NOCASE collating sequence. */
SELECT x FROM t1 ORDER BY c COLLATE NOCASE, x;
--result 2 4 3 1


This page last modified on  2022-04-12 00:22:10 UTC


c2: SQLite FTS5 Extension
c5: fts5.html
c4:
1. Overview of FTS5

FTS5 is an SQLite virtual table module that provides 
full-text search
functionality to database applications. In their most elementary form, 
full-text search engines allow the user to efficiently search a large 
collection of documents for the subset that contain one or more instances of a
search term. The search functionality provided to world wide web users by
Google is, among other things, a full-text search
engine, as it allows users to search for all documents on the web that contain,
for example, the term "fts5".

To use FTS5, the user creates an FTS5 virtual table with one or more
columns. For example:

CREATE VIRTUAL TABLE email USING fts5(sender, title, body);


It is an error to add types, constraints or PRIMARY KEY declarations to 
a CREATE VIRTUAL TABLE statement used to create an FTS5 table. Once created,
an FTS5 table may be populated using INSERT, UPDATE or DELETE statements
like any other table. Like any other table with no PRIMARY KEY declaration, an
FTS5 table has an implicit INTEGER PRIMARY KEY field named rowid. 

Not shown in the example above is that there are also 
various options that may be provided to FTS5 as
part of the CREATE VIRTUAL TABLE statement to configure various aspects of the
new table. These may be used to modify the way in which the FTS5 table extracts
terms from documents and queries, to create extra indexes on disk to speed up
prefix queries, or to create an FTS5 table that acts as an index on content
stored elsewhere.

Once populated, there are three ways to execute a full-text query against
the contents of an FTS5 table:

 Using a MATCH operator in the WHERE clause of a SELECT statement, or
     Using an equals ("=") operator in the WHERE clause of a SELECT statement, or
     using the table-valued function syntax.


If using the MATCH or = operators, the expression to the left of the MATCH
   operator is usually the name of the FTS5 table (the exception is when 
   specifying a column-filter). The expression on the right
   must be a text value specifying the term to search for. For the table-valued
   function syntax, the term to search for is specified as the first table argument.
   For example:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). The following three queries are equivalent.
SELECT * FROM email WHERE email MATCH 'fts5';
SELECT * FROM email WHERE email = 'fts5';
SELECT * FROM email('fts5');


 By default, FTS5 full-text searches are case-independent. Like any other
SQL query that does not contain an ORDER BY clause, the example above returns
results in an arbitrary order. To sort results by relevance (most to least
relevant), an ORDER BY may be added to a full-text query as follows:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). Return results in order from best to worst
-- match.  
SELECT * FROM email WHERE email MATCH 'fts5' ORDER BY rank;


 As well as the column values and rowid of a matching row, an application
may use FTS5 auxiliary functions to retrieve extra information regarding
the matched row. For example, an auxiliary function may be used to retrieve
a copy of a column value for a matched row with all instances of the matched
term surrounded by html &lt;b&gt;&lt;/b&gt; tags. Auxiliary functions are
invoked in the same way as SQLite scalar functions, except that the name
of the FTS5 table is specified as the first argument. For example:

-- Query for rows that match "fts5". Return a copy of the "body" column
-- of each row with the matches surrounded by &lt;b&gt;&lt;/b&gt; tags.
SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email('fts5');


A description of the available auxiliary functions, and more details
regarding configuration of the special "rank" column, are 
available below. Custom auxiliary functions may also be implemented in C and registered with
FTS5, just as custom SQL functions may be registered with the SQLite core.

 As well as searching for all rows that contain a term, FTS5 allows 
the user to search for rows that contain:


   any terms that begin with a specified prefix,
   "phrases" - sequences of terms or prefix terms that must feature in a
       document for it to match the query, 
   sets of terms, prefix terms or phrases that appear within a specified
       proximity of each other (these are called "NEAR queries"), or
   boolean combinations of any of the above.


 Such advanced searches are requested by providing a more complicated 
FTS5 query string as the text to the right of the MATCH operator (or =
operator, or as the first argument to a table-valued function syntax). The 
full query syntax is described here.


2. Compiling and Using FTS5

2.1. Building FTS5 as part of SQLite

As of version 3.9.0 (2015-10-14),
FTS5 is included as part of the SQLite amalgamation.
If using one of the two autoconf build system, FTS5 is
enabled by specifying the "--enable-fts5" option when running the configure
script.  (FTS5 is currently disabled by default for the
source-tree configure script and enabled by default for
the amalgamation configure script, but these defaults might
change in the future.)

Or, if sqlite3.c is compiled using some other build system, by arranging for
the SQLITE_ENABLE_FTS5 pre-processor symbol to be defined.

2.2. Building a Loadable Extension

Alternatively, FTS5 may be built as a loadable extension.

The canonical FTS5 source code consists of a series of *.c and other files
in the "ext/fts5" directory of the SQLite source tree. A build process reduces
this to just two files - "fts5.c" and "fts5.h" - which may be used to build an
SQLite loadable extension.


   Obtain the latest SQLite code from fossil.
   Create a Makefile as described in How To Compile SQLite.
   Build the "fts5.c" target. Which also creates fts5.h.


$ wget -c http://www.sqlite.org/src/tarball/SQLite-trunk.tgz?uuid=trunk -O SQLite-trunk.tgz
.... output ...
$ tar -xzf SQLite-trunk.tgz
$ cd SQLite-trunk
$ ./configure && make fts5.c
... lots of output ...
$ ls fts5.&#91;ch]
fts5.c        fts5.h



  The code in "fts5.c" may then be compiled into a loadable extension or
  statically linked into an application as described in 
  Compiling Loadable Extensions. There are two entry points defined, both
  of which do the same thing:


   sqlite3_fts_init
   sqlite3_fts5_init



  The other file, "fts5.h", is not required to compile the FTS5 extension. 
  It is used by applications that implement custom FTS5 tokenizers or auxiliary functions.


3. Full-text Query Syntax


The following block contains a summary of the FTS query syntax in BNF form.
A detailed explanation follows.

&lt;phrase&gt;    := string &#91;*]
&lt;phrase&gt;    := &lt;phrase&gt; + &lt;phrase&gt;
&lt;neargroup&gt; := NEAR ( &lt;phrase&gt; &lt;phrase&gt; ... &#91;, N] )
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &#91;&#94;] &lt;phrase&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &lt;neargroup&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] ( &lt;query&gt; )
&lt;query&gt;     := &lt;query&gt; AND &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; OR &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; NOT &lt;query&gt;
&lt;colspec&gt;   := colname
&lt;colspec&gt;   := { colname1 colname2 ... }



3.1. FTS5 Strings

Within an FTS expression a string may be specified in one of two ways:


   By enclosing it in double quotes ("). Within a string, any embedded
       double quote characters may be escaped SQL-style - by adding a second
       double-quote character.

   As an FTS5 bareword that is not "AND", "OR" or "NOT" (case sensitive). 
       An FTS5 bareword is a string of one or more consecutive characters that
       are all either:
       
       
          Non-ASCII range characters (i.e. unicode codepoints greater 
              than 127), or 
          One of the 52 upper and lower case ASCII characters, or
          One of the 10 decimal digit ASCII characters, or
          The underscore character (unicode codepoint 96).
          The substitute character (unicode codepoint 26).
       

       Strings that include any other characters must be quoted. Characters
       that are not currently allowed in barewords, are not quote characters and
       do not currently serve any special purpose in FTS5 query expressions may
       at some point in the future be allowed in barewords or used to implement
       new query functionality. This means that queries that are currently
       syntax errors because they include such a character outside of a quoted
       string may be interpreted differently by some future version of FTS5.



3.2. FTS5 Phrases

FTS queries are made up of phrases. A phrase is an ordered list of 
one or more tokens. A string is transformed into a phrase by passing it to
the FTS table tokenizer. Two phrases can be concatenated into a single 
large phrase using the "+" operator. For example, assuming the tokenizer
module being used tokenizes the input "one.two.three" to three separate
tokens, the following four queries all specify the same phrase:

... MATCH '"one two three"'
... MATCH 'one + two + three'
... MATCH '"one two" + three'
... MATCH 'one.two.three'



A phrase matches a document if the document contains at least one sub-sequence
of tokens that matches the sequence of tokens that make up the phrase.


3.3. FTS5 Prefix Queries

If a "*" character follows a string within an FTS expression, then the final
token extracted from the string is marked as a prefix token. As you
might expect, a prefix token matches any document token of which it is a 
prefix. For example, the first two queries in the following block will match
any document that contains the token "one" immediately followed by the token
"two" and then any token that begins with "thr".

... MATCH '"one two thr" * '
... MATCH 'one + two + thr*'
... MATCH '"one two thr*"'      -- May not work as expected!


The final query in the block above may not work as expected. Because the
"*" character is inside the double-quotes, it will be passed to the tokenizer,
which will likely discard it (or perhaps, depending on the specific tokenizer
in use, include it as part of the final token) instead of recognizing it as
a special FTS character.




3.4. FTS5 Initial Token Queries

If a "&#94;" character appears immediately before a phrase that is not part of a
NEAR query, then that phrase only matches a document only if it starts at the
first token in a column. The "&#94;" syntax may be combined with a 
column filter, but may not be inserted into the middle of
a phrase.

... MATCH '&#94;one'              -- first token in any column must be "one"
... MATCH '&#94; one + two'       -- phrase "one two" must appear at start of a column
... MATCH '&#94; "one two"'       -- same as previous 
... MATCH 'a : &#94;two'          -- first token of column "a" must be "two"
... MATCH 'NEAR(&#94;one, two)'   -- syntax error! 
... MATCH 'one + &#94;two'        -- syntax error! 
... MATCH '"&#94;one two"'        -- May not work as expected!



3.5. FTS5 NEAR Queries

Two or more phrases may be grouped into a NEAR group. A NEAR group
is specified by the token "NEAR" (case sensitive) followed by an open
parenthesis character, followed by two or more whitespace separated phrases, optionally followed by a comma and the numeric parameter N, followed by
a close parenthesis. For example:

... MATCH 'NEAR("one two" "three four", 10)'
... MATCH 'NEAR("one two" thr* + four)'


If no N parameter is supplied, it defaults to 10. A NEAR group
matches a document if the document contains at least one clump of tokens that: 

 
   contains at least one instance of each phrase, and 
   for which the number of tokens between the end of the first phrase 
       and the beginning of the last phrase in the clump is less than or equal to N.


For example:

CREATE VIRTUAL TABLE f USING fts5(x);
INSERT INTO f(rowid, x) VALUES(1, 'A B C D x x x E F x');

... MATCH 'NEAR(e d, 4)';                      -- Matches!
... MATCH 'NEAR(e d, 3)';                      -- Matches!
... MATCH 'NEAR(e d, 2)';                      -- Does not match!

... MATCH 'NEAR("c d" "e f", 3)';              -- Matches!
... MATCH 'NEAR("c"   "e f", 3)';              -- Does not match!

... MATCH 'NEAR(a d e, 6)';                    -- Matches!
... MATCH 'NEAR(a d e, 5)';                    -- Does not match!

... MATCH 'NEAR("a b c d" "b c" "e f", 4)';    -- Matches!
... MATCH 'NEAR("a b c d" "b c" "e f", 3)';    -- Does not match!



3.6. FTS5 Column Filters


A single phrase or NEAR group may be restricted to matching text within a
specified column of the FTS table by prefixing it with the column name 
followed by a colon character. Or to a set of columns by prefixing it
with a whitespace separated list of column names enclosed in parenthesis
("curly brackets") followed by a colon character. Column names may be specified
using either of the two forms described for strings above. Unlike strings that
are part of phrases, column names are not passed to the tokenizer module.
Column names are case-insensitive in the usual way for SQLite column names -
upper/lower case equivalence is understood for ASCII-range characters only.

... MATCH 'colname : NEAR("one two" "three four", 10)'
... MATCH '"colname" : one + two + three'

... MATCH '{col1 col2} : NEAR("one two" "three four", 10)'
... MATCH '{col2 col1 col3} : one + two + three'



If a column filter specification is preceded by a "-" character, then
it is interpreted as a list of column not to match against. For example:

-- Search for matches in all columns except "colname"
... MATCH '- colname : NEAR("one two" "three four", 10)'

-- Search for matches in all columns except "col1", "col2" and "col3"
... MATCH '- {col2 col1 col3} : one + two + three'



Column filter specifications may also be applied to arbitrary expressions
enclosed in parenthesis. In this case the column filter applies to all 
phrases within the expression. Nested column filter operations may only 
further restrict the subset of columns matched, they can not be used to 
re-enable filtered columns. For example:

-- The following are equivalent:
... MATCH '{a b} : ( {b c} : "hello" AND "world" )'
... MATCH '(b : "hello") AND ({a b} : "world")'



Finally, a column filter for a single column may be specified by using
the column name as the LHS of a MATCH operator (instead of the usual
table name). For example:

-- Given the following table
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);

-- The following are equivalent
SELECT * FROM ft WHERE b MATCH 'uvw AND xyz';
SELECT * FROM ft WHERE ft MATCH 'b : (uvw AND xyz)';

-- This query cannot match any rows (since all columns are filtered out): 
SELECT * FROM ft WHERE b MATCH 'a : xyz';



3.7. FTS5 Boolean Operators


Phrases and NEAR groups may be arranged into expressions using boolean
operators. In order of precedence, from highest (tightest grouping) to
lowest (loosest grouping), the operators are:


  Operator Function

  &lt;query1&gt; NOT &lt;query2&gt; 
      Matches if query1 matches and query2 does not match.

  &lt;query1&gt; AND &lt;query2&gt; 
      Matches if both query1 and query2 match.

  &lt;query1&gt; OR &lt;query2&gt; 
      Matches if either query1 or query2 match.




Parenthesis may be used to group expressions in order to modify operator
precedence in the usual ways. For example:

-- Matches documents that contain at least one instance of either "one"
-- or "two", but do not contain any instances of token "three".
... MATCH 'one OR two NOT three'

-- Match all documents that contain the token "two" but not "three", or
-- contain the token "one".
... MATCH 'one OR (two NOT three)'



Phrases and NEAR groups may also be connected by implicit AND operators.
For simplicity, these are not shown in the BNF grammar above. Essentially, any
sequence of phrases or NEAR groups (including those restricted to matching
specified columns) separated only by whitespace are handled as if there were an
implicit AND operator between each pair of phrases or NEAR groups. Implicit
AND operators are never inserted after or before an expression enclosed in
parenthesis. For example:

... MATCH 'one two three'         -- 'one AND two AND three'
... MATCH 'three "one two"'       -- 'three AND "one two"'
... MATCH 'NEAR(one two) three'   -- 'NEAR(one two) AND three'
... MATCH 'one OR two three'      -- 'one OR two AND three'

... MATCH '(one OR two) three'    -- Syntax error!
... MATCH 'func(one two)'         -- Syntax error!



4. FTS5 Table Creation and Initialization

Each argument specified as part of a "CREATE VIRTUAL TABLE ... USING fts5 
..." statement is either a column declaration or a configuration option. A
column declaration consists of one or more whitespace separated FTS5
barewords or string literals quoted in any manner acceptable to SQLite.

The first string or bareword in a column declaration is the column name. It
is an error to attempt to name an fts5 table column "rowid" or "rank", or to
assign the same name to a column as is used by the table itself. This is not
supported.

Each subsequent string or bareword in a column declaration is a column
option that modifies the behaviour of that column. Column options are
case-independent. Unlike the SQLite core, FTS5 considers unrecognized column
options to be errors. Currently, the only option recognized is 
"UNINDEXED" (see below).

A configuration option consists of an FTS5 bareword - the option name -
followed by an "=" character, followed by the option value. The option value is
specified using either a single FTS5 bareword or a string literal, again quoted
in any manner acceptable to the SQLite core. For example:

CREATE VIRTUAL TABLE mail USING fts5(sender, title, body, tokenize = 'porter ascii');


 There are currently the following configuration options:


   The "tokenize" option, used to configure a custom tokenizer.
   The "prefix" option, used to add prefix indexes
       to an FTS5 table.
   The "content" option, used to make the FTS5 table an 
       external content or contentless table.
   The "content_rowid" option, used to set the rowid field of an 
       external content table.
   The "columnsize" option, used to configure
       whether or not the size in tokens of each value in the FTS5 table is
       stored separately within the database.
   The "detail" option. This option may be used 
       to reduce the size of the FTS index on disk by omitting some information
       from it.  



4.1. The UNINDEXED column option

The contents of columns qualified with the UNINDEXED column option are not
added to the FTS index. This means that for the purposes of MATCH queries and
FTS5 auxiliary functions, the column contains no matchable tokens. 

For example, to avoid adding the contents of the "uuid" field to the FTS
index:
CREATE VIRTUAL TABLE customers USING fts5(name, addr, uuid UNINDEXED);



4.2. Prefix Indexes

 By default, FTS5 maintains a single index recording the location of each
token instance within the document set. This means that querying for complete
tokens is fast, as it requires a single lookup, but querying for a prefix 
token can be slow, as it requires a range scan. For example, to query for
the prefix token "abc*" requires a range scan of all tokens greater than
or equal to "abc" and less than "abd".

 A prefix index is a separate index that records the location of all
instances of prefix tokens of a certain length in characters used to speed
up queries for prefix tokens. For example, optimizing a query for prefix
token "abc*" requires a prefix index of three-character prefixes.

 To add prefix indexes to an FTS5 table, the "prefix" option is set to
either a single positive integer or a text value containing a white-space
separated list of one or more positive integer values. A prefix index is
created for each integer specified. If more than one "prefix" option is
specified as part of a single CREATE VIRTUAL TABLE statement, all apply.

-- Two ways to create an FTS5 table that maintains prefix indexes for
-- two and three character prefix tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix='2 3');
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix=2, prefix=3);



4.3. Tokenizers

 The CREATE VIRTUAL TABLE "tokenize" option is used to configure the
specific tokenizer used by the FTS5 table. The option argument must be either
an FTS5 bareword, or an SQL text literal. The text of the argument is itself
treated as a white-space series of one or more FTS5 barewords or SQL text
literals. The first of these is the name of the tokenizer to use. The second
and subsequent list elements, if they exist, are arguments passed to the
tokenizer implementation.

 Unlike option values and column names, SQL text literals intended as
tokenizers must be quoted using single quote characters. For example:

-- The following are all equivalent
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter ascii');
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "porter ascii");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "'porter' 'ascii'");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '''porter'' ''ascii''');

-- But this will fail:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '"porter" "ascii"');

-- This will fail too:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter' 'ascii');




FTS5 features three built-in tokenizer modules, described in subsequent
sections:


   The unicode61 tokenizer, based on the Unicode 6.1 standard. This
       is the default.

   The ascii tokenizer, which assumes all characters outside of
  the ASCII codepoint range (0-127) are to be treated as token characters.

   The porter tokenizer, which implements the 
porter stemming algorithm.


 It is also possible to create custom tokenizers for FTS5. The API for doing so is described here.

4.3.1. Unicode61 Tokenizer

 The unicode tokenizer classifies all unicode characters as either 
"separator" or "token" characters. By default all space and punctuation
characters, as defined by Unicode 6.1, are considered separators, and all 
other characters as token characters. More specifically, all unicode 
characters assigned to a 

general category beginning with "L" or "N" (letters and numbers,
specifically) or to category "Co" ("other, private use") are considered tokens.
All other characters are separators.
 
Each contiguous run of one or more token characters is considered to be a
token. The tokenizer is case-insensitive according to the rules defined by
Unicode 6.1.

 By default, diacritics are removed from all Latin script characters. This
means, for example, that "A", "a", "&#192;", "&#224;", "&#194;" and "&#226;"
are all considered to be equivalent.

 Any arguments following "unicode61" in the token specification are treated
as a list of alternating option names and values. Unicode61 supports the
following options:


   Option  Usage
   remove_diacritics
  This option should be set to "0", "1" or "2". The default value is "1".
  If it is set to "1" or "2", then diacritics are removed from Latin script
  characters as described above. However, if it is set to "1", then diacritics
  are not removed in the fairly uncommon case where a single unicode codepoint
  is used to represent a character with more that one diacritic. For example,
  diacritics are not removed from codepoint 0x1ED9 ("LATIN SMALL LETTER O WITH
  CIRCUMFLEX AND DOT BELOW"). This is technically a bug, but cannot be fixed
  without creating backwards compatibility problems. If this option is set to
  "2", then diacritics are correctly removed from all Latin characters.

   categories
  This option may be used to modify the set of Unicode general categories
  that are considered to correspond to token characters. The argument must
  consist of a space separated list of two-character general category
  abbreviations (e.g. "Lu" or "Nd"), or of the same with the second character
  replaced with an asterisk ("*"), interpreted as a glob pattern. The default
  value is "L* N* Co".

   tokenchars
   This option is used to specify additional unicode characters that 
  should be considered token characters, even if they are white-space or
  punctuation characters according to Unicode 6.1. All characters in the
  string that this option is set to are considered token characters.

   separators
   This option is used to specify additional unicode characters that 
  should be considered as separator characters, even if they are token
  characters according to Unicode 6.1. All characters in the string that 
  this option is set to are considered separators.


 For example:

-- Create an FTS5 table that does not remove diacritics from Latin
-- script characters, and that considers hyphens and underscore characters
-- to be part of tokens. 
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 remove_diacritics 0 tokenchars '-_'"
);


 or:

-- Create an FTS5 table that, as well as the default token character classes,
-- considers characters in class "Mn" to be token characters.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 categories 'L* N* Co Mn'"
);


 The fts5 unicode61 tokenizer is byte-for-byte compatible with the fts3/4
unicode61 tokenizer.

4.3.2. Ascii Tokenizer

 The Ascii tokenizer is similar to the Unicode61 tokenizer, except that:


   All non-ASCII characters (those with codepoints greater than 127) are
  always considered token characters. If any non-ASCII characters are specified
  as part of the separators option, they are ignored.  

   Case-folding is only performed for ASCII characters. So while "A" and
  "a" are considered to be equivalent, "&#195;" and "&#227;" are distinct.

   The remove_diacritics option is not supported.


 For example:

-- Create an FTS5 table that uses the ascii tokenizer, but does not
-- consider numeric characters to be part of tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "ascii separators '0123456789'"
);


4.3.3. Porter Tokenizer

 The porter tokenizer is a wrapper tokenizer. It takes the output of some
other tokenizer and applies the 
porter stemming algorithm
to each token before it returns it to FTS5. This allows search terms like
"correction" to match similar words such as "corrected" or "correcting". The
porter stemmer algorithm is designed for use with English language terms 
only - using it with other languages may or may not improve search utility.

 By default, the porter tokenizer operates as a wrapper around the default
tokenizer (unicode61). Or, if one or more extra arguments are added to the
"tokenize" option following "porter", they are treated as a specification for
the underlying tokenizer that the porter stemmer uses. For example:

-- Two ways to create an FTS5 table that uses the porter tokenizer to
-- stem the output of the default tokenizer (unicode61). 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = porter); 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61');

-- A porter tokenizer used to stem the output of the unicode61 tokenizer,
-- with diacritics removed before stemming.
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61 remove_diacritics 1');




4.3.4. The Experimental Trigram Tokenizer


The experimental trigram tokenizer extends FTS5 to support substring
matching in general, instead of the usual token matching. When using the
trigram tokenizer, a query or phrase token may match any sequence of characters
within a row, not just a complete token. For example:

CREATE VIRTUAL TABLE tri USING fts5(a, tokenize="trigram");
INSERT INTO tri VALUES('abcdefghij KLMNOPQRST uvwxyz');

-- The following queries all match the single row in the table
SELECT * FROM tri('cdefg');
SELECT * FROM tri('cdefg AND pqr');
SELECT * FROM tri('"hij klm" NOT stuv');



The trigram tokenizer supports a single option - "case_sensitive". With the default 
value, 0, matching is case-insensitive. If this value is set to 1, then all matches
are case-sensitive.

-- A case-sensitive trigram index
CREATE VIRTUAL TABLE tri USING fts5(a, tokenize="trigram case_sensitive 1");



FTS5 tables that use the trigram tokenizer also support indexed GLOB and LIKE
pattern matching. For example:

SELECT * FROM tri WHERE a LIKE '%cdefg%';
SELECT * FROM tri WHERE a GLOB '*ij klm*xyz';



If an FTS5 trigram tokenizer is created with the case_sensitive option set to 1,
it may only index GLOB queries, not LIKE.


Notes:


   Substrings consisting of fewer than 3 unicode characters do not match any
       rows when used with a full-text query. If a LIKE or GLOB pattern does not
       contain at least one sequence of non-wildcard unicode characters, FTS5
       falls back to a linear scan of the entire table.

   If the FTS5 table is created with the detail=none or detail=column option
       specified, full-text queries may not contain any tokens longer than 3
       unicode characters. LIKE and GLOB pattern matching may be slightly slower,
       but still works. If the index is to be used only for LIKE and/or GLOB
       pattern matching, these options are worth experimenting with to reduce
       the index size.



4.4. External Content and Contentless Tables


Normally, when a row is inserted into an FTS5 table, as well as the various
full-text index entries and other data a copy of the row is stored in a private
table managed by the FTS5 module. When column values are requested from the
FTS5 table by the user or by an auxiliary function implementation, they are
read from this private table. The "content" option may be used to create an
FTS5 table that stores only FTS full-text index entries. Because the column
values themselves are usually much larger than the associated full-text index
entries, this can save significant database space.


There are two ways to use the "content" option:

   By setting it to an empty string to create a contentless FTS5 table. In
       this case FTS5 assumes that the original column values are unavailable
       to it when processing queries. Full-text queries and some auxiliary
       functions can still be used, but no column values apart from the rowid
       may be read from the table.

   By setting it to the name of a database object (table, virtual table or
       view) that may be queried by FTS5 at any time to retrieve the column
       values. This is known as an "external content" table. In this case all
       FTS5 functionality may be used, but it is the responsibility of the user
       to ensure that the contents of the full-text index are consistent with
       the named database object. If they are not, query results may be
       unpredictable.  



4.4.1. Contentless Tables

 A contentless FTS5 table is created by setting the "content" option to
an empty string. For example:

CREATE VIRTUAL TABLE f1 USING fts5(a, b, c, content='');


 Contentless FTS5 tables do not support UPDATE or DELETE statements, or
INSERT statements that do not supply a non-NULL value for the rowid field.
Contentless tables do not support REPLACE conflict handling. REPLACE 
and INSERT OR REPLACE statements are treated as regular INSERT statements.
Rows may be deleted from a contentless table using an FTS5 delete command.

 Attempting to read any column value except the rowid from a contentless
FTS5 table returns an SQL NULL value.


4.4.2. External Content Tables

 An external content FTS5 table is created by setting the content 
option to the name of a table, virtual table or view (hereafter the "content
table") within the same database. Whenever column values are required by
FTS5, it queries the content table as follows, with the rowid of the row
for which values are required bound to the SQL variable:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; WHERE &lt;content_rowid&gt; = ?;


 In the above, &lt;content&gt; is replaced by the name of the content table.
By default, &lt;content_rowid&gt; is replaced by the literal text "rowid". Or,
if the "content_rowid" option is set within the CREATE VIRTUAL TABLE statement,
by the value of that option. &lt;cols&gt; is replaced by a comma-separated list
of the FTS5 table column names. For example:

-- If the database schema is: 
CREATE TABLE tbl (a, b, c, d INTEGER PRIMARY KEY);
CREATE VIRTUAL TABLE fts USING fts5(a, c, content=tbl, content_rowid=d);

-- Fts5 may issue queries such as:
SELECT d, a, c FROM tbl WHERE d = ?;


 The content table may also be queried as follows:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; ASC;
SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; DESC;


 It is still the responsibility of the user to ensure that the contents of
an external content FTS5 table are kept up to date with the content table. 
One way to do this is with triggers. For example:

-- Create a table. And an external content fts5 table to index it.
CREATE TABLE tbl(a INTEGER PRIMARY KEY, b, c);
CREATE VIRTUAL TABLE fts_idx USING fts5(b, c, content='tbl', content_rowid='a');

-- Triggers to keep the FTS index up to date.
CREATE TRIGGER tbl_ai AFTER INSERT ON tbl BEGIN
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;
CREATE TRIGGER tbl_ad AFTER DELETE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
END;
CREATE TRIGGER tbl_au AFTER UPDATE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;


 Like contentless tables, external content tables do not support REPLACE
conflict handling. Any operations that specify REPLACE conflict handling are
handled using ABORT.


4.5. The Columnsize Option

Normally, FTS5 maintains a special backing table within the database that
stores the size of each column value in tokens inserted into the main FTS5
table in a separate table. This backing table is used by the
xColumnSize API function, which is in turn used by
the built-in bm25 ranking function (and is likely to be useful 
to other ranking functions as well).

In order to save space, this backing table may be omitted by setting the
columnsize option to zero. For example:

-- A table without the xColumnSize() values stored on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=0);

-- Three equivalent ways of creating a table that does store the
-- xColumnSize() values on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=1);
CREATE VIRTUAL TABLE ft USING fts5(a, b, columnsize='1', c);


 It is an error to set the columnsize option to any value other than
0 or 1.

 If an FTS5 table is configured with columnsize=0 but is not a
contentless table, the xColumnSize API function
still works, but runs much more slowly. In this case, instead of reading
the value to return directly from the database, it reads the text value
itself and count the tokens within it on demand.

Or, if the table is also a contentless table,
then the following apply:


   The xColumnSize API always returns -1. There is no way to determine 
       the number of tokens in a value stored within a contentless FTS5 table
       configured with columnsize=0.

   Each inserted row must be accompanied by an explicitly specified rowid
       value. If a contentless table is configured with columnsize=0,
       attempting to insert a NULL value into the rowid is an SQLITE_MISMATCH
       error.

   All queries on the table must be full-text queries. In other words,
       they must use the MATCH or = operator with the table-name column as the
       left-hand operand, or else use the table-valued function syntax. Any
       query that is not a full-text query results in an error.


 The name of the table in which the xColumnSize values are stored
(unless columnsize=0 is specified) is "&lt;name&gt;_docsize", where 
&lt;name&gt; is the name of the FTS5 table itself. The 
sqlite3_analyzer
tool may be used on an existing database in order to determine how much
space might be saved by recreating an FTS5 table using columnsize=0.


4.6. The Detail Option

 For each term in a document, the FTS index maintained by FTS5 
stores the rowid of the document, the column number of the column that contains
the term and the offset of the term within the column value. The "detail"
option may be used to omit some of this information. This reduces the space
that the index consumes within the database file, but also reduces the
capability and efficiency of the system.

 The detail option may be set to "full" (the default value), "column" or
"none". For example:

-- The following two lines are equivalent (because the default value
-- of "detail" is "full". 
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c, detail=full);

CREATE VIRTUAL TABLE ft2 USING fts5(a, b, c, detail=column);
CREATE VIRTUAL TABLE ft3 USING fts5(a, b, c, detail=none);


If the detail option is set to column, then for each term the FTS
index records the rowid and column number only, omitting the term offset
information. This results in the following restrictions:


   NEAR queries are not available.
   Phrase queries are not available.
   Assuming the table is not also a 
  contentless table, the 
  xInstCount, xInst, 
  xPhraseFirst and xPhraseNext
  are slower than usual. This is because instead of reading the required data
  directly from the FTS index they have to load and tokenize the document text 
  on demand.
   If the table is also a contentless table, the xInstCount, xInst, 
  xPhraseFirst and xPhraseNext APIs behave as if the current row contains no
  phrase matches at all (i.e. xInstCount() returns 0).

  
If the detail option is set to none, then for each term the FTS
index records just the rowid is stored. Both column and offset information
are omitted. As well as the restrictions itemized above for detail=column
mode, this imposes the following extra limitations:


   Column filter queries are not available.
   Assuming the table is not also a contentless table, the 
  xPhraseFirstColumn and 
  xPhraseNextColumn are slower than usual. 

   If the table is also a contentless table, the xPhraseFirstColumn and
  xPhraseNextColumn APIs behave as if the current row contains no phrase
  matches at all (i.e. xPhraseFirstColumn() sets the iterator to EOF).


 In one test that indexed a large set of emails (1636 MiB on disk), the FTS
index was 743 MiB on disk with detail=full, 340 MiB with detail=column and 134
MiB with detail=none.


5.  Auxiliary Functions 

 Auxiliary functions are similar to SQL scalar functions,
except that they may only be used within full-text queries (those that use
the MATCH operator) on an FTS5 table. Their results are calculated based not
only on the arguments passed to them, but also on the current match and 
matched row. For example, an auxiliary function may return a numeric value
indicating the accuracy of the match (see the bm25() function), 
or a fragment of text from the matched row that contains one or more 
instances of the search terms (see the snippet() function).

To invoke an auxiliary function, the name of the FTS5 table should be
specified as the first argument. Other arguments may follow the first,
depending on the specific auxiliary function being invoked. For example, to
invoke the "highlight" function:

SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email WHERE email MATCH 'fts5'


The built-in auxiliary functions provided as part of FTS5 are described in
the following section. Applications may also implement 
custom auxiliary functions in C.

5.1. Built-in Auxiliary Functions

 FTS5 provides three built-in auxiliary functions:


   The bm25() auxiliary function returns a real value
       reflecting the accuracy of the current match. Better matches are
       assigned numerically lower values.

   The highlight() auxiliary function returns a copy
       of the text from one of the columns of the current match with each
       instance of a queried term within the result surrounded by specified
       markup (for example "&lt;b&gt;" and "&lt;/b&gt;").

   The snippet() auxiliary function selects a short
       fragment of text from one of the columns of the matched row and returns
       it with each instance of a queried term surrounded by markup in
       the same manner as the highlight() function. The fragment of text is
       selected so as to maximize the number of queried terms it contains.



5.1.1. The bm25() function

 The built-in auxiliary function bm25() returns a real value indicating
how well the current row matches the full-text query. The better the match,
the numerically smaller the value returned. A query such as the following may
be used to return matches in order from best to worst match:

SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts)


 In order to calculate a documents score, the full-text query is separated
    into its component phrases. The bm25 score for document D and 
    query Q is then calculated as follows:

 

 In the above, nPhrase is the number of phrases in the query.
    |D| is the number of tokens in the current document, and
    avgdl is the average number of tokens in all documents within the
    FTS5 table.  k1 and b are both constants,
    hard-coded at 1.2 and 0.75 respectively.

 The "-1" term at the start of the formula is not found in most
implementations of the BM25 algorithm. Without it, a better match is assigned
a numerically higher BM25 score. Since the default sorting order is
"ascending", this means that appending "ORDER BY bm25(fts)" to a query would
cause results to be returned in order from worst to best. The "DESC" keyword
would be required in order to return the best matches first. In order to
avoid this pitfall, the FTS5 implementation of BM25 multiplies the result
by -1 before returning it, ensuring that better matches are assigned
numerically lower scores.

 IDF(qi) is the inverse-document-frequency of query 
    phrase i. It is calculated as follows, where N is the total
    number of rows in the FTS5 table and n(qi) is the total
    number of rows that contain at least one instance of phrase i:

 

 Finally, f(qi,D) is the phrase frequency of phrase 
i. By default, this is simply the number of occurrences of the phrase
within the current row. However, by passing extra real value arguments to 
the bm25() SQL function, each column of the table may be assigned a different
weight and the phrase frequency calculated as follows:

 

 where wc is the weight assigned to column c and
n(qi,c) is the number of occurrences of phrase i in
column c of the current row. The first argument passed to bm25()
following the table name is the weight assigned to the leftmost column of
the FTS5 table. The second is the weight assigned to the second leftmost
column, and so on. If there are not enough arguments for all table columns,
remaining columns are assigned a weight of 1.0. If there are too many 
trailing arguments, the extras are ignored. For example:

-- Assuming the following schema:
CREATE VIRTUAL TABLE email USING fts5(sender, title, body);

-- Return results in bm25 order, with each phrase hit in the "sender"
-- column considered the equal of 10 hits in the "body" column, and
-- each hit in the "title" column considered as valuable as 5 hits in
-- the "body" column.
SELECT * FROM email WHERE email MATCH ? ORDER BY bm25(email, 10.0, 5.0);


Refer to wikipedia for 
more information regarding
BM25 and its variants.


5.1.2. The highlight() function

 The highlight() function returns a copy of the text from a specified 
column of the current row with extra markup text inserted to mark the start 
and end of phrase matches. 

The highlight() must be invoked with exactly three arguments following 
the table name. To be interpreted as follows:


   An integer indicating the index of the FTS table column to read the 
       text from. Columns are numbered from left to right starting at zero.

   The text to insert before each phrase match.

   The text to insert after each phrase match.


For example:

-- Return a copy of the text from the leftmost column of the current
-- row, with phrase matches marked using html "b" tags.
SELECT highlight(fts, 0, '&lt;b&gt;', '&lt;/b&gt;') FROM fts WHERE fts MATCH ?


In cases where two or more phrase instances overlap (share one or more
tokens in common), a single open and close marker is inserted for each set
of overlapping phrases. For example:

-- Assuming this:
CREATE VIRTUAL TABLE ft USING fts5(a);
INSERT INTO ft VALUES('a b c x c d e');
INSERT INTO ft VALUES('a b c c d e');
INSERT INTO ft VALUES('a b c d e');

-- The following SELECT statement returns these three rows:
--   '&#91;a b c&#93; x &#91;c d e&#93;'
--   '&#91;a b c&#93; &#91;c d e&#93;'
--   '&#91;a b c d e&#93;'
SELECT highlight(ft, 0, '&#91;', '&#93;') FROM ft WHERE ft MATCH 'a+b+c AND c+d+e';



5.1.3. The snippet() function

The snippet() function is similar to highlight(), except that instead of
returning entire column values, it automatically selects and extracts a
short fragment of document text to process and return. The snippet() function
must be passed five parameters following the table name argument:


   An integer indicating the index of the FTS table column to select
       the returned text from. Columns are numbered from left to right 
       starting at zero. A negative value indicates that the column should
       be automatically selected.

   The text to insert before each phrase match within the returned text.

   The text to insert after each phrase match within the returned text.

   The text to add to the start or end of the selected text to indicate
       that the returned text does not occur at the start or end of its column,
       respectively.

   The maximum number of tokens in the returned text. This must be greater
       than zero and equal to or less than 64. 



5.2. Sorting by Auxiliary Function Results

 All FTS5 tables feature a special hidden column named "rank". If the
current query is not a full-text query (i.e. if it does not include a MATCH
operator), the value of the "rank" column is always NULL. Otherwise, in a
full-text query, column rank contains by default the same value as would be
returned by executing the bm25() auxiliary function with no trailing 
arguments.

 The difference between reading from the rank column and using the bm25()
function directly within the query is only significant when sorting by the
returned value. In this case, using "rank" is faster than using bm25().

-- The following queries are logically equivalent. But the second may
-- be faster, particularly if the caller abandons the query before
-- all rows have been returned (or if the queries were modified to 
-- include LIMIT clauses).
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts);
SELECT * FROM fts WHERE fts MATCH ? ORDER BY rank;


 Instead of using bm25() with no trailing arguments, the specific auxiliary
function mapped to the rank column may be configured either on a per-query
basis, or by setting a different persistent default for the FTS table.

 In order to change the mapping of the rank column for a single query, 
a term similar to either of the following is added to the WHERE clause of a 
query:

rank MATCH 'auxiliary-function-name(arg1, arg2, ...)'
rank = 'auxiliary-function-name(arg1, arg2, ...)'


 The right-hand-side of the MATCH or = operator must be a constant
expression that evaluates to a string consisting of the auxiliary function to
invoke, followed by zero or more comma separated arguments within parenthesis.
Arguments must be SQL literals. For example:

-- The following queries are logically equivalent. But the second may
-- be faster. See above. 
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts, 10.0, 5.0);
SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;


 The table-valued function syntax may also be used to specify an alternative
ranking function. In this case the text describing the ranking function should
be specified as the second table-valued function argument. The following three
queries are equivalent:

SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts = ? AND rank = 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts(?, 'bm25(10.0, 5.0)') ORDER BY rank;


 The default mapping of the rank column for a table may be modified 
using the FTS5 rank configuration option.

6. Special INSERT Commands


6.1. The 'automerge' Configuration Option


  Instead of using a single data structure on disk to store the full-text
  index, FTS5 uses a series of b-trees. Each time a new transaction is
  committed, a new b-tree containing the contents of the committed transaction
  is written into the database file. When the full-text index is queried, each
  b-tree must be queried individually and the results merged before being
  returned to the user.


  In order to prevent the number of b-trees in the database from becoming too
  large (slowing down queries), smaller b-trees are periodically merged into
  single larger b-trees containing the same data. By default, this happens
  automatically within INSERT, UPDATE or DELETE statements that modify the
  full-text index. The 'automerge' parameter determines how many smaller
  b-trees are merged together at a time. Setting it to a small value can
  speed up queries (as they have to query and merge the results from fewer 
  b-trees), but can also slow down writing to the database (as each INSERT,
  UPDATE or DELETE statement has to do more work as part of the automatic
  merging process).


  Each of the b-trees that make up the full-text index is assigned to a "level"
  based on its size. Level-0 b-trees are the smallest, as they contain the
  contents of a single transaction. Higher level b-trees are the result of
  merging two or more level-0 b-trees together and so they are larger. FTS5
  begins to merge b-trees together once there exist M or more b-trees 
  with the same level, where M is the value of the 'automerge' 
  parameter.


  The maximum allowed value for the 'automerge' parameter is 16. The default
  value is 4. Setting the 'automerge' parameter to 0 disables the automatic 
  incremental merging of b-trees altogether.

INSERT INTO ft(ft, rank) VALUES('automerge', 8);


6.2. The 'crisismerge' Configuration Option

The 'crisismerge' option is similar to 'automerge', in that it determines
how and how often the component b-trees that make up the full-text index are
merged together. Once there exist C or more b-trees on a single level
within the full-text index, where C is the value of the 'crisismerge'
option, all b-trees on the level are immediately merged into a single b-tree.

The difference between this option and the 'automerge' option is that when
the 'automerge' limit is reached FTS5 only begins to merge the b-trees
together. Most of the work is performed as part of subsequent INSERT, 
UPDATE or DELETE operations. Whereas when the 'crisismerge' limit is reached,
the offending b-trees are all merged immediately. This means that an INSERT,
UPDATE or DELETE that triggers a crisis-merge may take a long time to 
complete.

The default 'crisismerge' value is 16. There is no maximum limit. Attempting
to set the 'crisismerge' parameter to a value of 0 or 1 is equivalent to
setting it to the default value (16). It is an error to attempt to set the
'crisismerge' option to a negative value.

INSERT INTO ft(ft, rank) VALUES('crisismerge', 16);



6.3. The 'delete' Command

 This command is only available with external content and contentless tables. It
is used to delete the index entries associated with a single row from the
full-text index. This command and the delete-all
command are the only ways to remove entries from the full-text index of a
contentless table.

 In order to use this command to delete a row, the text value 'delete' 
must be inserted into the special column with the same name as the table.
The rowid of the row to delete is inserted into the rowid column. The
values inserted into the other columns must match the values currently
stored in the table. For example:

-- Insert a row with rowid=14 into the fts5 table.
INSERT INTO ft(rowid, a, b, c) VALUES(14, $a, $b, $c);

-- Remove the same row from the fts5 table.
INSERT INTO ft(ft, rowid, a, b, c) VALUES('delete', 14, $a, $b, $c);


 If the values "inserted" into the text columns as part of a 'delete'
command are not the same as those currently stored within the table, the
results may be unpredictable.

 The reason for this is easy to understand: When a document is inserted
into the FTS5 table, an entry is added to the full-text index to record the
position of each token within the new document. When a document is removed,
the original data is required in order to determine the set of entries that
need to be removed from the full-text index. So if the data supplied to FTS5
when a row is deleted using this command is different from that used to
determine the set of token instances when it was inserted, some full-text 
index entries may not be correctly deleted, or FTS5 may try to remove index 
entries that do not exist. This can leave the full-text index in an
unpredictable state, making future query results unreliable.


6.4. The 'delete-all' Command

 This command is only available with external content and contentless tables. It
deletes all entries from the full-text index.

INSERT INTO ft(ft) VALUES('delete-all');


6.5. The 'integrity-check' Command

 This command is used to verify that the full-text index is internally
consistent, and, optionally, that it is consistent with any 
external content table.

The integrity-check command is invoked by inserting the text value
'integrity-check' into the special column with the same name as the FTS5
table. If a value is supplied for the "rank" column, it must be either
0 or 1. For example:

INSERT INTO ft(ft) VALUES('integrity-check');
INSERT INTO ft(ft, rank) VALUES('integrity-check', 0);
INSERT INTO ft(ft, rank) VALUES('integrity-check', 1);


The three forms above are equivalent for all FTS tables that are
not external content tables. They check that the index data structures are 
not corrupt, and, if the FTS table is not contentless, that the contents of
the index match the contents of the table itself.

For an external content table, the contents of the index are only
compared to the contents of the external content table if the value 
specified for the rank column is 1.

In all cases, if any discrepancies are found, the command fails 
with an SQLITE_CORRUPT_VTAB error.


6.6. The 'merge' Command

INSERT INTO ft(ft, rank) VALUES('merge', 500);


 This command merges b-tree structures together until roughly N pages
of merged data have been written to the database, where N is the absolute
value of the parameter specified as part of the 'merge' command. The size of
each page is as configured by the FTS5 pgsz option.

 If the parameter is a positive value, B-tree structures are only eligible
for merging if one of the following is true:


   There are U or more such b-trees on a
       single level (see the documentation for the FTS5 automerge option
       for an explanation of b-tree levels), where U is the value assigned
       to the FTS5 usermerge option option.
   A merge has already been started (perhaps by a 'merge' command that
       specified a negative parameter).


 It is possible to tell whether or not the 'merge' command found any 
b-trees to merge together by checking the value returned by the
sqlite3_total_changes() API before and after the command is executed. If
the difference between the two values is 2 or greater, then work was performed.
If the difference is less than 2, then the 'merge' command was a no-op. In this
case there is no reason to execute the same 'merge' command again, at least
until after the FTS table is next updated.

 If the parameter is negative, and there are B-tree structures on more than
one level within the FTS index, all B-tree structures are assigned to the same
level before the merge operation is commenced. Additionally, if the parameter
is negative, the value of the usermerge configuration option is not 
respected - as few as two b-trees from the same level may be merged together.

 The above means that executing the 'merge' command with a negative
parameter until the before and after difference in the return value of
sqlite3_total_changes() is less than two optimizes the FTS index in the
same way as the FTS5 optimize command. However, if a new b-tree is added
to the FTS index while this process is ongoing, FTS5 will move the new 
b-tree to the same level as the existing b-trees and restart the merge. To
avoid this, only the first call to 'merge' should specify a negative parameter.
Each subsequent call to 'merge' should specify a positive value so that the
merge started by the first call is run to completion even if new b-trees are
added to the FTS index.


6.7. The 'optimize' Command

This command merges all individual b-trees that currently make up the
full-text index into a single large b-tree structure. This ensures that the
full-text index consumes the minimum space within the database and is in the
fastest form to query.

Refer to the documentation for the FTS5 automerge option for more details
regarding the relationship between the full-text index and its component
b-trees.

INSERT INTO ft(ft) VALUES('optimize');


Because it reorganizes the entire FTS index, the optimize command can 
take a long time to run. The FTS5 merge command can be used to divide
the work of optimizing the FTS index into multiple steps. To do this:


   Invoke the 'merge' command once with the parameter set to -N, then
   Invoke the 'merge' command zero or more times with the parameter set to N.


where N is the number of pages of data to merge within each invocation of
the merge command. The application should stop invoking merge when the
difference in the value returned by the sqlite3_total_changes() function before
and after the merge command drops to below two. The merge commands may be
issued as part of the same or separate transactions, and by the same or
different database clients. Refer to the documentation for the 
merge command for further details.


6.8. The 'pgsz' Configuration Option

 This command is used to set the persistent "pgsz" option.

 The full-text index maintained by FTS5 is stored as a series of fixed-size
blobs in a database table. It is not strictly necessary for all blobs that make
up a full-text index to be the same size. The pgsz option determines the size
of all blobs created by subsequent index writers. The default value is 1000.

INSERT INTO ft(ft, rank) VALUES('pgsz', 4072);



6.9. The 'rank' Configuration Option

 This command is used to set the persistent "rank" option.

 The rank option is used to change the default auxiliary function mapping
for the rank column. The option should be set to a text value in the same
format as described for "rank MATCH ?" terms 
above. For example:

INSERT INTO ft(ft, rank) VALUES('rank', 'bm25(10.0, 5.0)');



6.10. The 'rebuild' Command

 This command first deletes the entire full-text index, then rebuilds it
based on the contents of the table or content
table.  It is not available with contentless
tables.

INSERT INTO ft(ft) VALUES('rebuild');



6.11. The 'usermerge' Configuration Option

 This command is used to set the persistent "usermerge" option.

 The usermerge option is similar to the automerge and crisismerge options.
It is the minimum number of b-tree segments that will be merged together by
a 'merge' command with a positive parameter. For example:

INSERT INTO ft(ft, rank) VALUES('usermerge', 4);


 The default value of the usermerge option is 4. The minimum allowed value
is 2, and the maximum 16.


7. Extending FTS5

FTS5 features APIs allowing it to be extended by:


   Adding new auxiliary functions implemented in C, and
   Adding new tokenizers, also implemented in C.


 The built-in tokenizers and auxiliary functions described in this
document are all implemented using the publicly available API described
below.

 Before a new auxiliary function or tokenizer implementation may be 
registered with FTS5, an application must obtain a pointer to the "fts5_api"
structure. There is one fts5_api structure for each database connection with
which the FTS5 extension is registered. To obtain the pointer, the application
invokes the SQL user-defined function fts5() with a single argument.  That
argument must be set to a pointer to a pointer to an fts5_api object
using the sqlite3_bind_pointer() interface.
The following example code demonstrates the technique:

/*
** Return a pointer to the fts5_api pointer for database connection db.
** If an error occurs, return NULL and leave an error in the database 
** handle (accessible using sqlite3_errcode()/errmsg()).
*/
fts5_api *fts5_api_from_db(sqlite3 *db){
  fts5_api *pRet = 0;
  sqlite3_stmt *pStmt = 0;

  if( SQLITE_OK==sqlite3_prepare(db, "SELECT fts5(?1)", -1, &pStmt, 0) ){
    sqlite3_bin_pointer(pStmt, (void*)&pRet, "fts5_api_ptr", NULL);
    sqlite3_step(pStmt);
  }
  sqlite3_finalize(pStmt);
  return pRet;
}


Backwards Compatibility Warning:
Prior to SQLite version 3.20.0 (2017-08-01), the fts5() worked slightly
differently.  Older applications that extend FTS5 must be revised to use 
the new technique shown above.

 The fts5_api structure is defined as follows. It exposes three methods, 
one each for registering new auxiliary functions and tokenizers, and one for
retrieving existing tokenizer. The latter is intended to facilitate the
implementation of "tokenizer wrappers" similar to the built-in
porter tokenizer.

typedef struct fts5_api fts5_api;
struct fts5_api {
  int iVersion;                   /* Currently always set to 2 */

  /* Create a new tokenizer */
  int (*xCreateTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_tokenizer *pTokenizer,
    void (*xDestroy)(void*)
  );

  /* Find an existing tokenizer */
  int (*xFindTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void **ppContext,
    fts5_tokenizer *pTokenizer
  );

  /* Create a new auxiliary function */
  int (*xCreateFunction)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_extension_function xFunction,
    void (*xDestroy)(void*)
  );
};


 To invoke a method of the fts5_api object, the fts5_api pointer itself
should be passed as the methods first argument followed by the other, method
specific, arguments. For example:

rc = pFts5Api->xCreateTokenizer(pFts5Api, ... other args ...);


 The fts5_api structure methods are described individually in the following
sections.


7.1. Custom Tokenizers

 To create a custom tokenizer, an application must implement three
functions: a tokenizer constructor (xCreate), a destructor (xDelete) and a
function to do the actual tokenization (xTokenize). The type of each
function is as for the member variables of the fts5_tokenizer struct:

typedef struct Fts5Tokenizer Fts5Tokenizer;
typedef struct fts5_tokenizer fts5_tokenizer;
struct fts5_tokenizer {
  int (*xCreate)(void*, const char **azArg, int nArg, Fts5Tokenizer **ppOut);
  void (*xDelete)(Fts5Tokenizer*);
  int (*xTokenize)(Fts5Tokenizer*, 
      void *pCtx,
      int flags,            /* Mask of FTS5_TOKENIZE_* flags */
      const char *pText, int nText, 
      int (*xToken)(
        void *pCtx,         /* Copy of 2nd argument to xTokenize() */
        int tflags,         /* Mask of FTS5_TOKEN_* flags */
        const char *pToken, /* Pointer to buffer containing token */
        int nToken,         /* Size of token in bytes */
        int iStart,         /* Byte offset of token within input text */
        int iEnd            /* Byte offset of end of token within input text */
      )
  );
};

/* Flags that may be passed as the third argument to xTokenize() */
#define FTS5_TOKENIZE_QUERY     0x0001
#define FTS5_TOKENIZE_PREFIX    0x0002
#define FTS5_TOKENIZE_DOCUMENT  0x0004
#define FTS5_TOKENIZE_AUX       0x0008

/* Flags that may be passed by the tokenizer implementation back to FTS5
** as the third argument to the supplied xToken callback. */
#define FTS5_TOKEN_COLOCATED    0x0001      /* Same position as prev. token */


 The implementation is registered with the FTS5 module by calling the
xCreateTokenizer() method of the fts5_api object. If there is already a
tokenizer with the same name, it is replaced.  If a non-NULL xDestroy parameter
is passed to xCreateTokenizer(), it is invoked with a copy of the pContext
pointer passed as the only argument when the database handle is closed or when
the tokenizer is replaced.

 If successful, xCreateTokenizer() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 When an FTS5 table uses the custom tokenizer, the FTS5 core calls xCreate()
once to create a tokenizer, then xTokenize() zero or more times to tokenize
strings, then xDelete() to free any resources allocated by xCreate(). More
specifically:


xCreate:
   This function is used to allocate and initialize a tokenizer instance.
   A tokenizer instance is required to actually tokenize text.

   The first argument passed to this function is a copy of the (void*)
   pointer provided by the application when the fts5_tokenizer object
   was registered with FTS5 (the third argument to xCreateTokenizer()). 
   The second and third arguments are an array of nul-terminated strings
   containing the tokenizer arguments, if any, specified following the
   tokenizer name as part of the CREATE VIRTUAL TABLE statement used
   to create the FTS5 table.

   The final argument is an output variable. If successful, (*ppOut) 
   should be set to point to the new tokenizer handle and SQLITE_OK
   returned. If an error occurs, some value other than SQLITE_OK should
   be returned. In this case, fts5 assumes that the final value of *ppOut 
   is undefined.

 xDelete:
   This function is invoked to delete a tokenizer handle previously
   allocated using xCreate(). Fts5 guarantees that this function will
   be invoked exactly once for each successful call to xCreate().

 xTokenize:
   This function is expected to tokenize the nText byte string indicated 
   by argument pText. pText may or may not be nul-terminated. The first
   argument passed to this function is a pointer to an Fts5Tokenizer object
   returned by an earlier call to xCreate().

   The second argument indicates the reason that FTS5 is requesting
   tokenization of the supplied text. This is always one of the following
   four values:

    FTS5_TOKENIZE_DOCUMENT - A document is being inserted into
            or removed from the FTS table. The tokenizer is being invoked to
            determine the set of tokens to add to (or delete from) the
            FTS index.

        FTS5_TOKENIZE_QUERY - A MATCH query is being executed 
            against the FTS index. The tokenizer is being called to tokenize 
            a bareword or quoted string specified as part of the query.

        (FTS5_TOKENIZE_QUERY | FTS5_TOKENIZE_PREFIX) - Same as
            FTS5_TOKENIZE_QUERY, except that the bareword or quoted string is
            followed by a "*" character, indicating that the last token
            returned by the tokenizer will be treated as a token prefix.

        FTS5_TOKENIZE_AUX - The tokenizer is being invoked to 
            satisfy an fts5_api.xTokenize() request made by an auxiliary
            function. Or an fts5_api.xColumnSize() request made by the same
            on a columnsize=0 database.  
   

   For each token in the input string, the supplied callback xToken() must
   be invoked. The first argument to it should be a copy of the pointer
   passed as the second argument to xTokenize(). The third and fourth
   arguments are a pointer to a buffer containing the token text, and the
   size of the token in bytes. The 4th and 5th arguments are the byte offsets
   of the first byte of and first byte immediately following the text from
   which the token is derived within the input.

   The second argument passed to the xToken() callback ("tflags") should
   normally be set to 0. The exception is if the tokenizer supports 
   synonyms. In this case see the discussion below for details.

   FTS5 assumes the xToken() callback is invoked for each token in the 
   order that they occur within the input text.

   If an xToken() callback returns any value other than SQLITE_OK, then
   the tokenization should be abandoned and the xTokenize() method should
   immediately return a copy of the xToken() return value. Or, if the
   input buffer is exhausted, xTokenize() should return SQLITE_OK. Finally,
   if an error occurs with the xTokenize() implementation itself, it
   may abandon the tokenization and return any error code other than
   SQLITE_OK or SQLITE_DONE.

7.1.1. Synonym Support

   Custom tokenizers may also support synonyms. Consider a case in which a
   user wishes to query for a phrase such as "first place". Using the 
   built-in tokenizers, the FTS5 query 'first + place' will match instances
   of "first place" within the document set, but not alternative forms
   such as "1st place". In some applications, it would be better to match
   all instances of "first place" or "1st place" regardless of which form
   the user specified in the MATCH query text.

   There are several ways to approach this in FTS5:

    By mapping all synonyms to a single token. In this case, using
            the above example, this means that the tokenizer returns the
            same token for inputs "first" and "1st". Say that token is in
            fact "first", so that when the user inserts the document "I won
            1st place" entries are added to the index for tokens "i", "won",
            "first" and "place". If the user then queries for '1st + place',
            the tokenizer substitutes "first" for "1st" and the query works
            as expected.

        By querying the index for all synonyms of each query term
            separately. In this case, when tokenizing query text, the
            tokenizer may provide multiple synonyms for a single term 
            within the document. FTS5 then queries the index for each 
            synonym individually. For example, faced with the query:

   ... MATCH 'first place'


            the tokenizer offers both "1st" and "first" as synonyms for the
            first token in the MATCH query and FTS5 effectively runs a query 
            similar to:

   ... MATCH '(first OR 1st) place'


            except that, for the purposes of auxiliary functions, the query
            still appears to contain just two phrases - "(first OR 1st)" 
            being treated as a single phrase.

        By adding multiple synonyms for a single term to the FTS index.
            Using this method, when tokenizing document text, the tokenizer
            provides multiple synonyms for each token. So that when a 
            document such as "I won first place" is tokenized, entries are
            added to the FTS index for "i", "won", "first", "1st" and
            "place".

            This way, even if the tokenizer does not provide synonyms
            when tokenizing query text (it should not - to do so would be
            inefficient), it doesn't matter if the user queries for 
            'first + place' or '1st + place', as there are entries in the
            FTS index corresponding to both forms of the first token.
   

   Whether it is parsing document or query text, any call to xToken that
   specifies a tflags argument with the FTS5_TOKEN_COLOCATED bit
   is considered to supply a synonym for the previous token. For example,
   when parsing the document "I won first place", a tokenizer that supports
   synonyms would call xToken() 5 times, as follows:

   xToken(pCtx, 0, "i",                      1,  0,  1);
xToken(pCtx, 0, "won",                    3,  2,  5);
xToken(pCtx, 0, "first",                  5,  6, 11);
xToken(pCtx, FTS5_TOKEN_COLOCATED, "1st", 3,  6, 11);
xToken(pCtx, 0, "place",                  5, 12, 17);


   It is an error to specify the FTS5_TOKEN_COLOCATED flag the first time
   xToken() is called. Multiple synonyms may be specified for a single token
   by making multiple calls to xToken(FTS5_TOKEN_COLOCATED) in sequence. 
   There is no limit to the number of synonyms that may be provided for a
   single token.

   In many cases, method (1) above is the best approach. It does not add 
   extra data to the FTS index or require FTS5 to query for multiple terms,
   so it is efficient in terms of disk space and query speed. However, it
   does not support prefix queries very well. If, as suggested above, the
   token "first" is substituted for "1st" by the tokenizer, then the query:

   ... MATCH '1s*'


   will not match documents that contain the token "1st" (as the tokenizer
   will probably not map "1s" to any prefix of "first").

   For full prefix support, method (3) may be preferred. In this case, 
   because the index contains entries for both "first" and "1st", prefix
   queries such as 'fi*' or '1s*' will match correctly. However, because
   extra entries are added to the FTS index, this method uses more space
   within the database.

   Method (2) offers a midpoint between (1) and (3). Using this method,
   a query such as '1s*' will match documents that contain the literal 
   token "1st", but not "first" (assuming the tokenizer is not able to
   provide synonyms for prefixes). However, a non-prefix query like '1st'
   will match against "1st" and "first". This method does not require
   extra disk space, as no extra entries are added to the FTS index. 
   On the other hand, it may require more CPU cycles to run MATCH queries,
   as separate queries of the FTS index are required for each synonym.

   When using methods (2) or (3), it is important that the tokenizer only
   provide synonyms when tokenizing document text (method (2)) or query
   text (method (3)), not both. Doing so will not cause any errors, but is
   inefficient.




7.2. Custom Auxiliary Functions

 Implementing a custom auxiliary function is similar to implementing a
scalar SQL function. The implementation
should be a C function of type fts5_extension_function, defined as follows:

typedef struct Fts5ExtensionApi Fts5ExtensionApi;
typedef struct Fts5Context Fts5Context;
typedef struct Fts5PhraseIter Fts5PhraseIter;

typedef void (*fts5_extension_function)(
  const Fts5ExtensionApi *pApi,   /* API offered by current FTS version */
  Fts5Context *pFts,              /* First arg to pass to pApi functions */
  sqlite3_context *pCtx,          /* Context for returning result/error */
  int nVal,                       /* Number of values in apVal[] array */
  sqlite3_value **apVal           /* Array of trailing arguments */
);


 The implementation is registered with the FTS5 module by calling the
xCreateFunction() method of the fts5_api object. If there is already an
auxiliary function with the same name, it is replaced by the new function.
If a non-NULL xDestroy parameter is passed to xCreateFunction(), it is invoked
with a copy of the pContext pointer passed as the only argument when the
database handle is closed or when the registered auxiliary function is
replaced.

 If successful, xCreateFunction() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 The final three arguments passed to the auxiliary function callback are
similar to the three arguments passed to the implementation of a scalar SQL
function. All arguments except the first passed to the auxiliary function are
available to the implementation in the apVal&#91;&#93; array. The
implementation should return a result or error via the content handle pCtx.

 The first argument passed to an auxiliary function callback is a pointer
to a structure containing methods that may be invoked in order to obtain
information regarding the current query or row. The second argument is an
opaque handle that should be passed as the first argument to any such method 
invocation. For example, the following auxiliary function definition returns
the total number of tokens in all columns of the current row:

/*
** Implementation of an auxiliary function that returns the number
** of tokens in the current row (including all columns).
*/
static void column_size_imp(
  const Fts5ExtensionApi *pApi,
  Fts5Context *pFts,
  sqlite3_context *pCtx,
  int nVal,
  sqlite3_value **apVal
){
  int rc;
  int nToken;
  rc = pApi->xColumnSize(pFts, -1, &nToken);
  if( rc==SQLITE_OK ){
    sqlite3_result_int(pCtx, nToken);
  }else{
    sqlite3_result_error_code(pCtx, rc);
  }
}


The following section describes the API offered to auxiliary function
implementations in detail. Further examples may be found in the "fts5_aux.c"
file of the source code.


7.2.1. 
  Custom Auxiliary Functions API Reference


struct Fts5ExtensionApi {
  int iVersion;                   /* Currently always set to 3 */

  void *(*xUserData)(Fts5Context*);

  int (*xColumnCount)(Fts5Context*);
  int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow);
  int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken);

  int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, /* Text to tokenize */
    void *pCtx,                   /* Context passed to xToken() */
    int (*xToken)(void*, int, const char*, int, int, int)       /* Callback */
  );

  int (*xPhraseCount)(Fts5Context*);
  int (*xPhraseSize)(Fts5Context*, int iPhrase);

  int (*xInstCount)(Fts5Context*, int *pnInst);
  int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff);

  sqlite3_int64 (*xRowid)(Fts5Context*);
  int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn);
  int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken);

  int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
  );
  int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*));
  void *(*xGetAuxdata)(Fts5Context*, int bClear);

  int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*);
  void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff);

  int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*);
  void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol);
};




void *(*xUserData)(Fts5Context*)

Return a copy of the context pointer the extension function was 
   registered with.



int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the FTS5 table. Or, if iCol is
   non-negative but less than the number of columns in the table, return
   the total number of tokens in column iCol, considering all rows in 
   the FTS5 table.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



int (*xColumnCount)(Fts5Context*)

Return the number of columns in the table.



int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the current row. Or, if iCol is
   non-negative but less than the number of columns in the table, set
   *pnToken to the number of tokens in column iCol of the current row.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



   This function may be quite inefficient if used with an FTS5 table
   created with the "columnsize=0" option.



int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn)

This function attempts to retrieve the text of column iCol of the
   current document. If successful, (*pz) is set to point to a buffer
   containing the text in utf-8 encoding, (*pn) is set to the size in bytes
   (not characters) of the buffer and SQLITE_OK is returned. Otherwise,
   if an error occurs, an SQLite error code is returned and the final values
   of (*pz) and (*pn) are undefined.



int (*xPhraseCount)(Fts5Context*)

Returns the number of phrases in the current query expression.



int (*xPhraseSize)(Fts5Context*, int iPhrase)

Returns the number of tokens in phrase iPhrase of the query. Phrases
   are numbered starting from zero.



int (*xInstCount)(Fts5Context*, int *pnInst)

Set *pnInst to the total number of occurrences of all phrases within
   the query within the current row. Return SQLITE_OK if successful, or
   an error code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always returns 0.



int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff)

Query for the details of phrase match iIdx within the current row.
   Phrase matches are numbered starting from zero, so the iIdx argument
   should be greater than or equal to zero and smaller than the value
   output by xInstCount().



   Usually, output parameter *piPhrase is set to the phrase number, *piCol
   to the column in which it occurs and *piOff the token offset of the
   first token of the phrase. Returns SQLITE_OK if successful, or an error
   code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option.



sqlite3_int64 (*xRowid)(Fts5Context*)

Returns the rowid of the current row.



int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, 
    void *pCtx,                   
    int (*xToken)(void*, int, const char*, int, int, int)       
)

Tokenize text using the tokenizer belonging to the FTS5 table.



int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
)

This API function is used to query the FTS table for phrase iPhrase
   of the current query. Specifically, a query equivalent to:


... FROM ftstable WHERE ftstable MATCH $p ORDER BY rowid



   with $p set to a phrase equivalent to the phrase iPhrase of the
   current query is executed. Any column filter that applies to
   phrase iPhrase of the current query is included in $p. For each 
   row visited, the callback function passed as the fourth argument 
   is invoked. The context and API objects passed to the callback 
   function may be used to access the properties of each matched row.
   Invoking Api.xUserData() returns a copy of the pointer passed as 
   the third argument to pUserData.



   If the callback function returns any value other than SQLITE_OK, the
   query is abandoned and the xQueryPhrase function returns immediately.
   If the returned value is SQLITE_DONE, xQueryPhrase returns SQLITE_OK.
   Otherwise, the error code is propagated upwards.



   If the query runs to completion without incident, SQLITE_OK is returned.
   Or, if some error occurs before the query completes or is aborted by
   the callback, an SQLite error code is returned.



int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*))

Save the pointer passed as the second argument as the extension function's 
   "auxiliary data". The pointer may then be retrieved by the current or any
   future invocation of the same fts5 extension function made as part of
   the same MATCH query using the xGetAuxdata() API.



   Each extension function is allocated a single auxiliary data slot for
   each FTS query (MATCH expression). If the extension function is invoked 
   more than once for a single FTS query, then all invocations share a 
   single auxiliary data context.



   If there is already an auxiliary data pointer when this function is
   invoked, then it is replaced by the new pointer. If an xDelete callback
   was specified along with the original pointer, it is invoked at this
   point.



   The xDelete callback, if one is specified, is also invoked on the
   auxiliary data pointer after the FTS5 query has finished.



   If an error (e.g. an OOM condition) occurs within this function,
   the auxiliary data is set to NULL and an error code returned. If the
   xDelete parameter was not NULL, it is invoked on the auxiliary data
   pointer before returning.



void *(*xGetAuxdata)(Fts5Context*, int bClear)

Returns the current auxiliary data pointer for the fts5 extension 
   function. See the xSetAuxdata() method for details.



   If the bClear argument is non-zero, then the auxiliary data is cleared
   (set to NULL) before this function returns. In this case the xDelete,
   if any, is not invoked.



int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow)

This function is used to retrieve the total number of rows in the table.
   In other words, the same value that would be returned by:


SELECT count(*) FROM ftstable;



int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*)

This function is used, along with type Fts5PhraseIter and the xPhraseNext
   method, to iterate through all instances of a single query phrase within
   the current row. This is the same information as is accessible via the
   xInstCount/xInst APIs. While the xInstCount/xInst APIs are more convenient
   to use, this API may be faster under some circumstances. To iterate 
   through instances of phrase iPhrase, use the following code:


Fts5PhraseIter iter;
int iCol, iOff;
for(pApi->xPhraseFirst(pFts, iPhrase, &iter, &iCol, &iOff);
    iCol>=0;
    pApi->xPhraseNext(pFts, &iter, &iCol, &iOff)
){
  // An instance of phrase iPhrase at offset iOff of column iCol
}



   The Fts5PhraseIter structure is defined above. Applications should not
   modify this structure directly - it should only be used as shown above
   with the xPhraseFirst() and xPhraseNext() API methods (and by
   xPhraseFirstColumn() and xPhraseNextColumn() as illustrated below).



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always iterates
   through an empty set (all calls to xPhraseFirst() set iCol to -1).



void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff)

See xPhraseFirst above.



int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*)

This function and xPhraseNextColumn() are similar to the xPhraseFirst()
   and xPhraseNext() APIs described above. The difference is that instead
   of iterating through all instances of a phrase in the current row, these
   APIs are used to iterate through the set of columns in the current row
   that contain one or more instances of a specified phrase. For example:


Fts5PhraseIter iter;
int iCol;
for(pApi->xPhraseFirstColumn(pFts, iPhrase, &iter, &iCol);
    iCol>=0;
    pApi->xPhraseNextColumn(pFts, &iter, &iCol)
){
  // Column iCol contains at least one instance of phrase iPhrase
}



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" option. If the FTS5 table is created with either 
   "detail=none" "content=" option (i.e. if it is a contentless table), 
   then this API always iterates through an empty set (all calls to 
   xPhraseFirstColumn() set iCol to -1).



   The information accessed using this API and its companion
   xPhraseFirstColumn() may also be obtained using xPhraseFirst/xPhraseNext
   (or xInst/xInstCount). The chief advantage of this API is that it is
   significantly more efficient than those alternatives when used with
   "detail=column" tables.



void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol)

See xPhraseFirstColumn above.






8. The fts5vocab Virtual Table Module

 The fts5vocab virtual table module allows users to extract information from
an FTS5 full-text index directly. The fts5vocab module is a part of FTS5 - it 
is available whenever FTS5 is.

 Each fts5vocab table is associated with a single FTS5 table. An fts5vocab
table is usually created by specifying two arguments in place of column names
in the CREATE VIRTUAL TABLE statement - the name of the associated FTS5 table
and the type of fts5vocab table. Currently there are three types of fts5vocab
table; "row", "col" and "instance". Unless the fts5vocab table is created
within the "temp" database, it must be part of the same database as the
associated FTS5 table.

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1".
CREATE VIRTUAL TABLE ft1_v USING fts5vocab('ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2".
CREATE VIRTUAL TABLE ft2_v USING fts5vocab(ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index
-- belonging to FTS5 table "ft3".
CREATE VIRTUAL TABLE ft3_v USING fts5vocab(ft3, instance);


 If an fts5vocab table is created in the temp database, it may be associated
with an FTS5 table in any attached database. In order to attach the fts5vocab
table to an FTS5 table located in a database other than "temp", the name of the
database is inserted before the FTS5 table name in the CREATE VIRTUAL TABLE 
arguments. For example:

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1" in database "main".
CREATE VIRTUAL TABLE temp.ft1_v USING fts5vocab(main, 'ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2" in attached database "aux".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index 
-- belonging to FTS5 table "ft3" in attached database "other".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft3, 'instance');


 Specifying three arguments when creating an fts5vocab table in any database
other than "temp" results in an error.

 An fts5vocab table of type "row" contains one row for each distinct term
in the associated FTS5 table. The table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  doc  The number of rows that contain at least one instance of the term.
  cnt  The total number of instances of the term in the entire FTS5 table.


 An fts5vocab table of type "col" contains one row for each distinct term/column
combination in the associated FTS5 table. Table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  col  The name of the FTS5 table column that contains the term.
  doc  The number of rows in the FTS5 table for which column $col
                   contains at least one instance of the term.
  cnt  The total number of instances of the term that appear in
                   column $col of the FTS5 table (considering all rows). 


 An fts5vocab table of type "instance" contains one row for each term
instance stored in the associated FTS index. Assuming the FTS5 table is
created with the 'detail' option set to 'full', table columns are as follows:


  ColumnContents
  term   The term, as stored in the FTS5 index.
  doc    The rowid of the document that contains the term instance.
  col    The name of the column that contains the term instance.
  offset The index of the term instance within its column. Terms 
                     are numbered in order of occurrence starting from 0.


 If the FTS5 table is created with the 'detail' option set to 'col', then
the offset column of an instance virtual table always contains NULL.
In this case there is one row in the table for each unique term/doc/col 
combination. Or, if the FTS5 table is created with 'detail' set to 'none',
then both offset and col always contain NULL values. For
detail=none FTS5 tables, there is one row in the fts5vocab table for each
unique term/doc combination.

Example:

-- Assuming a database created using:
CREATE VIRTUAL TABLE ft1 USING fts5(c1, c2);
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');

-- Then querying the following fts5vocab table (type "col") returns:
--
--    apple  | c1 | 1 | 1
--    banana | c1 | 1 | 1
--    banana | c2 | 1 | 2
--    cherry | c1 | 2 | 4
--    cherry | c2 | 1 | 1
--    date   | c3 | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_col USING fts5vocab(ft1, col);

-- Querying an fts5vocab table of type "row" returns:
--
--    apple  | 1 | 1
--    banana | 1 | 3
--    cherry | 2 | 5
--    date   | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_row USING fts5vocab(ft1, row);

-- And, for type "instance"
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');
--
--    apple  | 1 | c1 | 0
--    banana | 1 | c1 | 1
--    banana | 1 | c2 | 0
--    banana | 1 | c2 | 1
--    cherry | 1 | c1 | 2
--    cherry | 1 | c2 | 2
--    cherry | 2 | c1 | 0
--    cherry | 2 | c1 | 1
--    cherry | 2 | c1 | 2
--    date   | 2 | c2 | 0
--    date   | 2 | c2 | 1
--    date   | 2 | c2 | 2
--
CREATE VIRTUAL TABLE ft1_v_instance USING fts5vocab(ft1, instance);




9. FTS5 Data Structures

This section describes at a high-level the way the FTS module stores its
index and content in the database. It is not necessary to read or understand
the material in this section in order to use FTS in an application. However, it
may be useful to application developers attempting to analyze and understand
FTS performance characteristics, or to developers contemplating enhancements to
the existing FTS feature set.


When an FTS5 virtual table is created in a database, between 3 and 5 real
tables are created in the database. These are known as "shadow tables", and
are used by the virtual table module to store persistent data. They should not
be accessed directly by the user. Many other virtual table modules, including
FTS3 and rtree, also create and use shadow tables.

FTS5 creates the following shadow tables. In each case the actual table name
is based on the name of the FTS5 virtual table (in the following, replace
% with the name of the virtual table to find the actual shadow table name).

-- This table contains most of the full-text index data. 
CREATE TABLE %_data(id INTEGER PRIMARY KEY, block BLOB);

-- This table contains the remainder of the full-text index data. 
-- It is almost always much smaller than the %_data table. 
CREATE TABLE %_idx(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;

-- Contains the values of persistent configuration parameters.
CREATE TABLE %_config(k PRIMARY KEY, v) WITHOUT ROWID;

-- Contains the size of each column of each row in the virtual table
-- in tokens. This shadow table is not present if the "columnsize"
-- option is set to 0.
CREATE TABLE %_docsize(id INTEGER PRIMARY KEY, sz BLOB);

-- Contains the actual data inserted into the FTS5 table. There
-- is one "cN" column for each indexed column in the FTS5 table.
-- This shadow table is not present for contentless or external 
-- content FTS5 tables. 
CREATE TABLE %_content(id INTEGER PRIMARY KEY, c0, c1...);


The following sections describe in more detail how these five tables are
used to store FTS5 data.

9.1. Varint Format

The sections below refer to 64-bit signed integers stored in "varint" form.
FTS5 uses the same varint format as used in various places by the SQLite core.

A varint is between 1 and 9 bytes in length. The varint consists of either
zero or more bytes which have the high-order bit set followed by a single byte
with the high-order bit clear, or nine bytes, whichever is shorter. The lower
seven bits of each of the first eight bytes and all 8 bits of the ninth byte
are used to reconstruct the 64-bit twos-complement integer. Varints are
big-endian: bits taken from the earlier byte of the varint are more significant
than bits taken from the later bytes.

9.2. The FTS Index (%_idx and %_data tables)

The FTS index is an ordered key-value store where the keys are document
terms or term prefixes and the associated values are "doclists". A doclist is a
packed array of varints that encodes the position of each instance of the term
within the FTS5 table. The position of a single term instance is defined as the
combination of:


   The rowid of the FTS5 table row it appears in,
   The index of the column the term instance appears in (columns are
       numbered from left to right starting from zero), and
   The offset of the term within the column value (i.e. the number of 
       tokens that appear within the column value before this one).


The FTS index contains up to (nPrefix+1) entries for each token in the 
data set, where nPrefix is the number of defined 
prefix indexes.

Keys associated with the main FTS index (the one that is not a prefix
index) are prefixed with the character "0". Keys for the first prefix 
index are prefixed with "1". Keys for the second prefix index are
prefixed with "2", and so on. For example, if the token "document" is 
inserted into an FTS5 table with prefix indexes
specified by prefix="2 4", then the keys added to the FTS index would be 
"0document", "1do" and "2docu".

The FTS index entries are not stored in a single tree or hash table
structure. Instead, they are stored in a series of immutable b-tree like
structures referred to as "segment b-trees". Each time a write to the FTS5
table is committed, one or more (but usually just one) new segment b-trees
are added containing both the new entries and tombstones for any deleted
entries. When the FTS index is queried, the reader queries each segment 
b-tree in turn and merges the results, giving priority to newer data.

Each segment b-tree is assigned a numerical level. When a new segment 
b-tree is written to the database as part of committing a transaction,
it is assigned to level 0. Segment b-trees belonging to a single level are
periodically merged together to create a single, larger segment b-tree
that is assigned to the next level (i.e. level 0 segment b-trees are 
merged to become a single level 1 segment b-tree). Thus the numerically
larger levels contain older data in (usually) larger segment b-trees. 
Refer to the 
'automerge',
'crisismerge' and
'usermerge' options, along
with the 
'merge' and
'optimize' commands for details on how to
control the merging.

In cases where the doclist associated with a term or term prefix is very
large, there may be an associated doclist
index. A doclist index is similar to the set of internal nodes of a b-tree.
It allows a large doclist to be efficiently queried for rowids or ranges of
rowids. For example, when processing a query like:

SELECT ... FROM fts_table('term') WHERE rowid BETWEEN ? AND ?


FTS5 uses the segment b-tree index to locate the doclist for term "term",
then uses its doclist index (assuming it is present) to efficiently identify
the subset of matches with rowids in the required range.

9.2.1. The %_data Table Rowid Space

CREATE TABLE %_data(
  id INTEGER PRIMARY KEY, 
  block BLOB
);


The %_data table is used to store three types of records:


   The special structure record, 
       stored with id=1.
   The special averages record, 
       stored with id=10.
   A record to store each segment b-tree
       leaf and doclist index leaf and
       internal node. See below for how id values are calculated for these
       records.


Each segment b-tree in the system is assigned a unique 16-bit segment id.
Segment ids may only be reused after the original owner segment b-tree is
completely merged into a higher level segment b-tree. Within a segment b-tree,
each leaf page is assigned a unique page number - 1 for the first leaf page, 2
for the second, and so on.

Each doclist index leaf page is also assigned a page number. The first
(leftmost) leaf page in a doclist index is assigned the same page number as 
the segment b-tree leaf page on which its term appears (because doclist indexes
are only created for terms with very long doclists, at most one term per
segment b-tree leaf has an associated doclist index). Call this page number P.
If the doclist is so large that it requires a second leaf, the second leaf is
assigned page number P+1. The third leaf P+2. Each tier of a doclist index
b-tree (leaves, parents of leaves, grandparents etc.) is assigned page numbers
in this fashion, starting with page number P.

The "id" value used in the %_data table to store any given segment b-tree
leaf or doclist index leaf or node is composed as follows:


Rowid&nbsp;Bits Contents
38..43  (16 bit) Segment b-tree id value.
37  (1 bit) Doclist index flag. Set for doclist index pages, clear
                for segment b-tree leaves.
32..36  (5 bits) Height in tree. This is set to 0 for segment b-tree
                    and doclist index leaves, to 1 for the parents of doclist
                    index leaves, 2 for the grandparents, etc.
0..31  (32 bits) Page number


9.2.2. Structure Record Format

The structure record identifies the set of segment b-trees that make up the 
current FTS index, along with details of any ongoing incremental merge
operations. It is stored in the %_data table with id=1.

A structure record begins with a single 32-bit unsigned value - the cookie
value.  This value is incremented each time the structure is modified.
Following the cookie value are three varint values, as follows:


   The number of levels in the index (i.e. the maximum level associated
       with any segment b-tree plus one).
   The total number of segment b-trees in the index.
   The total number of segment b-tree leaves written to level 0 trees
       since the FTS5 table was created.


Then, for each level from 0 to nLevel:


   The number of input segments from the previous level being used as
       inputs for the current incremental merge, or zero if there is no
       ongoing incremental merge to create a new segment b-tree for this level.
   The total number of segment b-trees on the level.
   Then, for each segment b-tree, from oldest to newest:
  
     The segment id.
     Page number of first leaf (often 1, always >0).
     Page number of last leaf (always >0).
  


9.2.3. Averages Record Format

The averages record, which is always stored with id=10 in the %_data table,
does not store the average of anything. Instead, it contains a vector of
(nCol+1) packed varint values, where nCol is the number of columns in the FTS5
table, including unindexed columns. The first varint contains the total 
number of rows in the FTS5 table. The second contains the total number of
tokens in all values stored in the leftmost FTS5 table column. The third the
number of tokens in all values for the next leftmost, and so on. The value for
unindexed columns is always zero.

9.2.4. Segment B-Tree Format

9.2.4.1. The Key/Doclist Format

The key/doclist format is a format used to store a series of keys (document
terms or term prefixes prefixed by a single character to indentify the specific
index to which they belong) in sorted order, each with their associated
doclist. The format consists of alternating keys and doclists packed together.

The first key is stored as:

   A varint indicating the number of bytes in the key (N), followed by
   The key data itself (N bytes).


Each subsequent key is stored as:

   A varint indicating the size of the prefix that the key has in common
       with the previous key in bytes,
   A varint indicating the number of bytes in the key following the
       common prefix (N), followed by
   The key suffix data itself (N bytes).


For example, if the first two keys in an FTS5 key/doclist record are
"0challenger" and "0chandelier", then the first key is stored as varint 11
followed by the 11 bytes "0challenger", and the second key is stored as varints
4 and 7, followed by the 7 bytes "ndelier".



Figure 1 - Term/Doclist Format


Each doclist identifies the rows (by their rowid values) that contain at
least one instance of the term or term prefix and an associated position list,
or "poslist" enumerating the position of each term instance within the row. In
this sense a "position" is defined as a column number and term offset within
the column value. 

Within a doclist, documents are always stored in order sorted by rowid.  The
first rowid in a doclist is stored as is, as a varint. It is immediately
followed by its associated position list. Following this, the difference
between the first rowid and the second, as a varint, followed by the doclist
associated with the second rowid in the doclist. And so on.

There is no way to determine the size of a doclist by parsing it. This must
be stored externally. See the section below for
details of how this is accomplished in FTS5.



Figure 2 - Doclist Format


A position list - often shortened to "poslist" - identifies the column
and token offset within the row of each instance of the token in question.
The format of a poslist is:


   Varint set to twice the size of the poslist, not including this field,
       plus one if the "delete" flag is set on the entry.
   A (possibly empty) list of offsets for column 0 (the leftmost column) of
       the row. Each offset is stored as a varint. The first varint contains
       the value of the first offset, plus 2. The second variant contains the
       difference between the second and first offsets, plus 2. etc. For
       example, if the offset list is to contain offsets 0, 10, 15 and 16, it
       is encoded by packing the following values, encoded as varints, end to
       end:
       
           2, 12, 7, 3

   For each column other than column 0 that contains one of more instances
       of the token:
  
     Byte value 0x01.
     The column number, as a varint.
     An offset list, in the same format as the offset list for column 0.
  




Figure 3 - Position List (poslist) With Offsets in Columns 0 and i


9.2.4.2. Pagination

If it is small enough (by default this means smaller than 4000 bytes), the
entire contents of a segment b-tree may be stored in the key/doclist format
described in the previous section as a single blob within the %_data table.
Otherwise, the key/doclist is split into pages (by default, of approximately
4000 bytes each) and stored in a contiguous set of entries in the %_data table
(see above for details).

When a key/doclist is divided into pages, the following modifications are
made to the format:


   A single varint or key data field never spans two pages.
   The first key on each page is not prefix-compressed. It is stored in
       the format described above for the first key of a doclist - its size as
       a varint followed by the key data.
   If there are one or more rowids on a page before the first key, then
       the first of them is not delta compressed. It is stored as is, just as
       if it were the first rowid of its doclist (which it may or may not be).


Each page also has fixed-size 4-byte header and a variably-sized footer.
The header is divided into 2 16-bit big-endian integer fields. They
contain:


   The byte offset of the first rowid value on the page, if it occurs
       before the first key, or 0 otherwise.
   The byte offset of the page footer.


The page footer consists of a series of varints containing the byte offset
of each key that appears on the page. The page footer is zero bytes in size
if there are no keys on the page.



Figure 4 - Page Format


9.2.4.3. Segment Index Format

The result of formatting the contents of the segment b-tree in the 
key/doclist format and then splitting it into pages is something very
similar to the leaves of a b+tree. Instead of creating a format for
the internal nodes of this b+tree and storing them in the %_data table
alongside the leaves, the keys that would have been stored on such nodes are
added to the %_idx table, defined as:

CREATE TABLE %_idx(
  segid INTEGER,              -- segment id
  term TEXT,                  -- prefix of first key on page
  pgno INTEGER,               -- (2*pgno + bDoclistIndex)
  PRIMARY KEY(segid, term)
);


For each "leaf" page that contains at least one key, an entry is added
to the %_idx table. Fields are set as follows:


  ColumnContents
  segid   The integer segment id.
  term    The smallest prefix of the first key on the page that
                      is larger than all keys on the previous page. For the
                      first page in a segment, this prefix is zero bytes in
                      size.
  pgno    This field encodes both the page number (within the
                      segment - starting from 1) and the doclist index flag.
                      The doclist index flag is set if the final key on the
                      page has an associated 
                      doclist index. The value of this field is:
                      
       (pgno*2 + bDoclistIndexFlag)



Then, to find the leaf for segment i that may contain term t, instead of
searching through internal nodes, FTS5 runs the query:

SELECT pgno FROM %_idx WHERE segid=$i AND term>=$t ORDER BY term LIMIT 1


9.2.4.4. Doclist Index Format

The segment index described in the previous
section allows a segment b-tree to be efficiently queried by term or,
assuming there is a prefix index of the required size, a term prefix. The data
structure described in this section, doclist indexes, allows FTS5 to
efficiently search for a rowid or range or rowids within the doclist associated
with a single term or term prefix.

Not all keys have associated doclists indexes. By default, a doclist index
is only added for a key if its doclist spans more than 4 segment b-tree leaf
pages. Doclist indexes are themselves b-trees, with both leaves and internal
nodes stored as entries in the %_data table, but in practice most doclists are
small enough to fit on a single leaf. FTS5 uses the same rough size for doclist
index node and leaves as it does for segment b-tree leaves (by default 4000
bytes).

Doclist index leaves and internal nodes use the same page format. The first
byte is a "flags" byte. This is set to 0x00 for the root page of the doclist
index b-tree, and 0x01 for all other pages. The remainder of the page is a
series of tightly packed varints, as follows:


   page number of leftmost child page, followed by
   the smallest rowid value on the left most child page, followed by
   one varint for each subsequent child page, containing the value:
  
     0x00 if there are no rowids on the child page (this can only happen
         when the "child" page is actually a segment b-tree leaf), or
     the difference between the smallest rowid on the child page and
         the previous rowid value stored on the doclist index page.
  


For the leftmost doclist index leaf in a doclist index, the leftmost child
page is the first segment b-tree leaf after the one that contains the key
itself.

9.3. Document Sizes Table (%_docsize table)

CREATE TABLE %_docsize(
    id INTEGER PRIMARY KEY,   -- id of FTS5 row this record pertains to
    sz BLOB                   -- blob containing nCol packed varints
);


Many common search result ranking functions require as an input the size 
in tokens of the result document (as a search term hit in a short document is
considered more significant than one in a long document). To provide fast
access to this information, for each row in the FTS5 table there exists a 
corresponding record (with the same rowid) in the %_docsize shadow table
that contains the size of each column value in the row, in tokens.

The column value sizes are stored in a blob containing one packed varint for
each column of the FTS5 table, from left to right. The varint contains, of
course, the total number of tokens in the corresponding column value. Unindexed
columns are included in this vector of varints; for them the value is always
set to zero.

This table is used by the xColumnSize API. It can
be omitted altogether by specifying the 
columnsize=0 option. In that case the
xColumnSize API is still available to auxiliary functions, but runs much more
slowly.

9.4. The Table Contents (%_content table)

CREATE TABLE %_content(id INTEGER PRIMARY KEY, c0, c1...);


The actual table content - the values inserted into the FTS5 table, is
stored in the %_content table. This table is created with one "c*" column for
each column of the FTS5 table, including any unindexed columns. The values for
the leftmost FTS5 table column are stored in column "c0" of the %_content
table, the values from the next FTS5 table column in column "c1", and so on.

This table is omitted completely for 
external content or contentless FTS5 tables.
tables.

9.5. Configuration Options (%_config table)

CREATE TABLE %_config(k PRIMARY KEY, v) WITHOUT ROWID;


This table stores the values of any persistent configuration options. 
Column "k" stores the name of the option (text) and column "v" the value.
Example contents:

sqlite> SELECT * FROM fts_tbl_config;
┌─────────────┬──────┐
│      k      │  v   │
├─────────────┼──────┤
│ crisismerge │ 8    │
│ pgsz        │ 8000 │
│ usermerge   │ 4    │
│ version     │ 4    │
└─────────────┴──────┘




  Appendix A: Comparison with FTS3/4


 Also available is the similar but more mature FTS3/4 module. 
FTS5 is a new version of FTS4 that includes various fixes and solutions for 
problems that could not be fixed in FTS4 without sacrificing backwards 
compatibility. Some of these problems are 
described below.

 Application Porting Guide 

 In order to use FTS5 instead of FTS3 or FTS4, applications usually require
minimal modifications. Most of these fall into three categories - changes
required to the CREATE VIRTUAL TABLE statement used to create the FTS table,
changes required to SELECT queries used to execute queries against the table,
and changes required to applications that use FTS auxiliary functions.

 Changes to CREATE VIRTUAL TABLE statements 


 The module name must be changed from "fts3" or "fts4" to "fts5".

 All type information or constraint specifications must be removed from
     column definitions. FTS3/4 ignores everything following the column name in
     a column definition, FTS5 attempts to parse it (and will report an error
     if it fails to).

 The "matchinfo=fts3" option is not available. The 
     "columnsize=0" option is equivalent.

 The notindexed= option is not available. Adding UNINDEXED
     to the column definition is equivalent.

 The ICU tokenizer is not available.

 The compress=, uncompress= and languageid= options are not available.
     There is as of yet no equivalent for their functionality.


 -- FTS3/4 statement 
CREATE VIRTUAL TABLE t1 USING fts4(
  linkid INTEGER,
  header CHAR(20),
  text VARCHAR,
  notindexed=linkid,
  matchinfo=fts3,
  tokenizer=unicode61
);

 -- FTS5 equivalent (note - the "tokenizer=unicode61" option is not
 -- required as this is the default for FTS5 anyway)
CREATE VIRTUAL TABLE t1 USING fts5(
  linkid UNINDEXED,
  header,
  text,
  columnsize=0
);


 Changes to SELECT statements 


   The "docid" alias does not exist. Applications must use "rowid"
          instead.

   The behaviour of queries when a column-filter is specified both as
          part of the FTS query and by using a column as the LHS of a MATCH
          operator is slightly different. For a table with columns "a" and "b"
          and a query similar to:
... a MATCH 'b: string'

       FTS3/4 searches for matches in column "b". However, FTS5 always
          returns zero rows, as results are first filtered for column "b", then
          for column "a", leaving no results. In other words, in FTS3/4 the
          inner filter overrides the outer, in FTS5 both filters are applied.

   The FTS query syntax (right hand side of the MATCH operator) has
          changed in some ways. The FTS5 syntax is quite close to the FTS4
          "enhanced syntax". The main difference is that FTS5 is fussier 
          about unrecognized punctuation characters and similar within query
          strings. Most queries that work with FTS3/4 should also work with
          FTS5, and those that do not should return parse errors.


 Auxiliary Function Changes 

 FTS5 has no matchinfo() or offsets() function, and the snippet() function
is not as fully-featured as in FTS3/4. However, since FTS5 does provide 
an API allowing applications to create custom auxiliary functions, any
required functionality may be implemented within the application code.

 The set of built-in auxiliary functions provided by FTS5 may be 
improved upon in the future.

 Other Issues


   The functionality provided by the fts4aux module is now provided
          by fts5vocab. The schema of these two tables is slightly different.

   The FTS3/4 "merge=X,Y" command has been replaced by the 
          FTS5 merge command.

   The FTS3/4 "automerge=X" command has been replaced by the 
          FTS5 automerge option.



 
  Summary of Technical Differences 


FTS5 is similar to FTS3/4 in that the primary task of each is to maintain
an index mapping from each unique token to a list of instances of that token 
within a set of documents, where each instance is identified by the document 
in which it appears and its position within that document. For example:

-- Given the following SQL:
CREATE VIRTUAL TABLE ft USING fts5(a, b);
INSERT INTO ft(rowid, a, b) VALUES(1, 'X Y', 'Y Z');
INSERT INTO ft(rowid, a, b) VALUES(2, 'A Z', 'Y Y');

-- The FTS5 module creates the following mapping on disk:
A --&gt; (2, 0, 0)
X --&gt; (1, 0, 0)
Y --&gt; (1, 0, 1) (1, 1, 0) (2, 1, 0) (2, 1, 1)
Z --&gt; (1, 1, 1) (2, 0, 1)


In the example above, each triple identifies the location of a token
instance by rowid, column number (columns are numbered sequentially
starting at 0 from left to right) and position within the column value (the
first token in a column value is 0, the second is 1, and so on). Using this
index, FTS5 is able to provide timely answers to queries such as "the set
of all documents that contain the token 'A'", or "the set of all documents
that contain the sequence 'Y Z'". The list of instances associated with a
single token is called an "instance-list".

The principle difference between FTS3/4 and FTS5 is that in FTS3/4,
each instance-list is stored as a single large database record, whereas
in FTS5 large instance-lists are divided between multiple database records.
This has the following implications for dealing with large databases that
contain large lists:


   FTS5 is able to load instance-lists into memory incrementally in
       order to reduce memory usage and peak allocation size. FTS3/4 very
       often loads entire instance-lists into memory.

   When processing queries that feature more than one token, FTS5 is
       sometimes able to determine that the query can be answered by
       inspecting a subset of a large instance-list. FTS3/4 almost always
       has to traverse entire instance-lists.

   If an instance-list grows so large that it exceeds
       the SQLITE_MAX_LENGTH limit, FTS3/4 is unable to handle it. FTS5
       does not have this problem. 


For these reasons, many complex queries may use less memory and run faster 
using FTS5.

Some other ways in which FTS5 differs from FTS3/4 are:


   FTS5 supports "ORDER BY rank" for returning results in order of
       decreasing relevancy.

   FTS5 features an API allowing users to create custom auxiliary 
       functions for advanced ranking and text processing applications. The
       special "rank" column may be mapped to a custom auxiliary function
       so that adding "ORDER BY rank" to a query works as expected.

   FTS5 recognizes unicode separator characters and case equivalence by
       default. This is also possible using FTS3/4, but must be explicitly
       enabled.

   The query syntax has been revised where necessary to remove
       ambiguities and to make it possible to escape special characters
       in query terms.

   By default, FTS3/4 occasionally merges together two or more of the
       b-trees that make up its full-text index within an INSERT, UPDATE or
       DELETE statement executed by the user. This means that any operation
       on an FTS3/4 table may turn out to be surprisingly slow, as FTS3/4 
       may unpredictably choose to merge together two or more large b-trees
       within it. FTS5 uses incremental merging by default, which limits
       the amount of processing that may take place within any given 
       INSERT, UPDATE or DELETE operation.

This page last modified on  2022-09-24 16:31:11 UTC


c2: The RBU Extension
c5: rbu.html
c4:
1. The RBU Extension

The RBU extension is an add-on for SQLite designed for use with large 
SQLite database files on low-power devices at the edge of a network. RBU
may be used for two separate tasks:


   RBU Update operations. An RBU Update is a bulk update of a
          database file that may include many insert, update and delete
          operations on one or more tables.
   RBU Vacuum operations. An RBU Vacuum optimizes and rebuilds an
          entire database file, with results similar to SQLite's native VACUUM
          command.  


The acronym RBU stands for "Resumable Bulk Update".

Both of the RBU functions may be accomplished using SQLite's built-in 
SQL commands - RBU update via a series of INSERT, DELETE and 
UPDATE commands within a single transaction, and RBU vacuum by a single
VACUUM command.  The RBU module provides the following advantages over
these simpler approaches:



RBU may be more efficient

The most efficient way to apply changes to a B-Tree (the data structure
that SQLite uses to store each table and index on disk) is to make the
changes in key order. But if an SQL table has one or more indexes, the key
order for each index may be different from the main table and the other
auxiliary indexes. As a result, when executing a series of INSERT,
UPDATE and DELETE statements it is not generally possible to order the
operations so that all b-trees are updated in key order. The RBU update
process works around this by applying all changes to the main table in one 
pass, then applying changes to each index in separate passes, ensuring each
B-Tree is updated optimally. For a large database file (one that does not
fit in the OS disk cache) this procedure can result in two orders of
magnitude faster updates.

An RBU Vacuum operation requires less temporary disk space and writes
less data to disk than an SQLite VACUUM. An SQLite VACUUM requires roughly
twice the size of the final database file in temporary disk space to run.
The total amount of data written is around three times the size of the
final database file. By contrast, an RBU Vacuum requires roughly the size
of the final database file in temporary disk space and writes a total of
twice that to disk.

On the other hand, an RBU Vacuum uses more CPU than a regular SQLite
VACUUM - in one test as much as five times as much. For this reason, an RBU
Vacuum is often significantly slower than an SQLite VACUUM under the same
conditions.

RBU runs in the background

An ongoing RBU operation (either an update or a vacuum) does not
interfere with read access to the database file.

RBU runs incrementally

RBU operations may be suspended and then later resumed, perhaps with
intervening power outages and/or system resets. For an RBU update, the
original database content remains visible to all database readers until 
the entire update has been applied - even if the update is suspended and
then later resumed.



The RBU extension is not enabled by default. To enable it, compile the
amalgamation with the SQLITE_ENABLE_RBU compile-time option.



2. RBU Updates
2.1. RBU Update Limitations

The following limitations apply to RBU updates:


The changes must consist of INSERT, UPDATE, and DELETE
    operations only.  CREATE and DROP operations are not
    supported.
INSERT statements may not use default values.
UPDATE and DELETE statements must identify the target rows
    by rowid or by non-NULL PRIMARY KEY values.
UPDATE statements may not modify PRIMARY KEY or rowid values.
    
RBU updates cannot be applied to any tables that contain a column
       named "rbu_control".
The RBU update will not fire any triggers.
The RBU update will not detect or prevent foreign key or
       CHECK constraint violations.
All RBU updates use the "OR ROLLBACK" constraint handling mechanism.
    
The target database may not be in WAL mode.
The target database may not contain indexes on expressions.
    Indexes on expressions are supported beginning with SQLite 3.30.0
    (2019-10-04).
No other writes may occur on the target database while the
       RBU update is being applied.  A read-lock is held on the target
       database to prevent this.



2.2. Preparing an RBU Update File

All changes to be applied by RBU are stored in a separate SQLite database
called the "RBU database".  The database that is to be modified is called
the "target database".

For each table in the target database that will be modified by the update,
a corresponding table is created within the RBU database. The RBU database
table schema is not the same as that of the target database, but is derived
from it as described below.

The RBU database table contains a single row for each target database 
row inserted, updated or deleted by the update. Populating the RBU database
tables is described in the following section.




2.2.1. The RBU Database Schema


For each table in the target database, the RBU database should contain a table
named "data&lt;integer&gt;_&lt;target-table-name&gt;" where
&lt;target-table-name&gt; is the name of the table in the target
database and &lt;integer&gt; is any sequence of zero or more numeric
characters (0-9). Tables within the RBU database are processed in order by 
name (from smallest to largest according to the BINARY collation sequence),
so the order in which target tables are updated is influenced by the selection 
of the &lt;integer&gt; portion of the data_% table name. While this can
be useful when using RBU to update 
certain types of virtual tables, there is normally no
reason to use anything other than an empty string in place of
&lt;integer&gt;.

The data_% table must have all the same columns as the target table, plus
one additional column named "rbu_control". The data_% table should have no
PRIMARY KEY or UNIQUE constraints, but each column should have the same type as
the corresponding column in the target database. The rbu_control column should
have no type at all. For example, if the target database contains:

CREATE TABLE t1(a INTEGER PRIMARY KEY, b TEXT, c UNIQUE);


Then the RBU database should contain:

CREATE TABLE data_t1(a INTEGER, b TEXT, c, rbu_control);


The order of the columns in the data_% table does not matter.

If the target database table is a virtual table or a table that has no
PRIMARY KEY declaration, the data_% table must also contain a column 
named "rbu_rowid". The rbu_rowid column is mapped to the tables ROWID.
For example, if the target database contains either of the following:

CREATE VIRTUAL TABLE x1 USING fts3(a, b);
CREATE TABLE x1(a, b);


then the RBU database should contain:

CREATE TABLE data_x1(a, b, rbu_rowid, rbu_control);


Virtual tables for which the "rowid" column does 
not function like a primary key value cannot be updated using RBU.


All non-hidden columns (i.e. all columns matched by "SELECT *") of the
target table must be present in the input table. For virtual tables,
hidden columns are optional - they are updated by RBU if present in
the input table, or not otherwise. For example, to write to an fts4
table with a hidden languageid column such as:

CREATE VIRTUAL TABLE ft1 USING fts4(a, b, languageid='langid');


Either of the following input table schemas may be used:

CREATE TABLE data_ft1(a, b, langid, rbu_rowid, rbu_control);
CREATE TABLE data_ft1(a, b, rbu_rowid, rbu_control);




2.2.2. RBU Database Contents

For each row to INSERT into the target database as part of the RBU 
update, the corresponding data_% table should contain a single record
with the "rbu_control" column set to contain integer value 0. The
other columns should be set to the values that make up the new record 
to insert. 

The "rbu_control" column may also be set to integer value 2 for 
an INSERT. In this case, the new row silently replaces any existing row that
has the same primary key values. This is equivalent to a DELETE followed by an
INSERT with the same primary key values. It is not the same as an SQL REPLACE
command, as in that case the new row may replace any conflicting rows (i.e.
those that conflict due to UNIQUE constraints or indexes), not just those with
conflicting primary keys.

If the target database table has an INTEGER PRIMARY KEY, it is not 
possible to insert a NULL value into the IPK column. Attempting to 
do so results in an SQLITE_MISMATCH error.

For each row to DELETE from the target database as part of the RBU 
update, the corresponding data_% table should contain a single record
with the "rbu_control" column set to contain integer value 1. The
real primary key values of the row to delete should be stored in the
corresponding columns of the data_% table. The values stored in the
other columns are not used.

For each row to UPDATE from the target database as part of the RBU 
update, the corresponding data_% table should contain a single record
with the "rbu_control" column set to contain a value of type text.
The real primary key values identifying the row to update should be 
stored in the corresponding columns of the data_% table row, as should
the new values of all columns being update. The text value in the 
"rbu_control" column must contain the same number of characters as
there are columns in the target database table, and must consist entirely
of 'x' and '.' characters (or in some special cases 'd' - see below). For 
each column that is being updated, the corresponding character is set to
'x'. For those that remain as they are, the corresponding character of the
rbu_control value should be set to '.'. For example, given the tables 
above, the update statement:

UPDATE t1 SET c = 'usa' WHERE a = 4;


is represented by the data_t1 row created by:

INSERT INTO data_t1(a, b, c, rbu_control) VALUES(4, NULL, 'usa', '..x');


If RBU is used to update a large BLOB value within a target database, it
may be more efficient to store a patch or delta that can be used to modify
the existing BLOB instead of an entirely new value within the RBU database. 
RBU allows deltas to be specified in two ways:


   In the "fossil delta" format - the format used for blob deltas by the 
  Fossil source-code management system, or

   In a custom format defined by the RBU application.


 The fossil delta format may only be used to update BLOB values. Instead
of storing the new BLOB within the data_% table, the fossil delta is stored
instead. And instead of specifying an 'x' as part of the rbu_control string
for the column to be updated, an 'f' character is stored. When processing
an 'f' update, RBU loads the original BLOB data from disk, applies the fossil
delta to it and stores the results back into the database file. The RBU
databases generated by sqldiff --rbu make use of fossil deltas wherever
doing so would save space in the RBU database.

 To use a custom delta format, the RBU application must register a
user-defined SQL function named "rbu_delta" before beginning to process the
update. rbu_delta() will be invoked with two arguments - the original value
stored in the target table column and the delta value provided as part of
the RBU update. It should return the result of applying the delta to the
original value. To use the custom delta function, the character of the
rbu_control value corresponding to the target column to update must be
set to 'd' instead of 'x'. Then, instead of updating the target table with the
value stored in the corresponding data_% column, RBU invokes the user-defined
SQL function "rbu_delta()" and the store in the target table column.

For example, this row:

INSERT INTO data_t1(a, b, c, rbu_control) VALUES(4, NULL, 'usa', '..d');


causes RBU to update the target database table in a way similar to:

UPDATE t1 SET c = rbu_delta(c, 'usa') WHERE a = 4;


If the target database table is a virtual table or a table with no PRIMARY
KEY, the rbu_control value should not include a character corresponding 
to the rbu_rowid value. For example, this:

INSERT INTO data_ft1(a, b, rbu_rowid, rbu_control) 
  VALUES(NULL, 'usa', 12, '.x');



causes a result similar to:

UPDATE ft1 SET b = 'usa' WHERE rowid = 12;


The data_% tables themselves should have no PRIMARY KEY declarations.
However, RBU is more efficient if reading the rows in from each data_%
table in "rowid" order is roughly the same as reading them sorted by
the PRIMARY KEY of the corresponding target database table. In other 
words, rows should be sorted using the destination table PRIMARY KEY 
fields before they are inserted into the data_% tables.



2.2.3. Using RBU with FTS3/4 Tables

Usually, an FTS3 or FTS4 table is an example of a virtual table 
with a rowid that works like a PRIMARY KEY. So, for the following FTS4 tables:

CREATE VIRTUAL TABLE ft1 USING fts4(addr, text);
CREATE VIRTUAL TABLE ft2 USING fts4;             -- implicit "content" column


The data_% tables may be created as follows:

CREATE TABLE data_ft1 USING fts4(addr, text, rbu_rowid, rbu_control);
CREATE TABLE data_ft2 USING fts4(content, rbu_rowid, rbu_control);


And populated as if the target table were an ordinary SQLite table with no
explicit PRIMARY KEY columns.

Contentless FTS4 tables are handled similarly,
except that any attempt to update or delete rows will cause an error when
applying the update.

External content FTS4 tables may also be 
updated using RBU. In this case the user is required to configure the RBU
database so that the same set of UPDATE, DELETE and INSERT operations are
applied to the FTS4 index as to the underlying content table. As for all
updates of external content FTS4 tables, the user is also required to ensure
that any UPDATE or DELETE operations are applied to the FTS4 index before
they are applied to the underlying content table (refer to FTS4 documentation
for a detailed explanation). In RBU, this is done by ensuring that the name
of the data_% table used to write to the FTS4 table sorts before the name
of the data_% table used to update the underlying content table using the
BINARY collation sequence. In order to avoid duplicating data within the
RBU database, an SQL view may be used in place of one of the data_% tables.
For example, for the target database schema:

CREATE TABLE ccc(addr, text);
CREATE VIRTUAL TABLE ccc_fts USING fts4(addr, text, content=ccc);



  The following RBU database schema may be used: 

CREATE TABLE data_ccc(addr, text, rbu_rowid, rbu_control);
CREATE VIEW data0_ccc_fts AS SELECT * FROM data_ccc;

 

  The data_ccc table may then be populated as normal with the updates intended
  for target database table ccc. The same updates will be read by RBU from
  the data0_ccc_fts view and applied to FTS table ccc_fts. Because
  "data0_ccc_fts" is smaller than "data_ccc", the FTS table will be updated
  first, as required.


  Cases in which the underlying content table has an explicit INTEGER PRIMARY
  KEY column are slightly more difficult, as the text values stored in the
  rbu_control column are slightly different for the FTS index and its
  underlying content table. For the underlying content table, a character
  must be included in any rbu_control text values for the explicit IPK, but
  for the FTS table itself, which has an implicit rowid, it should not. This
  is inconvenient, but can be solved using a more complicated view, as follows:

-- Target database schema
CREATE TABLE ddd(i INTEGER PRIMARY KEY, k TEXT);
CREATE VIRTUAL TABLE ddd_fts USING fts4(k, content=ddd);

-- RBU database schema
CREATE TABLE data_ccc(i, k, rbu_control);
CREATE VIEW data0_ccc_fts AS SELECT i AS rbu_rowid, k, CASE 
  WHEN rbu_control IN (0,1) THEN rbu_control ELSE substr(rbu_control, 2) END
FROM data_ccc;



  The substr() function in the SQL view above returns the text of the
  rbu_control argument with the first character (the one corresponding to
  column "i", which is not required by the FTS table) removed.



2.2.4. Automatically Generating RBU Updates with sqldiff


  As of SQLite version 3.9.0 (2015-10-14), 
  the sqldiff utility is able to generate
  RBU databases representing the difference between two databases with
  identical schemas. For example, the following command:

sqldiff --rbu t1.db t2.db



  Outputs an SQL script to create an RBU database which, if used to update
  database t1.db, patches it so that its contents are identical to that of
  database t2.db.


  By default, sqldiff attempts to process all non-virtual tables within
  the two databases provided to it. If any table appears in one database
  but not the other, or if any table has a slightly different schema in
  one database it is an error. The "--table" option may be useful if this
  causes a problem
  

  Virtual tables are ignored by default by sqldiff. However, it is possible 
  to explicitly create an RBU data_% table for a virtual table that features
  a rowid that functions like a primary key using a command such as:

sqldiff --rbu --table &lt;virtual-table-name&gt; t1.db t2.db



  Unfortunately, even though virtual tables are ignored by default, any
  underlying database tables that they create in order to
  store data within the database are not, and sqldiff will include add these
  to any RBU database. For this reason, users attempting to use sqldiff to
  create RBU updates to apply to target databases with one or more virtual
  tables will likely have to run sqldiff using the --table option separately
  for each table to update in the target database.

2.3. RBU Update C/C++ Programming

The RBU extension interface allows an application to apply an RBU update 
stored in an RBU database to an existing target database.
The procedure is as follows:



Open an RBU handle using the sqlite3rbu_open(T,A,S) function.

The T argument is the name of the target database file.
The A argument is the name of the RBU database file.
The S argument is the name of a "state database" used to store
state information needed to resume the update after an interruption.
The S argument can be NULL in which case the state information
is stored in the RBU database in various tables whose names all
begin with "rbu_".

The sqlite3rbu_open(T,A,S) function returns a pointer to
an "sqlite3rbu" object, which is then passed into the subsequent
interfaces.



Register any required virtual table modules with the database
handle returned by sqlite3rbu_db(X) (where argument X is the sqlite3rbu
pointer returned from sqlite3rbu_open()).  Also, if required, register
the rbu_delta() SQL function using 
sqlite3_create_function_v2().


Invoke the sqlite3rbu_step(X) function one or more times on
the sqlite3rbu object pointer X. Each call to sqlite3rbu_step() 
performs a single b-tree operation, so thousands of calls may be 
required to apply a complete update.  The sqlite3rbu_step() 
interface will return SQLITE_DONE when the update has been
completely applied.


Call sqlite3rbu_close(X) to destroy the sqlite3rbu object pointer.
If sqlite3rbu_step(X) has been called enough times to completely
apply the update to the target database, then the RBU database
is marked as fully applied. Otherwise, the state of the RBU 
update application is saved in the state database (or in the RBU
database if the name of the state database file in sqlite3rbu_open()
is NULL) for later resumption of the update.


If an update is only partially applied to the target database by the
time sqlite3rbu_close() is called, state information is saved 
within the state database if it exists, or otherwise in the RBU database. 
This allows subsequent processes to automatically
resume the RBU update from where it left off.
If state information is stored in the RBU database, it can be removed
by dropping all tables whose names begin with "rbu_".

For more details, refer to the comments in 
header file
sqlite3rbu.h.



3. RBU Vacuum

3.1. RBU Vacuum Limitations

When compared with SQLite's built-in VACUUM command, RBU Vacuum has the
following limitations:


  It may not be used on a database that contains indexes on expressions.
  The database being vacuumed may not be in WAL mode.


3.2. RBU Vacuum C/C++ Programming

 This section provides an overview of and example code demonstrating the
    integration of RBU Vacuum into an application program.  For full details,
    refer to the comments in 
    header file
    sqlite3rbu.h.

 RBU Vacuum applications all implement some variation of the following
procedure:


   An RBU handle is created by calling sqlite3rbu_vacuum(T, S).

       Argument T is the name of the database file to vacuum. Argument S is
      the name of a database in which the RBU module will save its state if the
      vacuum operation is suspended.

       If state database S does not exist when sqlite3rbu_vacuum() is
      invoked, it is automatically created and populated with the single table
      used to store the state of an RBU vacuum - "rbu_state". If an ongoing RBU
      vacuum is suspended, this table is populated with state data. The next
      time sqlite3rbu_vacuum() is called with the same S parameter, it detects
      this data and attempts to resume the suspended vacuum operation. When
      an RBU vacuum operation is completed or encounters an error, RBU 
      automatically deletes the contents of the rbu_state table. In this case,
      the next call to sqlite3rbu_vacuum() starts an entirely new vacuum
      operation from scratch.

       It is a good idea to establish a convention for determining the RBU
      vacuum state database name based on the target database name. The
      example code below uses "&lt;target&gt;-vacuum", where &lt;target&gt; is
      the name of the database being vacuumed.

   Any custom collation sequences used by indexes within the database
      being vacuumed are registered with both of the database handles returned
      by the sqlite3rbu_db() function.

   Function sqlite3rbu_step() is called on the RBU handle until either
      the RBU vacuum is finished, an error occurs or the application wishes to
      suspend the RBU vacuum.

       Each call to sqlite3rbu_step() does a small amount of work towards
      completing the vacuum operation. Depending on the size of the database, a
      single vacuum may require thousands of calls to sqlite3rbu_step().
      sqlite3rbu_step() returns SQLITE_DONE if the vacuum operation has
      finished, SQLITE_OK if the vacuum operation has not finished but no error
      has occurred, and an SQLite error code if an error is encountered. If
      an error does occur, all subsequent calls to sqlite3rbu_step() immediately
      return the same error code.

   Finally, sqlite3rbu_close() is called to close the RBU handle. If the
      application stopped calling sqlite3rbu_step() before either the vacuum
      finished or an error occurred, the state of the vacuum is saved in the
      state database so that it may be resumed later on.

       Like sqlite3rbu_step(), if the vacuum operation has finished,
      sqlite3rbu_close() returns SQLITE_DONE. If the vacuum has not finished
      but no error has occurred, SQLITE_OK is returned. Or, if an error has
      occurred, an SQLite error code is returned. If an error occurred as part
      of a prior call to sqlite3rbu_step(), sqlite3rbu_close() returns the
      same error code.


The following example code illustrates the techniques described above.  

/*
** Either start a new RBU vacuum or resume a suspended RBU vacuum on 
** database zTarget. Return when either an error occurs, the RBU 
** vacuum is finished or when the application signals an interrupt
** (code not shown).
**
** If the RBU vacuum is completed successfully, return SQLITE_DONE.
** If an error occurs, return SQLite error code. Or, if the application
** signals an interrupt, suspend the RBU vacuum operation so that it
** may be resumed by a subsequent call to this function and return
** SQLITE_OK.
**
** This function uses the database named "&lt;zTarget&gt;-vacuum" for
** the state database, where &lt;zTarget&gt; is the name of the database 
** being vacuumed.
*/
int do_rbu_vacuum(const char *zTarget){
  int rc;
  char *zState;                   /* Name of state database */
  sqlite3rbu *pRbu;               /* RBU vacuum handle */

  zState = sqlite3_mprintf("%s-vacuum", zTarget);
  if( zState==0 ) return SQLITE_NOMEM;
  pRbu = sqlite3rbu_vacuum(zTarget, zState);
  sqlite3_free(zState);

  if( pRbu ){
    sqlite3 *dbTarget = sqlite3rbu_db(pRbu, 0);
    sqlite3 *dbState = sqlite3rbu_db(pRbu, 1);

    /* Any custom collation sequences used by the target database must
    ** be registered with both database handles here.  */

    while( sqlite3rbu_step(pRbu)==SQLITE_OK ){
      if( &lt;application has signaled interrupt&gt; ) break;
    }
  }
  rc = sqlite3rbu_close(pRbu);
  return rc;
}

This page last modified on  2019-09-26 19:34:22 UTC


c2: SQLite Shared-Cache Mode
c5: sharedcache.html
c4:
1. SQLite Shared-Cache Mode

Starting with version 3.3.0 (2006-01-11), 
SQLite includes a special "shared-cache"
mode (disabled by default) intended for use in embedded servers. If
shared-cache mode is enabled and a thread establishes multiple connections
to the same database, the connections share a single data and schema cache.
This can significantly reduce the quantity of memory and IO required by
the system.

In version 3.5.0 (2007-09-04), 
shared-cache mode was modified so that the same
cache can be shared across an entire process rather than just within
a single thread.  Prior to this change, there were restrictions on
passing database connections between threads.  Those restrictions were
dropped in 3.5.0 update.  This document describes shared-cache mode
as of version 3.5.0.

Shared-cache mode changes the semantics
of the locking model in some cases. The details are described by
this document. A basic understanding of the normal SQLite locking model (see
File Locking And Concurrency In SQLite Version 3
for details) is assumed.



1.1. Use of shared-cache is discouraged

Shared-cache mode is an obsolete feature.  The use of shared-cache mode
is discouraged.  Most use cases for shared-cache are better served by
WAL mode.

Shared-cache mode was invented in 2006 at the request of developers
of Symbian.  Their problem was that
if the contacts database on the phone was being synced, that would lock the
database file.  Then if a call came in, the database lock would prevent them
from querying the contacts database in order to find the appropriate 
ring-tone for the incoming call, or a photo of the caller to show on screen,
and so forth.
WAL mode (circa 2010) is a better solution to this problem as it permits 
simultaneous access without breaking transaction isolation.

Applications that build their own copy of SQLite from source code
are encouraged to use the -DSQLITE_OMIT_SHARED_CACHE compile-time option,
as the resulting binary will be both smaller and faster.

The shared-cache interfaces described here will continue to be supported
in SQLite, to insure full backwards compatibility.  However, the use of
shared-cache is discouraged.

2. Shared-Cache Locking Model

Externally, from the point of view of another process or thread, two
or more database connections using a shared-cache appear as a single 
connection. The locking protocol used to arbitrate between multiple 
shared-caches or regular database users is described elsewhere.







Figure 1

Figure 1 depicts an example runtime configuration where three 
database connections have been established. Connection 1 is a normal
SQLite database connection. Connections 2 and 3 share a cache 
The normal locking
protocol is used to serialize database access between connection 1 and
the shared cache. The internal protocol used to serialize (or not, see
"Read-Uncommitted Isolation Mode" below) access to the shared-cache by
connections 2 and 3 is described in the remainder of this section.


There are three levels to the shared-cache locking model, 
transaction level locking, table level locking and schema level locking. 
They are described in the following three sub-sections.

2.1. Transaction Level Locking

SQLite connections can open two kinds of transactions, read and write
transactions. This is not done explicitly, a transaction is implicitly a
read-transaction until it first writes to a database table, at which point
it becomes a write-transaction.

At most one connection to a single shared cache may open a 
write transaction at any one time. This may co-exist with any number of read 
transactions. 


2.2. Table Level Locking

When two or more connections use a shared-cache, locks are used to 
serialize concurrent access attempts on a per-table basis. Tables support 
two types of locks, "read-locks" and "write-locks". Locks are granted to
connections - at any one time, each database connection has either a
read-lock, write-lock or no lock on each database table.


At any one time, a single table may have any number of active read-locks
or a single active write lock. To read data from a table, a connection must 
first obtain a read-lock. To write to a table, a connection must obtain a 
write-lock on that table. If a required table lock cannot be obtained,
the query fails and SQLITE_LOCKED is returned to the caller.
 

Once a connection obtains a table lock, it is not released until the
current transaction (read or write) is concluded.


2.2.1. Read-Uncommitted Isolation Mode

The behaviour described above may be modified slightly by using the 
read_uncommitted pragma to change the isolation level from serialized 
(the default), to read-uncommitted.

 A database connection in read-uncommitted mode does not attempt 
to obtain read-locks before reading from database tables as described 
above. This can lead to inconsistent query results if another database
connection modifies a table while it is being read, but it also means that
a read-transaction opened by a connection in read-uncommitted mode can
neither block nor be blocked by any other connection.

Read-uncommitted mode has no effect on the locks required to write to
database tables (i.e. read-uncommitted connections must still obtain 
write-locks and hence database writes may still block or be blocked). 
Also, read-uncommitted mode has no effect on the sqlite_schema
locks required by the rules enumerated below (see section 
"Schema (sqlite_schema) Level Locking").



  /* Set the value of the read-uncommitted flag:
  **
  **   True  -> Set the connection to read-uncommitted mode.
  **   False -> Set the connection to serialized (the default) mode.
  */
  PRAGMA read_uncommitted = &lt;boolean&gt;;

  /* Retrieve the current value of the read-uncommitted flag */
  PRAGMA read_uncommitted;


2.3. Schema (sqlite_schema) Level Locking

The sqlite_schema table supports shared-cache read and write 
locks in the same way as all other database tables (see description 
above). The following special rules also apply:



A connection must obtain a read-lock on sqlite_schema before 
accessing any database tables or obtaining any other read or write locks.
Before executing a statement that modifies the database schema (i.e. 
a CREATE or DROP TABLE statement), a connection must obtain a write-lock on 
sqlite_schema.

A connection may not compile an SQL statement if any other connection
is holding a write-lock on the sqlite_schema table of any attached
database (including the default database, "main"). 



3. Thread Related Issues

In SQLite versions 3.3.0 through 3.4.2 when shared-cache mode is enabled, 
a database connection may only be
used by the thread that called sqlite3_open() to create it.
And a connection could only share cache with another connection in the
same thread.
These restrictions were dropped beginning with SQLite 
version 3.5.0 (2007-09-04).


4. Shared Cache And Virtual Tables


In older versions of SQLite,
shared cache mode could not be used together with virtual tables.
This restriction was removed in SQLite version 3.6.17 (2009-08-10).

5. Enabling Shared-Cache Mode

Shared-cache mode is enabled on a per-process basis. Using the C 
interface, the following API can be used to globally enable or disable
shared-cache mode:



int sqlite3_enable_shared_cache(int);


Each call to sqlite3_enable_shared_cache() affects subsequent database
connections created using sqlite3_open(), sqlite3_open16(), or
sqlite3_open_v2().  Database connections that already exist are
unaffected.  Each call to sqlite3_enable_shared_cache() overrides
all previous calls within the same process.


Individual database connections created using sqlite3_open_v2() can
choose to participate or not participate in shared cache mode by using
the SQLITE_OPEN_SHAREDCACHE or SQLITE_OPEN_PRIVATECACHE flags the
third parameter.  The use of either of these flags overrides the
global shared cache mode setting established by sqlite3_enable_shared_cache().
No more than one of the flags should be used; if both SQLITE_OPEN_SHAREDCACHE
and SQLITE_OPEN_PRIVATECACHE flags are used in the third argument to
sqlite3_open_v2() then the behavior is undefined.

When URI filenames are used, the "cache" query parameter can be used
to specify whether or not the database will use shared cache.  Use
"cache=shared" to enable shared cache and "cache=private" to disable
shared cache.   The ability to use URI query parameters to specify the
cache sharing behavior of a database connection allows cache sharing to
be controlled in ATTACH statements.  For example:


ATTACH 'file:aux.db?cache=shared' AS aux;




6. Shared Cache And In-Memory Databases


Beginning with SQLite version 3.7.13 (2012-06-11), 
shared cache can be used on
in-memory databases, provided that the database is created using
a URI filename.  For backwards compatibility, shared cache is always
disabled for in-memory
databases if the unadorned name ":memory:" is used to open the database.
Prior to version 3.7.13, shared cache was always
disabled for in-memory databases regardless of the database name used,
current system shared cache setting, or query parameters or flags.



Enabling shared-cache for an in-memory database allows two or more
database connections in the same process to have access to the same
in-memory database.  An in-memory database in shared cache is automatically
deleted and memory is reclaimed when the last connection to that database
closes.

This page last modified on  2022-08-15 11:37:22 UTC


c2: 8+3 Filenames
c5: shortnames.html
c4:
SQLite And 8+3 Filenames


The default configuration of SQLite assumes the underlying filesystem
supports long filenames.



SQLite does not impose any naming requirements on database files.
SQLite will happily work with a database file that has any filename extension
or with no extension at all.
When auxiliary files are needed for a rollback journal or
a write-ahead log or for one of the other kinds of
temporary disk files, then the name for the auxiliary file is normally
constructed by appending a suffix onto the end of the database file name.
For example, if the original database is call "app.db" then
the rollback journal will be called "app.db-journal"
and the write-ahead log will be called "app.db-wal".
This approach to auxiliary file naming works great on systems that
support long filenames.  But on systems that impose 8+3 filename
constraints, the auxiliary files do not fit the 8+3 format even though
the original database file does.



Changing Filesystems


The recommended fix for this problem is to select a different
filesystem.  These days, there is a huge selection of high-performance, 
reliable, patent-free filesystems that support long filenames.  
Where possible, it is recommended that embedded devices use one
of these other filesystems.  This will avoid compatibility issues
and the danger of
database corruption caused by inconsistent use of 8+3 filenames.


Adjusting SQLite To Use 8+3 Filenames


Some devices are compelled to use an older filesystem with 8+3
filename restrictions for backwards compatibility, or due
to other non-technical factors.  In such situations, SQLite can be
coerced into using auxiliary files that fit the 8+3 pattern as follows:



Compile the SQLite library with the either the compile-time
       options SQLITE_ENABLE_8_3_NAMES=1 or
       SQLITE_ENABLE_8_3_NAMES=2.
       Support for 8+3 filenames is not included in SQLite by default
       because it does introduce some overhead.  The overhead is tiny,
       but even so, we do not want to burden the billions of SQLite
       applications that do not need 8+3 filename support.

If the SQLITE_ENABLE_8_3_NAMES=1 option
       is used, then SQLite is capable of using 8+3 filenames but that
       capabilities is disabled and must be enabled separately for each
       database connection by using
       using URI filenames when opening or
       ATTACH-ing the database files and include the
       "8_3_names=1" query parameter in the URI.  If SQLite
       is compiled with
       SQLITE_ENABLE_8_3_NAMES=2 then
       8+3 filenames are enabled by default and this step can be
       skipped.

Make sure that database filenames follow the 8+3 filename
       format and that they do not have an empty name or extension.
       In other words, the database filename must contain between
       1 and 8 characters in the base name and between 1 and 3 characters
       in the extension.  Blank extensions are not allowed.



When the steps above are used, SQLite will shorten filename extensions
by only using the last 3 characters of
the extension.  Thus, for example, a file that would normally be called
"app.db-journal" is shortened to just "app.nal".
Similarly, "app.db-wal" will become "app.wal" and
"app.db-shm" becomes "app.shm".



Note that it is very important that the database filename have some kind
of extension.  If there is no extension, then SQLite creates auxiliary
filenames by appending to the base name of the file.  Thus, a database
named "db01" would have a rollback journal file named
"db01-journal".  And as this filename has no extension to shorten
to 3 characters, it will be used as-is, and will violate 8+3 naming rules.




Database Corruption Warning


If a database file is accessed using 8+3 naming rather than the default
long filename, then it must be consistently accessed 
using 8+3 naming by every database
connection every time it is opened, or else there is a risk of database
corruption.
The auxiliary rollback journal and write-ahead log files are essential
to SQLite for being about to recover from a crash.  If an application is
using 8+3 names and crashes, then the information needed to safely recover
from the crash is stored in files with the ".nal" or
".wal" extension.  If the next application to open the database
does not specify the "8_3_names=1" URI parameter, then SQLite
will use the long filenames to try to locate the rollback journal or
write-ahead log files.  It will not find them, since they were saved using
8+3 names by the application that crashed, and hence the database will not
be properly recovered and will likely go corrupt.



Using a database file with 8+3 filenames in some cases while in 
other cases using long filenames is equivalent to
deleting a hot journal.

This page last modified on  2011-08-15 23:00:56 UTC


c2: The SQLite Bytecode Engine
c5: opcode.html
c4:
1. Executive Summary

SQLite works by translating SQL statements into bytecode and
then running that bytecode in a virtual machine.  This document
describes how the bytecode engine works.

This document describes SQLite internals.  The information provided
here is not needed for routine application development using SQLite.
This document is intended for people who want to delve more deeply into
the internal operation of SQLite.

The bytecode engine is not an API of SQLite.  Details
about the bytecode engine change from one release of SQLite to the next.
Applications that use SQLite should not depend on any of the details
found in this document.



2. Introduction

SQLite works by translating each SQL statement into bytecode and
then running that bytecode.
A prepared statement in SQLite is mostly just the bytecode needed to
implement the corresponding SQL.  The sqlite3_prepare_v2() interface
is a compiler that translates SQL into bytecode.
The sqlite3_step() interface is the virtual machine that runs the
bytecode contained within the prepared statement.

The bytecode virtual machine is the heart of SQLite.
Programmers who want to understand how SQLite operates internally
must be familiar with the bytecode engine.

Historically, the bytecode engine in SQLite is called the
"Virtual DataBase Engine" or "VDBE".  This website uses the terms
"bytecode engine", "VDBE", "virtual machine", and "bytecode virtual
machine" interchangeably, as they all mean the same thing.


This article also uses the terms "bytecode program" and
"prepared statement" interchangeably, as they are mostly the same thing.

2.1. VDBE Source Code

The source code to the bytecode engine is in the 
vdbe.c source
file.  The opcode definitions in this document are derived
from comments in that source file. The 
source code comments are the canonical source of information
about the bytecode engine.  When in doubt, refer to the source code.

In addition to the primary vdbe.c source code file, there are 
other helper code files in the source tree, all of whose names
begin with "vdbe" - short for "Virtual DataBase Engine".

Remember that the names and meanings of opcodes often change from
one release of SQLite to the next.  So if you are studying the EXPLAIN
output from SQLite, you should reference the version of this document
(or the vdbe.c source code)
that corresponds to the version of SQLite that ran the EXPLAIN.
Otherwise, the description of the opcodes may not be accurate.
This document is derived from SQLite
 version 3.40.1 check-in
df5c253c0b3dd dated 2022-12-28.




2.2. Instruction Format


A bytecoded program in SQLite consists of one or more instructions.
Each instruction has an opcode and
five operands named P1, P2  P3, P4, and P5.  The P1, P2, and P3 
operands are 32-bit signed integers.  These operands often refer to 
registers.  For instructions that operate on b-tree cursors,
the P1 operand is usually the cursor number.
For jump instructions, P2 is usually the jump destination.
P4 may be a 32-bit signed integer, a 64-bit signed integer, a
64-bit floating point value, a string literal, a Blob literal,
a pointer to a collating sequence comparison function, or a
pointer to the implementation of an application-defined SQL
function, or various other things.  P5 is an 16-bit unsigned integer
normally used to hold flags.  Bits of the P5 flag can sometimes affect
the opcode in subtle ways.  For example, if the 
SQLITE_NULLEQ (0x0080) bit of the P5 operand
is set on the Eq opcode, then the NULL values compare
equal to one another.  Otherwise NULL values compare different
from one another.


Some opcodes use all five operands.  Some opcodes use
one or two.  Some opcodes use none of the operands.


The bytecode engine begins execution on instruction number 0.
Execution continues until a Halt instruction is seen, or until
the program counter becomes greater than the address of
last instruction, or until there is an error.
When the bytecode engine halts, all memory
that it allocated is released and all database cursors it may
have had open are closed.  If the execution stopped due to an
error, any pending transactions are terminated and changes made
to the database are rolled back.



The ResultRow opcode causes the
bytecode engine to pause and the corresponding sqlite3_step()
call to return SQLITE_ROW.  Before invoking
ResultRow, the bytecoded program will
have loaded the results for a single row of a query into a series
of registers.  C-language APIs such as sqlite3_column_int()
or sqlite3_column_text() extract the query results from those
registers.  The bytecode engine resumes with the next instruction
after the ResultRow on the next call
to sqlite3_step().


2.3. Registers


Every bytecode program has a fixed (but potentially large) number of
registers.  A single register can hold a variety of objects:

 A NULL value
 A signed 64-bit integer
 An IEEE double-precision (64-bit) floating point number
 An arbitrary length strings
 An arbitrary length BLOB
 A RowSet object (See the RowSetAdd, RowSetRead, and
                      RowSetTest opcodes)
 A Frame object (Used by subprograms - see Program)



A register can also be "Undefined" meaning that it holds no value
at all.  Undefined is different from NULL.  Depending on compile-time
options, an attempt to read an undefined register will usually cause
a run-time error.  If the code generator (sqlite3_prepare_v2())
ever generates a prepared statement that reads an Undefined register,
that is a bug in the code generator.


Registers are numbered beginning with 0.
Most opcodes refer to at least one register.

The number of registers in a single prepared statement is fixed
at compile-time.  The content of all registers is cleared when
a prepared statement is reset or
finalized.

The internal Mem object stores the value for a single register.
The abstract sqlite3_value object that is exposed in the API is really
just a Mem object or register.

2.4. B-Tree Cursors


A prepared statement can have
zero or more open cursors.  Each cursor is identified by a
small integer, which is usually the P1 parameter to the opcode
that uses the cursor.
There can be multiple cursors open on the same index or table.
All cursors operate independently, even cursors pointing to the same
indices or tables.
The only way for the virtual machine to interact with a database
file is through a cursor.
Instructions in the virtual machine can create a new cursor 
(ex: OpenRead or OpenWrite),
read data from a cursor (Column),
advance the cursor to the next entry in the table
(ex: Next or Prev), and so forth.
All cursors are automatically
closed when the prepared statement is reset or
finalized.




2.5. Subroutines, Coroutines, and Subprograms

The bytecode engine has no stack on which to store the return address
of a subroutine.  Return addresses must be stored in registers.
Hence, bytecode subroutines are not reentrant.


The Gosub opcode stores the current program counter into
register P1 then jumps to address P2.  The Return opcode jumps
to address P1+1.  Hence, every subroutine is associated with two integers:
the address of the entry point in the subroutine and the register number
that is used to hold the return address.

The Yield opcode swaps the value of the program counter with
the integer value in register P1.  This opcode is used to implement
coroutines.  Coroutines are often used to implement subqueries from
which content is pulled on an as-needed basis.


Triggers need to be reentrant.


Since bytecode
subroutines are not reentrant a different mechanism must be used to
implement triggers.  Each trigger is implemented using a separate bytecode
program with its own opcodes, program counter, and register set.  The
Program opcode invokes the trigger subprogram.  The Program instruction
allocates and initializes a fresh register set for each invocation of the
subprogram, so subprograms can be reentrant and recursive.  The
Param opcode is used by subprograms to access content in registers
of the calling bytecode program.


2.6. Self-Altering Code


Some opcodes are self-altering.
For example, the Init opcode (which is always the first opcode
in every bytecode program) increments its P1 operand.  Subsequent
Once opcodes compare their P1 operands to the P1 value for
the Init opcode in order to determine if the one-time initialization
code that follows should be skipped.
Another example is the String8 opcode which converts its P4
operand from UTF-8 into the correct database string encoding, then
converts itself into a String opcode.



3. Viewing The Bytecode

Every SQL statement that SQLite interprets results in a program
for the virtual machine.  But if the SQL statement begins with
the keyword EXPLAIN the virtual machine will not execute the
program.  Instead, the instructions of the program will be returned,
one instruction per row,
like a query result.  This feature is useful for debugging and
for learning how the virtual machine operates.  For example:


$&nbsp;sqlite3&nbsp;ex1.db
sqlite&gt;&nbsp;explain&nbsp;delete&nbsp;from&nbsp;tbl1&nbsp;where&nbsp;two&lt;20;
addr&nbsp;&nbsp;opcode&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;p1&nbsp;&nbsp;&nbsp;&nbsp;p2&nbsp;&nbsp;&nbsp;&nbsp;p3&nbsp;&nbsp;&nbsp;&nbsp;p4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;p5&nbsp;&nbsp;comment&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
----&nbsp;&nbsp;-------------&nbsp;&nbsp;----&nbsp;&nbsp;----&nbsp;&nbsp;----&nbsp;&nbsp;-------------&nbsp;&nbsp;--&nbsp;&nbsp;-------------
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Init&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;Start&nbsp;at&nbsp;12&nbsp;&nbsp;
1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Null&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;r[1]=NULL&nbsp;&nbsp;&nbsp;&nbsp;
2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OpenWrite&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;root=2&nbsp;iDb=0;&nbsp;tbl1
3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Rewind&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Column&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;r[2]=tbl1.two
5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Ge&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(BINARY)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;51&nbsp;&nbsp;if&nbsp;r[2]&gt;=r[3]&nbsp;goto&nbsp;9
6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Rowid&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;r[4]=rowid&nbsp;&nbsp;&nbsp;
7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Once&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Delete&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tbl1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Next&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
10&nbsp;&nbsp;&nbsp;&nbsp;Noop&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
11&nbsp;&nbsp;&nbsp;&nbsp;Halt&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
12&nbsp;&nbsp;&nbsp;&nbsp;Transaction&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;01&nbsp;&nbsp;usesStmtJournal=0
13&nbsp;&nbsp;&nbsp;&nbsp;TableLock&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tbl1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;iDb=0&nbsp;root=2&nbsp;write=1
14&nbsp;&nbsp;&nbsp;&nbsp;Integer&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;20&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00&nbsp;&nbsp;r[3]=20&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
15&nbsp;&nbsp;&nbsp;&nbsp;Goto&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;00

Any application can run an EXPLAIN query to get output similar to 
the above.
However, indentation to show the loop structure is not generated
by the SQLite core.  The command-line shell contains extra logic
for indenting loops.
Also, the "comment" column in the EXPLAIN output
is only provided if SQLite is compiled with the
-DSQLITE_ENABLE_EXPLAIN_COMMENTS options.

When SQLite is compiled with the SQLITE_DEBUG compile-time option,
extra PRAGMA commands are available that are useful for debugging and
for exploring the operation of the VDBE.  For example the vdbe_trace
pragma can be enabled to cause a disassembly of each VDBE opcode to be
printed on standard output as the opcode is executed.  These debugging
pragmas include:

 PRAGMA parser_trace
 PRAGMA vdbe_addoptrace
 PRAGMA vdbe_debug
 PRAGMA vdbe_listing
 PRAGMA vdbe_trace



4. The Opcodes

There are currently 186
opcodes defined by the virtual machine.
All currently defined opcodes are described in the table below.
This table was generated automatically by scanning the source code
from the file
vdbe.c.


Remember: The VDBE opcodes are not part of the interface 
definition for SQLite.  The number of opcodes and their names and meanings
change from one release of SQLite to the next.
The opcodes shown in the table below are valid for SQLite
 version 3.40.1 check-in
df5c253c0b3dd dated 2022-12-28.



  
  
  
  
  Opcode NameDescription

Abortable
Verify that an Abort can happen.  Assert if an Abort at this point
might cause database corruption.  This opcode only appears in debugging
builds.

An Abort is safe if either there have been no writes, or if there is
an active statement journal.

Add
Add the value in register P1 to the value in register P2
and store the result in register P3.
If either input is NULL, the result is NULL.

AddImm
Add the constant P2 to the value in register P1.
The result is always an integer.

To force any register to be an integer, just add 0.

Affinity
Apply affinities to a range of P2 registers starting with P1.

P4 is a string that is P2 characters long. The N-th character of the
string indicates the column affinity that should be used for the N-th
memory cell in the range.

AggFinal
P1 is the memory location that is the accumulator for an aggregate
or window function.  Execute the finalizer function
for an aggregate and store the result in P1.

P2 is the number of arguments that the step function takes and
P4 is a pointer to the FuncDef for this function.  The P2
argument is not used by this opcode.  It is only there to disambiguate
functions that can take varying numbers of arguments.  The
P4 argument is only needed for the case where
the step function was not previously called.

AggInverse
Execute the xInverse function for an aggregate.
The function has P5 arguments.  P4 is a pointer to the
FuncDef structure that specifies the function.  Register P3 is the
accumulator.

The P5 arguments are taken from register P2 and its
successors.

AggStep
Execute the xStep function for an aggregate.
The function has P5 arguments.  P4 is a pointer to the
FuncDef structure that specifies the function.  Register P3 is the
accumulator.

The P5 arguments are taken from register P2 and its
successors.

AggStep1
Execute the xStep (if P1==0) or xInverse (if P1!=0) function for an
aggregate.  The function has P5 arguments.  P4 is a pointer to the
FuncDef structure that specifies the function.  Register P3 is the
accumulator.

The P5 arguments are taken from register P2 and its
successors.

This opcode is initially coded as OP_AggStep0.  On first evaluation,
the FuncDef stored in P4 is converted into an sqlite3_context and
the opcode is changed.  In this way, the initialization of the
sqlite3_context only happens once, instead of on each call to the
step function.

AggValue
Invoke the xValue() function and store the result in register P3.

P2 is the number of arguments that the step function takes and
P4 is a pointer to the FuncDef for this function.  The P2
argument is not used by this opcode.  It is only there to disambiguate
functions that can take varying numbers of arguments.  The
P4 argument is only needed for the case where
the step function was not previously called.

And
Take the logical AND of the values in registers P1 and P2 and
write the result into register P3.

If either P1 or P2 is 0 (false) then the result is 0 even if
the other input is NULL.  A NULL and true or two NULLs give
a NULL output.

AutoCommit
Set the database auto-commit flag to P1 (1 or 0). If P2 is true, roll
back any currently active btree transactions. If there are any active
VMs (apart from this one), then a ROLLBACK fails.  A COMMIT fails if
there are active writing VMs or active VMs that use shared cache.

This instruction causes the VM to halt.

BeginSubrtn
Mark the beginning of a subroutine that can be entered in-line
or that can be called using Gosub.  The subroutine should
be terminated by an Return instruction that has a P1 operand that
is the same as the P2 operand to this opcode and that has P3 set to 1.
If the subroutine is entered in-line, then the Return will simply
fall through.  But if the subroutine is entered using Gosub, then
the Return will jump back to the first instruction after the Gosub.

This routine works by loading a NULL into the P2 register.  When the
return address register contains a NULL, the Return instruction is
a no-op that simply falls through to the next instruction (assuming that
the Return opcode has a P3 value of 1).  Thus if the subroutine is
entered in-line, then the Return will cause in-line execution to
continue.  But if the subroutine is entered via Gosub, then the
Return will cause a return to the address following the Gosub.

This opcode is identical to Null.  It has a different name
only to make the byte code easier to read and verify.

BitAnd
Take the bit-wise AND of the values in register P1 and P2 and
store the result in register P3.
If either input is NULL, the result is NULL.

BitNot
Interpret the content of register P1 as an integer.  Store the
ones-complement of the P1 value into register P2.  If P1 holds
a NULL then store a NULL in P2.

BitOr
Take the bit-wise OR of the values in register P1 and P2 and
store the result in register P3.
If either input is NULL, the result is NULL.

Blob
P4 points to a blob of data P1 bytes long.  Store this
blob in register P2.  If P4 is a NULL pointer, then construct
a zero-filled blob that is P1 bytes long in P2.

Cast
Force the value in register P1 to be the type defined by P2.


 P2=='A' &rarr; BLOB
 P2=='B' &rarr; TEXT
 P2=='C' &rarr; NUMERIC
 P2=='D' &rarr; INTEGER
 P2=='E' &rarr; REAL


A NULL value is not changed by this routine.  It remains NULL.

Checkpoint
Checkpoint database P1. This is a no-op if P1 is not currently in
WAL mode. Parameter P2 is one of SQLITE_CHECKPOINT_PASSIVE, FULL,
RESTART, or TRUNCATE.  Write 1 or 0 into mem&#91;P3&#93; if the checkpoint returns
SQLITE_BUSY or not, respectively.  Write the number of pages in the
WAL after the checkpoint into mem&#91;P3+1&#93; and the number of pages
in the WAL that have been checkpointed after the checkpoint
completes into mem&#91;P3+2&#93;.  However on an error, mem&#91;P3+1&#93; and
mem&#91;P3+2&#93; are initialized to -1.

Clear
Delete all contents of the database table or index whose root page
in the database file is given by P1.  But, unlike Destroy, do not
remove the table or index from the database file.

The table being clear is in the main database file if P2==0.  If
P2==1 then the table to be clear is in the auxiliary database file
that is used to store tables create using CREATE TEMPORARY TABLE.

If the P3 value is non-zero, then the row change count is incremented
by the number of rows in the table being cleared. If P3 is greater
than zero, then the value stored in register P3 is also incremented
by the number of rows in the table being cleared.

See also: Destroy

Close
Close a cursor previously opened as P1.  If P1 is not
currently open, this instruction is a no-op.

ClrSubtype
Clear the subtype from register P1.

CollSeq
P4 is a pointer to a CollSeq object. If the next call to a user function
or aggregate calls sqlite3GetFuncCollSeq(), this collation sequence will
be returned. This is used by the built-in min(), max() and nullif()
functions.

If P1 is not zero, then it is a register that a subsequent min() or
max() aggregate will set to 1 if the current row is not the minimum or
maximum.  The P1 register is initialized to 0 by this instruction.

The interface used by the implementation of the aforementioned functions
to retrieve the collation sequence set by this opcode is not available
publicly.  Only built-in functions have access to this feature.

Column
Interpret the data that cursor P1 points to as a structure built using
the MakeRecord instruction.  (See the MakeRecord opcode for additional
information about the format of the data.)  Extract the P2-th column
from this record.  If there are less than (P2+1)
values in the record, extract a NULL.

The value extracted is stored in register P3.

If the record contains fewer than P2 fields, then extract a NULL.  Or,
if the P4 argument is a P4_MEM use the value of the P4 argument as
the result.

If the OPFLAG_LENGTHARG bit is set in P5 then the result is guaranteed
to only be used by the length() function or the equivalent.  The content
of large blobs is not loaded, thus saving CPU cycles.  If the
OPFLAG_TYPEOFARG bit is set then the result will only be used by the
typeof() function or the IS NULL or IS NOT NULL operators or the
equivalent.  In this case, all content loading can be omitted.

ColumnsUsed
This opcode (which only exists if SQLite was compiled with
SQLITE_ENABLE_COLUMN_USED_MASK) identifies which columns of the
table or index for cursor P1 are used.  P4 is a 64-bit integer
(P4_INT64) in which the first 63 bits are one for each of the
first 63 columns of the table or index that are actually used
by the cursor.  The high-order bit is set if any column after
the 64th is used.

Compare
Compare two vectors of registers in reg(P1)..reg(P1+P3-1) (call this
vector "A") and in reg(P2)..reg(P2+P3-1) ("B").  Save the result of
the comparison for use by the next Jump instruct.

If P5 has the OPFLAG_PERMUTE bit set, then the order of comparison is
determined by the most recent Permutation operator.  If the
OPFLAG_PERMUTE bit is clear, then register are compared in sequential
order.

P4 is a KeyInfo structure that defines collating sequences and sort
orders for the comparison.  The permutation applies to registers
only.  The KeyInfo elements are used sequentially.

The comparison is a sort comparison, so NULLs compare equal,
NULLs are less than numbers, numbers are less than strings,
and strings are less than blobs.

This opcode must be immediately followed by an Jump opcode.

Concat
Add the text in register P1 onto the end of the text in
register P2 and store the result in register P3.
If either the P1 or P2 text are NULL then store NULL in P3.

P3 = P2 || P1

It is illegal for P1 and P3 to be the same register. Sometimes,
if P3 is the same register as P2, the implementation is able
to avoid a memcpy().

Copy
Make a copy of registers P1..P1+P3 into registers P2..P2+P3.

If the 0x0002 bit of P5 is set then also clear the MEM_Subtype flag in the
destination.  The 0x0001 bit of P5 indicates that this Copy opcode cannot
be merged.  The 0x0001 bit is used by the query planner and does not
come into play during query execution.

This instruction makes a deep copy of the value.  A duplicate
is made of any string or blob constant.  See also SCopy.

Count
Store the number of entries (an integer value) in the table or index
opened by cursor P1 in register P2.

If P3==0, then an exact count is obtained, which involves visiting
every btree page of the table.  But if P3 is non-zero, an estimate
is returned based on the current cursor position.

CreateBtree
Allocate a new b-tree in the main database file if P1==0 or in the
TEMP database file if P1==1 or in an attached database if
P1&gt;1.  The P3 argument must be 1 (BTREE_INTKEY) for a rowid table
it must be 2 (BTREE_BLOBKEY) for an index or WITHOUT ROWID table.
The root page number of the new b-tree is stored in register P2.

CursorHint
Provide a hint to cursor P1 that it only needs to return rows that
satisfy the Expr in P4.  TK_REGISTER terms in the P4 expression refer
to values currently held in registers.  TK_COLUMN terms in the P4
expression refer to columns in the b-tree to which cursor P1 is pointing.

CursorLock
Lock the btree to which cursor P1 is pointing so that the btree cannot be
written by an other cursor.

CursorUnlock
Unlock the btree to which cursor P1 is pointing so that it can be
written by other cursors.

DecrJumpZero
Register P1 must hold an integer.  Decrement the value in P1
and jump to P2 if the new value is exactly zero.

DeferredSeek
P1 is an open index cursor and P3 is a cursor on the corresponding
table.  This opcode does a deferred seek of the P3 table cursor
to the row that corresponds to the current row of P1.

This is a deferred seek.  Nothing actually happens until
the cursor is used to read a record.  That way, if no reads
occur, no unnecessary I/O happens.

P4 may be an array of integers (type P4_INTARRAY) containing
one entry for each column in the P3 table.  If array entry a(i)
is non-zero, then reading column a(i)-1 from cursor P3 is
equivalent to performing the deferred seek and then reading column i
from P1.  This information is stored in P3 and used to redirect
reads against P3 over to P1, thus possibly avoiding the need to
seek and read cursor P3.

Delete
Delete the record at which the P1 cursor is currently pointing.

If the OPFLAG_SAVEPOSITION bit of the P5 parameter is set, then
the cursor will be left pointing at  either the next or the previous
record in the table. If it is left pointing at the next record, then
the next Next instruction will be a no-op. As a result, in this case
it is ok to delete a record from within a Next loop. If
OPFLAG_SAVEPOSITION bit of P5 is clear, then the cursor will be
left in an undefined state.

If the OPFLAG_AUXDELETE bit is set on P5, that indicates that this
delete one of several associated with deleting a table row and all its
associated index entries.  Exactly one of those deletes is the "primary"
delete.  The others are all on OPFLAG_FORDELETE cursors or else are
marked with the AUXDELETE flag.

If the OPFLAG_NCHANGE flag of P2 (NB: P2 not P5) is set, then the row
change count is incremented (otherwise not).

P1 must not be pseudo-table.  It has to be a real table with
multiple rows.

If P4 is not NULL then it points to a Table object. In this case either
the update or pre-update hook, or both, may be invoked. The P1 cursor must
have been positioned using NotFound prior to invoking this opcode in
this case. Specifically, if one is configured, the pre-update hook is
invoked if P4 is not NULL. The update-hook is invoked if one is configured,
P4 is not NULL, and the OPFLAG_NCHANGE flag is set in P2.

If the OPFLAG_ISUPDATE flag is set in P2, then P3 contains the address
of the memory cell that contains the value that the rowid of the row will
be set to by the update.

Destroy
Delete an entire database table or index whose root page in the database
file is given by P1.

The table being destroyed is in the main database file if P3==0.  If
P3==1 then the table to be clear is in the auxiliary database file
that is used to store tables create using CREATE TEMPORARY TABLE.

If AUTOVACUUM is enabled then it is possible that another root page
might be moved into the newly deleted root page in order to keep all
root pages contiguous at the beginning of the database.  The former
value of the root page that moved - its value before the move occurred -
is stored in register P2. If no page movement was required (because the
table being dropped was already the last one in the database) then a
zero is stored in register P2.  If AUTOVACUUM is disabled then a zero
is stored in register P2.

This opcode throws an error if there are any active reader VMs when
it is invoked. This is done to avoid the difficulty associated with
updating existing cursors when a root page is moved in an AUTOVACUUM
database. This error is thrown even if the database is not an AUTOVACUUM
db in order to avoid introducing an incompatibility between autovacuum
and non-autovacuum modes.

See also: Clear

Divide
Divide the value in register P1 by the value in register P2
and store the result in register P3 (P3=P2/P1). If the value in
register P1 is zero, then the result is NULL. If either input is
NULL, the result is NULL.

DropIndex
Remove the internal (in-memory) data structures that describe
the index named P4 in database P1.  This is called after an index
is dropped from disk (using the Destroy opcode)
in order to keep the internal representation of the
schema consistent with what is on disk.

DropTable
Remove the internal (in-memory) data structures that describe
the table named P4 in database P1.  This is called after a table
is dropped from disk (using the Destroy opcode) in order to keep
the internal representation of the
schema consistent with what is on disk.

DropTrigger
Remove the internal (in-memory) data structures that describe
the trigger named P4 in database P1.  This is called after a trigger
is dropped from disk (using the Destroy opcode) in order to keep
the internal representation of the
schema consistent with what is on disk.

ElseEq
This opcode must follow an Lt or Gt comparison operator.  There
can be zero or more OP_ReleaseReg opcodes intervening, but no other
opcodes are allowed to occur between this instruction and the previous
Lt or Gt.

If result of an Eq comparison on the same two operands as the
prior Lt or Gt would have been true, then jump to P2.
If the result of an Eq comparison on the two previous
operands would have been false or NULL, then fall through.

EndCoroutine
The instruction at the address in register P1 is a Yield.
Jump to the P2 parameter of that Yield.
After the jump, register P1 becomes undefined.

See also: InitCoroutine

Eq
Compare the values in register P1 and P3.  If reg(P3)==reg(P1) then
jump to address P2.

The SQLITE_AFF_MASK portion of P5 must be an affinity character -
SQLITE_AFF_TEXT, SQLITE_AFF_INTEGER, and so forth. An attempt is made
to coerce both inputs according to this affinity before the
comparison is made. If the SQLITE_AFF_MASK is 0x00, then numeric
affinity is used. Note that the affinity conversions are stored
back into the input registers P1 and P3.  So this opcode can cause
persistent changes to registers P1 and P3.

Once any conversions have taken place, and neither value is NULL,
the values are compared. If both values are blobs then memcmp() is
used to determine the results of the comparison.  If both values
are text, then the appropriate collating function specified in
P4 is used to do the comparison.  If P4 is not specified then
memcmp() is used to compare text string.  If both values are
numeric, then a numeric comparison is used. If the two values
are of different types, then numbers are considered less than
strings and strings are considered less than blobs.

If SQLITE_NULLEQ is set in P5 then the result of comparison is always either
true or false and is never NULL.  If both operands are NULL then the result
of comparison is true.  If either operand is NULL then the result is false.
If neither operand is NULL the result is the same as it would be if
the SQLITE_NULLEQ flag were omitted from P5.

This opcode saves the result of comparison for use by the new
Jump opcode.

Expire
Cause precompiled statements to expire.  When an expired statement
is executed using sqlite3_step() it will either automatically
reprepare itself (if it was originally created using sqlite3_prepare_v2())
or it will fail with SQLITE_SCHEMA.

If P1 is 0, then all SQL statements become expired. If P1 is non-zero,
then only the currently executing statement is expired.

If P2 is 0, then SQL statements are expired immediately.  If P2 is 1,
then running SQL statements are allowed to continue to run to completion.
The P2==1 case occurs when a CREATE INDEX or similar schema change happens
that might help the statement run faster but which does not affect the
correctness of operation.

Filter
Compute a hash on the key contained in the P4 registers starting
with r&#91;P3&#93;.  Check to see if that hash is found in the
bloom filter hosted by register P1.  If it is not present then
maybe jump to P2.  Otherwise fall through.

False negatives are harmless.  It is always safe to fall through,
even if the value is in the bloom filter.  A false negative causes
more CPU cycles to be used, but it should still yield the correct
answer.  However, an incorrect answer may well arise from a
false positive - if the jump is taken when it should fall through.

FilterAdd
Compute a hash on the P4 registers starting with r&#91;P3&#93; and
add that hash to the bloom filter contained in r&#91;P1&#93;.

FinishSeek
If cursor P1 was previously moved via DeferredSeek, complete that
seek operation now, without further delay.  If the cursor seek has
already occurred, this instruction is a no-op.

FkCheck
Halt with an SQLITE_CONSTRAINT error if there are any unresolved
foreign key constraint violations.  If there are no foreign key
constraint violations, this is a no-op.

FK constraint violations are also checked when the prepared statement
exits.  This opcode is used to raise foreign key constraint errors prior
to returning results such as a row change count or the result of a
RETURNING clause.

FkCounter
Increment a "constraint counter" by P2 (P2 may be negative or positive).
If P1 is non-zero, the database constraint counter is incremented
(deferred foreign key constraints). Otherwise, if P1 is zero, the
statement counter is incremented (immediate foreign key constraints).

FkIfZero
This opcode tests if a foreign key constraint-counter is currently zero.
If so, jump to instruction P2. Otherwise, fall through to the next
instruction.

If P1 is non-zero, then the jump is taken if the database constraint-counter
is zero (the one that counts deferred constraint violations). If P1 is
zero, the jump is taken if the statement constraint-counter is zero
(immediate foreign key constraint violations).

Found
If P4==0 then register P3 holds a blob constructed by MakeRecord.  If
P4&gt;0 then register P3 is the first of P4 registers that form an unpacked
record.

Cursor P1 is on an index btree.  If the record identified by P3 and P4
is a prefix of any entry in P1 then a jump is made to P2 and
P1 is left pointing at the matching entry.

This operation leaves the cursor in a state where it can be
advanced in the forward direction.  The Next instruction will work,
but not the Prev instruction.

See also: NotFound, NoConflict, NotExists. SeekGe

Function
Invoke a user function (P4 is a pointer to an sqlite3_context object that
contains a pointer to the function to be run) with arguments taken
from register P2 and successors.  The number of arguments is in
the sqlite3_context object that P4 points to.
The result of the function is stored
in register P3.  Register P3 must not be one of the function inputs.

P1 is a 32-bit bitmask indicating whether or not each argument to the
function was determined to be constant at compile time. If the first
argument was constant then bit 0 of P1 is set. This is used to determine
whether meta data associated with a user function argument using the
sqlite3_set_auxdata() API may be safely retained until the next
invocation of this opcode.

See also: AggStep, AggFinal, PureFunc

Ge
This works just like the Lt opcode except that the jump is taken if
the content of register P3 is greater than or equal to the content of
register P1.  See the Lt opcode for additional information.

Gosub
Write the current address onto register P1
and then jump to address P2.

Goto
An unconditional jump to address P2.
The next instruction executed will be
the one at index P2 from the beginning of
the program.

The P1 parameter is not actually used by this opcode.  However, it
is sometimes set to 1 instead of 0 as a hint to the command-line shell
that this Goto is the bottom of a loop and that the lines from P2 down
to the current line should be indented for EXPLAIN output.

Gt
This works just like the Lt opcode except that the jump is taken if
the content of register P3 is greater than the content of
register P1.  See the Lt opcode for additional information.

Halt
Exit immediately.  All open cursors, etc are closed
automatically.

P1 is the result code returned by sqlite3_exec(), sqlite3_reset(),
or sqlite3_finalize().  For a normal halt, this should be SQLITE_OK (0).
For errors, it can be some other value.  If P1!=0 then P2 will determine
whether or not to rollback the current transaction.  Do not rollback
if P2==OE_Fail. Do the rollback if P2==OE_Rollback.  If P2==OE_Abort,
then back out all changes that have occurred during this execution of the
VDBE, but do not rollback the transaction.

If P4 is not null then it is an error message string.

P5 is a value between 0 and 4, inclusive, that modifies the P4 string.

0:  (no change)
1:  NOT NULL contraint failed: P4
2:  UNIQUE constraint failed: P4
3:  CHECK constraint failed: P4
4:  FOREIGN KEY constraint failed: P4

If P5 is not zero and P4 is NULL, then everything after the ":" is
omitted.

There is an implied "Halt 0 0 0" instruction inserted at the very end of
every program.  So a jump past the last instruction of the program
is the same as executing Halt.

HaltIfNull
Check the value in register P3.  If it is NULL then Halt using
parameter P1, P2, and P4 as if this were a Halt instruction.  If the
value in register P3 is not NULL, then this routine is a no-op.
The P5 parameter should be 1.

IdxDelete
The content of P3 registers starting at register P2 form
an unpacked index key. This opcode removes that entry from the
index opened by cursor P1.

If P5 is not zero, then raise an SQLITE_CORRUPT_INDEX error
if no matching index entry is found.  This happens when running
an UPDATE or DELETE statement and the index entry to be updated
or deleted is not found.  For some uses of IdxDelete
(example:  the EXCEPT operator) it does not matter that no matching
entry is found.  For those cases, P5 is zero.  Also, do not raise
this (self-correcting and non-critical) error if in writable_schema mode.

IdxGE
The P4 register values beginning with P3 form an unpacked index
key that omits the PRIMARY KEY.  Compare this key value against the index
that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID
fields at the end.

If the P1 index entry is greater than or equal to the key value
then jump to P2.  Otherwise fall through to the next instruction.

IdxGT
The P4 register values beginning with P3 form an unpacked index
key that omits the PRIMARY KEY.  Compare this key value against the index
that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID
fields at the end.

If the P1 index entry is greater than the key value
then jump to P2.  Otherwise fall through to the next instruction.

IdxInsert
Register P2 holds an SQL index key made using the
MakeRecord instructions.  This opcode writes that key
into the index P1.  Data for the entry is nil.

If P4 is not zero, then it is the number of values in the unpacked
key of reg(P2).  In that case, P3 is the index of the first register
for the unpacked key.  The availability of the unpacked key can sometimes
be an optimization.

If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer
that this insert is likely to be an append.

If P5 has the OPFLAG_NCHANGE bit set, then the change counter is
incremented by this instruction.  If the OPFLAG_NCHANGE bit is clear,
then the change counter is unchanged.

If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might
run faster by avoiding an unnecessary seek on cursor P1.  However,
the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior
seeks on the cursor or if the most recent seek used a key equivalent
to P2.

This instruction only works for indices.  The equivalent instruction
for tables is Insert.

IdxLE
The P4 register values beginning with P3 form an unpacked index
key that omits the PRIMARY KEY or ROWID.  Compare this key value against
the index that P1 is currently pointing to, ignoring the PRIMARY KEY or
ROWID on the P1 index.

If the P1 index entry is less than or equal to the key value then jump
to P2. Otherwise fall through to the next instruction.

IdxLT
The P4 register values beginning with P3 form an unpacked index
key that omits the PRIMARY KEY or ROWID.  Compare this key value against
the index that P1 is currently pointing to, ignoring the PRIMARY KEY or
ROWID on the P1 index.

If the P1 index entry is less than the key value then jump to P2.
Otherwise fall through to the next instruction.

IdxRowid
Write into register P2 an integer which is the last entry in the record at
the end of the index key pointed to by cursor P1.  This integer should be
the rowid of the table entry to which this index entry points.

See also: Rowid, MakeRecord.

If
Jump to P2 if the value in register P1 is true.  The value
is considered true if it is numeric and non-zero.  If the value
in P1 is NULL then take the jump if and only if P3 is non-zero.

IfNoHope
Register P3 is the first of P4 registers that form an unpacked
record.  Cursor P1 is an index btree.  P2 is a jump destination.
In other words, the operands to this opcode are the same as the
operands to NotFound and IdxGT.

This opcode is an optimization attempt only.  If this opcode always
falls through, the correct answer is still obtained, but extra works
is performed.

A value of N in the seekHit flag of cursor P1 means that there exists
a key P3:N that will match some record in the index.  We want to know
if it is possible for a record P3:P4 to match some record in the
index.  If it is not possible, we can skips some work.  So if seekHit
is less than P4, attempt to find out if a match is possible by running
NotFound.

This opcode is used in IN clause processing for a multi-column key.
If an IN clause is attached to an element of the key other than the
left-most element, and if there are no matches on the most recent
seek over the whole key, then it might be that one of the key element
to the left is prohibiting a match, and hence there is "no hope" of
any match regardless of how many IN clause elements are checked.
In such a case, we abandon the IN clause search early, using this
opcode.  The opcode name comes from the fact that the
jump is taken if there is "no hope" of achieving a match.

See also: NotFound, SeekHit

IfNot
Jump to P2 if the value in register P1 is False.  The value
is considered false if it has a numeric value of zero.  If the value
in P1 is NULL then take the jump if and only if P3 is non-zero.

IfNotOpen
If cursor P1 is not open or if P1 is set to a NULL row using the
NullRow opcode, then jump to instruction P2. Otherwise, fall through.

IfNotZero
Register P1 must contain an integer.  If the content of register P1 is
initially greater than zero, then decrement the value in register P1.
If it is non-zero (negative or positive) and then also jump to P2.
If register P1 is initially zero, leave it unchanged and fall through.

IfNullRow
Check the cursor P1 to see if it is currently pointing at a NULL row.
If it is, then set register P3 to NULL and jump immediately to P2.
If P1 is not on a NULL row, then fall through without making any
changes.

If P1 is not an open cursor, then this opcode is a no-op.

IfPos
Register P1 must contain an integer.
If the value of register P1 is 1 or greater, subtract P3 from the
value in P1 and jump to P2.

If the initial value of register P1 is less than 1, then the
value is unchanged and control passes through to the next instruction.

IfSmaller
Estimate the number of rows in the table P1.  Jump to P2 if that
estimate is less than approximately 2**(0.1*P3).

IncrVacuum
Perform a single step of the incremental vacuum procedure on
the P1 database. If the vacuum has finished, jump to instruction
P2. Otherwise, fall through to the next instruction.

Init
Programs contain a single instance of this opcode as the very first
opcode.

If tracing is enabled (by the sqlite3_trace()) interface, then
the UTF-8 string contained in P4 is emitted on the trace callback.
Or if P4 is blank, use the string returned by sqlite3_sql().

If P2 is not zero, jump to instruction P2.

Increment the value of P1 so that Once opcodes will jump the
first time they are evaluated for this run.

If P3 is not zero, then it is an address to jump to if an SQLITE_CORRUPT
error is encountered.

InitCoroutine
Set up register P1 so that it will Yield to the coroutine
located at address P3.

If P2!=0 then the coroutine implementation immediately follows
this opcode.  So jump over the coroutine implementation to
address P2.

See also: EndCoroutine

Insert
Write an entry into the table of cursor P1.  A new entry is
created if it doesn't already exist or the data for an existing
entry is overwritten.  The data is the value MEM_Blob stored in register
number P2. The key is stored in register P3. The key must
be a MEM_Int.

If the OPFLAG_NCHANGE flag of P5 is set, then the row change count is
incremented (otherwise not).  If the OPFLAG_LASTROWID flag of P5 is set,
then rowid is stored for subsequent return by the
sqlite3_last_insert_rowid() function (otherwise it is unmodified).

If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might
run faster by avoiding an unnecessary seek on cursor P1.  However,
the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior
seeks on the cursor or if the most recent seek used a key equal to P3.

If the OPFLAG_ISUPDATE flag is set, then this opcode is part of an
UPDATE operation.  Otherwise (if the flag is clear) then this opcode
is part of an INSERT operation.  The difference is only important to
the update hook.

Parameter P4 may point to a Table structure, or may be NULL. If it is
not NULL, then the update-hook (sqlite3.xUpdateCallback) is invoked
following a successful insert.

(WARNING/TODO: If P1 is a pseudo-cursor and P2 is dynamically
allocated, then ownership of P2 is transferred to the pseudo-cursor
and register P2 becomes ephemeral.  If the cursor is changed, the
value of register P2 will then change.  Make sure this does not
cause any problems.)

This instruction only works on tables.  The equivalent instruction
for indices is IdxInsert.

Int64
P4 is a pointer to a 64-bit integer value.
Write that value into register P2.

IntCopy
Transfer the integer value held in register P1 into register P2.

This is an optimized version of SCopy that works only for integer
values.

Integer
The 32-bit integer value P1 is written into register P2.

IntegrityCk
Do an analysis of the currently open database.  Store in
register P1 the text of an error message describing any problems.
If no problems are found, store a NULL in register P1.

The register P3 contains one less than the maximum number of allowed errors.
At most reg(P3) errors will be reported.
In other words, the analysis stops as soon as reg(P1) errors are
seen.  Reg(P1) is updated with the number of errors remaining.

The root page numbers of all tables in the database are integers
stored in P4_INTARRAY argument.

If P5 is not zero, the check is done on the auxiliary database
file, not the main database file.

This opcode is used to implement the integrity_check pragma.

IsNull
Jump to P2 if the value in register P1 is NULL.

IsTrue
This opcode implements the IS TRUE, IS FALSE, IS NOT TRUE, and
IS NOT FALSE operators.

Interpret the value in register P1 as a boolean value.  Store that
boolean (a 0 or 1) in register P2.  Or if the value in register P1 is
NULL, then the P3 is stored in register P2.  Invert the answer if P4
is 1.

The logic is summarized like this:


 If P3==0 and P4==0  then  r&#91;P2&#93; := r&#91;P1&#93; IS TRUE
 If P3==1 and P4==1  then  r&#91;P2&#93; := r&#91;P1&#93; IS FALSE
 If P3==0 and P4==1  then  r&#91;P2&#93; := r&#91;P1&#93; IS NOT TRUE
 If P3==1 and P4==0  then  r&#91;P2&#93; := r&#91;P1&#93; IS NOT FALSE


IsType
Jump to P2 if the type of a column in a btree is one of the types specified
by the P5 bitmask.

P1 is normally a cursor on a btree for which the row decode cache is
valid through at least column P3.  In other words, there should have been
a prior Column for column P3 or greater.  If the cursor is not valid,
then this opcode might give spurious results.
The the btree row has fewer than P3 columns, then use P4 as the
datatype.

If P1 is -1, then P3 is a register number and the datatype is taken
from the value in that register.

P5 is a bitmask of data types.  SQLITE_INTEGER is the least significant
(0x01) bit. SQLITE_FLOAT is the 0x02 bit. SQLITE_TEXT is 0x04.
SQLITE_BLOB is 0x08.  SQLITE_NULL is 0x10.

Take the jump to address P2 if and only if the datatype of the
value determined by P1 and P3 corresponds to one of the bits in the
P5 bitmask.

JournalMode
Change the journal mode of database P1 to P3. P3 must be one of the
PAGER_JOURNALMODE_XXX values. If changing between the various rollback
modes (delete, truncate, persist, off and memory), this is a simple
operation. No IO is required.

If changing into or out of WAL mode the procedure is more complicated.

Write a string containing the final journal-mode to register P2.

Jump
Jump to the instruction at address P1, P2, or P3 depending on whether
in the most recent Compare instruction the P1 vector was less than
equal to, or greater than the P2 vector, respectively.

This opcode must immediately follow an Compare opcode.

Last
The next use of the Rowid or Column or Prev instruction for P1
will refer to the last entry in the database table or index.
If the table or index is empty and P2&gt;0, then jump immediately to P2.
If P2 is 0 or if the table or index is not empty, fall through
to the following instruction.

This opcode leaves the cursor configured to move in reverse order,
from the end toward the beginning.  In other words, the cursor is
configured to use Prev, not Next.

Le
This works just like the Lt opcode except that the jump is taken if
the content of register P3 is less than or equal to the content of
register P1.  See the Lt opcode for additional information.

LoadAnalysis
Read the sqlite_stat1 table for database P1 and load the content
of that table into the internal index hash table.  This will cause
the analysis to be used when preparing all subsequent queries.

Lt
Compare the values in register P1 and P3.  If reg(P3)&lt;reg(P1) then
jump to address P2.

If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or
reg(P3) is NULL then the take the jump.  If the SQLITE_JUMPIFNULL
bit is clear then fall through if either operand is NULL.

The SQLITE_AFF_MASK portion of P5 must be an affinity character -
SQLITE_AFF_TEXT, SQLITE_AFF_INTEGER, and so forth. An attempt is made
to coerce both inputs according to this affinity before the
comparison is made. If the SQLITE_AFF_MASK is 0x00, then numeric
affinity is used. Note that the affinity conversions are stored
back into the input registers P1 and P3.  So this opcode can cause
persistent changes to registers P1 and P3.

Once any conversions have taken place, and neither value is NULL,
the values are compared. If both values are blobs then memcmp() is
used to determine the results of the comparison.  If both values
are text, then the appropriate collating function specified in
P4 is  used to do the comparison.  If P4 is not specified then
memcmp() is used to compare text string.  If both values are
numeric, then a numeric comparison is used. If the two values
are of different types, then numbers are considered less than
strings and strings are considered less than blobs.

This opcode saves the result of comparison for use by the new
Jump opcode.

MakeRecord
Convert P2 registers beginning with P1 into the record format
use as a data record in a database table or as a key
in an index.  The Column opcode can decode the record later.

P4 may be a string that is P2 characters long.  The N-th character of the
string indicates the column affinity that should be used for the N-th
field of the index key.

The mapping from character to affinity is given by the SQLITE_AFF_
macros defined in sqliteInt.h.

If P4 is NULL then all index fields have the affinity BLOB.

The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM
compile-time option is enabled:

* If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index
of the right-most table that can be null-trimmed.

* If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value
OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to
accept no-change records with serial_type 10.  This value is
only used inside an assert() and does not affect the end result.

MaxPgcnt
Try to set the maximum page count for database P1 to the value in P3.
Do not let the maximum page count fall below the current page count and
do not change the maximum page count value if P3==0.

Store the maximum page count after the change in register P2.

MemMax
P1 is a register in the root frame of this VM (the root frame is
different from the current frame if this instruction is being executed
within a sub-program). Set the value of register P1 to the maximum of
its current value and the value in register P2.

This instruction throws an error if the memory cell is not initially
an integer.

Move
Move the P3 values in register P1..P1+P3-1 over into
registers P2..P2+P3-1.  Registers P1..P1+P3-1 are
left holding a NULL.  It is an error for register ranges
P1..P1+P3-1 and P2..P2+P3-1 to overlap.  It is an error
for P3 to be less than 1.

Multiply
Multiply the value in register P1 by the value in register P2
and store the result in register P3.
If either input is NULL, the result is NULL.

MustBeInt
Force the value in register P1 to be an integer.  If the value
in P1 is not an integer and cannot be converted into an integer
without data loss, then jump immediately to P2, or if P2==0
raise an SQLITE_MISMATCH exception.

Ne
This works just like the Eq opcode except that the jump is taken if
the operands in registers P1 and P3 are not equal.  See the Eq opcode for
additional information.

NewRowid
Get a new integer record number (a.k.a "rowid") used as the key to a table.
The record number is not previously used as a key in the database
table that cursor P1 points to.  The new record number is written
written to register P2.

If P3&gt;0 then P3 is a register in the root frame of this VDBE that holds
the largest previously generated record number. No new record numbers are
allowed to be less than this value. When this value reaches its maximum,
an SQLITE_FULL error is generated. The P3 register is updated with the '
generated record number. This P3 mechanism is used to help implement the
AUTOINCREMENT feature.

Next
Advance cursor P1 so that it points to the next key/data pair in its
table or index.  If there are no more key/value pairs then fall through
to the following instruction.  But if the cursor advance was successful,
jump immediately to P2.

The Next opcode is only valid following an SeekGT, SeekGE, or
Rewind opcode used to position the cursor.  Next is not allowed
to follow SeekLT, SeekLE, or Last.

The P1 cursor must be for a real table, not a pseudo-table.  P1 must have
been opened prior to this opcode or the program will segfault.

The P3 value is a hint to the btree implementation. If P3==1, that
means P1 is an SQL index and that this instruction could have been
omitted if that index had been unique.  P3 is usually 0.  P3 is
always either 0 or 1.

If P5 is positive and the jump is taken, then event counter
number P5-1 in the prepared statement is incremented.

See also: Prev

NoConflict
If P4==0 then register P3 holds a blob constructed by MakeRecord.  If
P4&gt;0 then register P3 is the first of P4 registers that form an unpacked
record.

Cursor P1 is on an index btree.  If the record identified by P3 and P4
contains any NULL value, jump immediately to P2.  If all terms of the
record are not-NULL then a check is done to determine if any row in the
P1 index btree has a matching key prefix.  If there are no matches, jump
immediately to P2.  If there is a match, fall through and leave the P1
cursor pointing to the matching row.

This opcode is similar to NotFound with the exceptions that the
branch is always taken if any part of the search key input is NULL.

This operation leaves the cursor in a state where it cannot be
advanced in either direction.  In other words, the Next and Prev
opcodes do not work after this operation.

See also: NotFound, Found, NotExists

Noop
Do nothing.  This instruction is often useful as a jump
destination.

Not
Interpret the value in register P1 as a boolean value.  Store the
boolean complement in register P2.  If the value in register P1 is
NULL, then a NULL is stored in P2.

NotExists
P1 is the index of a cursor open on an SQL table btree (with integer
keys).  P3 is an integer rowid.  If P1 does not contain a record with
rowid P3 then jump immediately to P2.  Or, if P2 is 0, raise an
SQLITE_CORRUPT error. If P1 does contain a record with rowid P3 then
leave the cursor pointing at that record and fall through to the next
instruction.

The SeekRowid opcode performs the same operation but also allows the
P3 register to contain a non-integer value, in which case the jump is
always taken.  This opcode requires that P3 always contain an integer.

The NotFound opcode performs the same operation on index btrees
(with arbitrary multi-value keys).

This opcode leaves the cursor in a state where it cannot be advanced
in either direction.  In other words, the Next and Prev opcodes will
not work following this opcode.

See also: Found, NotFound, NoConflict, SeekRowid

NotFound
If P4==0 then register P3 holds a blob constructed by MakeRecord.  If
P4&gt;0 then register P3 is the first of P4 registers that form an unpacked
record.

Cursor P1 is on an index btree.  If the record identified by P3 and P4
is not the prefix of any entry in P1 then a jump is made to P2.  If P1
does contain an entry whose prefix matches the P3/P4 record then control
falls through to the next instruction and P1 is left pointing at the
matching entry.

This operation leaves the cursor in a state where it cannot be
advanced in either direction.  In other words, the Next and Prev
opcodes do not work after this operation.

See also: Found, NotExists, NoConflict, IfNoHope

NotNull
Jump to P2 if the value in register P1 is not NULL.

Null
Write a NULL into registers P2.  If P3 greater than P2, then also write
NULL into register P3 and every register in between P2 and P3.  If P3
is less than P2 (typically P3 is zero) then only register P2 is
set to NULL.

If the P1 value is non-zero, then also set the MEM_Cleared flag so that
NULL values will not compare equal even if SQLITE_NULLEQ is set on
Ne or Eq.

NullRow
Move the cursor P1 to a null row.  Any Column operations
that occur while the cursor is on the null row will always
write a NULL.

If cursor P1 is not previously opened, open it now to a special
pseudo-cursor that always returns NULL for every column.

Offset
Store in register r&#91;P3&#93; the byte offset into the database file that is the
start of the payload for the record at which that cursor P1 is currently
pointing.

P2 is the column number for the argument to the sqlite_offset() function.
This opcode does not use P2 itself, but the P2 value is used by the
code generator.  The P1, P2, and P3 operands to this opcode are the
same as for Column.

This opcode is only available if SQLite is compiled with the
-DSQLITE_ENABLE_OFFSET_SQL_FUNC option.

OffsetLimit
This opcode performs a commonly used computation associated with
LIMIT and OFFSET processing.  r&#91;P1&#93; holds the limit counter.  r&#91;P3&#93;
holds the offset counter.  The opcode computes the combined value
of the LIMIT and OFFSET and stores that value in r&#91;P2&#93;.  The r&#91;P2&#93;
value computed is the total number of rows that will need to be
visited in order to complete the query.

If r&#91;P3&#93; is zero or negative, that means there is no OFFSET
and r&#91;P2&#93; is set to be the value of the LIMIT, r&#91;P1&#93;.

if r&#91;P1&#93; is zero or negative, that means there is no LIMIT
and r&#91;P2&#93; is set to -1.

Otherwise, r&#91;P2&#93; is set to the sum of r&#91;P1&#93; and r&#91;P3&#93;.

Once
Fall through to the next instruction the first time this opcode is
encountered on each invocation of the byte-code program.  Jump to P2
on the second and all subsequent encounters during the same invocation.

Top-level programs determine first invocation by comparing the P1
operand against the P1 operand on the Init opcode at the beginning
of the program.  If the P1 values differ, then fall through and make
the P1 of this opcode equal to the P1 of Init.  If P1 values are
the same then take the jump.

For subprograms, there is a bitmask in the VdbeFrame that determines
whether or not the jump should be taken.  The bitmask is necessary
because the self-altering code trick does not work for recursive
triggers.

OpenAutoindex
This opcode works the same as OpenEphemeral.  It has a
different name to distinguish its use.  Tables created using
by this opcode will be used for automatically created transient
indices in joins.

OpenDup
Open a new cursor P1 that points to the same ephemeral table as
cursor P2.  The P2 cursor must have been opened by a prior OpenEphemeral
opcode.  Only ephemeral cursors may be duplicated.

Duplicate ephemeral cursors are used for self-joins of materialized views.

OpenEphemeral
Open a new cursor P1 to a transient table.
The cursor is always opened read/write even if
the main database is read-only.  The ephemeral
table is deleted automatically when the cursor is closed.

If the cursor P1 is already opened on an ephemeral table, the table
is cleared (all content is erased).

P2 is the number of columns in the ephemeral table.
The cursor points to a BTree table if P4==0 and to a BTree index
if P4 is not 0.  If P4 is not NULL, it points to a KeyInfo structure
that defines the format of keys in the index.

The P5 parameter can be a mask of the BTREE_* flags defined
in btree.h.  These flags control aspects of the operation of
the btree.  The BTREE_OMIT_JOURNAL and BTREE_SINGLE flags are
added automatically.

If P3 is positive, then reg&#91;P3&#93; is modified slightly so that it
can be used as zero-length data for Insert.  This is an optimization
that avoids an extra Blob opcode to initialize that register.

OpenPseudo
Open a new cursor that points to a fake table that contains a single
row of data.  The content of that one row is the content of memory
register P2.  In other words, cursor P1 becomes an alias for the
MEM_Blob content contained in register P2.

A pseudo-table created by this opcode is used to hold a single
row output from the sorter so that the row can be decomposed into
individual columns using the Column opcode.  The Column opcode
is the only cursor opcode that works with a pseudo-table.

P3 is the number of fields in the records that will be stored by
the pseudo-table.

OpenRead
Open a read-only cursor for the database table whose root page is
P2 in a database file.  The database file is determined by P3.
P3==0 means the main database, P3==1 means the database used for
temporary tables, and P3&gt;1 means used the corresponding attached
database.  Give the new cursor an identifier of P1.  The P1
values need not be contiguous but all P1 values should be small integers.
It is an error for P1 to be negative.

Allowed P5 bits:

  0x02 OPFLAG_SEEKEQ: This cursor will only be used for
equality lookups (implemented as a pair of opcodes SeekGE/IdxGT
of SeekLE/IdxLT)


The P4 value may be either an integer (P4_INT32) or a pointer to
a KeyInfo structure (P4_KEYINFO). If it is a pointer to a KeyInfo
object, then table being opened must be an index b-tree where the
KeyInfo object defines the content and collating
sequence of that index b-tree. Otherwise, if P4 is an integer
value, then the table being opened must be a table b-tree with a
number of columns no less than the value of P4.

See also: OpenWrite, ReopenIdx

OpenWrite
Open a read/write cursor named P1 on the table or index whose root
page is P2 (or whose root page is held in register P2 if the
OPFLAG_P2ISREG bit is set in P5 - see below).

The P4 value may be either an integer (P4_INT32) or a pointer to
a KeyInfo structure (P4_KEYINFO). If it is a pointer to a KeyInfo
object, then table being opened must be an index b-tree where the
KeyInfo object defines the content and collating
sequence of that index b-tree. Otherwise, if P4 is an integer
value, then the table being opened must be a table b-tree with a
number of columns no less than the value of P4.

Allowed P5 bits:

  0x02 OPFLAG_SEEKEQ: This cursor will only be used for
equality lookups (implemented as a pair of opcodes SeekGE/IdxGT
of SeekLE/IdxLT)
  0x08 OPFLAG_FORDELETE: This cursor is used only to seek
and subsequently delete entries in an index btree.  This is a
hint to the storage engine that the storage engine is allowed to
ignore.  The hint is not used by the official SQLite b*tree storage
engine, but is used by COMDB2.
  0x10 OPFLAG_P2ISREG: Use the content of register P2
as the root page, not the value of P2 itself.


This instruction works like OpenRead except that it opens the cursor
in read/write mode.

See also: OpenRead, ReopenIdx

Or
Take the logical OR of the values in register P1 and P2 and
store the answer in register P3.

If either P1 or P2 is nonzero (true) then the result is 1 (true)
even if the other input is NULL.  A NULL and false or two NULLs
give a NULL output.

Pagecount
Write the current number of pages in database P1 to memory cell P2.

Param
This opcode is only ever present in sub-programs called via the
Program instruction. Copy a value currently stored in a memory
cell of the calling (parent) frame to cell P2 in the current frames
address space. This is used by trigger programs to access the new.*
and old.* values.

The address of the cell in the parent frame is determined by adding
the value of the P1 argument to the value of the P1 argument to the
calling Program instruction.

ParseSchema
Read and parse all entries from the schema table of database P1
that match the WHERE clause P4.  If P4 is a NULL pointer, then the
entire schema for P1 is reparsed.

This opcode invokes the parser to create a new virtual machine,
then runs the new virtual machine.  It is thus a re-entrant opcode.

Permutation
Set the permutation used by the Compare operator in the next
instruction.  The permutation is stored in the P4 operand.

The permutation is only valid for the next opcode which must be
an Compare that has the OPFLAG_PERMUTE bit set in P5.

The first integer in the P4 integer array is the length of the array
and does not become part of the permutation.

Prev
Back up cursor P1 so that it points to the previous key/data pair in its
table or index.  If there is no previous key/value pairs then fall through
to the following instruction.  But if the cursor backup was successful,
jump immediately to P2.

The Prev opcode is only valid following an SeekLT, SeekLE, or
Last opcode used to position the cursor.  Prev is not allowed
to follow SeekGT, SeekGE, or Rewind.

The P1 cursor must be for a real table, not a pseudo-table.  If P1 is
not open then the behavior is undefined.

The P3 value is a hint to the btree implementation. If P3==1, that
means P1 is an SQL index and that this instruction could have been
omitted if that index had been unique.  P3 is usually 0.  P3 is
always either 0 or 1.

If P5 is positive and the jump is taken, then event counter
number P5-1 in the prepared statement is incremented.

Program
Execute the trigger program passed as P4 (type P4_SUBPROGRAM).

P1 contains the address of the memory cell that contains the first memory
cell in an array of values used as arguments to the sub-program. P2
contains the address to jump to if the sub-program throws an IGNORE
exception using the RAISE() function. Register P3 contains the address
of a memory cell in this (the parent) VM that is used to allocate the
memory required by the sub-vdbe at runtime.

P4 is a pointer to the VM containing the trigger program.

If P5 is non-zero, then recursive program invocation is enabled.

PureFunc
Invoke a user function (P4 is a pointer to an sqlite3_context object that
contains a pointer to the function to be run) with arguments taken
from register P2 and successors.  The number of arguments is in
the sqlite3_context object that P4 points to.
The result of the function is stored
in register P3.  Register P3 must not be one of the function inputs.

P1 is a 32-bit bitmask indicating whether or not each argument to the
function was determined to be constant at compile time. If the first
argument was constant then bit 0 of P1 is set. This is used to determine
whether meta data associated with a user function argument using the
sqlite3_set_auxdata() API may be safely retained until the next
invocation of this opcode.

This opcode works exactly like Function.  The only difference is in
its name.  This opcode is used in places where the function must be
purely non-deterministic.  Some built-in date/time functions can be
either determinitic of non-deterministic, depending on their arguments.
When those function are used in a non-deterministic way, they will check
to see if they were called using PureFunc instead of Function, and
if they were, they throw an error.

See also: AggStep, AggFinal, Function

ReadCookie
Read cookie number P3 from database P1 and write it into register P2.
P3==1 is the schema version.  P3==2 is the database format.
P3==3 is the recommended pager cache size, and so forth.  P1==0 is
the main database file and P1==1 is the database file used to store
temporary tables.

There must be a read-lock on the database (either a transaction
must be started or there must be an open cursor) before
executing this instruction.

Real
P4 is a pointer to a 64-bit floating point value.
Write that value into register P2.

RealAffinity
If register P1 holds an integer convert it to a real value.

This opcode is used when extracting information from a column that
has REAL affinity.  Such column values may still be stored as
integers, for space efficiency, but after extraction we want them
to have only a real value.

ReleaseReg
Release registers from service.  Any content that was in the
the registers is unreliable after this opcode completes.

The registers released will be the P2 registers starting at P1,
except if bit ii of P3 set, then do not release register P1+ii.
In other words, P3 is a mask of registers to preserve.

Releasing a register clears the Mem.pScopyFrom pointer.  That means
that if the content of the released register was set using SCopy,
a change to the value of the source register for the SCopy will no longer
generate an assertion fault in sqlite3VdbeMemAboutToChange().

If P5 is set, then all released registers have their type set
to MEM_Undefined so that any subsequent attempt to read the released
register (before it is reinitialized) will generate an assertion fault.

P5 ought to be set on every call to this opcode.
However, there are places in the code generator will release registers
before their are used, under the (valid) assumption that the registers
will not be reallocated for some other purpose before they are used and
hence are safe to release.

This opcode is only available in testing and debugging builds.  It is
not generated for release builds.  The purpose of this opcode is to help
validate the generated bytecode.  This opcode does not actually contribute
to computing an answer.

Remainder
Compute the remainder after integer register P2 is divided by
register P1 and store the result in register P3.
If the value in register P1 is zero the result is NULL.
If either operand is NULL, the result is NULL.

ReopenIdx
The ReopenIdx opcode works like OpenRead except that it first
checks to see if the cursor on P1 is already open on the same
b-tree and if it is this opcode becomes a no-op.  In other words,
if the cursor is already open, do not reopen it.

The ReopenIdx opcode may only be used with P5==0 or P5==OPFLAG_SEEKEQ
and with P4 being a P4_KEYINFO object.  Furthermore, the P3 value must
be the same as every other ReopenIdx or OpenRead for the same cursor
number.

Allowed P5 bits:

  0x02 OPFLAG_SEEKEQ: This cursor will only be used for
equality lookups (implemented as a pair of opcodes SeekGE/IdxGT
of SeekLE/IdxLT)


See also: OpenRead, OpenWrite

ResetCount
The value of the change counter is copied to the database handle
change counter (returned by subsequent calls to sqlite3_changes()).
Then the VMs internal change counter resets to 0.
This is used by trigger programs.

ResetSorter
Delete all contents from the ephemeral table or sorter
that is open on cursor P1.

This opcode only works for cursors used for sorting and
opened with OpenEphemeral or SorterOpen.

ResultRow
The registers P1 through P1+P2-1 contain a single row of
results. This opcode causes the sqlite3_step() call to terminate
with an SQLITE_ROW return code and it sets up the sqlite3_stmt
structure to provide access to the r(P1)..r(P1+P2-1) values as
the result row.

Return
Jump to the address stored in register P1.  If P1 is a return address
register, then this accomplishes a return from a subroutine.

If P3 is 1, then the jump is only taken if register P1 holds an integer
values, otherwise execution falls through to the next opcode, and the
Return becomes a no-op. If P3 is 0, then register P1 must hold an
integer or else an assert() is raised.  P3 should be set to 1 when
this opcode is used in combination with BeginSubrtn, and set to 0
otherwise.

The value in register P1 is unchanged by this opcode.

P2 is not used by the byte-code engine.  However, if P2 is positive
and also less than the current address, then the "EXPLAIN" output
formatter in the CLI will indent all opcodes from the P2 opcode up
to be not including the current Return.   P2 should be the first opcode
in the subroutine from which this opcode is returning.  Thus the P2
value is a byte-code indentation hint.  See tag-20220407a in
wherecode.c and shell.c.

Rewind
The next use of the Rowid or Column or Next instruction for P1
will refer to the first entry in the database table or index.
If the table or index is empty, jump immediately to P2.
If the table or index is not empty, fall through to the following
instruction.

This opcode leaves the cursor configured to move in forward order,
from the beginning toward the end.  In other words, the cursor is
configured to use Next, not Prev.

RowCell
P1 and P2 are both open cursors. Both must be opened on the same type
of table - intkey or index. This opcode is used as part of copying
the current row from P2 into P1. If the cursors are opened on intkey
tables, register P3 contains the rowid to use with the new record in
P1. If they are opened on index tables, P3 is not used.

This opcode must be followed by either an Insert or InsertIdx opcode
with the OPFLAG_PREFORMAT flag set to complete the insert operation.

RowData
Write into register P2 the complete row content for the row at
which cursor P1 is currently pointing.
There is no interpretation of the data.
It is just copied onto the P2 register exactly as
it is found in the database file.

If cursor P1 is an index, then the content is the key of the row.
If cursor P2 is a table, then the content extracted is the data.

If the P1 cursor must be pointing to a valid row (not a NULL row)
of a real table, not a pseudo-table.

If P3!=0 then this opcode is allowed to make an ephemeral pointer
into the database page.  That means that the content of the output
register will be invalidated as soon as the cursor moves - including
moves caused by other cursors that "save" the current cursors
position in order that they can write to the same table.  If P3==0
then a copy of the data is made into memory.  P3!=0 is faster, but
P3==0 is safer.

If P3!=0 then the content of the P2 register is unsuitable for use
in OP_Result and any OP_Result will invalidate the P2 register content.
The P2 register content is invalidated by opcodes like Function or
by any use of another cursor pointing to the same table.

Rowid
Store in register P2 an integer which is the key of the table entry that
P1 is currently point to.

P1 can be either an ordinary table or a virtual table.  There used to
be a separate OP_VRowid opcode for use with virtual tables, but this
one opcode now works for both table types.

RowSetAdd
Insert the integer value held by register P2 into a RowSet object
held in register P1.

An assertion fails if P2 is not an integer.

RowSetRead
Extract the smallest value from the RowSet object in P1
and put that value into register P3.
Or, if RowSet object P1 is initially empty, leave P3
unchanged and jump to instruction P2.

RowSetTest
Register P3 is assumed to hold a 64-bit integer value. If register P1
contains a RowSet object and that RowSet object contains
the value held in P3, jump to register P2. Otherwise, insert the
integer in P3 into the RowSet and continue on to the
next opcode.

The RowSet object is optimized for the case where sets of integers
are inserted in distinct phases, which each set contains no duplicates.
Each set is identified by a unique P4 value. The first set
must have P4==0, the final set must have P4==-1, and for all other sets
must have P4&gt;0.

This allows optimizations: (a) when P4==0 there is no need to test
the RowSet object for P3, as it is guaranteed not to contain it,
(b) when P4==-1 there is no need to insert the value, as it will
never be tested for, and (c) when a value that is part of set X is
inserted, there is no need to search to see if the same value was
previously inserted as part of set X (only if it was previously
inserted as part of some other set).

Savepoint
Open, release or rollback the savepoint named by parameter P4, depending
on the value of P1. To open a new savepoint set P1==0 (SAVEPOINT_BEGIN).
To release (commit) an existing savepoint set P1==1 (SAVEPOINT_RELEASE).
To rollback an existing savepoint set P1==2 (SAVEPOINT_ROLLBACK).

SCopy
Make a shallow copy of register P1 into register P2.

This instruction makes a shallow copy of the value.  If the value
is a string or blob, then the copy is only a pointer to the
original and hence if the original changes so will the copy.
Worse, if the original is deallocated, the copy becomes invalid.
Thus the program must guarantee that the original will not change
during the lifetime of the copy.  Use Copy to make a complete
copy.

SeekEnd
Position cursor P1 at the end of the btree for the purpose of
appending a new entry onto the btree.

It is assumed that the cursor is used only for appending and so
if the cursor is valid, then the cursor must already be pointing
at the end of the btree and so no changes are made to
the cursor.

SeekGE
If cursor P1 refers to an SQL table (B-Tree that uses integer keys),
use the value in register P3 as the key.  If cursor P1 refers
to an SQL index, then P3 is the first in an array of P4 registers
that are used as an unpacked index key.

Reposition cursor P1 so that  it points to the smallest entry that
is greater than or equal to the key value. If there are no records
greater than or equal to the key and P2 is not zero, then jump to P2.

If the cursor P1 was opened using the OPFLAG_SEEKEQ flag, then this
opcode will either land on a record that exactly matches the key, or
else it will cause a jump to P2.  When the cursor is OPFLAG_SEEKEQ,
this opcode must be followed by an IdxLE opcode with the same arguments.
The IdxGT opcode will be skipped if this opcode succeeds, but the
IdxGT opcode will be used on subsequent loop iterations.  The
OPFLAG_SEEKEQ flags is a hint to the btree layer to say that this
is an equality search.

This opcode leaves the cursor configured to move in forward order,
from the beginning toward the end.  In other words, the cursor is
configured to use Next, not Prev.

See also: Found, NotFound, SeekLt, SeekGt, SeekLe

SeekGT
If cursor P1 refers to an SQL table (B-Tree that uses integer keys),
use the value in register P3 as a key. If cursor P1 refers
to an SQL index, then P3 is the first in an array of P4 registers
that are used as an unpacked index key.

Reposition cursor P1 so that it points to the smallest entry that
is greater than the key value. If there are no records greater than
the key and P2 is not zero, then jump to P2.

This opcode leaves the cursor configured to move in forward order,
from the beginning toward the end.  In other words, the cursor is
configured to use Next, not Prev.

See also: Found, NotFound, SeekLt, SeekGe, SeekLe

SeekHit
Increase or decrease the seekHit value for cursor P1, if necessary,
so that it is no less than P2 and no greater than P3.

The seekHit integer represents the maximum of terms in an index for which
there is known to be at least one match.  If the seekHit value is smaller
than the total number of equality terms in an index lookup, then the
IfNoHope opcode might run to see if the IN loop can be abandoned
early, thus saving work.  This is part of the IN-early-out optimization.

P1 must be a valid b-tree cursor.

SeekLE
If cursor P1 refers to an SQL table (B-Tree that uses integer keys),
use the value in register P3 as a key. If cursor P1 refers
to an SQL index, then P3 is the first in an array of P4 registers
that are used as an unpacked index key.

Reposition cursor P1 so that it points to the largest entry that
is less than or equal to the key value. If there are no records
less than or equal to the key and P2 is not zero, then jump to P2.

This opcode leaves the cursor configured to move in reverse order,
from the end toward the beginning.  In other words, the cursor is
configured to use Prev, not Next.

If the cursor P1 was opened using the OPFLAG_SEEKEQ flag, then this
opcode will either land on a record that exactly matches the key, or
else it will cause a jump to P2.  When the cursor is OPFLAG_SEEKEQ,
this opcode must be followed by an IdxLE opcode with the same arguments.
The IdxGE opcode will be skipped if this opcode succeeds, but the
IdxGE opcode will be used on subsequent loop iterations.  The
OPFLAG_SEEKEQ flags is a hint to the btree layer to say that this
is an equality search.

See also: Found, NotFound, SeekGt, SeekGe, SeekLt

SeekLT
If cursor P1 refers to an SQL table (B-Tree that uses integer keys),
use the value in register P3 as a key. If cursor P1 refers
to an SQL index, then P3 is the first in an array of P4 registers
that are used as an unpacked index key.

Reposition cursor P1 so that  it points to the largest entry that
is less than the key value. If there are no records less than
the key and P2 is not zero, then jump to P2.

This opcode leaves the cursor configured to move in reverse order,
from the end toward the beginning.  In other words, the cursor is
configured to use Prev, not Next.

See also: Found, NotFound, SeekGt, SeekGe, SeekLe

SeekRowid
P1 is the index of a cursor open on an SQL table btree (with integer
keys).  If register P3 does not contain an integer or if P1 does not
contain a record with rowid P3 then jump immediately to P2.
Or, if P2 is 0, raise an SQLITE_CORRUPT error. If P1 does contain
a record with rowid P3 then
leave the cursor pointing at that record and fall through to the next
instruction.

The NotExists opcode performs the same operation, but with NotExists
the P3 register must be guaranteed to contain an integer value.  With this
opcode, register P3 might not contain an integer.

The NotFound opcode performs the same operation on index btrees
(with arbitrary multi-value keys).

This opcode leaves the cursor in a state where it cannot be advanced
in either direction.  In other words, the Next and Prev opcodes will
not work following this opcode.

See also: Found, NotFound, NoConflict, SeekRowid

SeekScan
This opcode is a prefix opcode to SeekGE.  In other words, this
opcode must be immediately followed by SeekGE. This constraint is
checked by assert() statements.

This opcode uses the P1 through P4 operands of the subsequent
SeekGE.  In the text that follows, the operands of the subsequent
SeekGE opcode are denoted as SeekOP.P1 through SeekOP.P4.   Only
the P1, P2 and P5 operands of this opcode are also used, and  are called
This.P1, This.P2 and This.P5.

This opcode helps to optimize IN operators on a multi-column index
where the IN operator is on the later terms of the index by avoiding
unnecessary seeks on the btree, substituting steps to the next row
of the b-tree instead.  A correct answer is obtained if this opcode
is omitted or is a no-op.

The SeekGE.P3 and SeekGE.P4 operands identify an unpacked key which
is the desired entry that we want the cursor SeekGE.P1 to be pointing
to.  Call this SeekGE.P3/P4 row the "target".

If the SeekGE.P1 cursor is not currently pointing to a valid row,
then this opcode is a no-op and control passes through into the SeekGE.

If the SeekGE.P1 cursor is pointing to a valid row, then that row
might be the target row, or it might be near and slightly before the
target row, or it might be after the target row.  If the cursor is
currently before the target row, then this opcode attempts to position
the cursor on or after the target row by invoking sqlite3BtreeStep()
on the cursor between 1 and This.P1 times.

The This.P5 parameter is a flag that indicates what to do if the
cursor ends up pointing at a valid row that is past the target
row.  If This.P5 is false (0) then a jump is made to SeekGE.P2.  If
This.P5 is true (non-zero) then a jump is made to This.P2.  The P5==0
case occurs when there are no inequality constraints to the right of
the IN constraing.  The jump to SeekGE.P2 ends the loop.  The P5!=0 case
occurs when there are inequality constraints to the right of the IN
operator.  In that case, the This.P2 will point either directly to or
to setup code prior to the IdxGT or IdxGE opcode that checks for
loop terminate.

Possible outcomes from this opcode:

 If the cursor is initally not pointed to any valid row, then
fall through into the subsequent SeekGE opcode.

 If the cursor is left pointing to a row that is before the target
row, even after making as many as This.P1 calls to
sqlite3BtreeNext(), then also fall through into SeekGE.

 If the cursor is left pointing at the target row, either because it
was at the target row to begin with or because one or more
sqlite3BtreeNext() calls moved the cursor to the target row,
then jump to This.P2..,

 If the cursor started out before the target row and a call to
to sqlite3BtreeNext() moved the cursor off the end of the index
(indicating that the target row definitely does not exist in the
btree) then jump to SeekGE.P2, ending the loop.

 If the cursor ends up on a valid row that is past the target row
(indicating that the target row does not exist in the btree) then
jump to SeekOP.P2 if This.P5==0 or to This.P2 if This.P5&gt;0.


Sequence
Find the next available sequence number for cursor P1.
Write the sequence number into register P2.
The sequence number on the cursor is incremented after this
instruction.

SequenceTest
P1 is a sorter cursor. If the sequence counter is currently zero, jump
to P2. Regardless of whether or not the jump is taken, increment the
the sequence value.

SetCookie
Write the integer value P3 into cookie number P2 of database P1.
P2==1 is the schema version.  P2==2 is the database format.
P2==3 is the recommended pager cache
size, and so forth.  P1==0 is the main database file and P1==1 is the
database file used to store temporary tables.

A transaction must be started before executing this opcode.

If P2 is the SCHEMA_VERSION cookie (cookie number 1) then the internal
schema version is set to P3-P5.  The "PRAGMA schema_version=N" statement
has P5 set to 1, so that the internal schema version will be different
from the database schema version, resulting in a schema reset.

ShiftLeft
Shift the integer value in register P2 to the left by the
number of bits specified by the integer in register P1.
Store the result in register P3.
If either input is NULL, the result is NULL.

ShiftRight
Shift the integer value in register P2 to the right by the
number of bits specified by the integer in register P1.
Store the result in register P3.
If either input is NULL, the result is NULL.

SoftNull
Set register P1 to have the value NULL as seen by the MakeRecord
instruction, but do not free any string or blob memory associated with
the register, so that if the value was a string or blob that was
previously copied using SCopy, the copies will continue to be valid.

Sort
This opcode does exactly the same thing as Rewind except that
it increments an undocumented global variable used for testing.

Sorting is accomplished by writing records into a sorting index,
then rewinding that index and playing it back from beginning to
end.  We use the Sort opcode instead of Rewind to do the
rewinding so that the global variable will be incremented and
regression tests can determine whether or not the optimizer is
correctly optimizing out sorts.

SorterCompare
P1 is a sorter cursor. This instruction compares a prefix of the
record blob in register P3 against a prefix of the entry that
the sorter cursor currently points to.  Only the first P4 fields
of r&#91;P3&#93; and the sorter record are compared.

If either P3 or the sorter contains a NULL in one of their significant
fields (not counting the P4 fields at the end which are ignored) then
the comparison is assumed to be equal.

Fall through to next instruction if the two records compare equal to
each other.  Jump to P2 if they are different.

SorterData
Write into register P2 the current sorter data for sorter cursor P1.
Then clear the column header cache on cursor P3.

This opcode is normally use to move a record out of the sorter and into
a register that is the source for a pseudo-table cursor created using
OpenPseudo.  That pseudo-table cursor is the one that is identified by
parameter P3.  Clearing the P3 column cache as part of this opcode saves
us from having to issue a separate NullRow instruction to clear that cache.

SorterInsert
Register P2 holds an SQL index key made using the
MakeRecord instructions.  This opcode writes that key
into the sorter P1.  Data for the entry is nil.

SorterNext
This opcode works just like Next except that P1 must be a
sorter object for which the SorterSort opcode has been
invoked.  This opcode advances the cursor to the next sorted
record, or jumps to P2 if there are no more sorted records.

SorterOpen
This opcode works like OpenEphemeral except that it opens
a transient index that is specifically designed to sort large
tables using an external merge-sort algorithm.

If argument P3 is non-zero, then it indicates that the sorter may
assume that a stable sort considering the first P3 fields of each
key is sufficient to produce the required results.

SorterSort
After all records have been inserted into the Sorter object
identified by P1, invoke this opcode to actually do the sorting.
Jump to P2 if there are no records to be sorted.

This opcode is an alias for Sort and Rewind that is used
for Sorter objects.

SqlExec
Run the SQL statement or statements specified in the P4 string.

String
The string value P4 of length P1 (bytes) is stored in register P2.

If P3 is not zero and the content of register P3 is equal to P5, then
the datatype of the register P2 is converted to BLOB.  The content is
the same sequence of bytes, it is merely interpreted as a BLOB instead
of a string, as if it had been CAST.  In other words:

if( P3!=0 and reg&#91;P3&#93;==P5 ) reg&#91;P2&#93; := CAST(reg&#91;P2&#93; as BLOB)

String8
P4 points to a nul terminated UTF-8 string. This opcode is transformed
into a String opcode before it is executed for the first time.  During
this transformation, the length of string P4 is computed and stored
as the P1 parameter.

Subtract
Subtract the value in register P1 from the value in register P2
and store the result in register P3.
If either input is NULL, the result is NULL.

TableLock
Obtain a lock on a particular table. This instruction is only used when
the shared-cache feature is enabled.

P1 is the index of the database in sqlite3.aDb[] of the database
on which the lock is acquired.  A readlock is obtained if P3==0 or
a write lock if P3==1.

P2 contains the root-page of the table to lock.

P4 contains a pointer to the name of the table being locked. This is only
used to generate an error message if the lock cannot be obtained.

Trace
Write P4 on the statement trace output if statement tracing is
enabled.

Operand P1 must be 0x7fffffff and P2 must positive.

Transaction
Begin a transaction on database P1 if a transaction is not already
active.
If P2 is non-zero, then a write-transaction is started, or if a
read-transaction is already active, it is upgraded to a write-transaction.
If P2 is zero, then a read-transaction is started.  If P2 is 2 or more
then an exclusive transaction is started.

P1 is the index of the database file on which the transaction is
started.  Index 0 is the main database file and index 1 is the
file used for temporary tables.  Indices of 2 or more are used for
attached databases.

If a write-transaction is started and the Vdbe.usesStmtJournal flag is
true (this flag is set if the Vdbe may modify more than one row and may
throw an ABORT exception), a statement transaction may also be opened.
More specifically, a statement transaction is opened iff the database
connection is currently not in autocommit mode, or if there are other
active statements. A statement transaction allows the changes made by this
VDBE to be rolled back after an error without having to roll back the
entire transaction. If no error is encountered, the statement transaction
will automatically commit when the VDBE halts.

If P5!=0 then this opcode also checks the schema cookie against P3
and the schema generation counter against P4.
The cookie changes its value whenever the database schema changes.
This operation is used to detect when that the cookie has changed
and that the current process needs to reread the schema.  If the schema
cookie in P3 differs from the schema cookie in the database header or
if the schema generation counter in P4 differs from the current
generation counter, then an SQLITE_SCHEMA error is raised and execution
halts.  The sqlite3_step() wrapper function might then reprepare the
statement and rerun it from the beginning.

TypeCheck
Apply affinities to the range of P2 registers beginning with P1.
Take the affinities from the Table object in P4.  If any value
cannot be coerced into the correct type, then raise an error.

This opcode is similar to Affinity except that this opcode
forces the register type to the Table column type.  This is used
to implement "strict affinity".

GENERATED ALWAYS AS ... STATIC columns are only checked if P3
is zero.  When P3 is non-zero, no type checking occurs for
static generated columns.  Virtual columns are computed at query time
and so they are never checked.

Preconditions:


 P2 should be the number of non-virtual columns in the
table of P4.
 Table P4 should be a STRICT table.


If any precondition is false, an assertion fault occurs.

Vacuum
Vacuum the entire database P1.  P1 is 0 for "main", and 2 or more
for an attached database.  The "temp" database may not be vacuumed.

If P2 is not zero, then it is a register holding a string which is
the file into which the result of vacuum should be written.  When
P2 is zero, the vacuum overwrites the original database.

Variable
Transfer the values of bound parameter P1 into register P2

If the parameter is named, then its name appears in P4.
The P4 value is used by sqlite3_bind_parameter_name().

VBegin
P4 may be a pointer to an sqlite3_vtab structure. If so, call the
xBegin method for that table.

Also, whether or not P4 is set, check that this is not being called from
within a callback to a virtual table xSync() method. If it is, the error
code will be set to SQLITE_LOCKED.

VColumn
Store in register P3 the value of the P2-th column of
the current row of the virtual-table of cursor P1.

If the VColumn opcode is being used to fetch the value of
an unchanging column during an UPDATE operation, then the P5
value is OPFLAG_NOCHNG.  This will cause the sqlite3_vtab_nochange()
function to return true inside the xColumn method of the virtual
table implementation.  The P5 column might also contain other
bits (OPFLAG_LENGTHARG or OPFLAG_TYPEOFARG) but those bits are
unused by VColumn.

VCreate
P2 is a register that holds the name of a virtual table in database
P1. Call the xCreate method for that table.

VDestroy
P4 is the name of a virtual table in database P1.  Call the xDestroy method
of that table.

VFilter
P1 is a cursor opened using VOpen.  P2 is an address to jump to if
the filtered result set is empty.

P4 is either NULL or a string that was generated by the xBestIndex
method of the module.  The interpretation of the P4 string is left
to the module implementation.

This opcode invokes the xFilter method on the virtual table specified
by P1.  The integer query plan parameter to xFilter is stored in register
P3. Register P3+1 stores the argc parameter to be passed to the
xFilter method. Registers P3+2..P3+1+argc are the argc
additional parameters which are passed to
xFilter as argv. Register P3+2 becomes argv[0] when passed to xFilter.

A jump is made to P2 if the result set after filtering would be empty.

VInitIn
Set register P2 to be a pointer to a ValueList object for cursor P1
with cache register P3 and output register P3+1.  This ValueList object
can be used as the first argument to sqlite3_vtab_in_first() and
sqlite3_vtab_in_next() to extract all of the values stored in the P1
cursor.  Register P3 is used to hold the values returned by
sqlite3_vtab_in_first() and sqlite3_vtab_in_next().

VNext
Advance virtual table P1 to the next row in its result set and
jump to instruction P2.  Or, if the virtual table has reached
the end of its result set, then fall through to the next instruction.

VOpen
P4 is a pointer to a virtual table object, an sqlite3_vtab structure.
P1 is a cursor number.  This opcode opens a cursor to the virtual
table and stores that cursor in P1.

VRename
P4 is a pointer to a virtual table object, an sqlite3_vtab structure.
This opcode invokes the corresponding xRename method. The value
in register P1 is passed as the zName argument to the xRename method.

VUpdate
P4 is a pointer to a virtual table object, an sqlite3_vtab structure.
This opcode invokes the corresponding xUpdate method. P2 values
are contiguous memory cells starting at P3 to pass to the xUpdate
invocation. The value in register (P3+P2-1) corresponds to the
p2th element of the argv array passed to xUpdate.

The xUpdate method will do a DELETE or an INSERT or both.
The argv[0] element (which corresponds to memory cell P3)
is the rowid of a row to delete.  If argv[0] is NULL then no
deletion occurs.  The argv[1] element is the rowid of the new
row.  This can be NULL to have the virtual table select the new
rowid for itself.  The subsequent elements in the array are
the values of columns in the new row.

If P2==1 then no insert is performed.  argv[0] is the rowid of
a row to delete.

P1 is a boolean flag. If it is set to true and the xUpdate call
is successful, then the value returned by sqlite3_last_insert_rowid()
is set to the value of the rowid for the row just inserted.

P5 is the error actions (OE_Replace, OE_Fail, OE_Ignore, etc) to
apply in the case of a constraint failure on an insert or update.

Yield
Swap the program counter with the value in register P1.  This
has the effect of yielding to a coroutine.

If the coroutine that is launched by this instruction ends with
Yield or Return then continue to the next instruction.  But if
the coroutine launched by this instruction ends with
EndCoroutine, then jump to P2 rather than continuing with the
next instruction.

See also: InitCoroutine

ZeroOrNull
If all both registers P1 and P3 are NOT NULL, then store a zero in
register P2.  If either registers P1 or P3 are NULL then put
a NULL in register P2.

  
  

This page last modified on  2022-08-10 18:45:48 UTC


c2: SQLite Over a Network, Caveats and Considerations
c5: useovernet.html
c4:
SQLite Over a Network,Caveats and Considerations


Introduction

  Users of the SQLite library, particularly application developers,
  who want to access a SQLite database
  from different systems connected by a network are often
  tempted to simply open a database connection by specifying
  a filename which references a database file somewhere within
  a network filesystem. ("remote database" here)
  This "file" is then accessed by means of
  OS API's which permit the illusion of I/O from/to a local file.
  The illusion is good but imperfect in important ways.

  This simple, "remote database" approach is usually not the best way
  to use a single SQLite database from multiple systems,
  (even if it appears to "work"),
  as it often leads to various kinds of trouble and grief.
  Because these problems are inevitable with some usages,
  but not frequent or repeatable,
  it behooves application developers to not rely
  on early testing success to decide
  that their remote database use will work as desired.

Issues Arising with Remote Database Files

This diagram shows components and their linkages
for reference in the discussion following:







  The issues arise from the properties and utilization
  of the two data/control channels between the above three blocks.


Channel Traffic Volume

  The "API Call" channel carries less information
  than the "File I/O" channel.
  API calls to submit queries or specify data modification
  normally require substantially fewer bits to be passed
  back and forth than are transferred to/from
  the database file to store or find the data.
  Query result retrieval will normally require much more file traffic
  than API traffic because the data to be returned is rarely
  to be found without reading unrequested data.

Channel Bandwidth

  The API Call channel operates at processor main memory speeds
  (Giga-words/second), with data often passed by reference (and so not copied.)
  In contrast, even the fastest File I/O channels are slower.
  They require the data to be copied, usually over a medium
  requiring bit-serialization. For spinning magnetic media,
  transfers await platter rotation and head movement, then
  are limited by spin velocity.


  When the File I/O channel includes a network connection,
  (in addition to some genuine File I/O at its far end),
  additional slowness is imposed. Even where raw transfer
  rate does not limit bandwidth, the traffic must still be
  packetized and buffered at both ends.
  Additional layers of I/O handlers add scheduling delays.
  However, slowed transfers are the least significant
  issue with network filesystems.

Channel Reliability

  The "API Call" channel is highly reliable, to the extent
  that error rates are unstated and ignored as negligible.
  The channel fails only when the system loses power
  (excepting meteorites, etc.)


  The "File I/O" channel, when it directly reaches a local storage device,
  is also highly reliable.
  (Spinning storage MTBF exceeds 1 million hours,
  and NVRAM lasts longer.)
  Local devices also have a characteristic
  which is critical for enabling database management software
  to be designed to ensure ACID behavior:
  When all process writes to the device have completed,
  (when POSIX fsync() or Windows FlushFileBuffers() calls return),
  the filesystem then either has
  stored the "written" data or will do so
  before storing any subsequently written data.


  When network filesystem apparatus and software layers are interposed
  between filesystem clients and a filesystem over an actual storage device,
  significant sources of failure and misbehavior are introduced.
  While network data transfers are error-checked well, transfer packets
  do not all reliably arrive at their destination once sent.
  Some packets are clobbered by other packets and must be resent.
  Under packet clobbering conditions, repeated retries
  can impose delays exceeding
  what is needed for similar data to reach local storage.
  Some portions of what a client writes can end up stored
  out of time order relative to other portions written.


  Because of the disordering and outright data loss
  which occur in network filesystem writes, it is critical
  that sets of file writes can be accurately known to be done
  before a subsequent set of file writes begins.
  This assurance is obtained by use of robustly designed
  and correctly implemented fsync() (or equivalent) OS functions.
  Unfortunately for some applications, network filesystem sync
  operation can be less robust than local filesystem sync.
  Attaining robust sync in the face of network packet transport errors
  is hard, and safeguards are sometimes relaxed in favor of performance.


  A similar hazard arises with file locking in network filesystems.
  SQLite relies on exclusive locks for write operations, and those have
  been known to operate incorrectly for some network filesystems. This
  has led to database corruption. That may happen again as the designers
  of such change their implementation to suit more common use cases.


  The bottom line is that network filesystem sync and locking reliability
  vary among implementations and installations. The design
  assumptions upon which it relies may hold more true where
  an application is tested than where it is relied upon.
  Rely upon it at your (and your customers') peril.
  See How To Corrupt Your Database Files.


Performance and Reliability Issues

  From the above diagram and discussion, it is obvious that
  performance (aka "speed") is degraded by insertion
  of a network link into one of the two channels.
  Consideration of relative traffic volumes between
  the API Call channel and the File I/O channel
  reveals that such insertion will have less performance
  impact at the API Call channel.


  Consideration of reliability impact is easier, with a clearer outcome:
  Inserting a network link into the API Call channel may also result
  in call failures at times. But if the Client Application
  has bothered to use SQL/SQLite transactions properly,
  such failures will only cause a transaction to fail
  and be rolled back, without compromising the integrity
  of the data. In contrast, if the network link is
  inserted into the File I/O channel, transactions may fail
  (as for the API Call insertion) but with the additional
  effect that the remote database is corrupted.


  These network unreliability issues can be mitigated,
  completely or to an acceptable degree,
  by using SQLite in rollback mode.
  However, the SQLite library is not tested in across-a-network
  scenarios, nor is that reasonably possible.
  Hence, use of a remote database is done at the user's risk.


Recommendations

  Generally, if your data is separated from the application
  by a network, you want to use a client/server database.
  This is due to the fact that the database engine acts
  as a bandwidth-reducing filter on the database traffic.

  If your data is separated from the application by a network,
  you want the low-traffic link to be across the network,
  not the high-traffic link. This means that the database engine
  needs to be on the same machine as the database itself.
  Such is the case with a client/server database like PostgreSQL.
  SQLite is different in that the database engine runs on
  the same machine as the application, which forces the
  higher-traffic link to traverse the network in remote
  database scenarios.  That normally results in lower performance.

  Network filesystems do not support the ability to do
  simultaneous reads and writes while at the same time
  keeping the database consistent.
  So if you have multiple clients on multiple different
  machines which need to do simultaneous database
  reads and writes, you have these choices:

  1.  Use a client/server database engine.
  PostgreSQL
  is an excellent choice. A variation of this is:

  2.  Host an SQLite database in WAL mode, but do
  all reads and writes from processes on the same machine
  that stores the database file.
  Implement a proxy that runs on the database machine that
  relays read/write requests from remote machines.

  3.  Use SQLite in rollback mode.
  This means you can have multiple simultaneous readers or one writer,
  but not simultaneous readers and writers.

  Application programmers should be cognizant of the possibility
  that their application's users will elect to use a remote database
  if they can do so. Unless one of the above choices
  has been effected, or one at a time, exclusive access is used,
  a programmer should consider blocking that
  election unless reliability is of little importance.

Summary

  Choose the technology that is right for you and your customers.
  If your data lives on a different machine from your application,
  then you should consider a client/server database.
  SQLite is designed for situations where the data and application
  coexist on the same machine.
  SQLite can still be made to work in many remote database
  situations, but a client/server solution will usually work
  better in that scenario.

This page last modified on  2022-06-22 21:14:29 UTC


c2: The Advantages Of Flexible Typing
c5: flextypegood.html
c4:
1. Introduction

SQLite provides developers with the freedom to store content in
any desired format, regardless of the declared datatype of the column.
Some people find this feature troublesome.  Some developers are shocked
to discover that it is possible to insert text into a column marked INTEGER.

This article advocates in favor of the flexible type rules
in SQLite.

2. About Flexible Typing

Details regarding the flexible type system of SQLite are found
in the separate Datatypes In SQLite document.  Here is a quick
summary:


 Datatype names on column definitions are optional.  A column definition
        can consist of just the column name and nothing else.
 When datatype names are provided, they can be just about any text.
        SQLite attempts to deduce the preferred datatype for the column based
        on the datatype name in the column definition, but that preferred
        datatype is advisory, not mandatory.  The preferred datatype is
        call the "column affinity".
 An attempt is made to transform incoming data into the preferred
        datatype of the column.  (All SQL database engines do this, not
        just SQLite.)  If this transformation is successful, all is well.
        But if unsuccessful, instead of raising an error, SQLite just stores
        the content using its original datatype.
 The above can lead to situations that advocates of rigid typing
        find incommodious:
        
        Column DatatypeTypes Allowed In That Column
        INTEGERINTEGER, REAL, TEXT, BLOB
        REALREAL, TEXT, BLOB
        TEXTTEXT, BLOB
        BLOBINTEGER, REAL, TEXT, BLOB
        
 Note that an INTEGER or REAL value will never end up being stored
        in a TEXT column, since an INTEGER or REAL value can and always will
        be converted into its equivalent TEXT representation.  Similarly,
        and INTEGER will never be stored in a REAL column because it will
        always be converted into a REAL.  But TEXT does not always look like
        and INTEGER or REAL value and so cannot always be converted.  And a
        BLOB cannot be converted into anything and nothing else can be
        converted into a BLOB.



3. Cases Where Flexible Typing Is Useful

Some readers, upon first encountering flexible typing in SQLite, ask
themselves "how could this ever be useful?"  Here is an
attempt to answer that question:

3.1. Attribute tables

 Many applications, especially those that use SQLite as an
application file format, need a place to store miscellaneous attributes
such as thumbnail images (as BLOB values), short pieces of text (such
as the user's name), as well as numeric, date, and JSON values.  It is
convenient to create a single table to handle this storage:

CREATE TABLE attribute(name TEXT PRIMARY KEY, value) WITHOUT ROWID;


Without flexible typing, such a table would need to be more complex,
with separate columns for each possible type of data.  Flexible typing
of the "value" column makes the table conceptually simpler, more
space-efficient, and easier to access and update.

In the Fossil version control system, each
repository has a CONFIG table that is used to store all kinds of settings
with every possible datatype.  The user-specific configuration file
for Fossil (the ~/.fossil file) is a separate SQLite database that contains
a single attribute table hold the user-specific state across all
repositories.

 Some applications use an SQLite database as a pure key-value store
The database schema contains a single table that looks something like this:

CREATE TABLE storage(name TEXT PRIMARY KEY, value ANYTHING);


3.2. The "value" column output from the json_tree virtual tables

The json_tree and json_each table-valued functions that are
built into SQLite both have a "value" column that can hold values of
type INTEGER, REAL, or TEXT depending on the type of the corresponding
JSON field.  For example:

SELECT typeof(value) FROM json_each('{"a":1,"b":2.5,"c":"hello"}');


The query above returns three rows of one column with values
"integer", "real", and "text", respectively.

3.3. Storage for dirty data

 Analysts sometimes encounter CSV files where some columns contain
a mixture of integer, real, and text data.  CSV file that are obtained
from Excel spreadsheet exports commonly have this trait, for example.
When importing such "dirty data" into an SQL database, it is convenient
to have flexible columns to import into.

 Dirty data is not restricted to CSV files coming out of Excel, of
course.  There are many data sources in which a single field might
contain a mix of types.  For example, a data column might the number
of seconds since 1970 sometimes, or as a text date string in other cases.
It is desirable to clean up these inconsistent representations,
but at the same time it is convenient to be able to store all the different
representations in the same column of the intermediate database while the
cleanup is underway.

3.4. Dynamic programming languages

SQLite began as a TCL extension that later escaped into the wild.
TCL is a dynamic language in the sense that the programmer does not need
to be aware of datatypes.  Under the hood, TCL keeps careful track of the
datatype of every value, but to the developer and user of a TCL program,
everything looks like a string.  Flexible typing is a natural fit for
use with dynamic programming languages like TCL and others, since with
a dynamic programming language, you can not always predict in advance what
datatype a variable will hold.  So when you need to store the value of that
variable into the database, having a database that is supports flexible
typing makes storage much easier.

3.5. Data typename cross-compatibility

Every SQL database engine seems to have its own unique set of supported
datatype names:


 BIGINT
 UNSIGNED SMALL INT
 TEXT
 VARCHAR
 VARYING CHARACTER
 NATIONAL VARYING CHARACTER
 NVARCHAR
 JSON
 REAL
 FLOAT
 DOUBLE PRECISION
  ... and so forth ...


The fact that SQLite will accept any of these names as a valid typename,
and let you store any kind of content into the column, increases the chances
that a script written to run on some other SQL database engine will also
work in SQLite.

3.6. Repurposing unused or disused columns in legacy databases

 Because an SQLite database file is a single file on disk, some
applications use SQLite as an application file format.  This means
that a single instance of the application might, over the course of its
life, talk to hundreds or thousands of separate databases, each in a separate
file.  When such applications evolve over years, some columns in the
underlying database will have their meanings altered subtly.  Or, it might
be desirable to repurpose an existing column to serve two or more purposes.
This is much easier to do if the column has a flexible datatype.

4. Perceived Disadvantages of Flexible Typing (With Rebuttals)

 The following perceived disadvantages of flexible typing were
    gleaned and compiled from countless posts on Hacker News and
    Reddit and similar forums where developers discuss these sorts of
    things.  If you can think of other reasons why flexible typing
    is a bad idea, please contact the SQLite developers or leave a
    post on the SQLite Forum so
    that your idea can be added to the list.


4.1. We've never done it that way before

 Many skeptics of flexible typing simply express
shock and disbelief, without offering any rationale for why they think
flexible typing is a bad idea.  Without supporting arguments, one must
assume their reason for not liking flexible typing is that it is different
from what they are used to.

 Presumably, many developers who are aghast at SQLite's flexible
typing feel this way because they have just never encountered anything
like it before.  All prior exposure to databases and especially SQL
databases has involved rigid typing, and the readers mental model of
SQL includes rigid typing as a fundamental feature.  Flexible typing
upsets their world-view.

 Yes, flexible typing is a new way of thinking about data in an
SQL database.  But new is not necessary bad.  Sometimes, and I think
especially in the case of flexible typing, innovation leads to improvement.

4.2. Rigid type enforcement helps prevent application bugs

 It has become a point of doctrine among many programmers that the
best way to prevent application bugs is strict type enforcement.  But I
find no evidence in support of this.

 To be sure, strict type enforcement does help prevent some kinds of
bugs in lower-level languages like C and C++ that present a model that is
close to machine hardware.  But this does not seem to
be the case for higher-abstraction languages in which all data is 
passed around in a "Value" superclass of some kind which is subclassed
for the various lower-level data types.  When everything is a Value
object, specific datatypes cease to be important.

 This technical note is being writing by the original author of SQLite.
I having been writing TCL programs for 27 years.  TCL has no type enforcement
whatsoever.  The "Value" class in TCL (called Tcl_Obj) can hold many
different datatypes, but it presents the content to the program and to
the application user as a string.  And I've had a lot of bugs in
those TCL programs over the years.  But I do not recall a single instance
where the bugs might have been caught by a rigid type system.  I have
also written a lot of C code over a span of 35 years, not the
least of which is SQLite itself.  I have found the type system in C
to be very helpful at finding and preventing problems.  For the
Fossil Version Control System, which is written
in C, I have even implemented supplemental static analysis programs that
scan the Fossil source code prior to compilation, looking for problems
that compilers miss.  This works well for compiled programs.

 The SQL language model is a higher-level abstraction that C/C++.
In SQLite, every data item is stored in memory as an "sqlite3_value" object.
There are subclasses of this object for strings, integer, floating-point
numbers, blobs, and other representations.  Everything is passed around
inside the SQL language implemented by SQLite as "sqlite3_value" objects
so the underlying datatype does not really matter.  I have never found
rigid type enforcement to be helpful in languages like TCL and SQLite
that have a single "Value" superclass used to represent any data element.
Fossil makes extensive use of SQLite in its implementation.  There have
been many bugs in Fossil over its 14-years history, but I cannot recall
a single bug that might have been prevented by rigid type enforcement in
the SQLite.  C-language bugs might have been caught by better type enforcement
(which is why I wrote the supplemental source code scanners), but no
SQL bugs.

 Based on decades of experience, I reject the thesis that rigid
type enforcement helps prevent application bugs.  I will accept and
believe a slightly modified thesis:  Rigid type enforcement helps to
prevent applications bugs in languages that lack a single
top-level "Value" superclass.  But SQLite does have the
single "sqlite3_value" superclass, so that proverb does not apply.

4.3. Rigid type enforcement prevents data pollution

 Some people contend that if you have rigorous constraints on the
schema, and especially strict enforcement of column datatypes, this
will help prevent incorrect data from being added to the database.
This is not true.  It is true that type enforcement might help prevent
egregiously incorrect data from getting into the system.
But type enforcement is no help in prevent subtly incorrect data
from being recorded.

 So, for example, rigid type enforcement can successfully prevent
the customer name (text) from being inserted into integer
Customer.creditScore column.  On the other hand, if that mistake occurs,
it is very easy to spot the problem and find all affected rows.
But type enforcement is no help in preventing a bug where the customer
family name and given name are reversed, since both are text fields.

 (Aside:  Decades ago, I worked on a team where there was a woman
named "Merritt Tracy".  "Merritt" was her given name and "Tracy" was
her family name.  She reported that she spend an inordinate amount of
time and energy trying to correct databases that had "Tracy" has her 
given name and "Merritt" as her family name.)

 By suppressing easy-to-detect errors and passing through only the
hard-to-detect errors, rigid type enforcement can actually make it more
difficult to find and fix bugs.  Data errors tend to cluster.  If you have
20 different data sources, most of the data errors will usually come
from just 2 or 3 of those sources.  The presence of egregious
errors (such as text in an integer column) is a convenient early warning
signal that something is amiss.  The source of the problem can be 
tracked quickly and extra scrutiny applied to the source of the
egregious errors, thus hopefully also fixing the subtle errors too.
When egregious errors are suppressed, you lose an important signal
that helps you to detect and fix the subtle errors.

 Data errors are inevitable.  They will happen regardless of how
much type checking is done.  Rigid type enforcement can catch only
a small subset of those cases - the most obvious cases.  It does
nothing to help find and fix the more subtle cases.  And, by suppressing
the signal of which data sources are problematic, it can sometimes
make the subtle errors more difficult to locate.

4.4. Other SQL database engines don't work this way

 Because SQLite is less restrictive and allows you to do more things,
SQL scripts that work on other database engines will also usually work
on SQLite, but script written initially for SQLite might not work
on more restrictive database engines.  This can cause problems when
developers user SQLite for prototyping and testing and then migrate their
application to a more restrictive SQL engine for deployment.  If the
application was (unintentionally) taking advantage of the flexible
typing available in SQLite, then it will fail when migrated.

 People use this problem to argue that SQLite should be more
restrictive about datatypes.  But you could just as easily turn
that argument around and say that other database engines should be
more flexible with regard to datatypes.  The application was working
correctly under SQLite, prior to be migrated, after all.  If rigid
type enforcement is really all that useful, why did it break an
application that was previously working?

5. If You Insist On Rigid Type Enforcement...

As of SQLite version 3.37.0 (2021-11-27), SQLite supports this
development style using STRICT tables.

If you find a real-world case where STRICT tables prevented or
would have prevented a bug in an application, please post a message to the
SQLite Forum so that we can add your story
to this document.

6. Embrace Freedom

If flexible typing in an SQL database is a new concept to you,
I encourage you to give it a try.  It probably will not cause you
any problems and it might make your program simpler and easier to
write and maintain.  I think that even if you are skeptical at first,
if you will just give flexible typing a try, you will eventually
come to realize that it is a better approach and will start
encouraging other database vendors to support at least an ANY
datatype if not complete SQLite-style type flexibility.

Most of the time, flexible typing does not matter because a column
stores a single well-defined type.  But occasionally you will run
across situations where having a flexible type system makes the
solution to your problem cleaner and easier.
This page last modified on  2021-08-24 21:13:00 UTC


c2: The SQLite Amalgamation
c5: amalgamation.html
c4:
1. Executive Summary

Over 100 separate source files are concatenated into a
single large file of C-code named "sqlite3.c" and
referred to as "the amalgamation". The amalgamation
contains everything an application needs to embed SQLite.

Combining all the code for SQLite into one big file makes SQLite
easier to deploy &mdash; there is just one file to keep track of.
And because all code is in
a single translation unit, compilers can do
better inter-procedure and inlining optimization
resulting in machine code that is between 5% and 10% faster.

2. The SQLite Amalgamation

The SQLite library consists of 111 files of C code
(as of Version 3.37.0 - 2021-11-27)
in the core with 22 additional files that
implement certain commonly used extensions.

Of the 133
main source files, about 75% are C code and about 25% are C header files.
Most of these are "source" files in the sense that they are stored
in the SQLite version control system
and are edited manually in an ordinary text editor.
But some of the C-language files are generated using scripts
or auxiliary programs.  For example, the
parse.y
file contains an LALR(1) grammar of the SQL language which is compiled,
by the Lemon parser generator, to produce a parser contained in the file
"parse.c" accompanied by token identifiers in "parse.h".


The makefiles for SQLite have an "sqlite3.c" target for building the
amalgamation, to contain all C code for the core SQLite library and the
FTS3, FTS5, RTREE, DBSTAT, JSON1,
RBU and SESSION
extensions.
This file contains about 238K lines of code
(or 145K if you omit blank lines and comments) and is over 8.4 megabytes
in size (as of 2021-12-29).

Though the various extensions are included in the
"sqlite3.c" amalgamation file, they are disabled using #ifdef statements.
Activate the extensions using compile-time options like:


 -DSQLITE_ENABLE_FTS3
 -DSQLITE_ENABLE_FTS5
 -DSQLITE_ENABLE_RTREE
 -DSQLITE_ENABLE_DBSTAT_VTAB
 -DSQLITE_ENABLE_JSON1
 -DSQLITE_ENABLE_RBU
 -DSQLITE_ENABLE_SESSION


The amalgamation contains everything you need to integrate SQLite 
into a larger project.  Just copy the amalgamation into your source 
directory and compile it along with the other C code files in your project.
(A more detailed discussion of the compilation process is
available.)
You may also want to make use of the "sqlite3.h" header file that 
defines the programming API for SQLite. 
The sqlite3.h header file is available separately. 
The sqlite3.h file is also contained within the amalgamation, in
the first few thousand lines. So if you have a copy of 
sqlite3.c but cannot seem to locate sqlite3.h, you can always 
regenerate the sqlite3.h by copying and pasting from the amalgamation.

In addition to making SQLite easier to incorporate into other
projects, the amalgamation also makes it run faster. Many
compilers are able to do additional optimizations on code when 
it is contained with in a single translation unit such as it 
is in the amalgamation. We have measured performance improvements
of between 5 and 10% when we use the amalgamation to compile 
SQLite rather than individual source files.  The downside of this
is that the additional optimizations often take the form of 
function inlining which tends to make the size of the resulting
binary image larger.



3. The Split Amalgamation

Developers sometimes experience trouble debugging the
quarter-million line amalgamation source file because some debuggers
are only able to handle source code line numbers less than 32,768.
The amalgamation source code runs fine.  One just cannot single-step
through it in a debugger.

To circumvent this limitation, the amalgamation is also available in
a split form, consisting of files "sqlite3-1.c", "sqlite3-2.c", and
so forth, where each file is less than 32,768 lines in length and
where the concatenation of the files contain all of the code for the
complete amalgamation.  Then there is a separate source file named
"sqlite3-all.c" which basically consists of code like this:

#include "sqlite3-1.c"
#include "sqlite3-2.c"
#include "sqlite3-3.c"
#include "sqlite3-4.c"
#include "sqlite3-5.c"
#include "sqlite3-6.c"
#include "sqlite3-7.c"


Applications using the split amalgamation simply compile against
"sqlite3-all.c" instead of "sqlite3.c".  The two files work exactly
the same.  But with "sqlite3-all.c", no single source file contains more
than 32,767 lines of code, and so it is more convenient to use some
debuggers.  The downside of the split amalgamation is that it consists
of 6 C source code files instead of just 1.

4. Download Copies Of The Amalgamation

The amalgamation and
the sqlite3.h header file are available on
the download page as a file 
named sqlite-amalgamation-X.zip
where the X is replaced by the appropriate version number.



5. Building The Amalgamation From Canonical Source Code

To build the amalgamation (either the full amalgamation or the
split amalgamation), first
get the canonical source code from one of the three servers.
Then, on both unix-like systems and on Windows systems that have the
free MinGW development environment
installed, the amalgamation can be built using the
following commands:

sh configure
make sqlite3.c


To build using Microsoft Visual C++, run this command:

nmake /f makefile.msc sqlite3.c


In both cases, the split amalgamation can be obtained by
substituting "sqlite3-all.c" for "sqlite3.c" as the make target.

5.1. Dependencies

The build process makes extensive use of the 
Tcl scripting language.  You will need to have a
copy of TCL installed in order for the make targets above to work.
Easy-to-use installers can be obtained from http://www.tcl-lang.org/.
Many unix workstations have Tcl installed by default.

5.2. See Also

Additional notes on compiling SQLite can be found on the
How To Compile SQLite page.
This page last modified on  2022-04-25 03:42:01 UTC


c2: SQLite FTS3 and FTS4 Extensions
c5: fts3.html
c4:
Overview


  FTS3 and FTS4 are SQLite virtual table modules that allows users to perform 
  full-text searches on a set of documents. The most common (and effective) 
  way to describe full-text searches is "what Google, Yahoo, and Bing do
  with documents placed on the World Wide Web". Users input a term, or series 
  of terms, perhaps connected by a binary operator or grouped together into a 
  phrase, and the full-text query system finds the set of documents that best 
  matches those terms considering the operators and groupings the user has 
  specified. This article describes the deployment and usage of FTS3 and FTS4.


  FTS1 and FTS2 are obsolete full-text search modules for SQLite.  There are known
  issues with these older modules and their use should be avoided.
  Portions of the original FTS3 code were contributed to the SQLite project 
  by Scott Hess of Google. It is now 
  developed and maintained as part of SQLite.

1. Introduction to FTS3 and FTS4


  The FTS3 and FTS4 extension modules allows users to create special tables with a 
  built-in full-text index (hereafter "FTS tables"). The full-text index
  allows the user to efficiently query the database for all rows that contain
  one or more words (hereafter "tokens"), even if the table
  contains many large documents.


  For example, if each of the 517430 documents in the 
  "Enron E-Mail Dataset"
  is inserted into both an FTS table and an ordinary SQLite table
  created using the following SQL script:

CREATE VIRTUAL TABLE enrondata1 USING fts3(content TEXT);     /* FTS3 table */
CREATE TABLE enrondata2(content TEXT);                        /* Ordinary table */



  Then either of the two queries below may be executed to find the number of
  documents in the database that contain the word "linux" (351). Using one
  desktop PC hardware configuration, the query on the FTS3 table returns in
  approximately 0.03 seconds, versus 22.5 for querying the ordinary table.

SELECT count(*) FROM enrondata1 WHERE content MATCH 'linux';  /* 0.03 seconds */
SELECT count(*) FROM enrondata2 WHERE content LIKE '%linux%'; /* 22.5 seconds */



  Of course, the two queries above are not entirely equivalent. For example
  the LIKE query matches rows that contain terms such as "linuxophobe"
  or "EnterpriseLinux" (as it happens, the Enron E-Mail Dataset does not
  actually contain any such terms), whereas the MATCH query on the FTS3 table
  selects only those rows that contain "linux" as a discrete token. Both 
  searches are case-insensitive. The FTS3 table consumes around 2006 MB on
  disk compared to just 1453 MB for the ordinary table. Using the same
  hardware configuration used to perform the SELECT queries above, the FTS3
  table took just under 31 minutes to populate, versus 25 for the ordinary
  table.

1.1. Differences between FTS3 and FTS4




  FTS3 and FTS4 are nearly identical. They share most of their code in common,
  and their interfaces are the same. The differences are:


   FTS4 contains query performance optimizations that may significantly
       improve the performance of full-text queries that contain terms that are
       very common (present in a large percentage of table rows).

   FTS4 supports some additional options that may used with the matchinfo()
       function. 

   Because it stores extra information on disk in two new 
       shadow tables in order to support the performance
       optimizations and extra matchinfo() options, FTS4 tables may consume more
       disk space than the equivalent table created using FTS3. Usually the overhead
       is 1-2% or less, but may be as high as 10% if the documents stored in the
       FTS table are very small. The overhead may be reduced by specifying the
       directive "matchinfo=fts3" as part of the FTS4 table
       declaration, but this comes at the expense of sacrificing some of the
       extra supported matchinfo() options.

   FTS4 provides hooks (the compress and uncompress 
       options) allowing data to be stored in a compressed 
       form, reducing disk usage and IO.



  FTS4 is an enhancement to FTS3. 
  FTS3 has been available since SQLite version 3.5.0 (2007-09-04)
  The enhancements for FTS4 were added with SQLite version 3.7.4
  (2010-12-07).


  Which module, FTS3 or FTS4, should you use in your application?  FTS4 is
  sometimes significantly faster than FTS3, even orders of magnitude faster
  depending on the query, though in the common case the performance of the two
  modules is similar. FTS4 also offers the enhanced matchinfo() outputs which
  can be useful in ranking the results of a MATCH operation.  On the
  other hand, in the absence of a matchinfo=fts3 directive FTS4 requires a little
  more disk space than FTS3, though only a percent of two in most cases.


  For newer applications, FTS4 is recommended; though if compatibility with older 
  versions of SQLite is important, then FTS3 will usually serve just as well.  

1.2. Creating and Destroying FTS Tables


  Like other virtual table types, new FTS tables are created using a 
  CREATE VIRTUAL TABLE statement. The module name, which follows
  the USING keyword, is either "fts3" or "fts4". The virtual table module arguments may
  be left empty, in which case an FTS table with a single user-defined 
  column named "content" is created. Alternatively, the module arguments
  may be passed a list of comma separated column names. 


  If column names are explicitly provided for the FTS table as part of
  the CREATE VIRTUAL TABLE statement, then a datatype name may be optionally 
  specified for each column. This is pure syntactic sugar, the
  supplied typenames are not used by FTS or the SQLite core for any
  purpose. The same applies to any constraints specified along with an
  FTS column name - they are parsed but not used or recorded by the system
  in any way.

-- Create an FTS table named "data" with one column - "content":
CREATE VIRTUAL TABLE data USING fts3();

-- Create an FTS table named "pages" with three columns:
CREATE VIRTUAL TABLE pages USING fts4(title, keywords, body);

-- Create an FTS table named "mail" with two columns. Datatypes
-- and column constraints are specified along with each column. These
-- are completely ignored by FTS and SQLite. 
CREATE VIRTUAL TABLE mail USING fts3(
  subject VARCHAR(256) NOT NULL,
  body TEXT CHECK(length(body)&lt;10240)
);



  As well as a list of columns, the module arguments passed to a CREATE
  VIRTUAL TABLE statement used to create an FTS table may be used to specify
  a tokenizer. This is done by specifying a string of the form
  "tokenize=&lt;tokenizer name&gt; &lt;tokenizer args&gt;" in place of a column
  name, where &lt;tokenizer name&gt; is the name of the tokenizer to use and
  &lt;tokenizer args&gt; is an optional list of whitespace separated qualifiers
  to pass to the tokenizer implementation. A tokenizer specification may be
  placed anywhere in the column list, but at most one tokenizer declaration is
  allowed for each CREATE VIRTUAL TABLE statement. See below for a 
  detailed description of using (and, if necessary, implementing) a tokenizer.

-- Create an FTS table named "papers" with two columns that uses
-- the tokenizer "porter".
CREATE VIRTUAL TABLE papers USING fts3(author, document, tokenize=porter);

-- Create an FTS table with a single column - "content" - that uses
-- the "simple" tokenizer.
CREATE VIRTUAL TABLE data USING fts4(tokenize=simple);

-- Create an FTS table with two columns that uses the "icu" tokenizer.
-- The qualifier "en_AU" is passed to the tokenizer implementation
CREATE VIRTUAL TABLE names USING fts3(a, b, tokenize=icu en_AU);



  FTS tables may be dropped from the database using an ordinary DROP TABLE
  statement. For example:

-- Create, then immediately drop, an FTS4 table.
CREATE VIRTUAL TABLE data USING fts4();
DROP TABLE data;


1.3. Populating FTS Tables

  
    FTS tables are populated using INSERT, UPDATE and DELETE
    statements in the same way as ordinary SQLite tables are.

  
    As well as the columns named by the user (or the "content" column if no
    module arguments were specified as part of the CREATE VIRTUAL TABLE 
    statement), each FTS table has a "rowid" column. The rowid of an FTS
    table behaves in the same way as the rowid column of an ordinary SQLite 
    table, except that the values stored in the rowid column of an FTS table 
    remain unchanged if the database is rebuilt using the VACUUM command. 
    For FTS tables, "docid" is allowed as an alias along with the usual "rowid",
    "oid" and "_oid_" identifiers. Attempting to insert or update a row with a 
    docid value that already exists in the table is an error, just as it would 
    be with an ordinary SQLite table.

  
    There is one other subtle difference between "docid" and the normal SQLite
    aliases for the rowid column. Normally, if an INSERT or UPDATE statement 
    assigns discrete values to two or more aliases of the rowid column, SQLite 
    writes the rightmost of such values specified in the INSERT or UPDATE
    statement to the database. However, assigning a non-NULL value to both
    the "docid" and one or more of the SQLite rowid aliases when inserting or
    updating an FTS table is considered an error. See below for an example.

-- Create an FTS table
CREATE VIRTUAL TABLE pages USING fts4(title, body);

-- Insert a row with a specific docid value.
INSERT INTO pages(docid, title, body) VALUES(53, 'Home Page', 'SQLite is a software...');

-- Insert a row and allow FTS to assign a docid value using the same algorithm as
-- SQLite uses for ordinary tables. In this case the new docid will be 54,
-- one greater than the largest docid currently present in the table.
INSERT INTO pages(title, body) VALUES('Download', 'All SQLite source code...');

-- Change the title of the row just inserted.
UPDATE pages SET title = 'Download SQLite' WHERE rowid = 54;

-- Delete the entire table contents.
DELETE FROM pages;

-- The following is an error. It is not possible to assign non-NULL values to both
-- the rowid and docid columns of an FTS table.
INSERT INTO pages(rowid, docid, title, body) VALUES(1, 2, 'A title', 'A document body');


  
    To support full-text queries, FTS maintains an inverted index that maps
    from each unique term or word that appears in the dataset to the locations
    in which it appears within the table contents. For the curious, a 
    complete description of the data structure used to store
    this index within the database file appears below. A feature of
    this data structure is that at any time the database may contain not
    one index b-tree, but several different b-trees that are incrementally
    merged as rows are inserted, updated and deleted. This technique improves 
    performance when writing to an FTS table, but causes some overhead for
    full-text queries that use the index. Evaluating the special "optimize" command, 
    an SQL statement of the
    form "INSERT INTO &lt;fts-table&gt;(&lt;fts-table&gt;) VALUES('optimize')",
    causes FTS to merge all existing index b-trees into a single large
    b-tree containing the entire index. This can be an expensive operation,
    but may speed up future queries. 

  
    For example, to optimize the full-text index for an FTS table named
    "docs":

-- Optimize the internal structure of FTS table "docs".
INSERT INTO docs(docs) VALUES('optimize');


  
    The statement above may appear syntactically incorrect to some. Refer to
    the section describing the simple fts queries for an explanation.

  
    There is another, deprecated, method for invoking the optimize 
    operation using a SELECT statement. New code should use statements
    similar to the INSERT above to optimize FTS structures.


1.4. Simple FTS Queries


  As for all other SQLite tables, virtual or otherwise, data is retrieved
  from FTS tables using a SELECT statement.


  FTS tables can be queried efficiently using SELECT statements of two
  different forms:


  
    Query by rowid. If the WHERE clause of the SELECT statement
    contains a sub-clause of the form "rowid = ?", where ? is an SQL expression,
    FTS is able to retrieve the requested row directly using the equivalent 
    of an SQLite INTEGER PRIMARY KEY index.

  
    Full-text query. If the WHERE clause of the SELECT statement contains
    a sub-clause of the form "&lt;column&gt; MATCH ?", FTS is able to use 
    the built-in full-text index to restrict the search to those documents 
    that match the full-text query string specified as the right-hand operand
    of the MATCH clause.



  If neither of these two query strategies can be used, all
  queries on FTS tables are implemented using a linear scan of the entire
  table. If the table contains large amounts of data, this may be an 
  impractical approach (the first example on this page shows that a linear
  scan of 1.5 GB of data takes around 30 seconds using a modern PC).

-- The examples in this block assume the following FTS table:
CREATE VIRTUAL TABLE mail USING fts3(subject, body);

SELECT * FROM mail WHERE rowid = 15;                -- Fast. Rowid lookup.
SELECT * FROM mail WHERE body MATCH 'sqlite';       -- Fast. Full-text query.
SELECT * FROM mail WHERE mail MATCH 'search';       -- Fast. Full-text query.
SELECT * FROM mail WHERE rowid BETWEEN 15 AND 20;   -- Fast. Rowid lookup.
SELECT * FROM mail WHERE subject = 'database';      -- Slow. Linear scan.
SELECT * FROM mail WHERE subject MATCH 'database';  -- Fast. Full-text query.



  In all of the full-text queries above, the right-hand operand of the MATCH
  operator is a string consisting of a single term. In this case, the MATCH
  expression evaluates to true for all documents that contain one or more 
  instances of the specified word ("sqlite", "search" or "database", depending 
  on which example you look at). Specifying a single term as the right-hand
  operand of the MATCH operator results in the simplest and most common type 
  of full-text query possible. However more complicated queries are possible,
  including phrase searches, term-prefix searches and searches for documents 
  containing combinations of terms occurring within a defined proximity of each
  other. The various ways in which the full-text index may be queried are
  described below.


  Normally, full-text queries are case-insensitive. However, this
  is dependent on the specific tokenizer used by the FTS table
  being queried. Refer to the section on tokenizers for details.


  The paragraph above notes that a MATCH operator with a simple term as the
  right-hand operand evaluates to true for all documents that contain the
  specified term. In this context, the "document" may refer to either the 
  data stored in a single column of a row of an FTS table, or to the contents
  of all columns in a single row, depending on the identifier used as the
  left-hand operand to the MATCH operator. If the identifier specified as
  the left-hand operand of the MATCH operator is an FTS table column name,
  then the document that the search term must be contained in is the value
  stored in the specified column. However, if the identifier is the name
  of the FTS table itself, then the MATCH operator evaluates to true
  for each row of the FTS table for which any column contains the search 
  term. The following example demonstrates this:

-- Example schema
CREATE VIRTUAL TABLE mail USING fts3(subject, body);

-- Example table population
INSERT INTO mail(docid, subject, body) VALUES(1, 'software feedback', 'found it too slow');
INSERT INTO mail(docid, subject, body) VALUES(2, 'software feedback', 'no feedback');
INSERT INTO mail(docid, subject, body) VALUES(3, 'slow lunch order',  'was a software problem');

-- Example queries
SELECT * FROM mail WHERE subject MATCH 'software';    -- Selects rows 1 and 2
SELECT * FROM mail WHERE body    MATCH 'feedback';    -- Selects row 2
SELECT * FROM mail WHERE mail    MATCH 'software';    -- Selects rows 1, 2 and 3
SELECT * FROM mail WHERE mail    MATCH 'slow';        -- Selects rows 1 and 3

  

  At first glance, the final two full-text queries in the example above seem
  to be syntactically incorrect, as there is a table name ("mail") used as
  an SQL expression. The reason this is acceptable is that each FTS table
  actually has a HIDDEN column with the same name
  as the table itself (in this case, "mail"). The value stored in this
  column is not meaningful to the application, but can be used as the 
  left-hand operand to a MATCH operator. This special column may also be
  passed as an argument to the FTS auxiliary functions.


  The following example illustrates the above. The expressions "docs", 
  "docs.docs" and "main.docs.docs" all refer to column "docs". However, the 
  expression "main.docs" does not refer to any column. It could be used to 
  refer to a table, but a table name is not allowed in the context in which
  it is used below.

-- Example schema
CREATE VIRTUAL TABLE docs USING fts4(content);

-- Example queries
SELECT * FROM docs WHERE docs MATCH 'sqlite';              -- OK.
SELECT * FROM docs WHERE docs.docs MATCH 'sqlite';         -- OK.
SELECT * FROM docs WHERE main.docs.docs MATCH 'sqlite';    -- OK.
SELECT * FROM docs WHERE main.docs MATCH 'sqlite';         -- Error.

 
1.5. Summary


  From the users point of view, FTS tables are similar to ordinary SQLite
  tables in many ways. Data may be added to, modified within and removed 
  from FTS tables using the INSERT, UPDATE and DELETE commands just as 
  it may be with ordinary tables. Similarly, the SELECT command may be used 
  to query data. The following list summarizes the differences between FTS
  and ordinary tables:


   
    As with all virtual table types, it is not possible to create indices or
    triggers attached to FTS tables. Nor is it possible to use the ALTER TABLE
    command to add extra columns to FTS tables (although it is possible to use
    ALTER TABLE to rename an FTS table).

   
    Data-types specified as part of the "CREATE VIRTUAL TABLE" statement
    used to create an FTS table are ignored completely. Instead of the
    normal rules for applying type affinity to inserted values, all
    values inserted into FTS table columns (except the special rowid
    column) are converted to type TEXT before being stored.

   
    FTS tables permit the special alias "docid" to be used to refer to the
    rowid column supported by all virtual tables.

   
    The FTS MATCH operator is supported for queries based on the built-in
    full-text index. 

   
    The FTS auxiliary functions, snippet(), offsets(), and matchinfo() are 
    available to support full-text queries.

  
    

    Every FTS table has a hidden column with the 
    same name as the table itself. The value contained in each row for the
    hidden column is a blob that is only useful as the left operand of a
    MATCH operator, or as the left-most argument to one
    of the FTS auxiliary functions.
    





2. Compiling and Enabling FTS3 and FTS4


  Although FTS3 and FTS4 are included with the SQLite core source code, they are not
  enabled by default. To build SQLite with FTS functionality enabled, define
  the preprocessor macro SQLITE_ENABLE_FTS3 when compiling. New applications
  should also define the SQLITE_ENABLE_FTS3_PARENTHESIS macro to enable the
  enhanced query syntax (see below). Usually, this is done by adding the 
  following two switches to the compiler command line:

-DSQLITE_ENABLE_FTS3
-DSQLITE_ENABLE_FTS3_PARENTHESIS



  Note that enabling FTS3 also makes FTS4 available.  There is not a separate
  SQLITE_ENABLE_FTS4 compile-time option.  A build of SQLite either supports
  both FTS3 and FTS4 or it supports neither.


  If using the amalgamation autoconf based build system, setting the CPPFLAGS
  environment variable while running the 'configure' script is an easy
  way to set these macros. For example, the following command:

CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS" ./configure &lt;configure options&gt;



  where &lt;configure options&gt; are those options normally passed to
  the configure script, if any.


  Because FTS3 and FTS4 are virtual tables, The SQLITE_ENABLE_FTS3 compile-time option
  is incompatible with the SQLITE_OMIT_VIRTUALTABLE option.


  If a build of SQLite does not include the FTS modules, then any attempt to prepare an
  SQL statement to create an FTS3 or FTS4 table or to drop or access an existing 
  FTS table in any way will fail. The error message returned will be similar 
  to "no such module: ftsN" (where N is either 3 or 4).


  If the C version of the ICU library
  is available, then FTS may also be compiled with the SQLITE_ENABLE_ICU
  pre-processor macro defined. Compiling with this macro enables an FTS
  tokenizer that uses the ICU library to split a document into terms
  (words) using the conventions for a specified language and locale.

-DSQLITE_ENABLE_ICU

  


3. Full-text Index Queries


  The most useful thing about FTS tables is the queries that may be 
  performed using the built-in full-text index. Full-text queries are 
  performed by specifying a clause of the form 
  "&lt;column&gt; MATCH &lt;full-text query expression&gt;" as part of the WHERE 
  clause of a SELECT statement that reads data from an FTS table. 
  Simple FTS queries that return all documents that 
  contain a given term are described above. In that discussion the right-hand
  operand of the MATCH operator was assumed to be a string consisting of a
  single term. This section describes the more complex query types supported 
  by FTS tables, and how they may be utilized by specifying a more
  complex query expression as the right-hand operand of a MATCH operator.


  FTS tables support three basic query types:


  

  Token or token prefix queries. 
    An FTS table may be queried for all documents that contain a specified
    term (the simple case described above), or for
    all documents that contain a term with a specified prefix. As we have
    seen, the query expression for a specific term is simply the term itself.
    The query expression used to search for a term prefix is the prefix
    itself with a '*' character appended to it. For example:


-- Virtual table declaration
CREATE VIRTUAL TABLE docs USING fts3(title, body);

-- Query for all documents containing the term "linux":
SELECT * FROM docs WHERE docs MATCH 'linux';

-- Query for all documents containing a term with the prefix "lin". This will match
-- all documents that contain "linux", but also those that contain terms "linear",
--"linker", "linguistic" and so on.
SELECT * FROM docs WHERE docs MATCH 'lin*';



  
    Normally, a token or token prefix query is matched against the FTS table 
    column specified as the left-hand side of the MATCH operator. Or, if the
    special column with the same name as the FTS table itself is specified,
    against all columns. This may be overridden by specifying a column-name
    followed by a ":" character before a basic term query. There may be space
    between the ":" and the term to query for, but not between the column-name
    and the ":" character. For example:

   
-- Query the database for documents for which the term "linux" appears in
-- the document title, and the term "problems" appears in either the title
-- or body of the document.
SELECT * FROM docs WHERE docs MATCH 'title:linux problems';

-- Query the database for documents for which the term "linux" appears in
-- the document title, and the term "driver" appears in the body of the document
-- ("driver" may also appear in the title, but this alone will not satisfy the
-- query criteria).
SELECT * FROM docs WHERE body MATCH 'title:linux driver';



  
    If the FTS table is an FTS4 table (not FTS3), a token may also be prefixed
    with a "&#94;" character. In this case, in order to match the token must
    appear as the very first token in any column of the matching row. Examples:


-- All documents for which "linux" is the first token of at least one
-- column.
SELECT * FROM docs WHERE docs MATCH '&#94;linux';

-- All documents for which the first token in column "title" begins with "lin".
SELECT * FROM docs WHERE body MATCH 'title: &#94;lin*';





  Phrase queries.
    A phrase query is a query that retrieves all documents that contain a
    nominated set of terms or term prefixes in a specified order with no
    intervening tokens. Phrase queries are specified by enclosing a space
    separated sequence of terms or term prefixes in double quotes (").
    For example:


-- Query for all documents that contain the phrase "linux applications".
SELECT * FROM docs WHERE docs MATCH '"linux applications"';

-- Query for all documents that contain a phrase that matches "lin* app*". As well as
-- "linux applications", this will match common phrases such as "linoleum appliances"
-- or "link apprentice".
SELECT * FROM docs WHERE docs MATCH '"lin* app*"';





  NEAR queries. 
    A NEAR query is a query that returns documents that contain a two or
    more nominated terms or phrases within a specified proximity of each 
    other (by default with 10 or less intervening terms). A NEAR query is 
    specified by putting the keyword "NEAR" between two phrase, token or 
    token prefix queries. To specify a proximity other than the default,
    an operator of the form "NEAR/&lt;N&gt;" may be used, where
    &lt;N&gt; is the maximum number of intervening terms allowed.
    For example:


-- Virtual table declaration.
CREATE VIRTUAL TABLE docs USING fts4();

-- Virtual table data.
INSERT INTO docs VALUES('SQLite is an ACID compliant embedded relational database management system');

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 10 intervening terms. This matches the only document in
-- table docs (since there are only six terms between "SQLite" and "database" 
-- in the document).
SELECT * FROM docs WHERE docs MATCH 'sqlite NEAR database';

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 6 intervening terms. This also matches the only document in
-- table docs. Note that the order in which the terms appear in the document
-- does not have to be the same as the order in which they appear in the query.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/6 sqlite';

-- Search for a document that contains the terms "sqlite" and "database" with
-- not more than 5 intervening terms. This query matches no documents.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/5 sqlite';

-- Search for a document that contains the phrase "ACID compliant" and the term
-- "database" with not more than 2 terms separating the two. This matches the
-- document stored in table docs.
SELECT * FROM docs WHERE docs MATCH 'database NEAR/2 "ACID compliant"';

-- Search for a document that contains the phrase "ACID compliant" and the term
-- "sqlite" with not more than 2 terms separating the two. This also matches
-- the only document stored in table docs.
SELECT * FROM docs WHERE docs MATCH '"ACID compliant" NEAR/2 sqlite';



  
    More than one NEAR operator may appear in a single query. In this case each
    pair of terms or phrases separated by a NEAR operator must appear within the
    specified proximity of each other in the document. Using the same table and
    data as in the block of examples above:



-- The following query selects documents that contains an instance of the term 
-- "sqlite" separated by two or fewer terms from an instance of the term "acid",
-- which is in turn separated by two or fewer terms from an instance of the term
-- "relational".
SELECT * FROM docs WHERE docs MATCH 'sqlite NEAR/2 acid NEAR/2 relational';

-- This query matches no documents. There is an instance of the term "sqlite" with
-- sufficient proximity to an instance of "acid" but it is not sufficiently close
-- to an instance of the term "relational".
SELECT * FROM docs WHERE docs MATCH 'acid NEAR/2 sqlite NEAR/2 relational';



  Phrase and NEAR queries may not span multiple columns within a row.


  The three basic query types described above may be used to query the full-text
  index for the set of documents that match the specified criteria. Using the
  FTS query expression language it is possible to perform various set 
  operations on the results of basic queries. There are currently three 
  supported operations:


   The AND operator determines the intersection of two sets of documents.

   The OR operator calculates the union of two sets of documents.

   The NOT operator (or, if using the standard syntax, a unary "-" operator)
       may be used to compute the relative complement of one set of
       documents with respect to another.



  The FTS modules may be compiled to use one of two slightly different versions
  of the full-text query syntax, the "standard" query syntax and the "enhanced" 
  query syntax. The basic term, term-prefix, phrase and NEAR queries described 
  above are the same in both versions of the syntax. The way in which set 
  operations are specified is slightly different. The following two sub-sections 
  describe the part of the two query syntaxes that pertains to set operations. 
  Refer to the description of how to compile fts for compilation notes.


3.1. 
  Set Operations Using The Enhanced Query Syntax


  The enhanced query syntax supports the AND, OR and NOT binary set operators.
  Each of the two operands to an operator may be a basic FTS query, or the
  result of another AND, OR or NOT set operation. Operators must be entered
  using capital letters. Otherwise, they are interpreted as basic term queries
  instead of set operators.


  The AND operator may be implicitly specified. If two basic queries appear 
  with no operator separating them in an FTS query string, the results are
  the same as if the two basic queries were separated by an AND operator.
  For example, the query expression "implicit operator" is a more succinct
  version of "implicit AND operator".

-- Virtual table declaration
CREATE VIRTUAL TABLE docs USING fts3();

-- Virtual table data
INSERT INTO docs(docid, content) VALUES(1, 'a database is a software system');
INSERT INTO docs(docid, content) VALUES(2, 'sqlite is a software system');
INSERT INTO docs(docid, content) VALUES(3, 'sqlite is a database');

-- Return the set of documents that contain the term "sqlite", and the
-- term "database". This query will return the document with docid 3 only.
SELECT * FROM docs WHERE docs MATCH 'sqlite AND database';

-- Again, return the set of documents that contain both "sqlite" and
-- "database". This time, use an implicit AND operator. Again, document
-- 3 is the only document matched by this query. 
SELECT * FROM docs WHERE docs MATCH 'database sqlite';

-- Query for the set of documents that contains either "sqlite" or "database".
-- All three documents in the database are matched by this query.
SELECT * FROM docs WHERE docs MATCH 'sqlite OR database';

-- Query for all documents that contain the term "database", but do not contain
-- the term "sqlite". Document 1 is the only document that matches this criteria.
SELECT * FROM docs WHERE docs MATCH 'database NOT sqlite';

-- The following query matches no documents. Because "and" is in lowercase letters,
-- it is interpreted as a basic term query instead of an operator. Operators must
-- be specified using capital letters. In practice, this query will match any documents
-- that contain each of the three terms "database", "and" and "sqlite" at least once.
-- No documents in the example data above match this criteria.
SELECT * FROM docs WHERE docs MATCH 'database and sqlite';



  The examples above all use basic full-text term queries as both operands of 
  the set operations demonstrated. Phrase and NEAR queries may also be used,
  as may the results of other set operations. When more than one set operation
  is present in an FTS query, the precedence of operators is as follows:


  OperatorEnhanced Query Syntax Precedence
  NOT  Highest precedence (tightest grouping).
  AND 
  OR   Lowest precedence (loosest grouping).



  When using the enhanced query syntax, parenthesis may be used to override
  the default precedence of the various operators. For example:

-- Return the docid values associated with all documents that contain the
-- two terms "sqlite" and "database", and/or contain the term "library".
SELECT docid FROM docs WHERE docs MATCH 'sqlite AND database OR library';

-- This query is equivalent to the above.
SELECT docid FROM docs WHERE docs MATCH 'sqlite AND database'
  UNION
SELECT docid FROM docs WHERE docs MATCH 'library';

-- Query for the set of documents that contains the term "linux", and at least
-- one of the phrases "sqlite database" and "sqlite library".
SELECT docid FROM docs WHERE docs MATCH '("sqlite database" OR "sqlite library") AND linux';

-- This query is equivalent to the above.
SELECT docid FROM docs WHERE docs MATCH 'linux'
  INTERSECT
SELECT docid FROM (
  SELECT docid FROM docs WHERE docs MATCH '"sqlite library"'
    UNION
  SELECT docid FROM docs WHERE docs MATCH '"sqlite database"'
);



3.2. Set Operations Using The Standard Query Syntax


  FTS query set operations using the standard query syntax are similar, but
  not identical, to set operations with the enhanced query syntax. There
  are four differences, as follows:


   Only the implicit version of the AND operator is supported.
    Specifying the string "AND" as part of a standard query syntax query is
    interpreted as a term query for the set of documents containing the term 
    "and".



   Parenthesis are not supported.



   The NOT operator is not supported. Instead of the NOT 
    operator, the standard query syntax supports a unary "-" operator that
    may be applied to basic term and term-prefix queries (but not to phrase
    or NEAR queries). A term or term-prefix that has a unary "-" operator
    attached to it may not appear as an operand to an OR operator. An FTS
    query may not consist entirely of terms or term-prefix queries with unary
    "-" operators attached to them.


-- Search for the set of documents that contain the term "sqlite" but do
-- not contain the term "database".
SELECT * FROM docs WHERE docs MATCH 'sqlite -database';



   The relative precedence of the set operations is different. 
   In particular, using the standard query syntax the "OR" operator has a
   higher precedence than "AND". The precedence of operators when using the
   standard query syntax is: 



  OperatorStandard Query Syntax Precedence
  Unary "-"  Highest precedence (tightest grouping).
  OR  
  AND  Lowest precedence (loosest grouping).



  The following example illustrates precedence of operators using the standard 
  query syntax:


-- Search for documents that contain at least one of the terms "database"
-- and "sqlite", and also contain the term "library". Because of the differences
-- in operator precedences, this query would have a different interpretation using
-- the enhanced query syntax.
SELECT * FROM docs WHERE docs MATCH 'sqlite OR database library';




4. Auxiliary Functions - Snippet, Offsets and Matchinfo


  The FTS3 and FTS4 modules provide three special SQL scalar functions that may be useful
  to the developers of full-text query systems: "snippet", "offsets" and
  "matchinfo". The purpose of the "snippet" and "offsets" functions is to allow
  the user to identify the location of queried terms in the returned documents.
  The "matchinfo" function provides the user with metrics that may be useful
  for filtering or sorting query results according to relevance.


  The first argument to all three special SQL scalar functions
  must be the FTS hidden column of the FTS table that the function is
  applied to.  The FTS hidden column is an automatically-generated column found on
  all FTS tables that has the same name as the FTS table itself.
  For example, given an FTS table named "mail":

SELECT offsets(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;
SELECT snippet(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;
SELECT matchinfo(mail) FROM mail WHERE mail MATCH &lt;full-text query expression&gt;;



  The three auxiliary functions are only useful within a SELECT statement that
  uses the FTS table's full-text index. If used within a SELECT that uses
  the "query by rowid" or "linear scan" strategies, then the snippet and
  offsets both return an empty string, and the matchinfo function returns
  a blob value zero bytes in size.


  All three auxiliary functions extract a set of "matchable phrases" from
  the FTS query expression to work with. The set of matchable phrases for
  a given query consists of all phrases (including unquoted tokens and
  token prefixes) in the expression except those that are prefixed with
  a unary "-" operator (standard syntax) or are part of a sub-expression 
  that is used as the right-hand operand of a NOT operator.


  With the following provisos, each series of tokens in the FTS table that
  matches one of the matchable phrases in the query expression is known as a
  "phrase match":


   If a matchable phrase is part of a series of phrases connected by
       NEAR operators in the FTS query expression, then each phrase match
       must be sufficiently close to other phrase matches of the relevant
       types to satisfy the NEAR condition.

   If the matchable phrase in the FTS query is restricted to matching
       data in a specified FTS table column, then only phrase matches that 
       occur within that column are considered.

 


4.1. The Offsets Function


  For a SELECT query that uses the full-text index, the offsets() function 
  returns a text value containing a series of space-separated integers. For
  each term in each phrase match of the current row, 
  there are four integers in the returned list. Each set of four integers is 
  interpreted as follows:


  Integer Interpretation
  0 
      The column number that the term instance occurs in (0 for the
          leftmost column of the FTS table, 1 for the next leftmost, etc.).
  1
      The term number of the matching term within the full-text query
          expression. Terms within a query expression are numbered starting
          from 0 in the order that they occur.
  2
      The byte offset of the matching term within the column.
  3
      The size of the matching term in bytes.



  The following block contains examples that use the offsets function.

CREATE VIRTUAL TABLE mail USING fts3(subject, body);
INSERT INTO mail VALUES('hello world', 'This message is a hello world message.');
INSERT INTO mail VALUES('urgent: serious', 'This mail is seen as a more serious mail');

-- The following query returns a single row (as it matches only the first
-- entry in table "mail". The text returned by the offsets function is
-- "0 0 6 5 1 0 24 5".
--
-- The first set of four integers in the result indicate that column 0
-- contains an instance of term 0 ("world") at byte offset 6. The term instance
-- is 5 bytes in size. The second set of four integers shows that column 1
-- of the matched row contains an instance of term 0 ("world") at byte offset
-- 24. Again, the term instance is 5 bytes in size.
SELECT offsets(mail) FROM mail WHERE mail MATCH 'world';

-- The following query returns also matches only the first row in table "mail".
-- In this case the returned text is "1 0 5 7 1 0 30 7".
SELECT offsets(mail) FROM mail WHERE mail MATCH 'message';

-- The following query matches the second row in table "mail". It returns the
-- text "1 0 28 7 1 1 36 4". Only those occurrences of terms "serious" and "mail"
-- that are part of an instance of the phrase "serious mail" are identified; the
-- other occurrences of "serious" and "mail" are ignored.
SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"';




4.2. The Snippet Function


  The snippet function is used to create formatted fragments of document text
  for display as part of a full-text query results report. The snippet function 
  may be passed between one and six arguments, as follows:


  Argument Default Value Description
  0 N/A
       The first argument to the snippet function must always be the FTS hidden column
           of the FTS table being queried and from which the snippet is to be taken.  The
           FTS hidden column is an automatically generated column with the same name as the
           FTS table itself.
  1 "&lt;b&gt;"
       The "start match" text.
  2 "&lt;/b&gt;"
       The "end match" text.
  3 "&lt;b&gt;...&lt;/b&gt;"
       The "ellipses" text.
  4 -1
       The FTS table column number to extract the returned fragments of
           text from. Columns are numbered from left to right starting with
           zero. A negative value indicates that the text may be extracted
           from any column.
  5 -15
       The absolute value of this integer argument is used as the 
           (approximate) number of tokens to include in the returned text 
           value. The maximum allowable absolute value is 64. The value of
           this argument is referred to as N in the discussion below.



  The snippet function first attempts to find a fragment of text consisting
  of |N| tokens within the current row that contains at least one phrase 
  match for each matchable phrase matched somewhere in the current row, 
  where |N| is the absolute value of the sixth argument passed to the
  snippet function. If the text stored in a single column contains less than
  |N| tokens, then the entire column value is considered. Text fragments 
  may not span multiple columns.


  If such a text fragment can be found, it is returned with the following
  modifications:


   If the text fragment does not begin at the start of a column value,
       the "ellipses" text is prepended to it.
   If the text fragment does not finish at the end of a column value,
       the "ellipses" text is appended to it.
   For each token in the text fragment that is part of a phrase match,
       the "start match" text is inserted into the fragment before the token,
       and the "end match" text is inserted immediately after it.



  If more than one such fragment can be found, then fragments that contain
  a larger number of "extra" phrase matches are favored. The start of
  the selected text fragment may be moved a few tokens forward or backward
  to attempt to concentrate the phrase matches toward the center of the
  fragment.


  Assuming N is a positive value, if no fragments can be found that
  contain a phrase match corresponding to each matchable phrase, the snippet
  function attempts to find two fragments of approximately N/2 tokens
  that between them contain at least one phrase match for each matchable phrase
  matched by the current row. If this fails, attempts are made to find three
  fragments of N/3 tokens each and finally four N/4 token
  fragments. If a set of four fragments cannot be found that encompasses the
  required phrase matches, the four fragments of N/4 tokens that provide
  the best coverage are selected.


  If N is a negative value, and no single fragment can be found 
  containing the required phrase matches, the snippet function searches
  for two fragments of |N| tokens each, then three, then four. In
  other words, if the specified value of N is negative, the sizes
  of the fragments is not decreased if more than one fragment is required
  to provide the desired phrase match coverage.


  After the M fragments have been located, where M is between
  two and four as described in the paragraphs above, they are joined together
  in sorted order with the "ellipses" text separating them. The three 
  modifications enumerated earlier are performed on the text before it is 
  returned.

Note: In this block of examples, newlines and whitespace characters have
been inserted into the document inserted into the FTS table, and the expected
results described in SQL comments. This is done to enhance readability only,
they would not be present in actual SQLite commands or output.

-- Create and populate an FTS table.
CREATE VIRTUAL TABLE text USING fts4();
INSERT INTO text VALUES('
  During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC 
  and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, 
  minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature 
  increases. Northeasterly winds 15-30 km/hr.     
');

-- The following query returns the text value:
--
--   "&lt;b&gt;...&lt;/b&gt;cool elsewhere, minimum temperature 17-20oC. &lt;b&gt;Cold&lt;/b&gt; to very 
--    &lt;b&gt;cold&lt;/b&gt; on mountaintops, minimum temperature 6&lt;b&gt;...&lt;/b&gt;".
--
SELECT snippet(text) FROM text WHERE text MATCH 'cold';

-- The following query returns the text value:
--
--   "...the upper portion, &#91;minimum&#93; &#91;temperature&#93; 14-16oC and cool elsewhere,
--    &#91;minimum&#93; &#91;temperature&#93; 17-20oC. Cold..."
--
SELECT snippet(text, '&#91;', '&#93;', '...') FROM text WHERE text MATCH '"min* tem*"'



4.3. The Matchinfo Function


  The matchinfo function returns a blob value. If it is used within a query
  that does not use the full-text index (a "query by rowid" or "linear scan"),
  then the blob is zero bytes in size. Otherwise, the blob consists of zero
  or more 32-bit unsigned integers in machine byte-order. The exact number
  of integers in the returned array depends on both the query and the value
  of the second argument (if any) passed to the matchinfo function.


  The matchinfo function is called with either one or two arguments. As for
  all auxiliary functions, the first argument must be the special 
  FTS hidden column. The second argument, if it is specified, must be a text value
  comprised only of the characters 'p', 'c', 'n', 'a', 'l', 's', 'x', 'y' and 'b'.
  If no second argument is explicitly supplied, it defaults to "pcx". The
  second argument is referred to as the "format string" below.


  Characters in the matchinfo format string are processed from left to right. 
  Each character in the format string causes one or more 32-bit unsigned
  integer values to be added to the returned array. The "values" column in
  the following table contains the number of integer values appended to the
  output buffer for each supported format string character. In the formula
  given, cols is the number of columns in the FTS table, and 
  phrases is the number of matchable phrases in 
  the query. 


  CharacterValuesDescription
  p 1 The number of matchable phrases in the query.
  c 1 The number of user defined columns in the FTS
    table (i.e. not including the docid or the FTS hidden column).
  x 3 * cols * phrases 
    

      For each distinct combination of a phrase and table column, the
      following three values:
      
         In the current row, the number of times the phrase appears in 
             the column.
         The total number of times the phrase appears in the column in
             all rows in the FTS table.
         The total number of rows in the FTS table for which the 
             column contains at least one instance of the phrase.
      
      The first set of three values corresponds to the left-most column
      of the table (column 0) and the left-most matchable phrase in the
      query (phrase 0). If the table has more than one column, the second
      set of three values in the output array correspond to phrase 0 and
      column 1. Followed by phrase 0, column 2 and so on for all columns of
      the table. And so on for phrase 1, column 0, then phrase 1, column 1
      etc. In other words, the data for occurrences of phrase p in
      column c may be found using the following formula:

          hits_this_row  = array&#91;3 * (c + p*cols) + 0&#93;
          hits_all_rows  = array&#91;3 * (c + p*cols) + 1&#93;
          docs_with_hits = array&#91;3 * (c + p*cols) + 2&#93;

  ycols * phrases 
    

      For each distinct combination of a phrase and table column, the
      number of usable phrase matches that appear in the column. This is
      usually identical to the first value in each set of three returned by the
      matchinfo 'x' flag. However, the number of hits reported by the
      'y' flag is zero for any phrase that is part of a sub-expression
      that does not match the current row. This makes a difference for
      expressions that contain AND operators that are descendants of OR
      operators. For example, consider the expression: 

          a OR (b AND c)

      and the document:

          "a c d"

      The matchinfo 'x' flag would report a single hit for the phrases "a" and "c".
      However, the 'y' directive reports the number of hits for "c" as zero, as
      it is part of a sub-expression that does not match the document - (b AND c).
      For queries that do not contain AND operators descended from OR
      operators, the result values returned by 'y' are always the same as
      those returned by 'x'.


      The first value in the array of integer values corresponds to the
      leftmost column of the table (column 0) and the first phrase in the query
      (phrase 0). The values corresponding to other column/phrase combinations
      may be located using the following formula:


          hits_for_phrase_p_column_c  = array&#91;c + p*cols&#93;

      For queries that use OR expressions, or those that use LIMIT or return
      many rows, the 'y' matchinfo option may be faster than 'x'.

b((cols+31)/32) * phrases 



  The matchinfo 'b' flag provides similar information to the
  matchinfo 'y' flag, but in a more
  compact form. Instead of the precise number of hits, 'b' provides a single
  boolean flag for each phrase/column combination. If the phrase is present in
  the column at least once (i.e. if the corresponding integer output of 'y' would
  be non-zero), the corresponding flag is set. Otherwise cleared.


  If the table has 32 or fewer columns, a single unsigned integer is output for
  each phrase in the query. The least significant bit of the integer is set if the
  phrase appears at least once in column 0. The second least significant bit is
  set if the phrase appears once or more in column 1. And so on.


  If the table has more than 32 columns, an extra integer is added to the output
  of each phrase for each extra 32 columns or part thereof. Integers
  corresponding to the same phrase are clumped together. For example, if a table
  with 45 columns is queried for two phrases, 4 integers are output. The first
  corresponds to phrase 0 and columns 0-31 of the table. The second integer
  contains data for phrase 0 and columns 32-44, and so on.


  For example, if nCol is the number of columns in the table, to determine if
  phrase p is present in column c:


    p_is_in_c = array&#91;p * ((nCol+31)/32)&#93; & (1 &lt;&lt; (c % 32))


  n 1 The number of rows in the FTS4 table. This value is
    only available when querying FTS4 tables, not FTS3.
  a cols For each column, the average number of
    tokens in the text values stored in the column (considering all rows in
    the FTS4 table). This value is only available when querying FTS4 tables,
    not FTS3.  
  l cols 
    For each column, the length of the value stored in the current row of the
    FTS4 table, in tokens.  This value is only available when querying
    FTS4 tables, not FTS3. And only if the "matchinfo=fts3" directive was not
    specified as part of the "CREATE VIRTUAL TABLE" statement used to create
    the FTS4 table.
  s cols For each column, the length of the longest 
    subsequence of phrase matches that the column value has in common
    with the query text. For example, if a table column contains the text
    'a b c d e' and the query is 'a c "d e"', then the length of the longest
    common subsequence is 2 (phrase "c" followed by phrase "d e").
    
    



  For example:

-- Create and populate an FTS4 table with two columns:
CREATE VIRTUAL TABLE t1 USING fts4(a, b);
INSERT INTO t1 VALUES('transaction default models default', 'Non transaction reads');
INSERT INTO t1 VALUES('the default transaction', 'these semantics present');
INSERT INTO t1 VALUES('single request', 'default data');

-- In the following query, no format string is specified and so it defaults
-- to "pcx". It therefore returns a single row consisting of a single blob
-- value 80 bytes in size (20 32-bit integers - 1 for "p", 1 for "c" and
-- 3*2*3 for "x"). If each block of 4 bytes in the blob is interpreted 
-- as an unsigned integer in machine byte-order, the values will be:
--
--     3 2  1 3 2  0 1 1  1 2 2  0 1 1  0 0 0  1 1 1
--
-- The row returned corresponds to the second entry inserted into table t1.
-- The first two integers in the blob show that the query contained three
-- phrases and the table being queried has two columns. The next block of
-- three integers describes column 0 (in this case column "a") and phrase
-- 0 (in this case "default"). The current row contains 1 hit for "default"
-- in column 0, of a total of 3 hits for "default" that occur in column
-- 0 of any table row. The 3 hits are spread across 2 different rows.
--
-- The next set of three integers (0 1 1) pertain to the hits for "default"
-- in column 1 of the table (0 in this row, 1 in all rows, spread across 
-- 1 rows).
--
SELECT matchinfo(t1) FROM t1 WHERE t1 MATCH 'default transaction "these semantics"';

-- The format string for this query is "ns". The output array will therefore
-- contain 3 integer values - 1 for "n" and 2 for "s". The query returns
-- two rows (the first two rows in the table match). The values returned are:
--
--     3  1 1
--     3  2 0
--
-- The first value in the matchinfo array returned for both rows is 3 (the 
-- number of rows in the table). The following two values are the lengths 
-- of the longest common subsequence of phrase matches in each column.
SELECT matchinfo(t1, 'ns') FROM t1 WHERE t1 MATCH 'default transaction';



  The matchinfo function is much faster than either the snippet or offsets
  functions. This is because the implementation of both snippet and offsets
  is required to retrieve the documents being analyzed from disk, whereas
  all data required by matchinfo is available as part of the same portions
  of the full-text index that are required to implement the full-text query
  itself. This means that of the following two queries, the first may be
  an order of magnitude faster than the second:

SELECT docid, matchinfo(tbl) FROM tbl WHERE tbl MATCH &lt;query expression&gt;;
SELECT docid, offsets(tbl) FROM tbl WHERE tbl MATCH &lt;query expression&gt;;



  The matchinfo function provides all the information required to calculate
  probabilistic "bag-of-words" relevancy scores such as 
  Okapi BM25/BM25F that may
  be used to order results in a full-text search application. Appendix A of this 
  document, "search application tips", contains an example of using the
  matchinfo() function efficiently.


5. Fts4aux - Direct Access to the Full-Text Index


  As of version 3.7.6 (2011-04-12), 
  SQLite includes a new virtual table module called 
  "fts4aux", which can be used to inspect the full-text index of an existing
  FTS table directly. Despite its name, fts4aux works just as well with FTS3
  tables as it does with FTS4 tables. Fts4aux tables are read-only. The only
  way to modify the contents of an fts4aux table is by modifying the
  contents of the associated FTS table. The fts4aux module is automatically
  included in all builds that include FTS.


  An fts4aux virtual table is constructed with one or two arguments.  When
  used with a single argument, that argument is the unqualified name of the
  FTS table that it will be used to access.  To access a table in a different
  database (for example, to create a TEMP fts4aux table that will access an
  FTS3 table in the MAIN database) use the two-argument form and give the
  name of the target database (ex: "main") in the first argument and the name
  of the FTS3/4 table as the second argument.  (The two-argument form of 
  fts4aux was added for SQLite version 3.7.17 (2013-05-20)
  and will throw an error in prior releases.)
  For example:

-- Create an FTS4 table
CREATE VIRTUAL TABLE ft USING fts4(x, y);

-- Create an fts4aux table to access the full-text index for table "ft"
CREATE VIRTUAL TABLE ft_terms USING fts4aux(ft);

-- Create a TEMP fts4aux table accessing the "ft" table in "main"
CREATE VIRTUAL TABLE temp.ft_terms_2 USING fts4aux(main,ft);



  For each term present in the FTS table, there are between 2 and N+1 rows
  in the fts4aux table, where N is the number of user-defined columns in
  the associated FTS table. An fts4aux table always has the same four columns, 
  as follows, from left to right:


  Column NameColumn Contents
  term 
    Contains the text of the term for this row.
  col 
    This column may contain either the text value '*' (i.e. a single 
    character, U+002a) or an integer between 0 and N-1, where N is
    again the number of user-defined columns in the corresponding FTS table.

  documents
    This column always contains an integer value greater than zero.
    
    If the "col" column contains the value '*', then this column
    contains the number of rows of the FTS table that contain at least one
    instance of the term (in any column). If col contains an integer
    value, then this column contains the number of rows of the FTS table that
    contain at least one instance of the term in the column identified by
    the col value. As usual, the columns of the FTS table are numbered
    from left to right, starting with zero.

  occurrences
    This column also always contains an integer value greater than zero.
    
    If the "col" column contains the value '*', then this column
    contains the total number of instances of the term in all rows of the 
    FTS table (in any column). Otherwise, if col contains an integer
    value, then this column contains the total number of instances of the
    term that appear in the FTS table column identified by the col
    value.

  languageid (hidden)
    

    This column determines which languageid is used to
    extract vocabulary from the FTS3/4 table.
    
    The default value for languageid is 0.  If an alternative language
    is specified in WHERE clause constraints, then that alternative is
    used instead of 0.  There can only be a single languageid per query.
    In other words, the WHERE clause cannot contain a range constraint
    or IN operator on the languageid.



  For example, using the tables created above:

INSERT INTO ft(x, y) VALUES('Apple banana', 'Cherry');
INSERT INTO ft(x, y) VALUES('Banana Date Date', 'cherry');
INSERT INTO ft(x, y) VALUES('Cherry Elderberry', 'Elderberry');

-- The following query returns this data:
--
--     apple       |  *  |  1  |  1
--     apple       |  0  |  1  |  1
--     banana      |  *  |  2  |  2
--     banana      |  0  |  2  |  2
--     cherry      |  *  |  3  |  3
--     cherry      |  0  |  1  |  1
--     cherry      |  1  |  2  |  2
--     date        |  *  |  1  |  2
--     date        |  0  |  1  |  2
--     elderberry  |  *  |  1  |  2
--     elderberry  |  0  |  1  |  1
--     elderberry  |  1  |  1  |  1
--
SELECT term, col, documents, occurrences FROM ft_terms;



  In the example, the values in the "term" column are all lower case, 
  even though they were inserted into table "ft" in mixed case. This is because
  an fts4aux table contains the terms as extracted from the document text
  by the tokenizer. In this case, since table "ft" uses the 
  simple tokenizer, this means all terms have been folded to
  lower case. Also, there is (for example) no row with column "term"
  set to "apple" and column "col" set to 1. Since there are no instances
  of the term "apple" in column 1, no row is present in the fts4aux table.


  During a transaction, some of the data written to an FTS table may be 
  cached in memory and written to the database only when the transaction is 
  committed. However the implementation of the fts4aux module is only able 
  to read data from the database. In practice this means that if an fts4aux 
  table is queried from within a transaction in which the associated 
  FTS table has been modified, the results of the query are likely to reflect 
  only a (possibly empty) subset of the changes made.


6. FTS4 Options


  If the "CREATE VIRTUAL TABLE" statement specifies module FTS4 (not FTS3), 
  then special directives - FTS4 options - similar to the "tokenize=*" option
  may also appear in place of column names. An FTS4 option consists of the
  option name, followed by an "=" character, followed by the option value.
  The option value may optionally be enclosed in single or double quotes, with
  embedded quote characters escaped in the same way as for SQL literals. There
  may not be whitespace on either side of the "=" character. For example,
  to create an FTS4 table with the value of option "matchinfo" set to "fts3":

-- Create a reduced-footprint FTS4 table.
CREATE VIRTUAL TABLE papers USING fts4(author, document, matchinfo=fts3);



  FTS4 currently supports the following options:


  OptionInterpretation
  compress
    The compress option is used to specify the compress function. It is an error to
    specify a compress function without also specifying an uncompress
    function. See below for details.

  content
    The content allows the text being indexed to be
    stored in a separate table distinct from the FTS4 table,
    or even outside of SQLite.

  languageid
    The languageid option causes the FTS4 table to have an additional hidden
    integer column that identifies the language of the text contained in
    each row.  The use of the languageid option allows the same FTS4 table
    to hold text in multiple languages or scripts, each with different tokenizer
    rules, and to query each language independently of the others.
    
  matchinfo 
    When set to the value "fts3", the matchinfo option reduces the amount of
    information stored by FTS4 with the consequence that the "l" option of
    matchinfo() is no longer available.

  notindexed 
    This option is used to specify the name of a column for which data is
    not indexed. Values stored in columns that are not indexed are not
    matched by MATCH queries. Nor are they recognized by auxiliary functions.
    A single CREATE VIRTUAL TABLE statement may have any number of notindexed 
    options.

  order
    

    The "order" option may be set to either "DESC" or "ASC" (in upper or
    lower case). If it is set to "DESC", then FTS4 stores its data in such
    a way as to optimize returning results in descending order by docid.
    If it is set to "ASC" (the default), then the data structures are 
    optimized for returning results in ascending order by docid.  In other
    words, if many of the queries run against the FTS4 table use "ORDER BY
    docid DESC", then it may improve performance to add the "order=desc" 
    option to the CREATE VIRTUAL TABLE statement.

  prefix
    This option may be set to a comma-separated list of positive non-zero 
    integers. For each integer N in the list, a separate index is created
    in the database file to optimize prefix queries where
    the query term is N bytes in length, not including the '*' character,
    when encoded using UTF-8. See below for details.

  uncompress
    This option is used to specify the uncompress function. It is an error to
    specify an uncompress function without also specifying a compress
    function. See below for details.



  When using FTS4, specifying a column name that contains an "=" character
  and is not either a "tokenize=*" specification or a recognized FTS4 option
  is an error. With FTS3, the first token in the unrecognized directive is 
  interpreted as a column name. Similarly, specifying multiple "tokenize=*"
  directives in a single table declaration is an error when using FTS4, whereas
  the second and subsequent "tokenize=*" directives are interpreted as column
  names by FTS3. For example:

-- An error. FTS4 does not recognize the directive "xyz=abc".
CREATE VIRTUAL TABLE papers USING fts4(author, document, xyz=abc);

-- Create an FTS3 table with three columns - "author", "document"
-- and "xyz".
CREATE VIRTUAL TABLE papers USING fts3(author, document, xyz=abc);

-- An error. FTS4 does not allow multiple tokenize=* directives
CREATE VIRTUAL TABLE papers USING fts4(tokenize=porter, tokenize=simple);

-- Create an FTS3 table with a single column named "tokenize". The
-- table uses the "porter" tokenizer.
CREATE VIRTUAL TABLE papers USING fts3(tokenize=porter, tokenize=simple);

-- An error. Cannot create a table with two columns named "tokenize".
CREATE VIRTUAL TABLE papers USING fts3(tokenize=porter, tokenize=simple, tokenize=icu);





6.1. The compress= and uncompress= options


  The compress and uncompress options allow FTS4 content to be stored in
  the database in a compressed form. Both options should be set to the name
  of an SQL scalar function registered using sqlite3_create_function()
  that accepts a single argument. 


  The compress function should return a compressed version of the value 
  passed to it as an argument. Each time data is written to the FTS4 table, 
  each column value is passed to the compress function and the result value 
  stored in the database. The compress function may return any type of SQLite 
  value (blob, text, real, integer or null).


  The uncompress function should uncompress data previously compressed by
  the compress function. In other words, for all SQLite values X, it should
  be true that uncompress(compress(X)) equals X. When data that has been
  compressed by the compress function is read from the database by FTS4, it
  is passed to the uncompress function before it is used.


  If the specified compress or uncompress functions do not exist, the table
  may still be created. An error is not returned until the FTS4 table is
  read (if the uncompress function does not exist) or written (if it is the 
  compress function that does not exist).

-- Create an FTS4 table that stores data in compressed form. This
-- assumes that the scalar functions zip() and unzip() have been (or
-- will be) added to the database handle.
CREATE VIRTUAL TABLE papers USING fts4(author, document, compress=zip, uncompress=unzip);



  When implementing the compress and uncompress functions it is important to
  pay attention to data types. Specifically, when a user reads a value from
  a compressed FTS table, the value returned by FTS is exactly the same
  as the value returned by the uncompress function, including the data type.
  If that data type is not the same as the data type of the original value as
  passed to the compress function (for example if the uncompress function is
  returning BLOB when compress was originally passed TEXT), then the users
  query may not function as expected.




6.2. The content= option 


  The content option allows FTS4 to forego storing the text being indexed.
  The content option can be used in two ways:


 The indexed documents are not stored within the SQLite database 
        at all (a "contentless" FTS4 table), or

 The indexed documents are stored in a database table created and
        managed by the user (an "external content" FTS4 table).



  Because the indexed documents themselves are usually much larger than 
  the full-text index, the content option can be used to achieve 
  significant space savings.


6.2.1.  Contentless FTS4 Tables 


  In order to create an FTS4 table that does not store a copy of the indexed
  documents at all, the content option should be set to an empty string.
  For example, the following SQL creates such an FTS4 table with three
  columns - "a", "b", and "c":

CREATE VIRTUAL TABLE t1 USING fts4(content="", a, b, c);



  Data can be inserted into such an FTS4 table using an INSERT statements.
  However, unlike ordinary FTS4 tables, the user must supply an explicit
  integer docid value. For example:

-- This statement is Ok:
INSERT INTO t1(docid, a, b, c) VALUES(1, 'a b c', 'd e f', 'g h i');

-- This statement causes an error, as no docid value has been provided:
INSERT INTO t1(a, b, c) VALUES('j k l', 'm n o', 'p q r');



  It is not possible to UPDATE or DELETE a row stored in a contentless FTS4
  table. Attempting to do so is an error.


  Contentless FTS4 tables also support SELECT statements. However, it is
  an error to attempt to retrieve the value of any table column other than
  the docid column. The auxiliary function matchinfo() may be used, but
  snippet() and offsets() may not. For example:

-- The following statements are Ok:
SELECT docid FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a MATCH 'xxx';
SELECT matchinfo(t1) FROM t1 WHERE t1 MATCH 'xxx';

-- The following statements all cause errors, as the value of columns
-- other than docid are required to evaluate them.
SELECT * FROM t1;
SELECT a, b FROM t1 WHERE t1 MATCH 'xxx';
SELECT docid FROM t1 WHERE a LIKE 'xxx%';
SELECT snippet(t1) FROM t1 WHERE t1 MATCH 'xxx';



  Errors related to attempting to retrieve column values other than docid
  are runtime errors that occur within sqlite3_step(). In some cases, for
  example if the MATCH expression in a SELECT query matches zero rows, there
  may be no error at all even if a statement does refer to column values 
  other than docid.


6.2.2.  External Content FTS4 Tables 


  An "external content" FTS4 table is similar to a contentless table, except
  that if evaluation of a query requires the value of a column other than 
  docid, FTS4 attempts to retrieve that value from a table (or view, or 
  virtual table) nominated by the user (hereafter referred to as the "content
  table"). The FTS4 module never writes to the content table, and writing
  to the content table does not affect the full-text index. It is the
  responsibility of the user to ensure that the content table and the 
  full-text index are consistent.


  An external content FTS4 table is created by setting the content option
  to the name of a table (or view, or virtual table) that may be queried by
  FTS4 to retrieve column values when required. If the nominated table does
  not exist, then an external content table behaves in the same way as
  a contentless table. For example:

CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", a, c);



  Assuming the nominated table does exist, then its columns must be the same 
  as or a superset of those defined for the FTS table.  The external table
  must also be in the same database file as the FTS table.  In other words,
  The external table cannot be in a different database file connected using
  ATTACH nor may one of the FTS table and the external content be in the
  TEMP database when the other is in a persistent database file such as MAIN.


  When a users query on the FTS table requires a column value other than
  docid, FTS attempts to read the requested value from the corresponding column of
  the row in the content table with a rowid value equal to the current FTS
  docid. Only the subset of content-table columns duplicated in the FTS/34
  table declaration can be queried for - to retrieve values from any other
  columns the content table must be queried directly. Or, if such a row cannot
  be found in the content table, a NULL value is used instead. For example:

CREATE TABLE t2(id INTEGER PRIMARY KEY, a, b, c);
CREATE VIRTUAL TABLE t3 USING fts4(content="t2", b, c);

INSERT INTO t2 VALUES(2, 'a b', 'c d', 'e f');
INSERT INTO t2 VALUES(3, 'g h', 'i j', 'k l');
INSERT INTO t3(docid, b, c) SELECT id, b, c FROM t2;
-- The following query returns a single row with two columns containing
-- the text values "i j" and "k l".
--
-- The query uses the full-text index to discover that the MATCH 
-- term matches the row with docid=3. It then retrieves the values
-- of columns b and c from the row with rowid=3 in the content table
-- to return.
--
SELECT * FROM t3 WHERE t3 MATCH 'k';

-- Following the UPDATE, the query still returns a single row, this
-- time containing the text values "xxx" and "yyy". This is because the
-- full-text index still indicates that the row with docid=3 matches
-- the FTS4 query 'k', even though the documents stored in the content
-- table have been modified.
--
UPDATE t2 SET b = 'xxx', c = 'yyy' WHERE rowid = 3;
SELECT * FROM t3 WHERE t3 MATCH 'k';

-- Following the DELETE below, the query returns one row containing two
-- NULL values. NULL values are returned because FTS is unable to find
-- a row with rowid=3 within the content table.
--
DELETE FROM t2;
SELECT * FROM t3 WHERE t3 MATCH 'k';



  When a row is deleted from an external content FTS4 table, FTS4 needs to
  retrieve the column values of the row being deleted from the content table.
  This is so that FTS4 can update the full-text index entries for each token
  that occurs within the deleted row to indicate that row has been 
  deleted. If the content table row cannot be found, or if it contains values
  inconsistent with the contents of the FTS index, the results can be difficult
  to predict. The FTS index may be left containing entries corresponding to the
  deleted row, which can lead to seemingly nonsensical results being returned
  by subsequent SELECT queries. The same applies when a row is updated, as
  internally an UPDATE is the same as a DELETE followed by an INSERT.


  This means that in order to keep an FTS in sync with an external content
  table, any UPDATE or DELETE operations must be applied first to the FTS
  table, and then to the external content table. For example:

CREATE TABLE t1_real(id INTEGER PRIMARY KEY, a, b, c, d);
CREATE VIRTUAL TABLE t1_fts USING fts4(content="t1_real", b, c);

-- This works. When the row is removed from the FTS table, FTS retrieves 
-- the row with rowid=123 and tokenizes it in order to determine the entries 
-- that must be removed from the full-text index.
-- 
DELETE FROM t1_fts WHERE rowid = 123;
DELETE FROM t1_real WHERE rowid = 123;

-- This does not work. By the time the FTS table is updated, the row
-- has already been deleted from the underlying content table. As a result
-- FTS is unable to determine the entries to remove from the FTS index and
-- so the index and content table are left out of sync.
--
DELETE FROM t1_real WHERE rowid = 123;
DELETE FROM t1_fts WHERE rowid = 123;


  
  Instead of writing separately to the full-text index and the content table,
  some users may wish to use database triggers to keep the full-text index
  up to date with respect to the set of documents stored in the content table.
  For example, using the tables from earlier examples:

CREATE TRIGGER t2_bu BEFORE UPDATE ON t2 BEGIN
  DELETE FROM t3 WHERE docid=old.rowid;
END;
CREATE TRIGGER t2_bd BEFORE DELETE ON t2 BEGIN
  DELETE FROM t3 WHERE docid=old.rowid;
END;

CREATE TRIGGER t2_au AFTER UPDATE ON t2 BEGIN
  INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;
CREATE TRIGGER t2_ai AFTER INSERT ON t2 BEGIN
  INSERT INTO t3(docid, b, c) VALUES(new.rowid, new.b, new.c);
END;



  The DELETE trigger must be fired before the actual delete takes place
  on the content table. This is so that FTS4 can still retrieve the original
  values in order to update the full-text index. And the INSERT trigger must
  be fired after the new row is inserted, so as to handle the case where the
  rowid is assigned automatically within the system. The UPDATE trigger must
  be split into two parts, one fired before and one after the update of the
  content table, for the same reasons.


  The FTS4 "rebuild" command
  deletes the entire full-text index and rebuilds it based on the current
  set of documents in the content table. Assuming again that "t3" is the
  name of the external content FTS4 table, the rebuild command looks like this:

INSERT INTO t3(t3) VALUES('rebuild');



  This command may also be used with ordinary FTS4 tables, for example if
  the implementation of the tokenizer changes.  It is an
  error to attempt to rebuild the full-text index maintained by a contentless
  FTS4 table, since no content will be available to do the rebuilding.





6.3. The languageid= option


  When the languageid option is present, it specifies the name of
  another hidden column that is added to the FTS4
  table and which is used to specify the language stored in each row
  of the FTS4 table.  The name of the languageid hidden column must
  be distinct from all other column names in the FTS4 table.  Example:

CREATE VIRTUAL TABLE t1 USING fts4(x, y, languageid="lid")



  The default value of a languageid column is 0. Any value inserted
  into a languageid column is converted to a 32-bit (not 64) signed
  integer.


  By default, FTS queries (those that use the MATCH operator)
  consider only those rows with the languageid column set to 0. To
  query for rows with other languageid values, a constraint of the
  form " = " must be added to the queries
  WHERE clause. For example:

SELECT * FROM t1 WHERE t1 MATCH 'abc' AND lid=5;



  It is not possible for a single FTS query to return rows with
  different languageid values. The results of adding WHERE clauses
  that use other operators (e.g.  lid!=5, or lid&lt;=5) are undefined.


  If the content option is used along with the languageid option,
  then the named languageid column must exist in the content= table
  (subject to the usual rules - if a query never needs to read the
  content table then this restriction does not apply).


  When the languageid option is used, SQLite invokes the xLanguageid()
  on the sqlite3_tokenizer_module object immediately after the object
  is created in order to pass in the language id that the
  tokenizer should use.  The xLanguageid() method will never be called
  more than once for any single tokenizer object.  The fact that different
  languages might be tokenized differently is one reason why no single
  FTS query can return rows with different languageid values.
  





6.4. The matchinfo= option


  The matchinfo option may only be set to the value "fts3". 
  Attempting to set matchinfo to anything other than "fts3" is an error.
  If this option is specified, then some of the extra information stored by
  FTS4 is omitted. This reduces the amount of disk space consumed by
  an FTS4 table until it is almost the same as the amount that would
  be used by the equivalent FTS3 table, but also means that the data
  accessed by passing the 'l' flag to the matchinfo() function is
  not available. 




6.5. The notindexed= option


  Normally, the FTS module maintains an inverted index of all terms in
  all columns of the table. This option is used to specify the name of
  a column for which entries should not be added to the index. Multiple
  "notindexed" options may be used to specify that multiple columns should
  be omitted from the index. For example:

-- Create an FTS4 table for which only the contents of columns c2 and c4
-- are tokenized and added to the inverted index.
CREATE VIRTUAL TABLE t1 USING fts4(c1, c2, c3, c4, notindexed=c1, notindexed=c3);



  Values stored in unindexed columns are not eligible to match MATCH 
  operators. They do not influence the results of the offsets() or matchinfo()
  auxiliary functions. Nor will the snippet() function ever return a
  snippet based on a value stored in an unindexed column.




6.6. The prefix= option


  The FTS4 prefix option causes FTS to index term prefixes of specified lengths
  in the same way that it always indexes complete terms.  The prefix option
  must be set to a comma separated list of positive non-zero integers. 
  For each value N in the list, prefixes of length N bytes (when encoded 
  using UTF-8) are indexed.  FTS4 uses term prefix indexes to speed up
  prefix queries. The cost, of course, is that indexing term prefixes as
  well as complete terms increases the database size and slows down write 
  operations on the FTS4 table.


  Prefix indexes may be used to optimize prefix queries in two cases.
  If the query is for a prefix of N bytes, then a prefix index created
  with "prefix=N" provides the best optimization. Or, if no "prefix=N"
  index is available, a "prefix=N+1" index may be used instead. 
  Using a "prefix=N+1" index is less
  efficient than a "prefix=N" index, but is better than no prefix index at all.

-- Create an FTS4 table with indexes to optimize 2 and 4 byte prefix queries.
CREATE VIRTUAL TABLE t1 USING fts4(c1, c2, prefix="2,4");

-- The following two queries are both optimized using the prefix indexes.
SELECT * FROM t1 WHERE t1 MATCH 'ab*';
SELECT * FROM t1 WHERE t1 MATCH 'abcd*';

-- The following two queries are both partially optimized using the prefix
-- indexes. The optimization is not as pronounced as it is for the queries
-- above, but still an improvement over no prefix indexes at all.
SELECT * FROM t1 WHERE t1 MATCH 'a*';
SELECT * FROM t1 WHERE t1 MATCH 'abc*';





7. Special Commands For FTS3 and FTS4


  Special INSERT operates can be used to issue commands to FTS3 and FTS4 tables.
  Every FTS3 and FTS4 has a hidden, read-only column which is the same name as
  the table itself.  INSERTs into this hidden column are interpreted as commands
  to the FTS3/4 table.  For a table with the name "xyz" the following commands
  are supported:


INSERT INTO xyz(xyz) VALUES('optimize');
INSERT INTO xyz(xyz) VALUES('rebuild');
INSERT INTO xyz(xyz) VALUES('integrity-check');
INSERT INTO xyz(xyz) VALUES('merge=X,Y');
INSERT INTO xyz(xyz) VALUES('automerge=N');




7.1. The "optimize" command


  The "optimize" command causes FTS3/4 to merge together all of its
  inverted index b-trees into one large and complete b-tree.  Doing
  an optimize will make subsequent queries run faster since there are
  fewer b-trees to search, and it may reduce disk usage by coalescing
  redundant entries.  However, for a large FTS table, running optimize
  can be as expensive as running VACUUM.  The optimize command
  essentially has to read and write the entire FTS table, resulting
  in a large transaction.


  In batch-mode operation, where an FTS table is initially built up
  using a large number of INSERT operations, then queried repeatedly
  without further changes, it is often a good idea
  to run "optimize" after the last INSERT and before the first query.



7.2. The "rebuild" command


  The "rebuild" command causes SQLite to discard the entire FTS3/4
  table and then rebuild it again from original text.  The concept
  is similar to REINDEX, only that it applies to an
  FTS3/4 table instead of an ordinary index.


  The "rebuild" command should be run whenever the implementation
  of a custom tokenizer changes, so that all content can be retokenized.
  The "rebuild" command is also useful when using the
  FTS4 content option after changes have been made to the original
  content table.



7.3. The "integrity-check" command


  The "integrity-check" command causes SQLite to read and verify
  the accuracy of all inverted indices in an FTS3/4 table by comparing
  those inverted indices against the original content. The 
  "integrity-check" command silently succeeds if the inverted
  indices are all ok, but will fail with an SQLITE_CORRUPT error
  if any problems are found.


  The "integrity-check" command is similar in concept to
 PRAGMA integrity_check.  In a working system, the "integrity-command"
 should always be successful.  Possible causes of integrity-check
 failures include:
  
   The application has made changes to the FTS shadow tables
       directly, without using the FTS3/4 virtual table, causing
       the shadow tables to become out of sync with each other.
   Using the FTS4 content option and failing to manually keep
       the content in sync with the FTS4 inverted indices.
   Bugs in the FTS3/4 virtual table.  (The "integrity-check"
       command was original conceived as part of the test suite
       for FTS3/4.)
   Corruption to the underlying SQLite database file.  (See
       documentation on how to corrupt and SQLite database for
       additional information.)
  



7.4. The "merge=X,Y" command


  The "merge=X,Y" command (where X and Y are integers) causes SQLite
  to do a limited amount of work toward merging the various inverted
  index b-trees of an FTS3/4 table together into one large b-tree.
  The X value is the target number of "blocks" to be merged, and Y is
  the minimum number of b-tree segments on a level required before
  merging will be applied to that level.  The value of Y should
  be between 2 and 16 with a recommended value of 8.  The value of X
  can be any positive integer but values on the order of 100 to 300
  are recommended.


  When an FTS table accumulates 16 b-tree segments at the same level,
  the next INSERT into that table will cause all 16 segments to be
  merged into a single b-tree segment at the next higher level.  The
  effect of these level merges is that most INSERTs into an FTS table
  are very fast and take minimal memory, but an occasional INSERT is
  slow and generates a large transaction because of the need to
  do merging. This results in "spiky" performance of INSERTs.


  To avoid spiky INSERT performance, an application can run the
  "merge=X,Y" command periodically, possibly in an idle thread or
  idle process, to ensure that the FTS table never accumulates
  too many b-tree segments at the same level.  INSERT performance
  spikes can generally be avoided, and performance of FTS3/4 can be
  maximized, by running "merge=X,Y" after every few thousand
  document inserts.  Each "merge=X,Y" command will run in a separate
  transaction (unless they are grouped together using BEGIN...COMMIT,
  of course).  The transactions can be kept small by choosing a value
  for X in the range of 100 to 300.  The idle thread that is running
  the merge commands can know when it is done by checking the difference
  in sqlite3_total_changes() before and after each "merge=X,Y"
  command and stopping the loop when the difference drops below two.



7.5. The "automerge=N" command


  The "automerge=N" command (where N is an integer between 0 and 15,
  inclusive) is used to configure an FTS3/4 tables "automerge" parameter,
  which controls automatic incremental inverted index merging. The default 
  automerge value for new tables is 0, meaning that automatic incremental 
  merging is completely disabled. If the value of the automerge parameter
  is modified using the "automerge=N" command, the new parameter value is
  stored persistently in the database and is used by all subsequently
  established database connections.


  Setting the automerge parameter to a non-zero value enables automatic
  incremental merging. This causes SQLite to do a small amount of inverted 
  index merging after every INSERT operation. The amount of merging 
  performed is designed so that the FTS3/4 table never reaches a point 
  where it has 16 segments at the same level and hence has to do a large 
  merge in order to complete an insert.  In other words, automatic 
  incremental merging is designed to prevent spiky INSERT performance.


  The downside of automatic incremental merging is that it makes
  every INSERT, UPDATE, and DELETE operation on an FTS3/4 table run
  a little slower, since extra time must be used to do the incremental
  merge.  For maximum performance, it is recommended that applications
  disable automatic incremental merge and instead use the 
  "merge" command in an idle process to keep the inverted indices
  well merged.  But if the structure of an application does not easily
  allow for idle processes, the use of automatic incremental merge is
  a very reasonable fallback solution.


  The actual value of the automerge parameter determines the number of
  index segments merged simultaneously by an automatic inverted index
  merge. If the value is set to N, the system waits until there are at
  least N segments on a single level before beginning to incrementally
  merge them. Setting a lower value of N causes segments to be merged more
  quickly, which may speed up full-text queries and, if the workload 
  contains UPDATE or DELETE operations as well as INSERTs, reduce the space
  on disk consumed by the full-text index. However, it also increases the
  amount of data written to disk.


  For general use in cases where the workload contains few UPDATE or DELETE
  operations, a good choice for automerge is 8. If the workload contains
  many UPDATE or DELETE commands, 
  or if query speed is a concern, it may be advantageous to reduce automerge
  to 2.


  For reasons of backwards compatibility, the "automerge=1" command sets
  the automerge parameter to 8, not 1 (a value of 1 would make no sense 
  anyway, as merging data from a single segment is a no-op).



8. Tokenizers


  An FTS tokenizer is a set of rules for extracting terms from a document 
  or basic FTS full-text query. 


  Unless a specific tokenizer is specified as part of the CREATE 
  VIRTUAL TABLE statement used to create the FTS table, the default 
  tokenizer, "simple", is used. The simple tokenizer extracts tokens from
  a document or basic FTS full-text query according to the following 
  rules:


   A term is a contiguous sequence of eligible characters, where 
    eligible characters are all alphanumeric characters and all characters with
    Unicode codepoint values greater than or equal to 128.
    All other characters are
    discarded when splitting a document into terms. Their only contribution is
    to separate adjacent terms.

   All uppercase characters within the ASCII range (Unicode codepoints
    less than 128), are transformed to their lowercase equivalents as part
    of the tokenization process. Thus, full-text queries are
    case-insensitive when using the simple tokenizer.



  For example, when a document containing the text "Right now, they're very
  frustrated.", the terms extracted from the document and added to the 
  full-text index are, in order, "right now they re very frustrated". Such
  a document would match a full-text query such as "MATCH 'Frustrated'", 
  as the simple tokenizer transforms the term in the query to lowercase
  before searching the full-text index.


  As well as the "simple" tokenizer, the FTS source code features a tokenizer 
  that uses the Porter 
  Stemming algorithm. This tokenizer uses the same rules to separate
  the input document into terms including folding all terms into lower case,
  but also uses the Porter Stemming algorithm to reduce related English language
  words to a common root. For example, using the same input document as in the
  paragraph above, the porter tokenizer extracts the following tokens:
  "right now thei veri frustrat". Even though some of these terms are not even
  English words, in some cases using them to build the full-text index is more
  useful than the more intelligible output produced by the simple tokenizer.
  Using the porter tokenizer, the document not only matches full-text queries
  such as "MATCH 'Frustrated'", but also queries such as "MATCH 'Frustration'",
  as the term "Frustration" is reduced by the Porter stemmer algorithm to 
  "frustrat" - just as "Frustrated" is. So, when using the porter tokenizer,
  FTS is able to find not just exact matches for queried terms, but matches
  against similar English language terms. For more information on the 
  Porter Stemmer algorithm, please refer to the page linked above.


  Example illustrating the difference between the "simple" and "porter"
  tokenizers:

-- Create a table using the simple tokenizer. Insert a document into it.
CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple);
INSERT INTO simple VALUES('Right now they''re very frustrated');

-- The first of the following two queries matches the document stored in
-- table "simple". The second does not.
SELECT * FROM simple WHERE simple MATCH 'Frustrated';
SELECT * FROM simple WHERE simple MATCH 'Frustration';

-- Create a table using the porter tokenizer. Insert the same document into it
CREATE VIRTUAL TABLE porter USING fts3(tokenize=porter);
INSERT INTO porter VALUES('Right now they''re very frustrated');

-- Both of the following queries match the document stored in table "porter".
SELECT * FROM porter WHERE porter MATCH 'Frustrated';
SELECT * FROM porter WHERE porter MATCH 'Frustration';



  If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor
  symbol defined, then there exists a built-in tokenizer named "icu"
  implemented using the ICU library. The first argument passed to the
  xCreate() method (see fts3_tokenizer.h) of this tokenizer may be
  an ICU locale identifier. For example "tr_TR" for Turkish as used
  in Turkey, or "en_AU" for English as used in Australia. For example:

CREATE VIRTUAL TABLE thai_text USING fts3(text, tokenize=icu th_TH)



  The ICU tokenizer implementation is very simple. It splits the input
  text according to the ICU rules for finding word boundaries and discards
  any tokens that consist entirely of white-space. This may be suitable
  for some applications in some locales, but not all. If more complex
  processing is required, for example to implement stemming or
  discard punctuation, this can be done by creating a tokenizer
  implementation that uses the ICU tokenizer as part of its implementation.




  The "unicode61" tokenizer is available beginning with SQLite version 3.7.13
  (2012-06-11).
  Unicode61 works very much like "simple" except that it does simple unicode
  case folding according to rules in Unicode Version 6.1 and it recognizes
  unicode space and punctuation characters and uses those to separate tokens.
  The simple tokenizer only does case folding of ASCII characters and only
  recognizes ASCII space and punctuation characters as token separators.


  By default, "unicode61" attempts to remove diacritics from Latin script
  characters. This behaviour can be overridden by adding the tokenizer argument
  "remove_diacritics=0". For example:

-- Create tables that remove alldiacritics from Latin script characters
-- as part of tokenization.
CREATE VIRTUAL TABLE txt1 USING fts4(tokenize=unicode61);
CREATE VIRTUAL TABLE txt2 USING fts4(tokenize=unicode61 "remove_diacritics=2");

-- Create a table that does not remove diacritics from Latin script
-- characters as part of tokenization.
CREATE VIRTUAL TABLE txt3 USING fts4(tokenize=unicode61 "remove_diacritics=0");


The remove_diacritics option may be set to "0", "1" or "2". The default
   value is "1".  If it is set to "1" or "2", then diacritics are removed from
   Latin script characters as described above. However, if it is set to "1",
   then diacritics are not removed in the fairly uncommon case where a single
   unicode codepoint is used to represent a character with more that one
   diacritic. For example, diacritics are not removed from codepoint 0x1ED9
   ("LATIN SMALL LETTER O WITH CIRCUMFLEX AND DOT BELOW"). This is technically
   a bug, but cannot be fixed without creating backwards compatibility
   problems. If this option is set to "2", then diacritics are correctly
   removed from all Latin characters.


  It is also possible to customize the set of codepoints that unicode61 treats
  as separator characters. The "separators=" option may be used to specify one
  or more extra characters that should be treated as separator characters, and
  the "tokenchars=" option may be used to specify one or more extra characters
  that should be treated as part of tokens instead of as separator characters.
  For example:

-- Create a table that uses the unicode61 tokenizer, but considers "."
-- and "=" characters to be part of tokens, and capital "X" characters to
-- function as separators.
CREATE VIRTUAL TABLE txt3 USING fts4(tokenize=unicode61 "tokenchars=.=" "separators=X");

-- Create a table that considers space characters (codepoint 32) to be
-- a token character
CREATE VIRTUAL TABLE txt4 USING fts4(tokenize=unicode61 "tokenchars= ");



  If a character specified as part of the argument to "tokenchars=" is considered
  to be a token character by default, it is ignored. This is true even if it has
  been marked as a separator by an earlier "separators=" option. Similarly, if
  a character specified as part of a "separators=" option is treated as a separator
  character by default, it is ignored. If multiple "tokenchars=" or "separators="
  options are specified, all are processed. For example:

-- Create a table that uses the unicode61 tokenizer, but considers "."
-- and "=" characters to be part of tokens, and capital "X" characters to
-- function as separators. Both of the "tokenchars=" options are processed
-- The "separators=" option ignores the "." passed to it, as "." is by
-- default a separator character, even though it has been marked as a token
-- character by an earlier "tokenchars=" option.
CREATE VIRTUAL TABLE txt5 USING fts4(
    tokenize=unicode61 "tokenchars=." "separators=X." "tokenchars=="
);



  The arguments passed to the "tokenchars=" or "separators=" options are 
  case-sensitive. In the example above, specifying that "X" is a separator
  character does not affect the way "x" is handled.



8.1. Custom (Application Defined) Tokenizers


  In addition to providing built-in "simple", "porter" and (possibly) "icu" and
  "unicode61" tokenizers,
  FTS provides an interface for applications to implement and register custom
  tokenizers written in C.  The interface used to create a new tokenizer is defined
  and described in the fts3_tokenizer.h source file.


  Registering a new FTS tokenizer is similar to registering a new
  virtual table module with SQLite. The user passes a pointer to a
  structure containing pointers to various callback functions that
  make up the implementation of the new tokenizer type. For tokenizers,
  the structure (defined in fts3_tokenizer.h) is called
  "sqlite3_tokenizer_module".


  FTS does not expose a C-function that users call to register new
  tokenizer types with a database handle. Instead, the pointer must
  be encoded as an SQL blob value and passed to FTS through the SQL
  engine by evaluating a special scalar function, "fts3_tokenizer()".
  The fts3_tokenizer() function may be called with one or two arguments,
  as follows:

SELECT fts3_tokenizer(&lt;tokenizer-name&gt;);
SELECT fts3_tokenizer(&lt;tokenizer-name&gt;, &lt;sqlite3_tokenizer_module ptr&gt;);



  Where &lt;tokenizer-name&gt; is parameter to which a string is bound using
  sqlite3_bind_text() where the string identifies the tokenizer and
  &lt;sqlite3_tokenizer_module ptr&gt; is a parameter to which a BLOB is
  bound using sqlite3_bind_blob() where the value of the BLOB is a
  pointer to an sqlite3_tokenizer_module structure.
  If the second argument is present,
  it is registered as tokenizer &lt;tokenizer-name&gt; and a copy of it
  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as &lt;tokenizer-name&gt; is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.


  Prior to SQLite version 3.11.0 (2016-02-15), the arguments to
  fts3_tokenizer() could be literal strings or BLOBs. They did not have to
  be bound parameters.  But that could lead to security problems in the
  event of an SQL injection.  Hence, the legacy behavior is now disabled
  by default.  But the old legacy behavior can be enabled, for backwards
  compatibility in applications that really need it, 
  by calling
  sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER,1,0).


  The following block contains an example of calling the fts3_tokenizer()
  function from C code:

/*
** Register a tokenizer implementation with FTS3 or FTS4.
*/
int registerTokenizer(
  sqlite3 *db,
  char *zName,
  const sqlite3_tokenizer_module *p
){
  int rc;
  sqlite3_stmt *pStmt;
  const char *zSql = "SELECT fts3_tokenizer(?1, ?2)";

  rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
  sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC);
  sqlite3_step(pStmt);

  return sqlite3_finalize(pStmt);
}

/*
** Query FTS for the tokenizer implementation named zName.
*/
int queryTokenizer(
  sqlite3 *db,
  char *zName,
  const sqlite3_tokenizer_module **pp
){
  int rc;
  sqlite3_stmt *pStmt;
  const char *zSql = "SELECT fts3_tokenizer(?)";

  *pp = 0;
  rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0);
  if( rc!=SQLITE_OK ){
    return rc;
  }

  sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
  if( SQLITE_ROW==sqlite3_step(pStmt) ){
    if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){
      memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
    }
  }

  return sqlite3_finalize(pStmt);
}





8.2. Querying Tokenizers

The "fts3tokenize" virtual table can be used to directly access any
   tokenizer.  The following SQL demonstrates how to create an instance 
   of the fts3tokenize virtual table:

CREATE VIRTUAL TABLE tok1 USING fts3tokenize('porter');


The name of the desired tokenizer should be substituted in place of
   'porter' in the example, of course.  If the tokenizer requires one or
   more arguments, they should be separated by commas in the fts3tokenize
   declaration (even though they are separated by spaces in declarations
   of regular fts4 tables). The following creates fts4 and fts3tokenize
   tables that use the same tokenizer:
CREATE VIRTUAL TABLE text1 USING fts4(tokenize=icu en_AU);
CREATE VIRTUAL TABLE tokens1 USING fts3tokenize(icu, en_AU);

CREATE VIRTUAL TABLE text2 USING fts4(tokenize=unicode61 "tokenchars=@." "separators=123");
CREATE VIRTUAL TABLE tokens2 USING fts3tokenize(unicode61, "tokenchars=@.", "separators=123");

   

   Once the virtual table is created, it can be queried as follows:

SELECT token, start, end, position 
  FROM tok1
 WHERE input='This is a test sentence.';


The virtual table will return one row of output for each token in the
   input string.  The "token" column is the text of the token.  The "start"
   and "end" columns are the byte offset to the beginning and end of the
   token in the original input string.  
   The "position" column is the sequence number
   of the token in the original input string.  There is also an "input"
   column which is simply a copy of the input string that is specified in
   the WHERE clause.  Note that a constraint of the form "input=?" must
   appear in the WHERE clause or else the virtual table will have no input
   to tokenize and will return no rows.  The example above generates
   the following output:

thi|0|4|0
is|5|7|1
a|8|9|2
test|10|14|3
sentenc|15|23|4


Notice that the tokens in the result set from the fts3tokenize virtual
   table have been transformed according to the rules of the tokenizer.
   Since this example used the "porter" tokenizer, the "This" token was
   converted into "thi".  If the original text of the token is desired,
   it can be retrieved using the "start" and "end" columns with the
   substr() function.  For example:

SELECT substr(input, start+1, end-start), token, position
  FROM tok1
 WHERE input='This is a test sentence.';


The fts3tokenize virtual table can be used on any tokenizer, regardless
   of whether or not there exists an FTS3 or FTS4 table that actually uses
   that tokenizer.

 

9. Data Structures


  This section describes at a high-level the way the FTS module stores its
  index and content in the database. It is not necessary to read or 
  understand the material in this section in order to use FTS in an 
  application. However, it may be useful to application developers attempting 
  to analyze and understand FTS performance characteristics, or to developers 
  contemplating enhancements to the existing FTS feature set.





9.1. Shadow Tables

  For each FTS virtual table in a database, three to five real (non-virtual) tables
  are created to store the underlying data.  These real tables are called "shadow tables".
  The real tables are named "%_content",
  "%_segdir", "%_segments", "%_stat", and "%_docsize", where "%" is replaced by the name
  of the FTS virtual table.


  The leftmost column of the "%_content" table is an INTEGER PRIMARY KEY field
  named "docid". Following this is one column for each column of the FTS
  virtual table as declared by the user, named by prepending the column name
  supplied by the user with "cN", where N is the index of the 
  column within the table, numbered from left to right starting with 0. Data
  types supplied as part of the virtual table declaration are not used as
  part of the %_content table declaration. For example:

-- Virtual table declaration
CREATE VIRTUAL TABLE abc USING fts4(a NUMBER, b TEXT, c);

-- Corresponding %_content table declaration
CREATE TABLE abc_content(docid INTEGER PRIMARY KEY, c0a, c1b, c2c);



  The %_content table contains the unadulterated data inserted by the user 
  into the FTS virtual table by the user. If the user does not explicitly
  supply a "docid" value when inserting records, one is selected automatically
  by the system.


  The %_stat and %_docsize tables are only created if the FTS table uses the
  FTS4 module, not FTS3. Furthermore, the %_docsize table is omitted if the
  FTS4 table is created with the "matchinfo=fts3" directive
  specified as part of the CREATE VIRTUAL TABLE statement. If they are created,
  the schema of the two tables is as follows:
CREATE TABLE %_stat(
  id INTEGER PRIMARY KEY, 
  value BLOB
);

CREATE TABLE %_docsize(
  docid INTEGER PRIMARY KEY,
  size BLOB
);



  For each row in the FTS table, the %_docsize table contains a corresponding
  row with the same "docid" value. The "size" field contains a blob consisting
  of N FTS varints, where N is the number of user-defined columns
  in the table. Each varint in the "size" blob is the number of tokens in the
  corresponding column of the associated row in the FTS table. The %_stat table
  always contains a single row with the "id" column set to 0. The "value" 
  column contains a blob consisting of N+1 FTS varints, where N
  is again the number of user-defined columns in the FTS table. The first
  varint in the blob is set to the total number of rows in the FTS table. The
  second and subsequent varints contain the total number of tokens stored in
  the corresponding column for all rows of the FTS table.


  The two remaining tables, %_segments and %_segdir, are used to store the 
  full-text index. Conceptually, this index is a lookup table that maps each 
  term (word) to the set of docid values corresponding to records in the 
  %_content table that contain one or more occurrences of the term. To
  retrieve all documents that contain a specified term, the FTS module
  queries this index to determine the set of docid values for records that
  contain the term, then retrieves the required documents from the %_content
  table. Regardless of the schema of the FTS virtual table, the %_segments
  and %_segdir tables are always created as follows:

CREATE TABLE %_segments(
  blockid INTEGER PRIMARY KEY,       -- B-tree node id
  block blob                         -- B-tree node data
);

CREATE TABLE %_segdir(
  level INTEGER,
  idx INTEGER,
  start_block INTEGER,               -- Blockid of first node in %_segments
  leaves_end_block INTEGER,          -- Blockid of last leaf node in %_segments
  end_block INTEGER,                 -- Blockid of last node in %_segments
  root BLOB,                         -- B-tree root node
  PRIMARY KEY(level, idx)
);



  The schema depicted above is not designed to store the full-text index 
  directly. Instead, it is used to store one or more b-tree structures. There
  is one b-tree for each row in the %_segdir table. The %_segdir table
  row contains the root node and various meta-data associated with the
  b-tree structure, and the %_segments table contains all other (non-root)
  b-tree nodes. Each b-tree is referred to as a "segment". Once it has
  been created, a segment b-tree is never updated (although it may be
  deleted altogether).


  The keys used by each segment b-tree are terms (words). As well as the
  key, each segment b-tree entry has an associated "doclist" (document list).
  A doclist consists of zero or more entries, where each entry consists of:


   A docid (document id), and
   A list of term offsets, one for each occurrence of the term within
       the document. A term offset indicates the number of tokens (words)
       that occur before the term in question, not the number of characters
       or bytes. For example, the term offset of the term "war" in the
       phrase "Ancestral voices prophesying war!" is 3.



  Entries within a doclist are sorted by docid. Positions within a doclist
  entry are stored in ascending order.


  The contents of the logical full-text index is found by merging the
  contents of all segment b-trees. If a term is present in more than one
  segment b-tree, then it maps to the union of each individual doclist. If,
  for a single term, the same docid occurs in more than one doclist, then only
  the doclist that is part of the most recently created segment b-tree is 
  considered valid. 


  Multiple b-tree structures are used instead of a single b-tree to reduce
  the cost of inserting records into FTS tables. When a new record is 
  inserted into an FTS table that already contains a lot of data, it is
  likely that many of the terms in the new record are already present in
  a large number of existing records. If a single b-tree were used, then
  large doclist structures would have to be loaded from the database,
  amended to include the new docid and term-offset list, then written back
  to the database. Using multiple b-tree tables allows this to be avoided
  by creating a new b-tree which can be merged with the existing b-tree
  (or b-trees) later on. Merging of b-tree structures can be performed as
  a background task, or once a certain number of separate b-tree structures
  have been accumulated. Of course, this scheme makes queries more expensive
  (as the FTS code may have to look up individual terms in more than one
  b-tree and merge the results), but it has been found that in practice this
  overhead is often negligible.
  
9.2. Variable Length Integer (varint) Format


  Integer values stored as part of segment b-tree nodes are encoded using the
  FTS varint format. This encoding is similar, but not identical, to
  the SQLite varint format.


  An encoded FTS varint consumes between one and ten bytes of space. The
  number of bytes required is determined by the sign and magnitude of the
  integer value encoded. More accurately, the number of bytes used to store
  the encoded integer depends on the position of the most significant set bit
  in the 64-bit twos-complement representation of the integer value. Negative
  values always have the most significant bit set (the sign bit), and so are
  always stored using the full ten bytes. Positive integer values may be
  stored using less space.


  The final byte of an encoded FTS varint has its most significant bit 
  cleared. All preceding bytes have the most significant bit set. Data
  is stored in the remaining seven least significant bits of each byte.
  The first byte of the encoded representation contains the least significant
  seven bits of the encoded integer value. The second byte of the encoded
  representation, if it is present, contains the seven next least significant
  bits of the integer value, and so on. The following table contains examples
  of encoded integer values:


  DecimalHexadecimalEncoded Representation
  430x000000000000002B0x2B 
  2008150x000000000003106F0xEF 0xA0 0x0C
  -10xFFFFFFFFFFFFFFFF0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0x01

  

9.3. Segment B-Tree Format


  Segment b-trees are prefix-compressed b+-trees. There is one segment b-tree
  for each row in the %_segdir table (see above). The root node of the segment
  b-tree is stored as a blob in the "root" field of the corresponding row
  of the %_segdir table. All other nodes (if any exist) are stored in the 
  "blob" column of the %_segments table. Nodes within the %_segments table are
  identified by the integer value in the blockid field of the corresponding
  row. The following table describes the fields of the %_segdir table:


  Column           Interpretation
  level             
    Between them, the contents of the "level" and "idx" fields define the
    relative age of the segment b-tree. The smaller the value stored in the
    "level" field, the more recently the segment b-tree was created. If two
    segment b-trees are of the same "level", the segment with the larger
    value stored in the "idx" column is more recent. The PRIMARY KEY constraint
    on the %_segdir table prevents any two segments from having the same value
    for both the "level" and "idx" fields.
  idx               See above.
  start_block      
    The blockid that corresponds to the node with the smallest blockid that 
    belongs to this segment b-tree. Or zero if the entire segment b-tree
    fits on the root node. If it exists, this node is always a leaf node.
  leaves_end_block 
    The blockid that corresponds to the leaf node with the largest blockid 
    that belongs to this segment b-tree. Or zero if the entire segment b-tree
    fits on the root node.
  end_block 
    This field may contain either an integer or a text field consisting of
    two integers separated by a space character (unicode codepoint 0x20).

    The first, or only, integer is the blockid that corresponds to the interior
    node with the largest blockid that belongs to this segment b-tree. Or zero
    if the entire segment b-tree fits on the root node. If it exists, this node
    is always an interior node.

    The second integer, if it is present, is the aggregate size of all data
    stored on leaf pages in bytes. If the value is negative, then the segment
    is the output of an unfinished incremental-merge operation, and the
    absolute value is current size in bytes.

  root             
    Blob containing the root node of the segment b-tree.



  Apart from the root node, the nodes that make up a single segment b-tree are
  always stored using a contiguous sequence of blockids. Furthermore, the
  nodes that make up a single level of the b-tree are themselves stored as
  a contiguous block, in b-tree order. The contiguous sequence of blockids
  used to store the b-tree leaves are allocated starting with the blockid
  value stored in the "start_block" column of the corresponding %_segdir row,
  and finishing at the blockid value stored in the "leaves_end_block"
  field of the same row. It is therefore possible to iterate through all the
  leaves of a segment b-tree, in key order, by traversing the %_segments 
  table in blockid order from "start_block" to "leaves_end_block".  

9.3.1. Segment B-Tree Leaf Nodes


  The following diagram depicts the format of a segment b-tree leaf node.


  
   Segment B-Tree Leaf Node Format



  The first term stored on each node ("Term 1" in the figure above) is
  stored verbatim. Each subsequent term is prefix-compressed with respect
  to its predecessor. Terms are stored within a page in sorted (memcmp)
  order.

9.3.2. Segment B-Tree Interior Nodes


  The following diagram depicts the format of a segment b-tree interior 
  (non-leaf) node.


  
   Segment B-Tree Interior Node Format



9.4. Doclist Format


  A doclist consists of an array of 64-bit signed integers, serialized using
  the FTS varint format. Each doclist entry is made up of a series of two 
  or more integers, as follows:


   The docid value. The first entry in a doclist contains the literal docid
       value. The first field of each subsequent doclist entry contains the 
       difference between the new docid and the previous one (always a positive 
       number).
   Zero or more term-offset lists. A term-offset list is present for each
       column of the FTS virtual table that contains the term. A term-offset
       list consists of the following:
     
        Constant value 1. This field is omitted for any term-offset list
            associated with column 0.
        The column number (1 for the second leftmost column, etc.). This
            field is omitted for any term-offset list associated with column 0.
        A list of term-offsets, sorted from smallest to largest. Instead
            of storing the term-offset value literally, each integer stored 
            is the difference between the current term-offset and the previous 
            one (or zero if the current term-offset is the first), plus 2.
     
   Constant value 0.



  
   FTS3 Doclist Format



  
   FTS Doclist Entry Format



  For doclists for which the term appears in more than one column of the FTS
  virtual table, term-offset lists within the doclist are stored in column 
  number order. This ensures that the term-offset list associated with 
  column 0 (if any) is always first, allowing the first two fields of the
  term-offset list to be omitted in this case.


10. Limitations

10.1.  UTF-16 byte-order-mark problem 

For UTF-16 databases, when using the "simple" tokenizer, it is possible to use
malformed unicode strings to cause the 
integrity-check special command to falsely report
corruption, or for auxiliary functions to return
incorrect results. More specifically, the bug can be triggered by any of the following:


  A UTF-16 byte-order-mark (BOM) is embedded at the beginning of an SQL string
       literal value inserted into an FTS3 table. For example:

INSERT INTO fts_table(col) VALUES(char(0xfeff)||'text...');


  Malformed UTF-8 that SQLite converts to a UTF-16 byte-order-mark is
       embedded at the beginning of an SQL string literal value inserted 
       into an FTS3 table.

  A text value created by casting a blob that begins with the two
       bytes 0xFF and 0xFE, in either possible order, is inserted into an
       FTS3 table. For example:
       
INSERT INTO fts_table(col) VALUES(CAST(X'FEFF' AS TEXT));



Everything works correctly if any of the following are true:

 The database encoding is UTF-8.
 All text strings are insert using one of the
     sqlite3_bind_text() family of functions.
 Literal strings contain no byte-order-marks.
 A tokenizer is used that recognizes byte-order-marks
     as whitespace.  (The default "simple" tokenizer for
     FTS3/4 does not think that BOMs are whitespace, but
     the unicode tokenizer does.)

All of the above conditions must be false in order for problems
to occur.  And even if all of the conditiona above are false,
most things will still operator correctly.  Only the
integrity-check command and the
auxiliary functions might given
unexpected results.



  Appendix A: Search Application Tips



  FTS is primarily designed to support Boolean full-text queries - queries
  to find the set of documents that match a specified criteria. However, many 
  (most?) search applications require that results are somehow ranked in order
  of "relevance", where "relevance" is defined as the likelihood that the user
  who performed the search is interested in a specific element of the returned
  set of documents. When using a search engine to find documents on the world
  wide web, the user expects that the most useful, or "relevant", documents 
  will be returned as the first page of results, and that each subsequent page 
  contains progressively less relevant results. Exactly how a machine can 
  determine document relevance based on a users query is a complicated problem
  and the subject of much ongoing research.


  One very simple scheme might be to count the number of instances of the 
  users search terms in each result document. Those documents that contain
  many instances of the terms are considered more relevant than those with
  a small number of instances of each term. In an FTS application, the 
  number of term instances in each result could be determined by counting
  the number of integers in the return value of the offsets function.
  The following example shows a query that could be used to obtain the
  ten most relevant results for a query entered by the user:

-- This example (and all others in this section) assumes the following schema
CREATE VIRTUAL TABLE documents USING fts3(title, content);

-- Assuming the application has supplied an SQLite user function named "countintegers"
-- that returns the number of space-separated integers contained in its only argument,
-- the following query could be used to return the titles of the 10 documents that contain
-- the greatest number of instances of the users query terms. Hopefully, these 10
-- documents will be those that the users considers more or less the most "relevant".
SELECT title FROM documents 
  WHERE documents MATCH &lt;query&gt;
  ORDER BY countintegers(offsets(documents)) DESC
  LIMIT 10 OFFSET 0



  The query above could be made to run faster by using the FTS matchinfo
  function to determine the number of query term instances that appear in each
  result. The matchinfo function is much more efficient than the offsets 
  function. Furthermore, the matchinfo function provides extra information
  regarding the overall number of occurrences of each query term in the entire
  document set (not just the current row) and the number of documents in which 
  each query term appears. This may be used (for example) to attach a higher
  weight to less common terms which may increase the overall computed relevancy 
  of those results the user considers more interesting.

-- If the application supplies an SQLite user function called "rank" that
-- interprets the blob of data returned by matchinfo and returns a numeric
-- relevancy based on it, then the following SQL may be used to return the
-- titles of the 10 most relevant documents in the dataset for a users query.
SELECT title FROM documents 
  WHERE documents MATCH &lt;query&gt;
  ORDER BY rank(matchinfo(documents)) DESC
  LIMIT 10 OFFSET 0



  The SQL query in the example above uses less CPU than the first example
  in this section, but still has a non-obvious performance problem. SQLite
  satisfies this query by retrieving the value of the "title" column and
  matchinfo data from the FTS module for every row matched by the users
  query before it sorts and limits the results. Because of the way SQLite's
  virtual table interface works, retrieving the value of the "title" column
  requires loading the entire row from disk (including the "content" field,
  which may be quite large). This means that if the users query matches
  several thousand documents, many megabytes of "title" and "content" data
  may be loaded from disk into memory even though they will never be used
  for any purpose. 


  The SQL query in the following example block is one solution to this 
  problem. In SQLite, when a sub-query 
  used in a join contains a LIMIT clause, the results of the sub-query are
  calculated and stored in temporary table before the main query is executed.
  This means that SQLite will load only the docid and matchinfo data for each
  row matching the users query into memory, determine the docid values
  corresponding to the ten most relevant documents, then load only the title
  and content information for those 10 documents only. Because both the matchinfo
  and docid values are gleaned entirely from the full-text index, this results
  in dramatically less data being loaded from the database into memory.

SELECT title FROM documents JOIN ( 
    SELECT docid, rank(matchinfo(documents)) AS rank 
    FROM documents
    WHERE documents MATCH &lt;query&gt;
    ORDER BY rank DESC 
    LIMIT 10 OFFSET 0
) AS ranktable USING(docid)
ORDER BY ranktable.rank DESC



  The next block of SQL enhances the query with solutions to two other problems
  that may arise in developing search applications using FTS:


   
       The snippet function cannot be used with the above query. Because
       the outer query does not include a "WHERE ... MATCH" clause, the snippet 
       function may not be used with it. One solution is to duplicate the WHERE
       clause used by the sub-query in the outer query. The overhead associated
       with this is usually negligible.
   
       The relevancy of a document may depend on something other than just
       the data available in the return value of matchinfo. For example
       each document in the database may be assigned a static weight based
       on factors unrelated to its content (origin, author, age, number
       of references etc.). These values can be stored by the application
       in a separate table that can be joined against the documents table
       in the sub-query so that the rank function may access them.



  This version of the query is very similar to that used by the 
  sqlite.org documentation search 
  application.

-- This table stores the static weight assigned to each document in FTS table
-- "documents". For each row in the documents table there is a corresponding row
-- with the same docid value in this table.
CREATE TABLE documents_data(docid INTEGER PRIMARY KEY, weight);

-- This query is similar to the one in the block above, except that:
--
--   1. It returns a "snippet" of text along with the document title for display. So
--      that the snippet function may be used, the "WHERE ... MATCH ..." clause from
--      the sub-query is duplicated in the outer query.
--
--   2. The sub-query joins the documents table with the document_data table, so that
--      implementation of the rank function has access to the static weight assigned
--      to each document.
SELECT title, snippet(documents) FROM documents JOIN ( 
    SELECT docid, rank(matchinfo(documents), documents_data.weight) AS rank
    FROM documents JOIN documents_data USING(docid)
    WHERE documents MATCH &lt;query&gt;
    ORDER BY rank DESC 
    LIMIT 10 OFFSET 0
) AS ranktable USING(docid)
WHERE documents MATCH &lt;query&gt;
ORDER BY ranktable.rank DESC



  All the example queries above return the ten most relevant query results.
  By modifying the values used with the OFFSET and LIMIT clauses, a query 
  to return (say) the next ten most relevant results is easy to construct. 
  This may be used to obtain the data required for a search applications second
  and subsequent pages of results.


  The next block contains an example rank function that uses matchinfo data 
  implemented in C. Instead of a single weight, it allows a weight to be 
  externally assigned to each column of each document. It may be registered
  with SQLite like any other user function using sqlite3_create_function.

 Security Warning: Because it is just an ordinary SQL function, 
  rank() may be invoked as part of any SQL query in any context. This means
  that the first argument passed may not be a valid matchinfo blob.
  Implementors should take care to handle this case without causing buffer
  overruns or other potential security problems.

/*
** SQLite user defined function to use with matchinfo() to calculate the
** relevancy of an FTS match. The value returned is the relevancy score
** (a real value greater than or equal to zero). A larger value indicates 
** a more relevant document.
**
** The overall relevancy returned is the sum of the relevancies of each 
** column value in the FTS table. The relevancy of a column value is the
** sum of the following for each reportable phrase in the FTS query:
**
**   (&lt;hit count&gt; / &lt;global hit count&gt;) * &lt;column weight&gt;
**
** where &lt;hit count&gt; is the number of instances of the phrase in the
** column value of the current row and &lt;global hit count&gt; is the number
** of instances of the phrase in the same column of all rows in the FTS
** table. The &lt;column weight&gt; is a weighting factor assigned to each
** column by the caller (see below).
**
** The first argument to this function must be the return value of the FTS 
** matchinfo() function. Following this must be one argument for each column 
** of the FTS table containing a numeric weight factor for the corresponding 
** column. Example:
**
**     CREATE VIRTUAL TABLE documents USING fts3(title, content)
**
** The following query returns the docids of documents that match the full-text
** query &lt;query&gt; sorted from most to least relevant. When calculating
** relevance, query term instances in the 'title' column are given twice the
** weighting of those in the 'content' column.
**
**     SELECT docid FROM documents 
**     WHERE documents MATCH &lt;query&gt; 
**     ORDER BY rank(matchinfo(documents), 1.0, 0.5) DESC
*/
static void rankfunc(sqlite3_context *pCtx, int nVal, sqlite3_value **apVal){
  int *aMatchinfo;                /* Return value of matchinfo() */
  int nMatchinfo;                 /* Number of elements in aMatchinfo&#x5B;&#x5D; */
  int nCol = 0;                   /* Number of columns in the table */
  int nPhrase = 0;                /* Number of phrases in the query */
  int iPhrase;                    /* Current phrase */
  double score = 0.0;             /* Value to return */

  assert( sizeof(int)==4 );

  /* Check that the number of arguments passed to this function is correct.
  ** If not, jump to wrong_number_args. Set aMatchinfo to point to the array
  ** of unsigned integer values returned by FTS function matchinfo. Set
  ** nPhrase to contain the number of reportable phrases in the users full-text
  ** query, and nCol to the number of columns in the table. Then check that the
  ** size of the matchinfo blob is as expected. Return an error if it is not.
  */
  if( nVal&lt;1 ) goto wrong_number_args;
  aMatchinfo = (unsigned int *)sqlite3_value_blob(apVal&#x5B;0&#x5D;);
  nMatchinfo = sqlite3_value_bytes(apVal&#x5B;0&#x5D;) / sizeof(int);
  if( nMatchinfo&gt;=2 ){
    nPhrase = aMatchinfo&#x5B;0&#x5D;;
    nCol = aMatchinfo&#x5B;1&#x5D;;
  }
  if( nMatchinfo!=(2+3*nCol*nPhrase) ){
    sqlite3_result_error(pCtx,
      "invalid matchinfo blob passed to function rank()", -1);
    return;
  }
  if( nVal!=(1+nCol) ) goto wrong_number_args;

  /* Iterate through each phrase in the users query. */
  for(iPhrase=0; iPhrase&lt;nPhrase; iPhrase++){
    int iCol;                     /* Current column */

    /* Now iterate through each column in the users query. For each column,
    ** increment the relevancy score by:
    **
    **   (&lt;hit count&gt; / &lt;global hit count&gt;) * &lt;column weight&gt;
    **
    ** aPhraseinfo&#x5B;&#x5D; points to the start of the data for phrase iPhrase. So
    ** the hit count and global hit counts for each column are found in 
    ** aPhraseinfo&#x5B;iCol*3&#x5D; and aPhraseinfo&#x5B;iCol*3+1&#x5D;, respectively.
    */
    int *aPhraseinfo = &aMatchinfo&#x5B;2 + iPhrase*nCol*3&#x5D;;
    for(iCol=0; iCol&lt;nCol; iCol++){
      int nHitCount = aPhraseinfo&#x5B;3*iCol&#x5D;;
      int nGlobalHitCount = aPhraseinfo&#x5B;3*iCol+1&#x5D;;
      double weight = sqlite3_value_double(apVal&#x5B;iCol+1&#x5D;);
      if( nHitCount>0 ){
        score += ((double)nHitCount / (double)nGlobalHitCount) * weight;
      }
    }
  }

  sqlite3_result_double(pCtx, score);
  return;

  /* Jump here if the wrong number of arguments are passed to this function */
wrong_number_args:
  sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}

This page last modified on  2022-08-10 18:11:15 UTC


