# page_content

c2: Recent SQLite News
c5: news.html
c4:
Recent News

2024-08-13 - Version 3.46.1
Version 3.46.1 is a patch release that fixes several minor
problems in version 3.46.0.  There are minimal changes from
the previous release.
2024-05-23 - Version 3.46.0
Version 3.46.0 is an enhancement release.  Enhancements include:

 Improvements to the PRAGMA optimize command
 Enhancements to the date and time functions
 The ability to put "_" between digits in numeric literals for readability
 Added the json_pretty() function
 Improvements to the query planner, especially a new faster implementation
     for VALUES clauses with many terms.


None of these enhancements are critical.  Update at your convenience.
2024-04-15 - Version 3.45.3
Version 3.45.3 is a patch release that fixes a few obscure problems,
including:

 The "old.*" values in an UPDATE trigger might be incorrect
     if the trigger fires in response to an UPSERT.
 The sum() function might return NULL in some cases where Infinity
     would be a better answer.

None of the problems are emergencies.  Upgrade at your convenience.
2024-03-12 - Version 3.45.2
Version 3.45.2 is a patch against SQLite versions 3.45.0 and 3.45.1.

The primary reason for this patch is to fix two bugs three-year-old
bugs identified by Forum posts
919c6579c8
and 440f2a2f17.
These problems could results in incorrect query results or corrupt
indexes.  See the associated forum threads for details.

Other trifling fixes are also included in the patch.
2024-01-30 - Version 3.45.1
Version 3.45.1 is a patch against SQLite version 3.45.0.

The main focus of this patch release is to restore certain
undocumented legacy behavior in the JSON SQL functions that
the developers were unaware of but which some applications
had come to depend on.  This undocumented behavior was "fixed"
in the 3.45.0, resulting in breakage for applications that were
using it.  So it has now been restored and documented.

Other obscure issues that have come up since the 3.45.0 release,
some related to the release itself and some going back years,
are also fixed.
2024-01-15 - Version 3.45.0
Version 3.45.0 is an enhancement release of SQLite.

The most impactful change is likely enhancements to
the JSON SQL functions such that they are now able to
store their internal JSON parse trees in the database
as BLOB values.  This can significantly increase the
performance of applications that process large JSON
strings as it omits the need to translate JSON text
to and from the internal binary format used by SQLite.

A second noteworthy change is that any
application-defined SQL functions that make use of
the sqlite3_result_subtype() interface must now include
the SQLITE_RESULT_SUBTYPE attribute when the function
is registered using sqlite3_create_function() or similar.
Failure to include the SQLITE_RESULT_SUBTYPE attribute on
functions that use sqlite3_result_subtype() might result
in incorrect answers.

See the change log
for additional enhancements that are part of the 3.45.0 release.
2023-11-24 - Version 3.44.2
The CLI fix in version 3.44.1 introduced a new bug, which is
fixed by patch release 3.44.2.  Version 3.44.2 also fixes an
FTS5 problem that was found by a fuzzer just minutes after the
3.44.1 release.
2023-11-22 - Version 3.44.1
Version 3.44.1 is a patch release that fixes various obscure
bugs.  There is no need to upgrade, unless you are having
problems with a prior release.
2023-11-01 - Version 3.44.0
Version 3.44.0 is new enhancement release of SQLite.

It has only been 69 days since the previous major
release (3.43.0).  The original plan was for version 3.44.0
to occur at a spacing of approximately 120 days from the
prior release.  However, the code accumulated so many important
enhancements that is seemed better to accelerate the release
of 3.44.0, thus getting those enhancements into circulation.
This means that some enhancements that where originally
planned to be in 3.44.0 have been deferred until subsequent
releases.
2023-10-10 - Version 3.43.2
Version 3.43.2 is a patch release that fixes a few small inaccuracies in
version 3.43.0 and 3.43.1.
2023-09-11 - Version 3.43.1
Version 3.43.1 is a patch release that fixes a few small inaccuracies that were
discovered in the the 3.43.0 and/or 3.42.0 releases after the 3.43.0 release was
published.
2023-08-24 - Version 3.43.0
Version 3.43.0 is a routine enhancement release of SQLite.
Key enhancements in this release include added support for
Contentless-Delete FTS5 Indexes, 
and performance improvements in JSON processing.
See the change log for
details.
2023-05-16 - Version 3.42.0
Version 3.42.0 is a routine enhancement release of SQLite.
Key enhancements in this release are added support for
JSON5 and the FTS5 secure-delete command.  See the
change log for a
summary of all enhancements in this release.
2023-03-22 - Version 3.41.2
Version 3.41.2 is a patch release that fixes multiple
fuzzer-found problems in prior releases.  The worst problems
include reads (not writes) past the end of a buffer.  Upgrading
is recommended.
2023-03-10 - Version 3.41.1
Version 3.41.1 is a patch release that fixes various obscure
problems found in 3.41.0 and reported by users.  Upgrading
is optional.
2023-02-21 - Version 3.41.0
Version 3.41.0 is a routine enhancement release.
2022-12-28 - Version 3.40.1
Version 3.40.1 is a patch release that fixes some obscure problems
in version 3.40.0.  The problems fixed have no impact on most applications.
Upgrading is only necessary if you encounter problems.

The two most important fixes are these:


Fix the safe command-line option on the CLI so that it
correctly disallows functions with side-effects.  This is a bug
in the CLI &mdash; not a bug in the
SQLite library &mdash; and it only affects the --safe command-line
option, making that option less than fully "safe".  As the number
of systems that use the --safe command-line option in the CLI is
approximately zero, this is not considered an important bug.  However,
a third-party wrote a CVE against it which caused considerable angst
among maintainers, so it seems good to get the fix into circulation
sooner rather than wait on the next major release.

The optional memsys5 memory allocator picked up a bug that might
put it into an infinite loop for very large (500MiB) allocations.
Almost all systems use their native memory allocator, not memsys5.
Memsys5 is only used if SQLite is compiled using SQLITE_ENABLE_MEMSYS5
and then initialized using sqlite3_config(SQLITE_CONFIG_HEAP).
Very, very few systems do this, and so the problem is not considered
to be serious.


See the branch-3.40 timeline
for a complete listing of changes that have gone into this patch release.
2022-11-16 - Version 3.40.0
Version 3.40.0 is a new feature release of SQLite.  See the
change log for details.
Key enhancements in this release include:


Official support for compiling 
SQLite to WASM
and running it in a web browser.

New and improved recovery extension for extracting
data from corrupted database files.


This release also includes performance enhancements and
improvements to the query planner.

A psychological milestone:  The performance benchmark that
the SQLite developers have used for many years has for
the first time dropped below 1 billion CPU cycles
(measured using cachegrind) when run in WAL mode.
This is less than half the number of CPU cycles used as
recently as 8 years ago.  (The graph below shows SQLite using
a rollback journal which uses fewer CPU cycles at the expense
of extra I/O.  The benchmark passed through the 1 billion cycle
barrier for rollback journals for the previous release.)



2022-09-29 - Version 3.39.4
Version 3.39.4 is a security release that addresses a single
long-standing problem in the FTS3 extension.  An attacker
who is able to execute arbitrary SQL statements and who can create
a corrupt database that is 2GB or larger in size might be able to trick
FTS3 into overflowing an integer used to size a memory allocation,
causing the allocation to be too small and ultimately resulting in
a buffer overrun.  The release also includes fixes for other
obscure weaknesses, as described in the release notes.
2022-09-05 - Version 3.39.3
Version 3.39.3 is a patch release that fixes a few obscure problems
in the 3.39.0 release.  See the release notes for details.
2022-07-21 - Version 3.39.2
Version 3.39.2 is a security release that addresses multiple long-standing
issues in SQLite.  The most severe problem is identified by CVE-2022-35737.
That issue is associated with the auxiliary C-language APIs
of SQLite and cannot be reached using SQL or database inputs, and hence is
unlikely to affect your application.  Nevertheless, upgrading is recommended.
The other issues are comparatively minor.
This release also fixes a performance regression that appeared in version
3.39.0 that might affect some multi-way joins that use LEFT JOIN.
2022-07-13 - Version 3.39.1
Version 3.39.1 is a patch release that fixes a few minor problem in version
3.39.0.  Upgrading is optional.
2022-06-25 - Version 3.39.0
Version 3.39.0 is regular maintenance release of SQLite.  The key enhancement
in this release is added support for RIGHT and FULL JOIN.  There are other
language and performance enhancements as well &mdash; see the
release notes for details.
2022-05-06 - Version 3.38.5
The 3.38.4 patch release included a minor change to the CLI source code
that did not work.  The release manager only ran a subset of the normal
release tests, and hence did not catch the problem. As a result, the CLI
will segfault when using columnar output modes in version 3.38.4.  This
blunder did not affect the core SQLite library.  It only affected the CLI.

Take-away lesson:  Always run all of your tests prior to
a release - even a trival patch release.  Always.

The 3.38.5 patch release fixes the 3.38.4 blunder.
2022-05-04 - Version 3.38.4
Another user-discovered problem in the new Bloom filter optimization
is fixed in this patch release.  Without the fix, it is possible for
a multi-way join that uses a Bloom filters for two or more tables in
the join to enter an infinite loop if the key constraint on one of those
tables contains a NULL value.
2022-04-27 - Version 3.38.3
Version 3.38.3 fixes a bug in the automatic-index and Bloom filter
construction logic that might cause SQLite to be overly aggressive
in the use of ON clause constraints, resulting in an incorrect
automatic-index or Bloom filter that excludes some valid rows from
output.  The bug was introduced in version 3.38.0.  Other minor
changes were tossed in to complete the patch.
2022-03-26 - Version 3.38.2
Version 3.38.2 fixes another bug in the new Bloom filter
optimization that might cause incorrect answers for a
LEFT JOIN that has an IS NULL constraint on the right-hand
table.
2022-03-12 - Version 3.38.1
Version 3.38.1 fixes a pair of bugs in the Bloom filter
optimization that was introduced in version 3.38.0.  These
bugs might cause incorrect answers for some obscure queries.
Various other minor problems and documentation typos were
fixed at the same time.
2022-02-22 - Version 3.38.0
Version 3.38.0 is a routine maintenance release of
SQLite.  There are various minor enhancements and
about a 0.5% reduction in the number of CPU cycles
used. See the
release notes for
more detail.
2022-01-06 - Version 3.37.2
Version 3.37.2 fixes a
database corruption bug.  You are
encouraged to upgrade, especially if you are using SAVEPOINT.

The problem first appeared in version 3.35.0 (2021-03-12) and
affects all subsequent releases through 3.37.1.
If temporary files are store in memory (which is not the default
behavior, but is sometimes selected by applications using either
-DSQLITE_TEMP_STORE or PRAGMA temp_store) and
if a SAVEPOINT is rolled back and then subsequent changes
within the same transaction are committed, the database file might
(with low but non-zero probability) go corrupt.
2021-12-30 - Version 3.37.1
Version 3.37.1 fixes a bug in the UPSERT logic, introduced by
the UPSERT enhancements of version 3.35.0, that can cause
incorrect byte-code to be generated in some cases, resulting
in an infinite loop in the byte code, or a NULL-pointer dereference.
This patch release also fixes some other minor problems with
assert() statements and in the CLI.
2021-11-27 - Version 3.37.0
Version 3.37.0 is a routine maintenance release of SQLite.
The biggest new feature in this release is support for
STRICT tables.  Other enhancements are described in
the release notes.
2021-06-18 - Version 3.36.0
Version 3.36.0 is a routine maintenance release of SQLite.
There are no new major features, only incremental improvements
to existing features and small performance improvements.
2021-04-19 - Patch release 3.35.5
The new ALTER TABLE DROP COLUMN capability that was added
in the 3.35.0 release contained a bug that might cause the
table content to go corrupt when the table was rewritten
to remove the dropped column.  Fixed by this patch.
2021-04-02 - Patch release 3.35.4
Version 3.35.4 is yet another patch release to fix
obscure problems in features associated with the 3.35.0.
2021-03-26 - Patch release 3.35.3
Version 3.35.3 contains patches for a handful of minor
problems discovered in prior releases.
2021-03-17 - Patch release 3.35.2
Version 3.35.2 is a small patch release to fix some minor problems
that were discovered shortly after the 3.35.1 release.
2021-03-15 - Patch release 3.35.1
A user discovered an issue with the new DROP COLUMN capability
in version 3.35.0, and so version 3.35.1 was created to fix it.
No need to upgrade if you are not using DROP COLUMN.
2021-03-12 - Release 3.35.0
SQLite version 3.35.0 is a routine maintenance release.  This
release adds a number of new language features, including
support for ALTER TABLE DROP COLUMN, built-in math functions,
generalized UPSERT, and the MATERIALIZED hint on common table
expressions.  There are also query planner optimizations and
incremental CLI improvements.
2020-01-20 - Release 3.34.1
SQLite version 3.34.1 is a patch releases that fixes a possible
use-after-free bug that can be provoked by malicious SQL.  Other
minor issues in extensions and documentation are also fixed.
2020-12-01 - Release 3.34.0
SQLite version 3.34.0 is a routine maintenance release.  This
release adds incremental improvements to performance and features,
including enhancements to the query planner, multiple recursive
SELECTS in recursive common table expressions, and better error messages
from CHECK constraint failures.  See the change log for details.
2020-08-14 - Release 3.33.0
SQLite version 3.33.0 is a routine maintenance release.  This
release features added support for "UPDATE FROM" following the
PostgreSQL syntax, and a doubling of the maximum database size
to 281 TB, as well as many other improvements.  See the change
log for details.
2020-06-18 - Release 3.32.3
The 3.32.3 release is a patch release that contains fixes for
various issues discovered by fuzzers.  None of the issues fixed
are likely to be encountered by applications that use SQLite in
ordinary ways, though upgrading never hurts.

Map of all changes since the 3.32.0 release:
https://www.sqlite.org/src/timeline?p=version-3.32.3&bt=version-3.32.0
2020-06-04 - Release 3.32.2
The 3.32.2 release is a one-line change relative to 3.32.1
that fixes a long-standing bug in the COMMIT command.  Since
version 3.17.0, if you were to retry a COMMIT command over
and over after it returns SQLITE_BUSY, it might eventually
report success, even though it was still blocked.  This patch
fixes the problem.
2020-05-25 - Release 3.32.1
Grey-hats published
information about two SQLite bugs approximately 24 hours after
the release of version 3.32.0.  These bugs enable maliciously
crafted SQL to crash the process that is running SQLite.  Both
bugs are long-standing problems that affect releases prior to
3.32.0. The 3.32.1 release fixes both problems.
2020-05-22 - Release 3.32.0
Version 3.32.0 is an ordinary maintenance release of SQLite.
This release features the ability to run an
approximate ANALYZE to gather database statistics for
use by the query planner, without having to scan every row
of every index.
See the change log for additional enhancements
and improvements.
2020-01-27 - Release 3.31.1
Applications that use SQLite should only interface with SQLite
through the officially published APIs.  Applications should not
depend upon or use the internal data structures of SQLite as those
structures might change from one release to another.  However, there
is a popular application that does depend on the details of the
internal layout of data in an internal SQLite data structure, and
those details changed in version 3.31.0, breaking the application.
This is, technically, a bug in the application, not in SQLite.
But it is within the power of SQLite to fix it, by reverting the
internal data structure change, and so that is what we have done
for the 3.31.1 release.
2020-01-22 - Release 3.31.0
Version 3.31.0 is an ordinary maintenance release of SQLite.
This release features the ability to define
generated columns for tables as well as many other enhancements.
See the change log for additional information.
2019-10-11 - Release 3.30.1
Version 3.30.1 is a bug-fix release that addresses a problem
that can occur when an aggregate function in a nested query
makes use of the new FILTER clause capability.  Some addition
patches for various obscure issues are also included, for
completeness.
2019-10-04 - Release 3.30.0
Version 3.30.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  This release adds support fo the NULLS FIRST
and NULLS LAST clauses on ORDER BY statements and the 
addition of FILTER clauses on all aggregate functions.
See the change log for details.
2019-07-10 - Release 3.29.0
Version 3.29.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

Beginning with this release, the
double-quoted string literal misfeature is deprecated.
The misfeature is still enabled by default, for legacy
compatibility, however developers are encouraged to
disable it at compile-time using the
-DSQLITE_DQS=0 option, or at run-time using
the SQLITE_DBCONFIG_DQS_DML and SQLITE_DBCONFIG_DQS_DDL
actions to the sqlite3_db_config() interface.  This is
especially true for double-quoted string literals in
CREATE TABLE and CREATE INDEX statements, as those elements
can cause unexpected problems following an ALTER TABLE.
See ticket 9b78184be266fd70
for an example.
2019-04-16 - Release 3.28.0
Version 3.28.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

This release fixes an obscure security issue. Applications
using older versions of SQLite may be vulnerable if

 SQLite is compiled with certain optional extensions
     enabled, and
 the attacker is able to inject arbitrary SQL.

We are not aware of any applications that are vulnerable to
this problem. On the other hand, we do not know about
every application that uses SQLite.  
If your application allows unauthenticated users on the internet
(and hence potential attackers) to run arbitrary SQL and if
you build SQLite with any of the optional extensions enabled,
then you should take this upgrade at your earliest opportunity.

For further information about improving SQLite's robustness
in internet-facing applications, see the our
security recommendations.
2019-02-25 - Release 3.27.2
Version 3.27.2 is a patch release that
fixes a two bugs and various documentation
errors found in the version 3.27.1.  The
changes from version 3.27.1 and 3.27.0 are
minimal.
2019-02-08 - Release 3.27.1
After release 3.27.0 was tagged, but before the build could
be completed and uploaded, a 
long-standing bug
in the query optimizer was reported via System.Data.SQLite.
Since people will be upgrading anyhow, we decided 
publish the fix for this newly discovered problems right
away.  Hence, 3.27.1 was released less than 24 hours after
3.27.0.

It would have been better if the query optimizer bug had come to
our attention one day earlier, so that we could have incorporated
a fix into 3.27.0, but sometimes that's the way things go.
2019-02-07 - Release 3.27.0
SQLite version 3.27.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-12-01 - Release 3.26.0
SQLite version 3.26.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-11-05 - Release 3.25.3
SQLite version 3.25.3 is a third patch against 3.25.0 that fixes various
problems that have come to light and which seem serious enough to 
justify a patch.
2018-09-25 - Release 3.25.2
SQLite version 3.25.2 is another patch against 3.25.0 that fixes still
more problems associated with the new window function feature and the
ALTER TABLE enhancements.  Of particular note is the new
PRAGMA legacy_alter_table=ON command, which causes the ALTER TABLE RENAME
command to behave in the same goofy way that it did before the enhancements
found in version 3.25.0 &rarr; references to renamed tables that are inside
the bodies of triggers and views are not updated.  The legacy behavior is
arguably a bug, but some programs depend on the older buggy behavior.  The
3.25.2 release also contains a fix to window function processing for
VIEWs.  There also a slew of other minor fixes that affect obscure
compile-time options.  See the
Fossil Timeline for
details.
2018-09-18 - Release 3.25.1
SQLite version 3.25.1 is a patch against version 3.25.0 that contains
two one-line fixes for bug that were introduced in version 3.25.0.
See the change log for details.  Upgrading from 3.25.0 is recommended.
2018-09-15 - Release 3.25.0
SQLite version 3.25.0 is a regularly scheduled maintenance release.
Two big enhancements in this release:


Support for window functions was added, using the PostgreSQL documentation
as the baseline.

The ALTER TABLE command was enhanced to support renaming of columns, and
so that column and table renames are propagated into trigger bodies and views.

In addition, there are various performance enhancements and minor fixes.
One bug of note is 
ticket 9936b2fa443fec
which describes a hard-to-reach condition where the ORDER BY LIMIT
optimization might cause an infinite loop during query evaluation.  
This ticket raised a lot of
concern on 
HackerNews and
Reddit,
probably due to my choice of the ticket
title.  "Infinite Loop" sounds scary.  But I argue that the bug isn't really
all that bad in that it is very difficult to reach, will show up during
testing (rather than magically appearing after a product is
deployed), does not cause any data loss, and does not return an
incorrect result.  It was an important error, but not nearly as dire
as many people interpreted it to be.  And, in any event, it is fixed now.
2018-06-04 - Release 3.24.0
SQLite version 3.24.0 is a regularly scheduled maintenance release.
Highlights of this release include support for PostgreSQL-style
UPSERT and improved performance, especially for ORDER BY LIMIT queries.
2018-04-10 - Release 3.23.1
The version 3.23.1 release fixes a bug in the new
LEFT JOIN strength reduction optimization added to version 3.23.0.
A few other minor and obscure fixes were also inserted, as well as
a small performance optimization.  Code changes relative to
version 3.23.0 are minimal.
2018-04-02 - Release 3.23.0
The version 3.23.0 release is a regularly scheduled maintenance release.
See the change log for a list of enhancements and bug
fixes.
2018-01-22 - Release 3.22.0
The version 3.22.0 release is a regularly scheduled maintenance release.
There are many minor, though interesting, enhancements in this release.
See the change log for details.
2017-10-24 - Release 3.21.0
The version 3.21.0 release is a regularly scheduled maintenance release.
There are lots of enhancements in this release.
See the change log for details.
2017-08-24 - Release 3.20.1
The version 3.20.1 patch release changes two lines of code in
the sqlite3_result_pointer() interface in order to fix a rare
memory leak.  There are no other changes relative to version 3.20.0.
2017-08-01 - Release 3.20.0
SQLite version 3.20.0 is a regularly scheduled maintenance release
of SQLite.

This release contains many minor enhancements, including:

 Several new extensions
 Enhancements to the "sqlite3.exe" command-line shell
 Query planner enhancements
 Miscellaneous code optimizations for improved performance
 Fixes for some obscure bugs


See the release notes for more information.
2017-06-17 - Release 3.18.2
SQLite version 3.18.2 is another backport of a bug fix found
in SQLite version 3.19.0, specifically the fix for
ticket 61fe9745.  Changes
against version 3.18.0 are minimal.
2017-06-16 - Release 3.18.1
SQLite version 3.18.1 is a bug-fix release against version 3.18.0
that fixes the auto_vacuum corruption bug described in ticket
fda22108.  This release was
created for users who need that bug fix but do not yet want to upgrade 
to version 3.19.3.  
2017-06-08 - Release 3.19.3
Version 3.19.3 is an emergency patch release to fix a 
bug in 
auto_vacuum logic that can lead to database corruption.
The bug was introduced in version 3.16.0 
(2017-01-02).  Though the bug is obscure and rarely
encountered, upgrading is recommended for all users, and
especially for users who turn on auto_vacuum.
2017-05-25 - Release 3.19.2
Still more problems have been found in the LEFT JOIN
flattening optimization
that was added in the 3.19.0 release.  This patch release fixes all known
issues with that optimization and adds new test cases.  Hopefully this
will be the last patch.
2017-05-24 - Release 3.19.1
One of the new query planner optimizations in the 3.19.0 release contained
bugs.  The 3.19.1 patch release fixes them.

Beginning with 3.19.0, subqueries and views on the right-hand side of
a LEFT JOIN operator could sometimes be
flattened into the
main query.  The new optimization worked well for all of the test cases
that the developers devised, and for millions of legacy test cases, but
once 3.19.0 was released, users found some other cases where the optimization
failed.  Ticket
cad1ab4cb7b0fc344 contains
examples.

These problems exist only in 3.19.0.  Users of SQLite 3.19.0 should
upgrade, but users of all prior versions of SQLite are safe.
2017-05-22 - Release 3.19.0
SQLite version 3.19.0 is a regularly scheduled maintenance release.

The emphasis on this release is improvements to the query planner.
There are also some obscure bug fixes.  There is no reason to upgrade
unless you are having problems with a prior release.
2017-03-30 - Release 3.18.0
SQLite version 3.18.0 is a regularly scheduled maintenance release.

This release features an initial implementation the 
"PRAGMA optimize" command.  This command can now be used to cause
ANALYZE to be run on an as-needed basis.  Applications should invoke
"PRAGMA optimize" just before closing the database connection.
The "PRAGMA optimize" statement will likely be enhanced to do other
kinds of automated database maintenance in future releases.

The Fossil version control system that is
used to manage the SQLite project has been upgraded to use SHA3-256 hashes
instead of SHA1.  Therefore, the version identifications for SQLite now
show a 64-hex-digit SHA3-256 hash rather than the 40-hex-digit SHA1 hash.

See the change log for other enhancements and optimizations
in this release.
2017-02-13 - Release 3.17.0
SQLite version 3.17.0 is a regularly scheduled maintenance release.

Most of the changes in this release are performance optimizations.
Optimizations to the R-Tree extension are especially noticeable.

In this release, the default size of the 
lookaside buffer allocated for each database connection
is increased from 64,000 to 120,000 bytes.  This provides improved
performance on many common workloads in exchange for a small increase
in memory usage.
Applications that value a small memory footprint over raw speed
can change the lookaside buffer size back to its old value (or to zero)
using the SQLITE_DEFAULT_LOOKASIDE compile-time option, or the
sqlite3_config(SQLITE_CONFIG_LOOKASIDE)
start-time setting, or the
sqlite3_db_config(SQLITE_DBCONFIG_LOOKASIDE)
run-time setting.
2017-01-06 - Release 3.16.2
One of the performance optimizations added in 3.16.0 caused triggers
and foreign keys to malfunction for the REPLACE statement on
WITHOUT ROWID tables that lack secondary indexes.  This patch
release fixes the problem.  See ticket 
30027b613b4 for details.
2017-01-03 - Release 3.16.1
SQLite version 3.16.1 fixes a bug in the row-value logic for UPDATE
statements inside of triggers.  The bug has been there since row-values
were added by release 3.15.0, but was not discovered until just a few
minutes after the 3.16.0 release was published, and so it was not fixed
by 3.16.0.  This patch release is version 3.16.0 with the row-value bug fix.
2017-01-02 - Release 3.16.0
SQLite version 3.16.0 is a regularly schedule maintenance release.


This release includes many microoptimizations that collectively reduce
the CPU cycle count by about 9%,
add there have been important enhancements to the command-line shell.

Support for PRAGMA functions is added,
so that many pragma statements can be used as part of a larger SQL query.
This is considered an experimental feature.
We do not anticipate any changes to the PRAGMA function interface, but
will keep continue to call this interface "experimental" for a few release
cycles in case unforeseen issues arise.

See the change log for other enhancements.
2016-11-28 - Release 3.15.2
SQLite version 3.15.2 is a bug-fix patch release that fixes several minor
issues in the 3.15.0 and 3.15.1 releases.
2016-11-04 - Release 3.15.1
SQLite version 3.15.1 is a bug-fix patch release that fixes some minor
issues in the 3.15.0 release.
2016-10-14 - Release 3.15.0
SQLite version 3.15.0 is a regularly scheduled maintenance release.
The key feature in this release is the added support for
row values.  There are also other enhancements and
fixes for a number of obscure bugs.

The 3.15.0 release uses about 7% fewer CPU cycles than 3.14.2.
Most of the improvement in this release
is in the SQL parser, query planner, and
byte-code generator (the front-end) corresponding
to the sqlite3_prepare_v2() interface.  Overall,
version 3.15.0 uses about half as much CPU time as
version 3.8.1 (2013-10-17).  These
performance measurements are made using the "speedtest1.c"
workload on x64 compiled with gcc and -Os. Performance
improvements may vary with different platforms and
workloads.
2016-09-12 - Release 3.14.2
SQLite version 3.14.2 fixes several obscure bugs and adds 
improved support for building SQLite using the STDCALL calling
convention on 32-bit windows systems.  Upgrading from versions
3.14 and 3.14.1 is optional.
2016-08-11 - Release 3.14.1
SQLite version 3.14.1 adds a small patch to improve the performance
of the pcache1TruncateUnsafe() routine for cases when the only a few
pages on the end of the cache are being removed.  This causes COMMITs
to run faster when there is a very large page cache.  Upgrading from
version 3.14 is optional.
2016-08-08 - Release 3.14
SQLite version 3.14 (the "&#960;" release)
is a regularly scheduled maintenance
release containing performance enhancements, new features, and fixes for
obscure bugs.
2016-05-18 - Release 3.13.0
SQLite version 3.13.0 is a regularly schedule maintenance release containing
performance enhancements and fixes for obscure bugs.
2016-04-18 - Release 3.12.2
Yikes!  The 3.12.0 and 3.12.1 releases contain a backwards compatibility bug!
   Tables that declare a column with type "INTEGER" PRIMARY KEY
   (where the datatype name INTEGER is quoted) generate an incompatible
   database file.  The mistake came about because the developers have never
   thought to put a typename in quotes before, and so there was no documentation 
   of that capability nor any tests.  (There are tests now, though, of course.)
   Instances of quoting the datatype name are probably infrequent in the wild,
   so we do not expect the impact of this bug to be too severe.
   Upgrading is still strongly recommended.
Fixes for three other minor issues were included in this patch release.
   The other issues would have normally been deferred until the next scheduled
   release, but since a patch release is being issued anyhow, they might as
   well be included.
2016-04-08 - Release 3.12.1
SQLite version 3.12.1 is an emergency patch release to address a 
   crash bug that snuck
   into version 3.12.0.  Upgrading from version 3.12.0 is highly
   recommended.
Another minor problem involving datatypes on view columns, and
   a query planner deficiency are fixed at the same time.  These two
   issues did not justify a new release on their own, but since a release
   is being issued to deal with the crash bug, we included these other
   fixes for good measure.
2016-03-29 - Release 3.12.0
SQLite version 3.12.0 is a regularly scheduled maintenance release.
   A notable change in this release is an
   increase in the default page size for newly created database files.
   There are also various performance improvements.
   See the change log for details.
2016-03-03 - Release 3.11.1
SQLite version 3.11.1 is a patch release that fixes problems in the
   new FTS5 extension and increases a default setting in the spellfix1
   extension, and implements enhancements to some of the Windows makefiles.
   The SQLite core is unchanged from 3.11.0. Upgrading is optional.
2016-02-15 - Release 3.11.0
SQLite version 3.11.0 is a regularly scheduled maintenance release.
2016-01-20 - Release 3.10.2
Yikes!  An optimization attempt gone bad resulted in a 
bug in the LIKE operator
which is fixed by this patch release.
Three other minor but low-risk fixes are also included in the patch.
2016-01-14 - Release 3.10.1
SQLite version 3.10.1 is a bug-fix release primarily targeting the
fix for the query planner bug
cb3aa0641d9a4 discovered by
Mapscape.  Also included is a minor API enhancement requested by
the Firefox developers at Mozilla.  The differences from version 
3.10.0 are minimal.
2016-01-06 - Release 3.10.0
SQLite version 3.10.0 is a regularly scheduled maintenance release.


Old news...
This page last modified on  2024-08-13 09:24:49 UTC


c2: SQLite Unlock-Notify API
c5: unlock_notify.html
c4:
Using the sqlite3_unlock_notify() API


/* This example uses the pthreads API */
#include &lt;pthread.h&gt;

/*
** A pointer to an instance of this structure is passed as the user-context
** pointer when registering for an unlock-notify callback.
*/
typedef struct UnlockNotification UnlockNotification;
struct UnlockNotification {
  int fired;                         /* True after unlock event has occurred */
  pthread_cond_t cond;               /* Condition variable to wait on */
  pthread_mutex_t mutex;             /* Mutex to protect structure */
};

/*
** This function is an unlock-notify callback registered with SQLite.
*/
static void unlock_notify_cb(void **apArg, int nArg){
  int i;
  for(i=0; i&lt;nArg; i++){
    UnlockNotification *p = (UnlockNotification *)apArg[i];
    pthread_mutex_lock(&p-&gt;mutex);
    p-&gt;fired = 1;
    pthread_cond_signal(&p-&gt;cond);
    pthread_mutex_unlock(&p-&gt;mutex);
  }
}

/*
** This function assumes that an SQLite API call (either sqlite3_prepare_v2() 
** or sqlite3_step()) has just returned SQLITE_LOCKED. The argument is the
** associated database connection.
**
** This function calls sqlite3_unlock_notify() to register for an 
** unlock-notify callback, then blocks until that callback is delivered 
** and returns SQLITE_OK. The caller should then retry the failed operation.
**
** Or, if sqlite3_unlock_notify() indicates that to block would deadlock 
** the system, then this function returns SQLITE_LOCKED immediately. In 
** this case the caller should not retry the operation and should roll 
** back the current transaction (if any).
*/
static int wait_for_unlock_notify(sqlite3 *db){
  int rc;
  UnlockNotification un;

  /* Initialize the UnlockNotification structure. */
  un.fired = 0;
  pthread_mutex_init(&un.mutex, 0);
  pthread_cond_init(&un.cond, 0);

  /* Register for an unlock-notify callback. */
  rc = sqlite3_unlock_notify(db, unlock_notify_cb, (void *)&un);
  assert( rc==SQLITE_LOCKED || rc==SQLITE_OK );

  /* The call to sqlite3_unlock_notify() always returns either SQLITE_LOCKED 
  ** or SQLITE_OK. 
  **
  ** If SQLITE_LOCKED was returned, then the system is deadlocked. In this
  ** case this function needs to return SQLITE_LOCKED to the caller so 
  ** that the current transaction can be rolled back. Otherwise, block
  ** until the unlock-notify callback is invoked, then return SQLITE_OK.
  */
  if( rc==SQLITE_OK ){
    pthread_mutex_lock(&un.mutex);
    if( !un.fired ){
      pthread_cond_wait(&un.cond, &un.mutex);
    }
    pthread_mutex_unlock(&un.mutex);
  }

  /* Destroy the mutex and condition variables. */
  pthread_cond_destroy(&un.cond);
  pthread_mutex_destroy(&un.mutex);

  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_step().
** It functions in the same way as step(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API step()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_step(sqlite3_stmt *pStmt){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_step(pStmt)) ){
    rc = wait_for_unlock_notify(sqlite3_db_handle(pStmt));
    if( rc!=SQLITE_OK ) break;
    sqlite3_reset(pStmt);
  }
  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_prepare_v2().
** It functions in the same way as prepare_v2(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API prepare_v2()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_prepare_v2(
  sqlite3 *db,              /* Database handle. */
  const char *zSql,         /* UTF-8 encoded SQL statement. */
  int nSql,                 /* Length of zSql in bytes. */
  sqlite3_stmt **ppStmt,    /* OUT: A pointer to the prepared statement */
  const char **pz           /* OUT: End of parsed string */
){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_prepare_v2(db, zSql, nSql, ppStmt, pz)) ){
    rc = wait_for_unlock_notify(db);
    if( rc!=SQLITE_OK ) break;
  }
  return rc;
}




  When two or more connections access the same database in shared-cache 
  mode, read and write (shared and exclusive) locks on individual tables
  are used to ensure that concurrently executing transactions are kept 
  isolated. Before writing to a table, a write (exclusive) lock must be 
  obtained on that table. Before reading, a read (shared) lock must be
  obtained. A connection releases all held table locks when it concludes
  its transaction. If a connection cannot obtain a required lock, then
  the call to sqlite3_step() returns SQLITE_LOCKED.


  Although it is less common, a call to sqlite3_prepare() or
  sqlite3_prepare_v2() may also return SQLITE_LOCKED if it cannot obtain
  a read-lock on the sqlite_schema table of each attached database. These
  APIs need to read the schema data contained in the sqlite_schema table
  in order to compile SQL statements to sqlite3_stmt* objects.


  This article presents a technique using the SQLite sqlite3_unlock_notify()
  interface such that calls to sqlite3_step() and sqlite3_prepare_v2()
  block until the required locks are available instead of returning 
  SQLITE_LOCKED immediately. If the
  sqlite3_blocking_step() or sqlite3_blocking_prepare_v2() functions presented 
  to the left return SQLITE_LOCKED, this indicates that to block would 
  deadlock the system.


  The sqlite3_unlock_notify() API, which is only available if the library is
  compiled with the pre-processor symbol SQLITE_ENABLE_UNLOCK_NOTIFY defined,
  is documented here. This article is not a
  substitute for reading the full API documentation!


  The sqlite3_unlock_notify() interface is designed for use in systems
  that have a separate thread assigned to each database connection.  There
  is nothing in the implementation that prevents a single thread from running
  multiple database connections.  However, the sqlite3_unlock_notify()
  interface only works on a single connection at a time, so the lock
  resolution logic presented here will only work for a single 
  database connection per thread.

The sqlite3_unlock_notify() API


  After a call to sqlite3_step() or sqlite3_prepare_v2() returns 
  SQLITE_LOCKED, the sqlite3_unlock_notify() API may be invoked to register 
  for an unlock-notify callback. The unlock-notify callback is invoked by 
  SQLite after the database connection holding the table-lock that prevented
  the call to sqlite3_step() or sqlite3_prepare_v2() from succeeding has
  finished its transaction and released all locks. For example, if a call to
  sqlite3_step() is an attempt to read from table X, and some other connection
  Y is holding a write-lock on table X, then sqlite3_step() will return
  SQLITE_LOCKED. If sqlite3_unlock_notify() is then called, the unlock-notify
  callback will be invoked after connection Y's transaction is concluded. The
  connection that the unlock-notify callback is waiting on, in this case
  connection Y, is known as the "blocking connection".


  If a call to sqlite3_step() that attempts to write to a database table
  returns SQLITE_LOCKED, then more than one other connection may be holding 
  a read-lock on the database table in question. In this case SQLite simply
  selects one of those other connections arbitrarily and issues the 
  unlock-notify callback when that connection's transaction is finished.
  Whether the call to sqlite3_step() was blocked by one or many connections, 
  when the corresponding unlock-notify callback is issued it is not 
  guaranteed that the required lock is available, only that it may be.


  When the unlock-notify callback is issued, it is issued from within a
  call to sqlite3_step() (or sqlite3_close()) associated with the blocking 
  connection. It is illegal to invoke any sqlite3_XXX() API functions from
  within an unlock-notify callback. The expected use is that the unlock-notify
  callback will signal some other waiting thread or schedule some action
  to take place later.


  The algorithm used by the sqlite3_blocking_step() function is as follows:


   Call sqlite3_step() on the supplied statement handle. If the call
       returns anything other than SQLITE_LOCKED, then return this value
       to the caller. Otherwise, continue.

   Invoke sqlite3_unlock_notify() on the database connection handle
       associated with the supplied statement handle to register for an
       unlock-notify callback. If the call to unlock_notify() returns
       SQLITE_LOCKED, then return this value to the caller.

   Block until the unlock-notify callback is invoked by another thread.

   Call sqlite3_reset() on the statement handle. Since an 
       SQLITE_LOCKED error may only occur on the first call to sqlite3_step()
       (it is not possible for one call to sqlite3_step() to return 
       SQLITE_ROW and then the next SQLITE_LOCKED), the statement handle may 
       be reset at this point without affecting the results of the query
       from the point of view of the caller. If sqlite3_reset() were not
       called at this point, the next call to sqlite3_step() would return
       SQLITE_MISUSE.

   Return to step 1.



  The algorithm used by the sqlite3_blocking_prepare_v2() function is similar,
  except that step 4 (resetting the statement handle) is omitted.


Writer Starvation


  Multiple connections may hold a read-lock simultaneously.  If many threads
  are acquiring overlapping read-locks, it might be the case that at least
  one thread is always holding a read lock.
  Then a table waiting for a write-lock will wait forever.
  This scenario is called "writer starvation."


  SQLite helps applications avoid writer starvation. After any attempt to
  obtain a write-lock on a table fails (because one or more other 
  connections are holding read-locks), all attempts to open new transactions
  on the shared-cache fail until one of the following is true:


   The current writer concludes its transaction, OR
   The number of open read-transactions on the shared-cache drops to zero.



  Failed attempts to open new read-transactions return SQLITE_LOCKED to the
  caller. If the caller then calls sqlite3_unlock_notify() to register for
  an unlock-notify callback, the blocking connection is the connection that
  currently has an open write-transaction on the shared-cache. This prevents
  writer-starvation since if no new read-transactions may be opened and 
  assuming all existing read-transactions are eventually concluded, the 
  writer will eventually have an opportunity to obtain the required
  write-lock.

The pthreads API

   By the time sqlite3_unlock_notify() is invoked by
      wait_for_unlock_notify(), it is possible that the blocking connection
      that prevented the sqlite3_step() or sqlite3_prepare_v2() call from 
      succeeding has already finished its transaction. In this case, the 
      unlock-notify callback is invoked immediately, before 
      sqlite3_unlock_notify() returns. Or, it is possible that the 
      unlock-notify callback is invoked by a second thread after 
      sqlite3_unlock_notify() is called but before the thread starts waiting 
      to be asynchronously signaled. 

   Exactly how such a potential race-condition is handled depends on the
      threads and synchronization primitives interface used by the application.
      This example uses pthreads, the interface provided by modern UNIX-like 
      systems, including Linux.

   The pthreads interface provides the pthread_cond_wait() function. 
      This function allows the caller to simultaneously release a mutex
      and start waiting for an asynchronous signal. Using this function,
      a "fired" flag and a mutex, the race-condition described above may
      be eliminated as follows:

   When the unlock-notify callback is invoked, which may be before the
      thread that called sqlite3_unlock_notify() begins waiting for the 
      asynchronous signal, it does the following:

    
       Obtains the mutex.
       Sets the "fired" flag to true.
       Attempts to signal a waiting thread.
       Releases the mutex.
    

   When the wait_for_unlock_notify() thread is ready to begin waiting for
      the unlock-notify callback to arrive, it:

    
       Obtains the mutex.
       Checks if the "fired" flag has been set. If so, the unlock-notify
           callback has already been invoked. Release the mutex and continue.
       Atomically releases the mutex and begins waiting for the
           asynchronous signal. When the signal arrives, continue.
    

   This way, it doesn't matter if the unlock-notify callback has already
      been invoked, or is being invoked, when the wait_for_unlock_notify() 
      thread begins blocking.

Possible Enhancements

   The code in this article could be improved in at least two ways:

  
     It could manage thread priorities.
     It could handle a special case of SQLITE_LOCKED that can occur
         when dropping a table or index.
  

  
    Even though the sqlite3_unlock_notify() function only allows the caller
    to specify a single user-context pointer, an unlock-notify callback 
    is passed an array of such context pointers. This is because if when
    a blocking connection concludes its transaction, if there is more
    than one unlock-notify registered to call the same C function, the
    context-pointers are marshaled into an array and a single callback
    issued. If each thread were assigned a priority, then instead of just
    signaling the threads in arbitrary order as this implementation does,
    higher priority threads could be signaled before lower priority threads.

  
    If a "DROP TABLE" or "DROP INDEX" SQL command is executed, and the
    same database connection currently has one or more actively
    executing SELECT statements, then SQLITE_LOCKED is returned. If
    sqlite3_unlock_notify() is called in this case, then the specified
    callback will be invoked immediately. Re-attempting the "DROP
    TABLE" or "DROP INDEX" statement will return another SQLITE_LOCKED 
    error. In the implementation of sqlite3_blocking_step() shown to the
    left, this could cause an infinite loop.

  
    The caller could distinguish between this special "DROP TABLE|INDEX" 
    case and other cases by using extended error codes. When it is appropriate
    to call sqlite3_unlock_notify(), the extended error code is
    SQLITE_LOCKED_SHAREDCACHE. Otherwise, in the "DROP TABLE|INDEX" case,
    it is just plain SQLITE_LOCKED. Another solution might be to limit
    the number of times that any single query could be reattempted (to say 
    100). Although this might be less efficient than one might wish, the 
    situation in question is not likely to occur often.


This page last modified on  2022-01-08 05:02:57 UTC


c2: Defense Against The Dark Arts
c5: security.html
c4:
1. SQLite Always Validates Its Inputs


SQLite should never crash, overflow a buffer, leak memory,
or exhibit any other harmful behavior, even when presented with
maliciously malformed SQL inputs or database files.  SQLite should
always detect erroneous inputs and raise an error, not crash or
corrupt memory.
Any malfunction caused by an SQL input or database file
is considered a serious bug and will be promptly addressed when
brought to the attention of the SQLite developers.  SQLite is
extensively fuzz-tested to help ensure that it is resistant
to these kinds of errors.


Nevertheless, bugs happen.
If you are writing an application that sends untrusted SQL inputs
or database files to SQLite, there are additional steps you can take
to help reduce the attack surface and
prevent zero-day exploits caused by undetected bugs.

1.1. Untrusted SQL Inputs

Applications that accept untrusted SQL inputs should take the following
precautions:



Set the SQLITE_DBCONFIG_DEFENSIVE flag.
This prevents ordinary SQL statements from deliberately corrupting the 
database file.  SQLite should be proof against attacks that involve both
malicious SQL inputs and a maliciously corrupted database file at the
same time.  Nevertheless, denying a script-only attacker access to 
corrupt database inputs provides an extra layer of defense.


Reduce the limits that SQLite imposes on inputs.  This can help prevent
denial of service attacks and other kinds of mischief that can occur
as a result of unusually large inputs.  You can do this either at compile-time
using -DSQLITE_MAX_... options, or at run-time using the
sqlite3_limit() interface.  Most applications can reduce limits
dramatically without impacting functionality.  The table below
provides some suggestions, though exact values will vary depending
on the application:


Limit SettingDefault ValueHigh-security Value
LIMIT_LENGTH1,000,000,0001,000,000
LIMIT_SQL_LENGTH1,000,000,000100,000
LIMIT_COLUMN2,000100
LIMIT_EXPR_DEPTH1,00010
LIMIT_COMPOUND_SELECT5003
LIMIT_VDBE_OP250,000,00025,000
LIMIT_FUNCTION_ARG1278
LIMIT_ATTACH100
LIMIT_LIKE_PATTERN_LENGTH50,00050
LIMIT_VARIABLE_NUMBER99910
LIMIT_TRIGGER_DEPTH1,00010



Consider using the sqlite3_set_authorizer() interface to limit
the scope of SQL that will be processed.  For example, an application
that does not need to change the database schema might add an
sqlite3_set_authorizer() callback that causes any CREATE or DROP
statement to fail.


The SQL language is very powerful, and so it is always possible for
malicious SQL inputs (or erroneous SQL inputs caused by an application
bug) to submit SQL that runs for a very long time.  To prevent this
from becoming a denial-of-service attack, consider using the
sqlite3_progress_handler() interface to invoke a callback periodically
as each SQL statement runs, and have that callback return non-zero to
abort the statement if the statement runs for too long.  Alternatively,
set a timer in a separate thread and invoke sqlite3_interrupt() when
the timer goes off to prevent the SQL statement from running forever.


Limit the maximum amount of memory that SQLite will allocate using
the sqlite3_hard_heap_limit64() interface.  This helps prevent
denial-of-service attacks.  To find out how much heap space an
application actually needs, run the it against typical inputs and
then measure the maximum instantaneous memory usage with the 
sqlite3_memory_highwater() interface.  Set the hard heap limit
to the maximum observed instantaneous memory usage plus some margin.


Consider setting the SQLITE_MAX_ALLOCATION_SIZE compile-time option
to something smaller than its default value of 2147483391 (0x7ffffeff).
A value of 100000000 (100 million) or even smaller would not be unreasonable,
depending on the application.


For embedded systems, consider compiling SQLite with the
-DSQLITE_ENABLE_MEMSYS5 option and then providing SQLite with
a fixed chunk of memory to use as its heap via the
sqlite3_config(SQLITE_CONFIG_HEAP) interface. This will
prevent malicious SQL from executing a denial-of-service attack
by using an excessive amount of memory. If (say) 5 MB of memory
is provided for SQLite to use, once that much has been consumed,
SQLite will start returning SQLITE_NOMEM errors rather than
soaking up memory needed by other parts of the application.
This also sandboxes SQLite's memory so that a write-after-free
error in some other part of the application will not cause
problems for SQLite, or vice versa.



To control memory usage in the printf() SQL function, compile
with "-DSQLITE_PRINTF_PRECISION_LIMIT=100000" or some similarly
reasonable value.
This #define limits the width and precision for %-substitutions in the
printf() function, and thus prevents a hostile SQL statement from
consuming large amounts of RAM via constructs such as
"printf('%1000000000s','hi')".

Note that SQLite uses its built-in printf() internally to help it
format the sql column in the sqlite_schema table.  For that reason,
no table, index, view, or trigger definition can be much larger than the
precision limit.  You can set a precision limit of less than 100000,
but be careful that whatever precision limit you use is at least as
long as the longest CREATE statement in your schema.






1.2. Untrusted SQLite Database Files

Applications that read or write SQLite database files of uncertain
provenance should take precautions enumerated below.

Even if the application does not deliberately accept database files 
from untrusted sources, beware of attacks in which a local 
database file is altered.  For best security, any database file which 
might have ever been writable by an agent in a different security domain
should be treated as suspect.



If the application includes any custom SQL functions or 
custom virtual tables that have side effects or that might leak
privileged information, then the application should use one or more
of the techniques below to prevent a maliciously crafted database
schema from surreptitiously running those SQL functions and/or
virtual tables for nefarious purposes:

 Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
     on each database connection as soon as it is opened.
 Run the PRAGMA trusted_schema=OFF statement on each database connection
     as soon as it is opened.
 Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
 Disable the surreptitious use of custom SQL functions and virtual tables
     by setting the SQLITE_DIRECTONLY flag on all custom SQL functions and
     the SQLITE_VTAB_DIRECTONLY flag on all custom virtual tables.



If the application does not use triggers or views, consider disabling the
unused capabilities with:

sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_TRIGGER,0,0);
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_VIEW,0,0);





For reading database files that are unusually high-risk, such as database
files that are received from remote machines, and possibly from anonymous
contributors, the following extra precautions
might be justified.  These added defenses come with performance costs,
however, and so may not be appropriate in every situation:



Run PRAGMA integrity_check or PRAGMA quick_check on the database
as the first SQL statement after opening the database files and
prior to running any other SQL statements.  Reject and refuse to
process any database file containing errors.


Enable the PRAGMA cell_size_check=ON setting.



Do not enable memory-mapped I/O.
In other words, make sure that PRAGMA mmap_size=0.



2. Summary


The precautions above are not required in order to use SQLite safely
with potentially hostile inputs.
However, they do provide an extra layer of defense against zero-day
exploits and are encouraged for applications that pass data from
untrusted sources into SQLite.
This page last modified on  2024-01-16 14:06:27 UTC


c2: Query Planning
c5: queryplanner.html
c4:
Overview 


The best feature of SQL (in all its implementations, not just SQLite)
is that it is a declarative language, not a procedural
language.  When programming in SQL you tell the system what you
want to compute, not how to compute it.  The task of figuring out
the how is delegated to the query planner subsystem within 
the SQL database engine.

For any given SQL statement, there might be hundreds or thousands or
even millions of different algorithms of performing the operation.  All
of these algorithms will get the correct answer, though some will run
faster than others.
The query planner is an 
AI that 
tries to pick the fastest and most efficient algorithm for each SQL
statement.



Most of the time, the query planner in SQLite does a good job.
However, the query planner needs indices to
work with.  
These indices must normally be added by programmers.
Rarely, the query planner AI will make a suboptimal algorithm
choice.
In those cases, programmers may want to provide additional
hints to help the query planner do a better job.



This document provides background information about how the
SQLite query planner and query engine work.
Programmers can use this information to help create better
indexes, and provide hints to help the query planner when
needed.



Additional information is provided in the
SQLite query planner and 
next generation query planner documents.




1.  Searching

1.1.  Tables Without Indices


Most tables in SQLite consist of zero or more rows with a unique integer
key (the rowid or INTEGER PRIMARY KEY) followed by content.  
(The exception is WITHOUT ROWID tables.)
The rows
are logically stored in order of increasing rowid.  As an example, this
article uses a table named "FruitsForSale" which relates various fruits 
to the state
where they are grown and their unit price at market.  The schema is this:



CREATE TABLE FruitsForSale(
  Fruit TEXT,
  State TEXT,
  Price REAL
);




With some (arbitrary) data, such a table might be logically stored on disk
as shown in figure 1:





Figure 1: Logical Layout Of Table "FruitsForSale"




In this example, the rowids are not
consecutive but they are ordered.  SQLite usually creates rowids beginning
with one and increasing by one with each added row.  But if rows are 
deleted, gaps can appear in the sequence.  And the application can control
the rowid assigned if desired, so that rows are not necessarily inserted 
at the bottom.  But regardless of what happens, the rowids are always 
unique and in strictly ascending order.



Suppose you want to look up the price of peaches.  The query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Peach';




To satisfy this query, SQLite reads every row out of the
table, checks to see if the "fruit" column has the value of "Peach" and if
so, outputs the "price" column from that row.  The process is illustrated
by figure 2 below.
This is algorithm is called a full table scan 
since the entire content of the
table must be read and examined in order to find the one row of interest.
With a table of only 7 rows, a full table scan is acceptable, 
but if the table contained 7 million rows, a full table scan might read 
megabytes of content in order to find a single 8-byte number.  
For that reason, one normally tries to avoid full table scans.





Figure 2: Full Table Scan



1.2.  Lookup By Rowid


One technique for avoiding a full table scan is to do lookups by
rowid (or by the equivalent INTEGER PRIMARY KEY).   To lookup the
price of peaches, one would query for the entry with a rowid of 4:



SELECT price FROM fruitsforsale WHERE rowid=4;




Since the information is stored in the table in rowid order, SQLite
can find the correct row using a binary search.
If the table contains N elements, the time required to look up the
desired row is proportional to logN rather than being proportional
to N as in a full table scan.  If the table contains 10 million elements,
that means the query will be on the order of N/logN or about 1 million
times faster.





Figure 3: Lookup By Rowid



1.3.  Lookup By Index

The problem with looking up information by rowid is that you probably
do not care what the price of "item 4" is - you want to know the price
of peaches.  And so a rowid lookup is not helpful.



To make the original query more efficient, we can add an index on the
"fruit" column of the "fruitsforsale" table like this:



CREATE INDEX Idx1 ON fruitsforsale(fruit);




An index is another table similar to the original "fruitsforsale" table
but with the content (the fruit column in this case) stored in front of the
rowid and with all rows in content order.
Figure 4 gives a logical view of the Idx1 index.
The "fruit" column is the primary key used to order the elements of the
table and the "rowid" is the secondary key used to break the tie when
two or more rows have the same "fruit".  In the example, the rowid
has to be used as a tie-breaker for the "Orange" rows.
Notice that since the rowid
is always unique over all elements of the original table, the composite key
of "fruit" followed by "rowid" will be unique over all elements of the index.





Figure 4: An Index On The Fruit Column




This new index can be used to implement a faster algorithm for the
original "Price of Peaches" query.



SELECT price FROM fruitsforsale WHERE fruit='Peach';




The query starts by doing a binary search on the Idx1 index for entries
that have fruit='Peach'.  SQLite can do this binary search on the Idx1 index
but not on the original FruitsForSale table because the rows in Idx1 are sorted
by the "fruit" column.  Having found a row in the Idx1 index that has
fruit='Peach', the database engine can extract the rowid for that row.
Then the database engines does a second binary search
on the original FruitsForSale table to find the
original row that contains fruit='Peach'.  
From the row in the FruitsForSale table,
SQLite can then extract the value of the price column.
This procedure is illustrated by figure 5.





Figure 5: Indexed Lookup For The Price Of Peaches




SQLite has to do two binary searches to find the price of peaches using
the method show above.  But for a table with a large number of rows, this
is still much faster than doing a full table scan.


1.4.  Multiple Result Rows


In the previous query the fruit='Peach' constraint narrowed the result
down to a single row.  But the same technique works even if multiple
rows are obtained.  Suppose we looked up the price of Oranges instead of
Peaches:



SELECT price FROM fruitsforsale WHERE fruit='Orange'




Figure 6: Indexed Lookup For The Price Of Oranges




In this case, SQLite still does a single binary search to find the first
entry of the index where fruit='Orange'.  Then it extracts the rowid from
the index and uses that rowid to lookup the original table entry via
binary search and output the price from the original table.  But instead
of quitting, the database engine then advances to the next row of index
to repeat the process for next fruit='Orange' entry.  Advancing to the
next row of an index (or table) is much less costly than doing a binary
search since the next row is often located on the same database page as
the current row.  In fact, the cost of advancing to the next row is so
cheap in comparison to a binary search that we usually ignore it.  So
our estimate for the total cost of this query is 3 binary searches.
If the number of rows of output is K and the number of rows in the table
is N, then in general the cost of doing the query is proportional
to (K+1)*logN.


1.5.  Multiple AND-Connected WHERE-Clause Terms


Next, suppose that you want to look up the price of not just any orange,
but specifically California-grown oranges.  The appropriate query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 7: Indexed Lookup Of California Oranges




One approach to this query is to use the fruit='Orange' term of the WHERE
clause to find all rows dealing with oranges, then filter those rows
by rejecting any that are from states other than California.  This
process is shown by figure 7 above.  This is a perfectly
reasonable approach in most cases.  Yes, the database engine did have
to do an extra binary search for the Florida orange row that was
later rejected, so it was not as efficient as we might hope, though
for many applications it is efficient enough.  



Suppose that in addition to the index on "fruit" there was also
an index on "state".



CREATE INDEX Idx2 ON fruitsforsale(state);




Figure 8: Index On The State Column




The "state" index works just like the "fruit" index in that it is a
new table with an extra column in front of the rowid and sorted by
that extra column as the primary key.  The only difference is that
in Idx2, the first column is "state" instead of "fruit" as it is with
Idx1.  In our example data set, there is more redundancy in the "state"
column and so they are more duplicate entries.  The ties are still
resolved using the rowid.



Using the new Idx2 index on "state", SQLite has another option for
lookup up the price of California oranges:  it can look up every row
that contains fruit from California and filter out those rows that
are not oranges.





Figure 9: Indexed Lookup Of California Oranges




Using Idx2 instead of Idx1 causes SQLite to examine a different set of
rows, but it gets the same answer in the end (which is very important -
remember that indices should never change the answer, only help SQLite to
get to the answer more quickly) and it does the same amount of work.
So the Idx2 index did not help performance in this case.



The last two queries take the same amount of time, in our example.
So which index, Idx1 or Idx2, will SQLite choose?  If the
ANALYZE command has been run on the database, so that SQLite has
had an opportunity to gather statistics about the available indices,
then SQLite will know that the Idx1 index usually narrows the search
down to a single item (our example of fruit='Orange' is the exception
to this rule) whereas the Idx2 index will normally only narrow the 
search down to two rows.  So, if all else is equal, SQLite will
choose Idx1 with the hope of narrowing the search to as small
a number of rows as possible.  This choice is only possible because
of the statistics provided by ANALYZE.  If ANALYZE has not been
run then the choice of which index to use is arbitrary.


1.6.  Multi-Column Indices


To get the maximum performance out of a query with multiple AND-connected
terms in the WHERE clause, you really want a multi-column index with
columns for each of the AND terms.  In this case we create a new index
on the "fruit" and "state" columns of FruitsForSale:



CREATE INDEX Idx3 ON FruitsForSale(fruit, state);




Figure 1: A Two-Column Index




A multi-column index follows the same pattern as a single-column index;
the indexed columns are added in front of the rowid.  The only difference
is that now multiple columns are added.  The left-most column is the
primary key used for ordering the rows in the index.  The second column is
used to break ties in the left-most column.  If there were a third column,
it would be used to break ties for the first two columns.  And so forth for
all columns in the index.  Because rowid is guaranteed
to be unique, every row of the index will be unique even if all of the
content columns for two rows are the same.  That case does not happen
in our sample data, but there is one case (fruit='Orange') where there
is a tie on the first column which must be broken by the second column.



Given the new multi-column Idx3 index, it is now possible for SQLite
to find the price of California oranges using only 2 binary searches:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 11: Lookup Using A Two-Column Index




With the Idx3 index on both columns that are constrained by the WHERE clause,
SQLite can do a single binary search against Idx3 to find the one rowid
for California oranges, then do a single binary search to find the price
for that item in the original table.  There are no dead-ends and no
wasted binary searches.  This is a more efficient query.



Note that Idx3 contains all the same information as the original 
Idx1.  And so if we have Idx3, we do not really need Idx1
any more.  The "price of peaches" query can be satisfied using Idx3
by simply ignoring the "state" column of Idx3:



SELECT price FROM fruitsforsale WHERE fruit='Peach'




Figure 12: Single-Column Lookup On A Multi-Column Index




Hence, a good rule of thumb is that your database schema should never
contain two indices where one index is a prefix of the other.  Drop the
index with fewer columns.  SQLite will still be able to do efficient
lookups with the longer index.




1.7.  Covering Indexes


The "price of California oranges" query was made more efficient through
the use of a two-column index.  But SQLite can do even better with a
three-column index that also includes the "price" column:



CREATE INDEX Idx4 ON FruitsForSale(fruit, state, price);




Figure 13: A Covering Index




This new index contains all the columns of the original FruitsForSale table that
are used by the query - both the search terms and the output.  We call
this a "covering index".  Because all of the information needed is in
the covering index, SQLite never needs to consult the original table
in order to find the price.



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA';




Figure 14: Query Using A Covering Index




Hence, by adding extra "output" columns onto the end of an index, one
can avoid having to reference the original table and thereby
cut the number of binary searches for a query in half.  This is a
constant-factor improvement in performance (roughly a doubling of
the speed).  But on the other hand, it is also just a refinement;
A two-fold performance increase is not nearly as dramatic as the
one-million-fold increase seen when the table was first indexed.
And for most queries, the difference between 1 microsecond and
2 microseconds is unlikely to be noticed.




1.8.  OR-Connected Terms In The WHERE Clause


Multi-column indices only work if the constraint terms in the WHERE
clause of the query are connected by AND.
So Idx3 and Idx4 are helpful when the search is for items that
are both Oranges and grown in California, but neither index would
be that useful if we wanted all items that were either oranges
or are grown in California.



SELECT price FROM FruitsForSale WHERE fruit='Orange' OR state='CA';




When confronted with OR-connected terms in a WHERE clause, SQLite 
examines each OR term separately and tries to use an index to
find the rowids associated with each term.
It then takes the union of the resulting rowid sets to find
the end result.  The following figure illustrates this process:





Figure 15: Query With OR Constraints




The diagram above implies that SQLite computes all of the rowids first
and then combines them with a union operation before starting to do
rowid lookups on the original table.  In reality, the rowid lookups
are interspersed with rowid computations.  SQLite uses one index at
a time to find rowids while remembering which rowids it has seen
before so as to avoid duplicates.  That is just an implementation
detail, though.  The diagram, while not 100% accurate, provides a good
overview of what is happening.



In order for the OR-by-UNION technique shown above to be useful, there
must be an index available that helps resolve every OR-connected term
in the WHERE clause.  If even a single OR-connected term is not indexed,
then a full table scan would have to be done in order to find the rowids
generated by the one term, and if SQLite has to do a full table scan, it
might as well do it on the original table and get all of the results in
a single pass without having to mess with union operations and follow-on
binary searches.



One can see how the OR-by-UNION technique could also be leveraged to
use multiple indices on queries where the WHERE clause has terms connected
by AND, by using an intersect operator in place of union.  Many SQL
database engines will do just that.  But the performance gain over using
just a single index is slight and so SQLite does not implement that technique
at this time.  However, a future version SQLite might be enhanced to support
AND-by-INTERSECT.




2.  Sorting


SQLite (like all other SQL database engines) can also use indices to
satisfy the ORDER BY clauses in a query, in addition to expediting
lookup.  In other words, indices can be used to speed up sorting as
well as searching.



When no appropriate indices are available, a query with an ORDER BY
clause must be sorted as a separate step.  Consider this query:



SELECT * FROM fruitsforsale ORDER BY fruit;




SQLite processes this by gathering all the output of query and then
running that output through a sorter.





Figure 16: Sorting Without An Index




If the number of output rows is K, then the time needed to sort is
proportional to KlogK.  If K is small, the sorting time is usually
not a factor, but in a query such as the above where K==N, the time
needed to sort can be much greater than the time needed to do a
full table scan.  Furthermore, the entire output is accumulated in
temporary storage (which might be either in main memory or on disk,
depending on various compile-time and run-time settings)
which can mean that a lot of temporary storage is required to complete
the query.


2.1.  Sorting By Rowid


Because sorting can be expensive, SQLite works hard to convert ORDER BY
clauses into no-ops.  If SQLite determines that output will
naturally appear in the order specified, then no sorting is done.
So, for example, if you request the output in rowid order, no sorting
will be done:



SELECT * FROM fruitsforsale ORDER BY rowid;




Figure 17: Sorting By Rowid




You can also request a reverse-order sort like this:



SELECT * FROM fruitsforsale ORDER BY rowid DESC;




SQLite will still omit the sorting step.  But in order for output to
appear in the correct order, SQLite will do the table scan starting at
the end and working toward the beginning, rather than starting at the
beginning and working toward the end as shown in 
figure 17.


2.2.  Sorting By Index


Of course, ordering the output of a query by rowid is seldom useful.
Usually one wants to order the output by some other column.



If an index is available on the ORDER BY column, that index can be used
for sorting.  Consider the request for all items sorted by "fruit":



SELECT * FROM fruitsforsale ORDER BY fruit;






Figure 18: Sorting With An Index




The Idx1 index is scanned from top to bottom (or from bottom to top if
"ORDER BY fruit DESC" is used) in order to find the rowids for each item
in order by fruit.  Then for each rowid, a binary search is done to lookup
and output that row.  In this way, the output appears in the requested order
without the need to gather the entire output and sort it using a separate step.



But does this really save time?  The number of steps in the 
original indexless sort is proportional to NlogN since
that is how much time it takes to sort N rows.  But when we use Idx1 as
shown here, we have to do N rowid lookups which take logN time each, so
the total time of NlogN is the same!



SQLite uses a cost-based query planner.  When there are two or more ways
of solving the same query, SQLite tries to estimate the total amount of
time needed to run the query using each plan, and then uses the plan with
the lowest estimated cost.  A cost is computed mostly from the estimated
time, and so this case could go either way depending on the table size and
what WHERE clause constraints were available, and so forth.  But generally
speaking, the indexed sort would probably be chosen, if for no other
reason, because it does not need to accumulate the entire result set in
temporary storage before sorting and thus uses much less temporary storage.


2.3.  Sorting By Covering Index


If a covering index can be used for a query, then the multiple rowid lookups
can be avoided and the cost of the query drops dramatically.





Figure 19: Sorting With A Covering Index




With a covering index, SQLite can simply walk the index from one end to the
other and deliver the output in time proportional to N and without having
allocate a large buffer to hold the result set.


3.  Searching And Sorting At The Same Time


The previous discussion has treated searching and sorting as separate
topics.  But in practice, it is often the case that one wants to search
and sort at the same time.  Fortunately, it is possible to do this
using a single index.


3.1.  Searching And Sorting With A Multi-Column Index


Suppose we want to find the prices of all kinds of oranges sorted in
order of the state where they are grown.  The query is this:



SELECT price FROM fruitforsale WHERE fruit='Orange' ORDER BY state




The query contains both a search restriction in the WHERE clause
and a sort order in the ORDER BY clause.  Both the search and the sort
can be accomplished at the same time using the two-column index Idx3.





Figure 20: Search And Sort By Multi-Column Index




The query does a binary search on the index to find the subset of rows
that have fruit='Orange'.  (Because the fruit column is the left-most column
of the index and the rows of the index are in sorted order, all such 
rows will be adjacent.)  Then it scans the matching index rows from top to
bottom to get the rowids for the original table, and for each rowid does
a binary search on the original table to find the price.



You will notice that there is no "sort" box anywhere in the above diagram.
The ORDER BY clause of the query has become a no-op.  No sorting has to be
done here because the output order is by the state column and the state
column also happens to be the first column after the fruit column in the
index.  So, if we scan entries of the index that have the same value for
the fruit column from top to bottom, those index entries are guaranteed to
be ordered by the state column.




3.2.  Searching And Sorting With A Covering Index


A covering index can also be used to search and sort at the same time.
Consider the following:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state




Figure 21: Search And Sort By Covering Index




As before, SQLite does single binary search
for the range of rows in the covering
index that satisfy the WHERE clause, the scans that range from top to 
bottom to get the desired results.  
The rows that satisfy the WHERE clause are guaranteed to be adjacent
since the WHERE clause is an equality constraint on the left-most
column of the index.  And by scanning the matching index rows from
top to bottom, the output is guaranteed to be ordered by state since the
state column is the very next column to the right of the fruit column.
And so the resulting query is very efficient.



SQLite can pull a similar trick for a descending ORDER BY:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state DESC




The same basic algorithm is followed, except this time the matching rows
of the index are scanned from bottom to top instead of from top to bottom,
so that the states will appear in descending order.




3.3.  Partial Sorting Using An Index (a.k.a. Block Sorting)


Sometimes only part of an ORDER BY clause can be satisfied using indexes.
Consider, for example, the following query:



SELECT * FROM fruitforsale ORDER BY fruit, price




If the covering index is used for the scan, the "fruit" column will appear
naturally in the correct order, but when there are two or more rows with
the same fruit, the price might be out of order.  When this occurs, SQLite
does many small sorts, one sort for each distinct value of fruit, rather
than one large sort.  Figure 22 below illustrates the concept.





Figure 22: Partial Sort By Index




In the example, instead of a single sort of 7 elements, there
are 5 sorts of one-element each and 1 sort of 2 elements for the
case of fruit=='Orange'.


The advantages of doing many smaller sorts instead of a single large sort
are:

Multiple small sorts collectively use fewer CPU cycles than a single
    large sort.
Each small sort is run independently, meaning that much less information
    needs to be kept in temporary storage at any one time.
Those columns of the ORDER BY that are already in the correct order
    due to indexes can be omitted from the sort key, further reducing
    storage requirements and CPU time.
Output rows can be returned to the application as each small sort
    completes, and well before the table scan is complete.
If a LIMIT clause is present, it might be possible to avoid scanning
    the entire table.


Because of these advantages, SQLite always tries to do a partial sort using an
index even if a complete sort by index is not possible.

4.  WITHOUT ROWID tables


The basic principals described above apply to both ordinary rowid tables
and WITHOUT ROWID tables.
The only difference is that the rowid column that serves as the key for
tables and that appears as the right-most term in indexes is replaced by
the PRIMARY KEY.

This page last modified on  2022-10-26 13:30:36 UTC


c2: Atomic Commit In SQLite
c5: atomiccommit.html
c4:
1.  Introduction

An important feature of transactional databases like SQLite
is "atomic commit".
Atomic commit means that either all database changes within a single
transaction occur or none of them occur.  With atomic commit, it
is as if many different writes to different sections of the database
file occur instantaneously and simultaneously.
Real hardware serializes writes to mass storage, and writing
a single sector takes a finite amount of time.
So it is impossible to truly write many different sectors of a
database file simultaneously and/or instantaneously.
But the atomic commit logic within
SQLite makes it appear as if the changes for a transaction
are all written instantaneously and simultaneously.

SQLite has the important property that transactions appear
to be atomic even if the transaction is interrupted by an
operating system crash or power failure.

This article describes the techniques used by SQLite to create the
illusion of atomic commit.

The information in this article applies only when SQLite is operating
in "rollback mode", or in other words when SQLite is not
using a write-ahead log.  SQLite still supports atomic commit when
write-ahead logging is enabled, but it accomplishes atomic commit by
a different mechanism from the one described in this article.  See
the write-ahead log documentation for additional information on how
SQLite supports atomic commit in that context.



2.  Hardware Assumptions

Throughout this article, we will call the mass storage device "disk"
even though the mass storage device might really be flash memory.

We assume that disk is written in chunks which we call a "sector".
It is not possible to modify any part of the disk smaller than a sector.
To change a part of the disk smaller than a sector, you have to read in
the full sector that contains the part you want to change, make the
change, then write back out the complete sector.

On a traditional spinning disk, a sector is the minimum unit of transfer
in both directions, both reading and writing.  On flash memory, however,
the minimum size of a read is typically much smaller than a minimum write.
SQLite is only concerned with the minimum write amount and so for the
purposes of this article, when we say "sector" we mean the minimum amount
of data that can be written to mass storage in a single go.


  Prior to SQLite version 3.3.14, a sector size of 512 bytes was
  assumed in all cases.  There was a compile-time option to change
  this but the code had never been tested with a larger value.  The
  512 byte sector assumption seemed reasonable since until very recently
  all disk drives used a 512 byte sector internally.  However, there
  has recently been a push to increase the sector size of disks to
  4096 bytes.  Also the sector size
  for flash memory is usually larger than 512 bytes.  For these reasons,
  versions of SQLite beginning with 3.3.14 have a method in the OS
  interface layer that interrogates the underlying filesystem to find
  the true sector size.  As currently implemented (version 3.5.0) this
  method still returns a hard-coded value of 512 bytes, since there
  is no standard way of discovering the true sector size on either
  Unix or Windows.  But the method is available for embedded device
  manufacturers to tweak according to their own needs.  And we have
  left open the possibility of filling in a more meaningful implementation
  on Unix and Windows in the future.

SQLite has traditionally assumed that a sector write is not atomic.
However, SQLite does always assume that a sector write is linear.  By "linear"
we mean that SQLite assumes that when writing a sector, the hardware begins
at one end of the data and writes byte by byte until it gets to
the other end.  The write might go from beginning to end or from
end to beginning.  If a power failure occurs in the middle of a
sector write it might be that part of the sector was modified
and another part was left unchanged.  The key assumption by SQLite
is that if any part of the sector gets changed, then either the
first or the last bytes will be changed.  So the hardware will
never start writing a sector in the middle and work towards the
ends.  We do not know if this assumption is always true but it
seems reasonable.

The previous paragraph states that SQLite does not assume that
sector writes are atomic.  This is true by default.  But as of
SQLite version 3.5.0, there is a new interface called the
Virtual File System (VFS) interface.  The VFS is the only means
by which SQLite communicates to the underlying filesystem.  The
code comes with default VFS implementations for Unix and Windows
and there is a mechanism for creating new custom VFS implementations
at runtime.  In this new VFS interface there is a method called
xDeviceCharacteristics.  This method interrogates the underlying
filesystem to discover various properties and behaviors that the
filesystem may or may not exhibit.  The xDeviceCharacteristics
method might indicate that sector writes are atomic, and if it does
so indicate, SQLite will try to take advantage of that fact.  But
the default xDeviceCharacteristics method for both Unix and Windows
does not indicate atomic sector writes and so these optimizations
are normally omitted.

SQLite assumes that the operating system will buffer writes and
that a write request will return before data has actually been stored
in the mass storage device.
SQLite further assumes that write operations will be reordered by
the operating system.
For this reason, SQLite does a "flush" or "fsync" operation at key
points.  SQLite assumes that the flush or fsync will not return until
all pending write operations for the file that is being flushed have
completed.  We are told that the flush and fsync primitives
are broken on some versions of Windows and Linux.  This is unfortunate.
It opens SQLite up to the possibility of database corruption following
a power loss in the middle of a commit.  However, there is nothing
that SQLite can do to test for or remedy the situation.  SQLite
assumes that the operating system that it is running on works as
advertised.  If that is not quite the case, well then hopefully you
will not lose power too often.

SQLite assumes that when a file grows in length that the new
file space originally contains garbage and then later is filled in
with the data actually written.  In other words, SQLite assumes that
the file size is updated before the file content.  This is a
pessimistic assumption and SQLite has to do some extra work to make
sure that it does not cause database corruption if power is lost
between the time when the file size is increased and when the
new content is written.  The xDeviceCharacteristics method of
the VFS might indicate that the filesystem will always write the
data before updating the file size.  (This is the
SQLITE_IOCAP_SAFE_APPEND property for those readers who are looking
at the code.)  When the xDeviceCharacteristics method indicates
that files content is written before the file size is increased,
SQLite can forego some of its pedantic database protection steps
and thereby decrease the amount of disk I/O needed to perform a
commit.  The current implementation, however, makes no such assumptions
for the default VFSes for Windows and Unix.

SQLite assumes that a file deletion is atomic from the
point of view of a user process.  By this we mean that if SQLite
requests that a file be deleted and the power is lost during the
delete operation, once power is restored either the file will
exist completely with all if its original content unaltered, or
else the file will not be seen in the filesystem at all.  If
after power is restored the file is only partially deleted,
if some of its data has been altered or erased,
or the file has been truncated but not completely removed, then
database corruption will likely result.

SQLite assumes that the detection and/or correction of
bit errors caused by cosmic rays, thermal noise, quantum
fluctuations, device driver bugs, or other mechanisms, is the
responsibility of the underlying hardware and operating system.
SQLite does not add any redundancy to the database file for
the purpose of detecting corruption or I/O errors.
SQLite assumes that the data it reads is exactly the same data
that it previously wrote.

By default, SQLite assumes that an operating system call to write
a range of bytes will not damage or alter any bytes outside of that range
even if a power loss or OS crash occurs during that write.  We
call this the "powersafe overwrite" property.
Prior to version 3.7.9 (2011-11-01),
SQLite did not assume powersafe overwrite.  But with the standard
sector size increasing from 512 to 4096 bytes on most disk drives, it
has become necessary to assume powersafe overwrite in order to maintain
historical performance levels and so powersafe overwrite is assumed by
default in recent versions of SQLite.  The assumption of powersafe
overwrite property can be disabled at compile-time or a run-time if
desired.  See the powersafe overwrite documentation for further
details.



3.  Single File Commit

We begin with an overview of the steps SQLite takes in order to
perform an atomic commit of a transaction against a single database
file.  The details of file formats used to guard against damage from
power failures and techniques for performing an atomic commit across
multiple databases are discussed in later sections.



3.1.  Initial State



The state of the computer when a database connection is
first opened is shown conceptually by the diagram at the
right.
The area of the diagram on the extreme right (labeled "Disk") represents
information stored on the mass storage device.  Each rectangle is
a sector.  The blue color represents that the sectors contain
original data.
The middle area is the operating systems disk cache.  At the
onset of our example, the cache is cold and this is represented
by leaving the rectangles of the disk cache empty.
The left area of the diagram shows the content of memory for
the process that is using SQLite.  The database connection has
just been opened and no information has been read yet, so the
user space is empty.






3.2.  Acquiring A Read Lock



Before SQLite can write to a database, it must first read
the database to see what is there already.  Even if it is just
appending new data, SQLite still has to read in the database
schema from the "sqlite_schema" table so that it can know
how to parse the INSERT statements and discover where in the
database file the new information should be stored.

The first step toward reading from the database file
is obtaining a shared lock on the database file.  A "shared"
lock allows two or more database connections to read from the
database file at the same time.  But a shared lock prevents
another database connection from writing to the database file
while we are reading it.  This is necessary because if another
database connection were writing to the database file at the
same time we are reading from the database file, we might read
some data before the change and other data after the change.
This would make it appear as if the change made by the other
process is not atomic.

Notice that the shared lock is on the operating system
disk cache, not on the disk itself.  File locks
really are just flags within the operating system kernel,
usually.  (The details depend on the specific OS layer
interface.)  Hence, the lock will instantly vanish if the
operating system crashes or if there is a power loss.  It
is usually also the case that the lock will vanish if the
process that created the lock exits.




3.3.  Reading Information Out Of The Database



After the shared lock is acquired, we can begin reading
information from the database file.  In this scenario, we
are assuming a cold cache, so information must first be
read from mass storage into the operating system cache then
transferred from operating system cache into user space.
On subsequent reads, some or all of the information might
already be found in the operating system cache and so only
the transfer to user space would be required.

Usually only a subset of the pages in the database file
are read.  In this example we are showing three
pages out of eight being read.  In a typical application, a
database will have thousands of pages and a query will normally
only touch a small percentage of those pages.





3.4.  Obtaining A Reserved Lock



Before making changes to the database, SQLite first
obtains a "reserved" lock on the database file.  A reserved
lock is similar to a shared lock in that both a reserved lock
and shared lock allow other processes to read from the database
file.  A single reserve lock can coexist with multiple shared
locks from other processes.  However, there can only be a
single reserved lock on the database file.  Hence only a
single process can be attempting to write to the database
at one time.

The idea behind a reserved lock is that it signals that
a process intends to modify the database file in the near
future but has not yet started to make the modifications.
And because the modifications have not yet started, other
processes can continue to read from the database.  However,
no other process should also begin trying to write to the
database.



3.5.  Creating A Rollback Journal File


Prior to making any changes to the database file, SQLite first
creates a separate rollback journal file and writes into the
rollback journal the original
content of the database pages that are to be altered.
The idea behind the rollback journal is that it contains
all information needed to restore the database back to
its original state.

The rollback journal contains a small header (shown in green
in the diagram) that records the original size of the database
file.  So if a change causes the database file to grow, we
will still know the original size of the database.  The page
number is stored together with each database page that is
written into the rollback journal.


  When a new file is created, most desktop operating systems
  (Windows, Linux, Mac OS X) will not actually write anything to
  disk.  The new file is created in the operating systems disk
  cache only.  The file is not created on mass storage until sometime
  later, when the operating system has a spare moment.  This creates
  the impression to users that I/O is happening much faster than
  is possible when doing real disk I/O.  We illustrate this idea in
  the diagram to the right by showing that the new rollback journal
  appears in the operating system disk cache only and not on the
  disk itself.



3.6.  Changing Database Pages In User Space


After the original page content has been saved in the rollback
journal, the pages can be modified in user memory.  Each database
connection has its own private copy of user space, so the changes
that are made in user space are only visible to the database connection
that is making the changes.  Other database connections still see
the information in operating system disk cache buffers which have
not yet been changed.  And so even though one process is busy
modifying the database, other processes can continue to read their
own copies of the original database content.



3.7.  Flushing The Rollback Journal File To Mass Storage


The next step is to flush the content of the rollback journal
file to nonvolatile storage.
As we will see later,
this is a critical step in insuring that the database can survive
an unexpected power loss.
This step also takes a lot of time, since writing to nonvolatile
storage is normally a slow operation.

This step is usually more complicated than simply flushing
the rollback journal to the disk.  On most platforms two separate
flush (or fsync()) operations are required.  The first flush writes
out the base rollback journal content.  Then the header of the
rollback journal is modified to show the number of pages in the
rollback journal.  Then the header is flushed to disk.  The details
on why we do this header modification and extra flush are provided
in a later section of this paper.



3.8.  Obtaining An Exclusive Lock


Prior to making changes to the database file itself, we must
obtain an exclusive lock on the database file.  Obtaining an
exclusive lock is really a two-step process.  First SQLite obtains
a "pending" lock.  Then it escalates the pending lock to an
exclusive lock.

A pending lock allows other processes that already have a
shared lock to continue reading the database file.  But it
prevents new shared locks from being established.  The idea
behind a pending lock is to prevent writer starvation caused
by a large pool of readers.  There might be dozens, even hundreds,
of other processes trying to read the database file.  Each process
acquires a shared lock before it starts reading, reads what it
needs, then releases the shared lock.  If, however, there are
many different processes all reading from the same database, it
might happen that a new process always acquires its shared lock before
the previous process releases its shared lock.  And so there is
never an instant when there are no shared locks on the database
file and hence there is never an opportunity for the writer to
seize the exclusive lock.  A pending lock is designed to prevent
that cycle by allowing existing shared locks to proceed but
blocking new shared locks from being established.  Eventually
all shared locks will clear and the pending lock will then be
able to escalate into an exclusive lock.



3.9.  Writing Changes To The Database File


Once an exclusive lock is held, we know that no other
processes are reading from the database file and it is
safe to write changes into the database file.  Usually
those changes only go as far as the operating systems disk
cache and do not make it all the way to mass storage.



3.10. 0 Flushing Changes To Mass Storage


Another flush must occur to make sure that all the
database changes are written into nonvolatile storage.
This is a critical step to ensure that the database will
survive a power loss without damage.  However, because
of the inherent slowness of writing to disk or flash memory,
this step together with the rollback journal file flush in section
3.7 above takes up most of the time required to complete a
transaction commit in SQLite.



3.11. 1 Deleting The Rollback Journal


After the database changes are all safely on the mass
storage device, the rollback journal file is deleted.
This is the instant where the transaction commits.
If a power failure or system crash occurs prior to this
point, then recovery processes to be described later make
it appear as if no changes were ever made to the database
file.  If a power failure or system crash occurs after
the rollback journal is deleted, then it appears as if
all changes have been written to disk.  Thus, SQLite gives
the appearance of having made no changes to the database
file or having made the complete set of changes to the
database file depending on whether or not the rollback
journal file exists.

Deleting a file is not really an atomic operation, but
it appears to be from the point of view of a user process.
A process is always able to ask the operating system "does
this file exist?" and the process will get back a yes or no
answer.  After a power failure that occurs during a
transaction commit, SQLite will ask the operating system
whether or not the rollback journal file exists.  If the
answer is "yes" then the transaction is incomplete and is
rolled back.  If the answer is "no" then it means the transaction
did commit.

The existence of a transaction depends on whether or
not the rollback journal file exists and the deletion
of a file appears to be an atomic operation from the point of
view of a user-space process.  Therefore,
a transaction appears to be an atomic operation.

The act of deleting a file is expensive on many systems.
As an optimization, SQLite can be configured to truncate
the journal file to zero bytes in length
or overwrite the journal file header with zeros.  In either
case, the resulting journal file is no longer capable of rolling
back and so the transaction still commits.  Truncating a file
to zero length, like deleting a file, is assumed to be an atomic
operation from the point of view of a user process.  Overwriting
the header of the journal with zeros is not atomic, but if any
part of the header is malformed the journal will not roll back.
Hence, one can say that the commit occurs as soon as the header
is sufficiently changed to make it invalid.  Typically this happens
as soon as the first byte of the header is zeroed.



3.12. 2 Releasing The Lock


The last step in the commit process is to release the
exclusive lock so that other processes can once again
start accessing the database file.

In the diagram at the right, we show that the information
that was held in user space is cleared when the lock is released.
This used to be literally true for older versions of SQLite.  But
more recent versions of SQLite keep the user space information
in memory in case it might be needed again at the start of the
next transaction.  It is cheaper to reuse information that is
already in local memory than to transfer the information back
from the operating system disk cache or to read it off of the
disk drive again.  Prior to reusing the information in user space,
we must first reacquire the shared lock and then we have to check
to make sure that no other process modified the database file while
we were not holding a lock.  There is a counter in the first page
of the database that is incremented every time the database file
is modified.  We can find out if another process has modified the
database by checking that counter.  If the database was modified,
then the user space cache must be cleared and reread.  But it is
commonly the case that no changes have been made and the user
space cache can be reused for a significant performance savings.




4.  Rollback

An atomic commit is supposed to happen instantaneously.  But the processing
described above clearly takes a finite amount of time.
Suppose the power to the computer were cut
part way through the commit operation described above.  In order
to maintain the illusion that the changes were instantaneous, we
have to "rollback" any partial changes and restore the database to
the state it was in prior to the beginning of the transaction.



4.1.  When Something Goes Wrong...


Suppose the power loss occurred
during step 3.10 above,
while the database changes were being written to disk.
After power is restored, the situation might be something
like what is shown to the right.  We were trying to change
three pages of the database file but only one page was
successfully written.  Another page was partially written
and a third page was not written at all.

The rollback journal is complete and intact on disk when
the power is restored.  This is a key point.  The reason for
the flush operation in step 3.7
is to make absolutely sure that
all of the rollback journal is safely on nonvolatile storage
prior to making any changes to the database file itself.



4.2.  Hot Rollback Journals


The first time that any SQLite process attempts to access
the database file, it obtains a shared lock as described in
section 3.2 above.
But then it notices that there is a
rollback journal file present.  SQLite then checks to see if
the rollback journal is a "hot journal".   A hot journal is
a rollback journal that needs to be played back in order to
restore the database to a sane state.  A hot journal only
exists when an earlier process was in the middle of committing
a transaction when it crashed or lost power.

A rollback journal is a "hot" journal if all of the following
are true:


The rollback journal exists.
The rollback journal is not an empty file.
There is no reserved lock on the main database file.
The header of the rollback journal is well-formed and in particular
    has not been zeroed out.
The rollback journal does not
contain the name of a super-journal file (see
section 5.5 below) or if does
contain the name of a super-journal, then that super-journal
file exists.


The presence of a hot journal is our indication
that a previous process was trying to commit a transaction but
it aborted for some reason prior to the completion of the
commit.  A hot journal means that
the database file is in an inconsistent state and needs to
be repaired (by rollback) prior to being used.




4.3.  Obtaining An Exclusive Lock On The Database


The first step toward dealing with a hot journal is to
obtain an exclusive lock on the database file.  This prevents two
or more processes from trying to rollback the same hot journal
at the same time.



4.4.  Rolling Back Incomplete Changes


Once a process obtains an exclusive lock, it is permitted
to write to the database file.  It then proceeds to read the
original content of pages out of the rollback journal and write
that content back to where it came from in the database file.
Recall that the header of the rollback journal records the original
size of the database file prior to the start of the aborted
transaction.  SQLite uses this information to truncate the
database file back to its original size in cases where the
incomplete transaction caused the database to grow.  At the
end of this step, the database should be the same size and
contain the same information as it did before the start of
the aborted transaction.




4.5.  Deleting The Hot Journal


After all information in the rollback journal has been
played back into the database file (and flushed to disk in case
we encounter yet another power failure), the hot rollback journal
can be deleted.

As in section 3.11, the journal
file might be truncated to zero length or its header might
be overwritten with zeros as an optimization on systems where
deleting a file is expensive.  Either way, the journal is no
longer hot after this step.




4.6.  Continue As If The Uncompleted Writes Had Never Happened


The final recovery step is to reduce the exclusive lock back
to a shared lock.  Once this happens, the database is back in the
state that it would have been if the aborted transaction had never
started.  Since all of this recovery activity happens completely
automatically and transparently, it appears to the program using
SQLite as if the aborted transaction had never begun.




5.  Multi-file Commit

SQLite allows a single
database connection to talk to
two or more database files simultaneously through the use of
the ATTACH DATABASE command.
When multiple database files are modified within a single
transaction, all files are updated atomically.
In other words, either all of the database files are updated or
else none of them are.
Achieving an atomic commit across multiple database files is
more complex that doing so for a single file.  This section
describes how SQLite works that bit of magic.



5.1.  Separate Rollback Journals For Each Database


When multiple database files are involved in a transaction,
each database has its own rollback journal and each database
is locked separately.  The diagram at the right shows a scenario
where three different database files have been modified within
one transaction.  The situation at this step is analogous to
the single-file transaction scenario at
step 3.6.  Each database file has
a reserved lock.  For each database, the original content of pages
that are being changed have been written into the rollback journal
for that database, but the content of the journals have not yet
been flushed to disk.  No changes have been made to the database
file itself yet, though presumably there are changes being held
in user memory.

For brevity, the diagrams in this section are simplified from
those that came before.  Blue color still signifies original content
and pink still signifies new content.  But the individual pages
in the rollback journal and the database file are not shown and
we are not making the distinction between information in the
operating system cache and information that is on disk.  All of
these factors still apply in a multi-file commit scenario.  They
just take up a lot of space in the diagrams and they do not add
any new information, so they are omitted here.




5.2.  The Super-Journal File


The next step in a multi-file commit is the creation of a
"super-journal" file.  The name of the super-journal file is
the same name as the original database filename (the database
that was opened using the
sqlite3_open() interface,
not one of the ATTACHed auxiliary
databases) with the text "-mjHHHHHHHH" appended where
HHHHHHHH is a random 32-bit hexadecimal number.  The
random HHHHHHHH suffix changes for every new super-journal.

(Nota bene: The formula for computing the super-journal filename
given in the previous paragraph corresponds to the implementation as
of SQLite version 3.5.0.  But this formula is not part of the SQLite
specification and is subject to change in future releases.)

Unlike the rollback journals, the super-journal does not contain
any original database page content.  Instead, the super-journal contains
the full pathnames for rollback journals for every database that is
participating in the transaction.

After the super-journal is constructed, its content is flushed
to disk before any further actions are taken.  On Unix, the directory
that contains the super-journal is also synced in order to make sure
the super-journal file will appear in the directory following a
power failure.

The purpose of the super-journal is to ensure that multi-file
transactions are atomic across a power-loss.  But if the database files
have other settings that compromise integrity across a power-loss event
(such as PRAGMA synchronous=OFF or PRAGMA journal_mode=MEMORY) then
the creation of the super-journal is omitted, as an optimization.




5.3.  Updating Rollback Journal Headers


The next step is to record the full pathname of the super-journal file
in the header of every rollback journal.  Space to hold the
super-journal filename was reserved at the beginning of each rollback journal
as the rollback journals were created.

The content of each rollback journal is flushed to disk both before
and after the super-journal filename is written into the rollback
journal header.  It is important to do both of these flushes.  Fortunately,
the second flush is usually inexpensive since typically only a single
page of the journal file (the first page) has changed.

This step is analogous to
step 3.7 in the single-file commit
scenario described above.




5.4.  Updating The Database Files


Once all rollback journal files have been flushed to disk, it
is safe to begin updating database files.  We have to obtain an
exclusive lock on all database files before writing the changes.
After all the changes are written, it is important to flush the
changes to disk so that they will be preserved in the event of
a power failure or operating system crash.

This step corresponds to steps
3.8,
3.9, and
3.10 in the single-file commit
scenario described previously.




5.5.  Delete The Super-Journal File


The next step is to delete the super-journal file.
This is the point where the multi-file transaction commits.
This step corresponds to
step 3.11 in the single-file
commit scenario where the rollback journal is deleted.

If a power failure or operating system crash occurs at this
point, the transaction will not rollback when the system reboots
even though there are rollback journals present.  The
difference is the super-journal pathname in the header of the
rollback journal.  Upon restart, SQLite only considers a journal
to be hot and will only playback the journal if there is no
super-journal filename in the header (which is the case for
a single-file commit) or if the super-journal file still
exists on disk.




5.6.  Clean Up The Rollback Journals


The final step in a multi-file commit is to delete the
individual rollback journals and drop the exclusive locks on
the database files so that other processes can see the changes.
This corresponds to
step 3.12 in the single-file
commit sequence.

The transaction has already committed at this point so timing
is not critical in the deletion of the rollback journals.
The current implementation deletes a single rollback journal
then unlocks the corresponding database file before proceeding
to the next rollback journal.  But in the future we might change
this so that all rollback journals are deleted before any database
files are unlocked.  As long as the rollback journal is deleted before
its corresponding database file is unlocked it does not matter in what
order the rollback journals are deleted or the database files are
unlocked.



6.  Additional Details Of The Commit Process

Section 3.0 above provides an overview of
how atomic commit works in SQLite.  But it glosses over a number of
important details.  The following subsections will attempt to fill
in the gaps.



6.1.  Always Journal Complete Sectors

When the original content of a database page is written into
the rollback journal (as shown in section 3.5),
SQLite always writes a complete sector of data, even if the
page size of the database is smaller than the sector size.
Historically, the sector size in SQLite has been hard coded to 512
bytes and since the minimum page size is also 512 bytes, this has never
been an issue.  But beginning with SQLite version 3.3.14, it is possible
for SQLite to use mass storage devices with a sector size larger than 512
bytes.  So, beginning with version 3.3.14, whenever any page within a
sector is written into the journal file, all pages in that same sector
are stored with it.

It is important to store all pages of a sector in the rollback
journal in order to prevent database corruption following a power
loss while writing the sector.  Suppose that pages 1, 2, 3, and 4 are
all stored in sector 1 and that page 2 is modified.  In order to write
the changes to page 2, the underlying hardware must also rewrite the
content of pages 1, 3, and 4 since the hardware must write the complete
sector.  If this write operation is interrupted by a power outage,
one or more of the pages 1, 3, or 4 might be left with incorrect data.
Hence, to avoid lasting corruption to the database, the original content
of all of those pages must be contained in the rollback journal.



6.2.  Dealing With Garbage Written Into Journal Files

When data is appended to the end of the rollback journal,
SQLite normally makes the pessimistic assumption that the file
is first extended with invalid "garbage" data and that afterwards
the correct data replaces the garbage.  In other words, SQLite assumes
that the file size is increased first and then afterwards the content
is written into the file.  If a power failure occurs after the file
size has been increased but before the file content has been written,
the rollback journal can be left containing garbage data.  If after
power is restored, another SQLite process sees the rollback journal
containing the garbage data and tries to roll it back into the original
database file, it might copy some of the garbage into the database file
and thus corrupt the database file.

SQLite uses two defenses against this problem.  In the first place,
SQLite records the number of pages in the rollback journal in the header
of the rollback journal.  This number is initially zero.  So during an
attempt to rollback an incomplete (and possibly corrupt) rollback
journal, the process doing the rollback will see that the journal
contains zero pages and will thus make no changes to the database.  Prior
to a commit, the rollback journal is flushed to disk to ensure that
all content has been synced to disk and there is no "garbage" left
in the file, and only then is the page count in the header changed from
zero to true number of pages in the rollback journal.  The rollback journal
header is always kept in a separate sector from any page data so that
it can be overwritten and flushed without risking damage to a data
page if a power outage occurs.  Notice that the rollback journal
is flushed to disk twice: once to write the page content and a second
time to write the page count in the header.

The previous paragraph describes what happens when the
synchronous pragma setting is "full".


PRAGMA synchronous=FULL;


The default synchronous setting is full so the above is what usually
happens.  However, if the synchronous setting is lowered to "normal",
SQLite only flushes the rollback journal once, after the page count has
been written.
This carries a risk of corruption because it might happen that the
modified (non-zero) page count reaches the disk surface before all
of the data does.  The data will have been written first, but SQLite
assumes that the underlying filesystem can reorder write requests and
that the page count can be burned into oxide first even though its
write request occurred last.  So as a second line of defense, SQLite
also uses a 32-bit checksum on every page of data in the rollback
journal.  This checksum is evaluated for each page during rollback
while rolling back a journal as described in
section 4.4.  If an incorrect checksum
is seen, the rollback is abandoned.  Note that the checksum does
not guarantee that the page data is correct since there is a small
but finite probability that the checksum might be right even if the data is
corrupt.  But the checksum does at least make such an error unlikely.


Note that the checksums in the rollback journal are not necessary
if the synchronous setting is FULL.  We only depend on the checksums
when synchronous is lowered to NORMAL.  Nevertheless, the checksums
never hurt and so they are included in the rollback journal regardless
of the synchronous setting.



6.3.  Cache Spill Prior To Commit

The commit process shown in section 3.0
assumes that all database changes fit in memory until it is time to
commit.  This is the common case.  But sometimes a larger change will
overflow the user-space cache prior to transaction commit.  In those
cases, the cache must spill to the database before the transaction
is complete.

At the beginning of a cache spill, the status of the database
connection is as shown in step 3.6.
Original page content has been saved in the rollback journal and
modifications of the pages exist in user memory.  To spill the cache,
SQLite executes steps 3.7 through
3.9.  In other words, the rollback journal
is flushed to disk, an exclusive lock is acquired, and changes are
written into the database.  But the remaining steps are deferred
until the transaction really commits.  A new journal header is
appended to the end of the rollback journal (in its own sector)
and the exclusive database lock is retained, but otherwise processing
returns to step 3.6.  When the transaction
commits, or if another cache spill occurs, steps
3.7 and 3.9 are
repeated.  (Step 3.8 is omitted on second
and subsequent passes since an exclusive database lock is already held
due to the first pass.)

A cache spill causes the lock on the database file to
escalate from reserved to exclusive.  This reduces concurrency.
A cache spill also causes extra disk flush or fsync operations to
occur and these operations are slow, hence a cache spill can
seriously reduce performance.
For these reasons a cache spill is avoided whenever possible.



7.  Optimizations

Profiling indicates that for most systems and in most circumstances
SQLite spends most of its time doing disk I/O.  It follows then that
anything we can do to reduce the amount of disk I/O will likely have a
large positive impact on the performance of SQLite.  This section
describes some of the techniques used by SQLite to try to reduce the
amount of disk I/O to a minimum while still preserving atomic commit.



7.1.  Cache Retained Between Transactions

Step 3.12 of the commit process shows
that once the shared lock has been released, all user-space cache
images of database content must be discarded.  This is done because
without a shared lock, other processes are free to modify the database
file content and so any user-space image of that content might become
obsolete.  Consequently, each new transaction would begin by rereading
data which had previously been read.  This is not as bad as it sounds
at first since the data being read is still likely in the operating
systems file cache.  So the "read" is really just a copy of data
from kernel space into user space.  But even so, it still takes time.

Beginning with SQLite version 3.3.14 a mechanism has been added
to try to reduce the needless rereading of data.  In newer versions
of SQLite, the data in the user-space pager cache is retained when
the lock on the database file is released.  Later, after the
shared lock is acquired at the beginning of the next transaction,
SQLite checks to see if any other process has modified the database
file.  If the database has been changed in any way since the lock
was last released, the user-space cache is erased at that point.
But commonly the database file is unchanged and the user-space cache
can be retained, and some unnecessary read operations can be avoided.

In order to determine whether or not the database file has changed,
SQLite uses a counter in the database header (in bytes 24 through 27)
which is incremented during every change operation.  SQLite saves a copy
of this counter prior to releasing its database lock.  Then after
acquiring the next database lock it compares the saved counter value
against the current counter value and erases the cache if the values
are different, or reuses the cache if they are the same.


7.2.  Exclusive Access Mode

SQLite version 3.3.14 adds the concept of "Exclusive Access Mode".
In exclusive access mode, SQLite retains the exclusive
database lock at the conclusion of each transaction.  This prevents
other processes from accessing the database, but in many deployments
only a single process is using a database so this is not a
serious problem.  The advantage of exclusive access mode is that
disk I/O can be reduced in three ways:


It is not necessary to increment the change counter in the
database header for transactions after the first transaction.  This
will often save a write of page one to both the rollback
journal and the main database file.

No other processes can change the database so there is never
a need to check the change counter and clear the user-space cache
at the beginning of a transaction.

Each transaction can be committed by overwriting the rollback
journal header with zeros rather than deleting the journal file.
This avoids having to modify the directory entry for the journal file
and it avoids having to deallocate disk sectors associated with the
journal.  Furthermore, the next transaction will overwrite existing
journal file content rather than append new content and on most systems
overwriting is much faster than appending.


The third optimization, zeroing the journal file header rather than
deleting the rollback journal file,
does not depend on holding an exclusive lock at all times.
This optimization can be set independently of exclusive lock mode
using the journal_mode pragma
as described in section 7.6 below.



7.3.  Do Not Journal Freelist Pages

When information is deleted from an SQLite database, the pages used
to hold the deleted information are added to a "freelist".  Subsequent
inserts will draw pages off of this freelist rather than expanding the
database file.

Some freelist pages contain critical data; specifically the locations
of other freelist pages.  But most freelist pages contain nothing useful.
These latter freelist pages are called "leaf" pages.  We are free to
modify the content of a leaf freelist page in the database without
changing the meaning of the database in any way.

Because the content of leaf freelist pages is unimportant, SQLite
avoids storing leaf freelist page content in the rollback journal
in step 3.5 of the commit process.
If a leaf freelist page is changed and that change does not get rolled back
during a transaction recovery, the database is not harmed by the omission.
Similarly, the content of a new freelist page is never written back
into the database at step 3.9 nor
read from the database at step 3.3.
These optimizations can greatly reduce the amount of I/O that occurs
when making changes to a database file that contains free space.



7.4.  Single Page Updates And Atomic Sector Writes

Beginning in SQLite version 3.5.0, the new Virtual File System (VFS)
interface contains a method named xDeviceCharacteristics which reports
on special properties that the underlying mass storage device
might have.  Among the special properties that
xDeviceCharacteristics might report is the ability of to do an
atomic sector write.

Recall that by default SQLite assumes that sector writes are
linear but not atomic.  A linear write starts at one end of the
sector and changes information byte by byte until it gets to the
other end of the sector.  If a power loss occurs in the middle of
a linear write then part of the sector might be modified while the
other end is unchanged.  In an atomic sector write, either the entire
sector is overwritten or else nothing in the sector is changed.

We believe that most modern disk drives implement atomic sector
writes.  When power is lost, the drive uses energy stored in capacitors
and/or the angular momentum of the disk platter to provide power to
complete any operation in progress.  Nevertheless, there are so many
layers in between the write system call and the on-board disk drive
electronics that we take the safe approach in both Unix and w32 VFS
implementations and assume that sector writes are not atomic.  On the
other hand, device
manufacturers with more control over their filesystems might want
to consider enabling the atomic write property of xDeviceCharacteristics
if their hardware really does do atomic writes.

When sector writes are atomic and the page size of a database is
the same as a sector size, and when there is a database change that
only touches a single database page, then SQLite skips the whole
journaling and syncing process and simply writes the modified page
directly into the database file.  The change counter in the first
page of the database file is modified separately since no harm is
done if power is lost before the change counter can be updated.



7.5.  Filesystems With Safe Append Semantics

Another optimization introduced in SQLite version 3.5.0 makes
use of "safe append" behavior of the underlying disk.
Recall that SQLite assumes that when data is appended to a file
(specifically to the rollback journal) that the size of the file
is increased first and that the content is written second.  So
if power is lost after the file size is increased but before the
content is written, the file is left containing invalid "garbage"
data.  The xDeviceCharacteristics method of the VFS might, however,
indicate that the filesystem implements "safe append" semantics.
This means that the content is written before the file size is
increased so that it is impossible for garbage to be introduced
into the rollback journal by a power loss or system crash.

When safe append semantics are indicated for a filesystem,
SQLite always stores the special value of -1 for the page count
in the header of the rollback journal.  The -1 page count value
tells any process attempting to rollback the journal that the
number of pages in the journal should be computed from the journal
size.  This -1 value is never changed.  So that when a commit
occurs, we save a single flush operation and a sector write of
the first page of the journal file.  Furthermore, when a cache
spill occurs we no longer need to append a new journal header
to the end of the journal; we can simply continue appending
new pages to the end of the existing journal.


7.6.  Persistent Rollback Journals

Deleting a file is an expensive operation on many systems.
So as an optimization, SQLite can be configured to avoid the
delete operation of section 3.11.
Instead of deleting the journal file in order to commit a transaction,
the file is either truncated to zero bytes in length or its
header is overwritten with zeros.  Truncating the file to zero
length saves having to make modifications to the directory containing
the file since the file is not removed from the directory.
Overwriting the header has the additional savings of not having
to update the length of the file (in the "inode" on many systems)
and not having to deal with newly freed disk sectors.  Furthermore,
at the next transaction the journal will be created by overwriting
existing content rather than appending new content onto the end
of a file, and overwriting is often much faster than appending.

SQLite can be configured to commit transactions by overwriting
the journal header with zeros instead of deleting the journal file
by setting the "PERSIST" journaling mode using the
journal_mode PRAGMA.
For example:


PRAGMA journal_mode=PERSIST;


The use of persistent journal mode provides a noticeable performance
improvement on many systems.  Of course, the drawback is that the
journal files remain on the disk, using disk space and cluttering
directories, long after the transaction commits.  The only safe way
to delete a persistent journal file is to commit a transaction
with journaling mode set to DELETE:


PRAGMA journal_mode=DELETE;
BEGIN EXCLUSIVE;
COMMIT;


Beware of deleting persistent journal files by any other means
since the journal file might be hot, in which case deleting it will
corrupt the corresponding database file.

Beginning in SQLite version 3.6.4 (2008-10-15),
the TRUNCATE journal mode is
also supported:


PRAGMA journal_mode=TRUNCATE;


In truncate journal mode, the transaction is committed by truncating
the journal file to zero length rather than deleting the journal file
(as in DELETE mode) or by zeroing the header (as in PERSIST mode).
TRUNCATE mode shares the advantage of PERSIST mode that the directory
that contains the journal file and database does not need to be updated.
Hence truncating a file is often faster than deleting it.  TRUNCATE has
the additional advantage that it is not followed by a
system call (ex: fsync()) to synchronize the change to disk.  It might
be safer if it did.
But on many modern filesystems, a truncate is an atomic and
synchronous operation and so we think that TRUNCATE will usually be safe
in the face of power failures.  If you are uncertain about whether or
not TRUNCATE will be synchronous and atomic on your filesystem and it is
important to you that your database survive a power loss or operating
system crash that occurs during the truncation operation, then you might
consider using a different journaling mode.

On embedded systems with synchronous filesystems, TRUNCATE results
in slower behavior than PERSIST.  The commit operation is the same speed.
But subsequent transactions are slower following a TRUNCATE because it is
faster to overwrite existing content than to append to the end of a file.
New journal file entries will always be appended following a TRUNCATE but
will usually overwrite with PERSIST.



8.  Testing Atomic Commit Behavior

The developers of SQLite are confident that it is robust
in the face of power failures and system crashes because the
automatic test procedures do extensive checks on
the ability of SQLite to recover from simulated power loss.
We call these the "crash tests".

Crash tests in SQLite use a modified VFS that can simulate
the kinds of filesystem damage that occur during a power
loss or operating system crash.  The crash-test VFS can simulate
incomplete sector writes, pages filled with garbage data because
a write has not completed, and out of order writes, all occurring
at varying points during a test scenario.  Crash tests execute
transactions over and over, varying the time at which a simulated
power loss occurs and the properties of the damage inflicted.
Each test then reopens the database after the simulated crash and
verifies that the transaction either occurred completely
or not at all and that the database is in a completely
consistent state.

The crash tests in SQLite have discovered a number of very
subtle bugs (now fixed) in the recovery mechanism.  Some of
these bugs were very obscure and unlikely to have been found
using only code inspection and analysis techniques.  From this
experience, the developers of SQLite feel confident that any other
database system that does not use a similar crash test system
likely contains undetected bugs that will lead to database
corruption following a system crash or power failure.



9.  Things That Can Go Wrong

The atomic commit mechanism in SQLite has proven to be robust,
but it can be circumvented by a sufficiently creative
adversary or a sufficiently broken operating system implementation.
This section describes a few of the ways in which an SQLite database
might be corrupted by a power failure or system crash.
(See also: How To Corrupt Your Database Files.)



9.1.  Broken Locking Implementations

SQLite uses filesystem locks to make sure that only one
process and database connection is trying to modify the database
at a time.  The filesystem locking mechanism is implemented
in the VFS layer and is different for every operating system.
SQLite depends on this implementation being correct.  If something
goes wrong and two or more processes are able to write the same
database file at the same time, severe damage can result.

We have received reports of implementations of both
Windows network filesystems and NFS in which locking was
subtly broken.  We can not verify these reports, but as
locking is difficult to get right on a network filesystem
we have no reason to doubt them.  You are advised to
avoid using SQLite on a network filesystem in the first place,
since performance will be slow.  But if you must use a
network filesystem to store SQLite database files, consider
using a secondary locking mechanism to prevent simultaneous
writes to the same database even if the native filesystem
locking mechanism malfunctions.

The versions of SQLite that come preinstalled on Apple
Mac OS X computers contain a version of SQLite that has been
extended to use alternative locking strategies that work on
all network filesystems that Apple supports.  These extensions
used by Apple work great as long as all processes are accessing
the database file in the same way.  Unfortunately, the locking
mechanisms do not exclude one another, so if one process is
accessing a file using (for example) AFP locking and another
process (perhaps on a different machine) is using dot-file locks,
the two processes might collide because AFP locks do not exclude
dot-file locks or vice versa.



9.2.  Incomplete Disk Flushes

SQLite uses the fsync() system call on Unix and the FlushFileBuffers()
system call on w32 in order to sync the file system buffers onto disk
oxide as shown in step 3.7 and
step 3.10.  Unfortunately, we have received
reports that neither of these interfaces works as advertised on many
systems.  We hear that FlushFileBuffers() can be completely disabled
using registry settings on some Windows versions.  Some historical
versions of Linux contain versions of fsync() which are no-ops on
some filesystems, we are told.  Even on systems where
FlushFileBuffers() and fsync() are said to be working, often
the IDE disk control lies and says that data has reached oxide
while it is still held only in the volatile control cache.

On the Mac, you can set this pragma:


PRAGMA fullfsync=ON;


Setting fullfsync on a Mac will guarantee that data really does
get pushed out to the disk platter on a flush.  But the implementation
of fullfsync involves resetting the disk controller.  And so not only
is it profoundly slow, it also slows down other unrelated disk I/O.
So its use is not recommended.



9.3.  Partial File Deletions

SQLite assumes that file deletion is an atomic operation from the
point of view of a user process.  If power fails in the middle of
a file deletion, then after power is restored SQLite expects to see
either the entire file with all of its original data intact, or it
expects not to find the file at all.  Transactions may not be atomic
on systems that do not work this way.



9.4.  Garbage Written Into Files

SQLite database files are ordinary disk files that can be
opened and written by ordinary user processes.  A rogue process
can open an SQLite database and fill it with corrupt data.
Corrupt data might also be introduced into an SQLite database
by bugs in the operating system or disk controller; especially
bugs triggered by a power failure.  There is nothing SQLite can
do to defend against these kinds of problems.



9.5.  Deleting Or Renaming A Hot Journal

If a crash or power loss does occur and a hot journal is left on
the disk, it is essential that the original database file and the hot
journal remain on disk with their original names until the database
file is opened by another SQLite process and rolled back.
During recovery at step 4.2 SQLite locates
the hot journal by looking for a file in the same directory as the
database being opened and whose name is derived from the name of the
file being opened.  If either the original database file or the
hot journal have been moved or renamed, then the hot journal will
not be seen and the database will not be rolled back.

We suspect that a common failure mode for SQLite recovery happens
like this:  A power failure occurs.  After power is restored, a well-meaning
user or system administrator begins looking around on the disk for
damage.  They see their database file named "important.data".  This file
is perhaps familiar to them.  But after the crash, there is also a
hot journal named "important.data-journal".  The user then deletes
the hot journal, thinking that they are helping to cleanup the system.
We know of no way to prevent this other than user education.

If there are multiple (hard or symbolic) links to a database file,
the journal will be created using the name of the link through which
the file was opened.  If a crash occurs and the database is opened again
using a different link, the hot journal will not be located and no
rollback will occur.

Sometimes a power failure will cause a filesystem to be corrupted
such that recently changed filenames are forgotten and the file is
moved into a "/lost+found" directory.  When that happens, the hot
journal will not be found and recovery will not occur.
SQLite tries to prevent this
by opening and syncing the directory containing the rollback journal
at the same time it syncs the journal file itself.  However, the
movement of files into /lost+found can be caused by unrelated processes
creating unrelated files in the same directory as the main database file.
And since this is out from under the control of SQLite, there is nothing
that SQLite can do to prevent it.  If you are running on a system that
is vulnerable to this kind of filesystem namespace corruption (most
modern journalling filesystems are immune, we believe) then you might
want to consider putting each SQLite database file in its own private
subdirectory.



10.  Future Directions And Conclusion

Every now and then someone discovers a new failure mode for
the atomic commit mechanism in SQLite and the developers have to
put in a patch.  This is happening less and less and the
failure modes are becoming more and more obscure.  But it would
still be foolish to suppose that the atomic commit logic of
SQLite is entirely bug-free.  The developers are committed to fixing
these bugs as quickly as they might be found.


The developers are also on the lookout for new ways to
optimize the commit mechanism.  The current VFS implementations
for Unix (Linux and Mac OS X) and Windows make pessimistic assumptions about
the behavior of those systems.  After consultation with experts
on how these systems work, we might be able to relax some of the
assumptions on these systems and allow them to run faster.  In
particular, we suspect that most modern filesystems exhibit the
safe append property and that many of them might support atomic
sector writes.  But until this is known for certain, SQLite will
take the conservative approach and assume the worst.
This page last modified on  2022-12-31 21:51:03 UTC


c2: SQLite In 5 Minutes Or Less
c5: quickstart.html
c4:
Here is what you do to start experimenting with SQLite without having
to do a lot of tedious reading and configuration:

Download The Code


Get a copy of the prebuilt binaries for your machine, or get a copy
of the sources and compile them yourself.  Visit
the download page for more information.


Create A New Database


At a shell or DOS prompt, enter: "sqlite3 test.db".  This will
create a new database named "test.db".  (You can use a different name if
you like.)
Enter SQL commands at the prompt to create and populate the
new database.
Additional documentation is available here.


Write Programs That Use SQLite


Below is a simple 
TCL program that demonstrates how to use
the TCL interface to SQLite.  The program executes the SQL statements
given as the second argument on the database defined by the first
argument.  The commands to watch for are the sqlite3 command
on line 7 which opens an SQLite database and creates
a new object named "db" to access that database, the
use of the eval method on the db object on line 8 to run
SQL commands against the database, and the closing of the database connection
on the last line of the script.


01  #!/usr/bin/tclsh
02  if {$argc!=2} {
03    puts stderr "Usage: %s DATABASE SQL-STATEMENT"
04    exit 1
05  }
06  package require sqlite3
07  sqlite3 db &#91;lindex $argv 0]
08  db eval &#91;lindex $argv 1] x {
09    foreach v $x(*) {
10      puts "$v = $x($v)"
11    }
12    puts ""
13  }
14  db close



Below is a simple C program that demonstrates how to use
the C/C++ interface to SQLite.  The name of a database is given by
the first argument and the second argument is one or more SQL statements
to execute against the database.  The function calls to pay attention
to here are the call to sqlite3_open() on line 22 which opens
the database, sqlite3_exec() on line 28 that executes SQL
commands against the database, and sqlite3_close() on line 33
that closes the database connection.

See also the Introduction To The SQLite C/C++ Interface for
an introductory overview and roadmap to the dozens of SQLite interface
functions.


01  #include &lt;stdio.h&gt;
02  #include &lt;sqlite3.h&gt;
03  
04  static int callback(void *NotUsed, int argc, char **argv, char **azColName){
05    int i;
06    for(i=0; i&lt;argc; i++){
07      printf("%s = %s\n", azColName&#91;i], argv&#91;i] ? argv&#91;i] : "NULL");
08    }
09    printf("\n");
10    return 0;
11  }
12  
13  int main(int argc, char **argv){
14    sqlite3 *db;
15    char *zErrMsg = 0;
16    int rc;
17  
18    if( argc!=3 ){
19      fprintf(stderr, "Usage: %s DATABASE SQL-STATEMENT\n", argv&#91;0]);
20      return(1);
21    }
22    rc = sqlite3_open(argv&#91;1], &db);
23    if( rc ){
24      fprintf(stderr, "Can't open database: %s\n", sqlite3_errmsg(db));
25      sqlite3_close(db);
26      return(1);
27    }
28    rc = sqlite3_exec(db, argv&#91;2], callback, 0, &zErrMsg);
29    if( rc!=SQLITE_OK ){
30      fprintf(stderr, "SQL error: %s\n", zErrMsg);
31      sqlite3_free(zErrMsg);
32    }
33    sqlite3_close(db);
34    return 0;
35  }

See the How To Compile SQLite document for instructions and hints on
how to compile the program shown above.


This page last modified on  2024-07-25 15:06:57 UTC


c2: Most Widely Deployed SQL Database Engine
c5: mostdeployed.html
c4:
Most Widely Deployed and Used Database Engine

SQLite is likely used more than all other database engines
combined.  Billions and billions of copies
of SQLite exist in the wild.  SQLite is found in:


Every Android device
Every iPhone and iOS device
Every Mac
Every Windows10 machine
Every Firefox, Chrome, and Safari web browser
Every instance of Skype
Every instance of iTunes
Every Dropbox client
Every TurboTax and QuickBooks
PHP and Python
Most television sets and set-top cable boxes
Most automotive multimedia systems
Countless millions of other applications


Since SQLite is used extensively in every smartphone, and there are
more than 4.0 billion (4.0e9) smartphones in active use, each holding
hundreds of
SQLite database files, it is seems likely that there are over one trillion
(1e12) SQLite databases in active use.

Most Widely Deployed Software Module of Any Type?

SQLite is probably one of the top five most deployed
software modules of any description.  
Other libraries with similar reach include:



The original zlib implementation by
Jean-loup Gailly and Mark Adler.

The original reference implementation for
libpng

Libjpeg from the Independent JPEG Group



Libc is omitted from the above list even though
it is more common than SQLite because
libc is not a single software component but rather several
competing implementations (ex: BSD vs. GNU) with similar interfaces.
There are also independent implementations of libjpeg and libpng,
though in those cases the canonical implementations are very popular.



Precise numbers are difficult to obtain and so exact rankings
are impossible.  But our best guess is that SQLite is the second
mostly widely deployed software library, after libz.
Some commentators observe that SQLite tends
to be statically linked and thus have multiple instances on
each machine, whereas libz tends to have just
a single instance per machine in the form of a shared library or DLL.
So even though the number of devices containing libz
may be greater than the number of
devices that contain SQLite, the total number of instances per device
might be higher for SQLite and so SQLite might
be the single most widely deployed and used software component.

This page last modified on  2022-01-08 05:02:57 UTC


c2: C/C++ Interface For SQLite Version 3 (old)
c5: capi3.html
c4:
Note:
This document was written in 2004 as a guide to helping programmers
move from using SQLite version 2 to SQLite version 3.  The information
in this document is still essentially correct, however there have been
many changes and enhancements over the years.  We recommend that the
following documents be used instead:

An Introduction To The SQLite C/C++ Interface
SQLite C/C++ Reference Guide




C/C++ Interface For SQLite Version 3

1.0 Overview


SQLite version 3.0 is a new version of SQLite, derived from
the SQLite 2.8.13 code base, but with an incompatible file format
and API.
SQLite version 3.0 was created to answer demand for the following features:



Support for UTF-16.
User-definable text collating sequences.
The ability to store BLOBs in indexed columns.



It was necessary to move to version 3.0 to implement these features because
each requires incompatible changes to the database file format.  Other
incompatible changes, such as a cleanup of the API, were introduced at the
same time under the theory that it is best to get your incompatible changes
out of the way all at once.  



The API for version 3.0 is similar to the version 2.X API,
but with some important changes.  Most noticeably, the "sqlite_"
prefix that occurs on the beginning of all API functions and data
structures are changed to "sqlite3_".  
This avoids confusion between the two APIs and allows linking against both
SQLite 2.X and SQLite 3.0 at the same time.



There is no agreement on what the C datatype for a UTF-16
string should be.  Therefore, SQLite uses a generic type of void*
to refer to UTF-16 strings.  Client software can cast the void* 
to whatever datatype is appropriate for their system.


2.0 C/C++ Interface


The API for SQLite 3.0 includes 83 separate functions in addition
to several data structures and #defines.  (A complete
API reference is provided as a separate 
document.)
Fortunately, the interface is not nearly as complex as its size implies.
Simple programs can still make do with only 3 functions:
sqlite3_open(), sqlite3_exec(), and sqlite3_close().
More control over the execution of the database engine is provided
using sqlite3_prepare_v2()
to compile an SQLite statement into byte code and
sqlite3_step() to execute that bytecode.
A family of routines with names beginning with 
sqlite3_column_
is used to extract information about the result set of a query.
Many interface functions come in pairs, with both a UTF-8 and
UTF-16 version.  And there is a collection of routines
used to implement user-defined SQL functions and user-defined
text collating sequences.



2.1 Opening and closing a database


   typedef struct sqlite3 sqlite3;
   int sqlite3_open(const char*, sqlite3**);
   int sqlite3_open16(const void*, sqlite3**);
   int sqlite3_close(sqlite3*);
   const char *sqlite3_errmsg(sqlite3*);
   const void *sqlite3_errmsg16(sqlite3*);
   int sqlite3_errcode(sqlite3*);



The sqlite3_open() routine returns an integer error code rather than
a pointer to the sqlite3 structure as the version 2 interface did.
The difference between sqlite3_open()
and sqlite3_open16() is that sqlite3_open16() takes UTF-16 (in host native
byte order) for the name of the database file.  If a new database file
needs to be created, then sqlite3_open16() sets the internal text
representation to UTF-16 whereas sqlite3_open() sets the text
representation to UTF-8.



The opening and/or creating of the database file is deferred until the
file is actually needed.  This allows options and parameters, such
as the native text representation and default page size, to be
set using PRAGMA statements.



The sqlite3_errcode() routine returns a result code for the most
recent major API call.  sqlite3_errmsg() returns an English-language
text error message for the most recent error.  The error message is
represented in UTF-8 and will be ephemeral - it could disappear on
the next call to any SQLite API function.  sqlite3_errmsg16() works like
sqlite3_errmsg() except that it returns the error message represented
as UTF-16 in host native byte order.



The error codes for SQLite version 3 are unchanged from version 2.
They are as follows:



#define SQLITE_OK           0   /* Successful result */
#define SQLITE_ERROR        1   /* SQL error or missing database */
#define SQLITE_INTERNAL     2   /* An internal logic error in SQLite */
#define SQLITE_PERM         3   /* Access permission denied */
#define SQLITE_ABORT        4   /* Callback routine requested an abort */
#define SQLITE_BUSY         5   /* The database file is locked */
#define SQLITE_LOCKED       6   /* A table in the database is locked */
#define SQLITE_NOMEM        7   /* A malloc() failed */
#define SQLITE_READONLY     8   /* Attempt to write a readonly database */
#define SQLITE_INTERRUPT    9   /* Operation terminated by sqlite_interrupt() */
#define SQLITE_IOERR       10   /* Some kind of disk I/O error occurred */
#define SQLITE_CORRUPT     11   /* The database disk image is malformed */
#define SQLITE_NOTFOUND    12   /* (Internal Only) Table or record not found */
#define SQLITE_FULL        13   /* Insertion failed because database is full */
#define SQLITE_CANTOPEN    14   /* Unable to open the database file */
#define SQLITE_PROTOCOL    15   /* Database lock protocol error */
#define SQLITE_EMPTY       16   /* (Internal Only) Database table is empty */
#define SQLITE_SCHEMA      17   /* The database schema changed */
#define SQLITE_TOOBIG      18   /* Too much data for one row of a table */
#define SQLITE_CONSTRAINT  19   /* Abort due to contraint violation */
#define SQLITE_MISMATCH    20   /* Data type mismatch */
#define SQLITE_MISUSE      21   /* Library used incorrectly */
#define SQLITE_NOLFS       22   /* Uses OS features not supported on host */
#define SQLITE_AUTH        23   /* Authorization denied */
#define SQLITE_ROW         100  /* sqlite_step() has another row ready */
#define SQLITE_DONE        101  /* sqlite_step() has finished executing */


2.2 Executing SQL statements


   typedef int (*sqlite_callback)(void*,int,char**, char**);
   int sqlite3_exec(sqlite3*, const char *sql, sqlite_callback, void*, char**);



The sqlite3_exec() function works much as it did in SQLite version 2.
Zero or more SQL statements specified in the second parameter are compiled
and executed.  Query results are returned to a callback routine.



In SQLite version 3, the sqlite3_exec routine is just a wrapper around
calls to the prepared statement interface.



   typedef struct sqlite3_stmt sqlite3_stmt;
   int sqlite3_prepare(sqlite3*, const char*, int, sqlite3_stmt**, const char**);
   int sqlite3_prepare16(sqlite3*, const void*, int, sqlite3_stmt**, const void**);
   int sqlite3_finalize(sqlite3_stmt*);
   int sqlite3_reset(sqlite3_stmt*);



The sqlite3_prepare interface compiles a single SQL statement into byte code
for later execution.  This interface is now the preferred way of accessing
the database.



The SQL statement is a UTF-8 string for sqlite3_prepare().
The sqlite3_prepare16() works the same way except
that it expects a UTF-16 string as SQL input.
Only the first SQL statement in the input string is compiled.
The fifth parameter is filled in with a pointer to the next (uncompiled)
SQLite statement in the input string, if any.
The sqlite3_finalize() routine deallocates a prepared SQL statement.
All prepared statements must be finalized before the database can be
closed.
The sqlite3_reset() routine resets a prepared SQL statement so that it
can be executed again.



The SQL statement may contain tokens of the form "?" or "?nnn" or ":aaa"
where "nnn" is an integer and "aaa" is an identifier.
Such tokens represent unspecified literal values (or "wildcards")
to be filled in later by the 
sqlite3_bind interface.
Each wildcard has an associated number which is its sequence in the
statement or the "nnn" in the case of a "?nnn" form. 
It is allowed for the same wildcard
to occur more than once in the same SQL statement, in which case
all instance of that wildcard will be filled in with the same value.
Unbound wildcards have a value of NULL.



   int sqlite3_bind_blob(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
   int sqlite3_bind_double(sqlite3_stmt*, int, double);
   int sqlite3_bind_int(sqlite3_stmt*, int, int);
   int sqlite3_bind_int64(sqlite3_stmt*, int, long long int);
   int sqlite3_bind_null(sqlite3_stmt*, int);
   int sqlite3_bind_text(sqlite3_stmt*, int, const char*, int n, void(*)(void*));
   int sqlite3_bind_text16(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
   int sqlite3_bind_value(sqlite3_stmt*, int, const sqlite3_value*);



There is an assortment of sqlite3_bind routines used to assign values
to wildcards in a prepared SQL statement.  Unbound wildcards
are interpreted as NULLs.  Bindings are not reset by sqlite3_reset().
But wildcards can be rebound to new values after an sqlite3_reset().



After an SQL statement has been prepared (and optionally bound), it
is executed using:



   int sqlite3_step(sqlite3_stmt*);



The sqlite3_step() routine return SQLITE_ROW if it is returning a single
row of the result set, or SQLITE_DONE if execution has completed, either
normally or due to an error.  It might also return SQLITE_BUSY if it is
unable to open the database file.  If the return value is SQLITE_ROW, then
the following routines can be used to extract information about that row
of the result set:



   const void *sqlite3_column_blob(sqlite3_stmt*, int iCol);
   int sqlite3_column_bytes(sqlite3_stmt*, int iCol);
   int sqlite3_column_bytes16(sqlite3_stmt*, int iCol);
   int sqlite3_column_count(sqlite3_stmt*);
   const char *sqlite3_column_decltype(sqlite3_stmt *, int iCol);
   const void *sqlite3_column_decltype16(sqlite3_stmt *, int iCol);
   double sqlite3_column_double(sqlite3_stmt*, int iCol);
   int sqlite3_column_int(sqlite3_stmt*, int iCol);
   long long int sqlite3_column_int64(sqlite3_stmt*, int iCol);
   const char *sqlite3_column_name(sqlite3_stmt*, int iCol);
   const void *sqlite3_column_name16(sqlite3_stmt*, int iCol);
   const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol);
   const void *sqlite3_column_text16(sqlite3_stmt*, int iCol);
   int sqlite3_column_type(sqlite3_stmt*, int iCol);



The sqlite3_column_count()
function returns the number of columns in
the results set.  sqlite3_column_count() can be called at any time after
sqlite3_prepare_v2().  sqlite3_data_count() works similarly to
sqlite3_column_count() except that it only works following sqlite3_step().
If the previous call to sqlite3_step() returned SQLITE_DONE or an error code,
then sqlite3_data_count() will return 0 whereas sqlite3_column_count() will
continue to return the number of columns in the result set.


Returned data is examined using the other 
sqlite3_column_***() functions, 
all of which take a column number as their second parameter. Columns are
zero-indexed from left to right. Note that this is different to parameters,
which are indexed starting at one.



The sqlite3_column_type() function returns the
datatype for the value in the Nth column.  The return value is one
of these:



   #define SQLITE_INTEGER  1
   #define SQLITE_FLOAT    2
   #define SQLITE_TEXT     3
   #define SQLITE_BLOB     4
   #define SQLITE_NULL     5



The sqlite3_column_decltype() routine returns text which is the
declared type of the column in the CREATE TABLE statement.  For an
expression, the return type is an empty string.  sqlite3_column_name()
returns the name of the Nth column.  sqlite3_column_bytes() returns
the number of bytes in a column that has type BLOB or the number of bytes
in a TEXT string with UTF-8 encoding.  sqlite3_column_bytes16() returns
the same value for BLOBs but for TEXT strings returns the number of bytes
in a UTF-16 encoding.
sqlite3_column_blob() return BLOB data.  
sqlite3_column_text() return TEXT data as UTF-8.
sqlite3_column_text16() return TEXT data as UTF-16.
sqlite3_column_int() return INTEGER data in the host machines native
integer format.
sqlite3_column_int64() returns 64-bit INTEGER data.
Finally, sqlite3_column_double() return floating point data.



It is not necessary to retrieve data in the format specify by
sqlite3_column_type().  If a different format is requested, the data
is converted automatically.



Data format conversions can invalidate the pointer returned by
prior calls to sqlite3_column_blob(), sqlite3_column_text(), and/or
sqlite3_column_text16().  Pointers might be invalided in the following
cases:



The initial content is a BLOB and sqlite3_column_text() 
or sqlite3_column_text16()
is called.  A zero-terminator might need to be added to the string.


The initial content is UTF-8 text and sqlite3_column_bytes16() or
sqlite3_column_text16() is called.  The content must be converted to UTF-16.


The initial content is UTF-16 text and sqlite3_column_bytes() or
sqlite3_column_text() is called.  The content must be converted to UTF-8.



Note that conversions between UTF-16be and UTF-16le 
are always done in place and do
not invalidate a prior pointer, though of course the content of the buffer
that the prior pointer points to will have been modified.  Other kinds
of conversion are done in place when it is possible, but sometime it is
not possible and in those cases prior pointers are invalidated.  



The safest and easiest to remember policy is this: assume that any
result from

sqlite3_column_blob(),
sqlite3_column_text(), or
sqlite3_column_text16()

is invalided by subsequent calls to 

sqlite3_column_bytes(),
sqlite3_column_bytes16(),
sqlite3_column_text(), or
sqlite3_column_text16().

This means that you should always call sqlite3_column_bytes() or
sqlite3_column_bytes16() before calling sqlite3_column_blob(),
sqlite3_column_text(), or sqlite3_column_text16().


2.3 User-defined functions


User defined functions can be created using the following routine:



   typedef struct sqlite3_value sqlite3_value;
   int sqlite3_create_function(
     sqlite3 *,
     const char *zFunctionName,
     int nArg,
     int eTextRep,
     void*,
     void (*xFunc)(sqlite3_context*,int,sqlite3_value**),
     void (*xStep)(sqlite3_context*,int,sqlite3_value**),
     void (*xFinal)(sqlite3_context*)
   );
   int sqlite3_create_function16(
     sqlite3*,
     const void *zFunctionName,
     int nArg,
     int eTextRep,
     void*,
     void (*xFunc)(sqlite3_context*,int,sqlite3_value**),
     void (*xStep)(sqlite3_context*,int,sqlite3_value**),
     void (*xFinal)(sqlite3_context*)
   );
   #define SQLITE_UTF8     1
   #define SQLITE_UTF16    2
   #define SQLITE_UTF16BE  3
   #define SQLITE_UTF16LE  4
   #define SQLITE_ANY      5



The nArg parameter specifies the number of arguments to the function.
A value of 0 indicates that any number of arguments is allowed.  The
eTextRep parameter specifies what representation text values are expected
to be in for arguments to this function.  The value of this parameter should
be one of the parameters defined above.  SQLite version 3 allows multiple
implementations of the same function using different text representations.
The database engine chooses the function that minimization the number
of text conversions required.



Normal functions specify only xFunc and leave xStep and xFinal set to NULL.
Aggregate functions specify xStep and xFinal and leave xFunc set to NULL.
There is no separate sqlite3_create_aggregate() API.



The function name is specified in UTF-8.  A separate sqlite3_create_function16()
API works the same as sqlite_create_function()
except that the function name is specified in UTF-16 host byte order.



Notice that the parameters to functions are now pointers to sqlite3_value
structures instead of pointers to strings as in SQLite version 2.X.
The following routines are used to extract useful information from these
"values":



   const void *sqlite3_value_blob(sqlite3_value*);
   int sqlite3_value_bytes(sqlite3_value*);
   int sqlite3_value_bytes16(sqlite3_value*);
   double sqlite3_value_double(sqlite3_value*);
   int sqlite3_value_int(sqlite3_value*);
   long long int sqlite3_value_int64(sqlite3_value*);
   const unsigned char *sqlite3_value_text(sqlite3_value*);
   const void *sqlite3_value_text16(sqlite3_value*);
   int sqlite3_value_type(sqlite3_value*);



Function implementations use the following APIs to acquire context and
to report results:



   void *sqlite3_aggregate_context(sqlite3_context*, int nbyte);
   void *sqlite3_user_data(sqlite3_context*);
   void sqlite3_result_blob(sqlite3_context*, const void*, int n, void(*)(void*));
   void sqlite3_result_double(sqlite3_context*, double);
   void sqlite3_result_error(sqlite3_context*, const char*, int);
   void sqlite3_result_error16(sqlite3_context*, const void*, int);
   void sqlite3_result_int(sqlite3_context*, int);
   void sqlite3_result_int64(sqlite3_context*, long long int);
   void sqlite3_result_null(sqlite3_context*);
   void sqlite3_result_text(sqlite3_context*, const char*, int n, void(*)(void*));
   void sqlite3_result_text16(sqlite3_context*, const void*, int n, void(*)(void*));
   void sqlite3_result_value(sqlite3_context*, sqlite3_value*);
   void *sqlite3_get_auxdata(sqlite3_context*, int);
   void sqlite3_set_auxdata(sqlite3_context*, int, void*, void (*)(void*));


2.4 User-defined collating sequences


The following routines are used to implement user-defined
collating sequences:



   sqlite3_create_collation(sqlite3*, const char *zName, int eTextRep, void*,
      int(*xCompare)(void*,int,const void*,int,const void*));
   sqlite3_create_collation16(sqlite3*, const void *zName, int eTextRep, void*,
      int(*xCompare)(void*,int,const void*,int,const void*));
   sqlite3_collation_needed(sqlite3*, void*, 
      void(*)(void*,sqlite3*,int eTextRep,const char*));
   sqlite3_collation_needed16(sqlite3*, void*,
      void(*)(void*,sqlite3*,int eTextRep,const void*));



The sqlite3_create_collation() function specifies a collating sequence name
and a comparison function to implement that collating sequence.  The
comparison function is only used for comparing text values.  The eTextRep
parameter is one of SQLITE_UTF8, SQLITE_UTF16LE, SQLITE_UTF16BE, or
SQLITE_ANY to specify which text representation the comparison function works
with.  Separate comparison functions can exist for the same collating
sequence for each of the UTF-8, UTF-16LE and UTF-16BE text representations.
The sqlite3_create_collation16() works like sqlite3_create_collation() except
that the collation name is specified in UTF-16 host byte order instead of
in UTF-8.



The sqlite3_collation_needed() routine registers a callback which the
database engine will invoke if it encounters an unknown collating sequence.
The callback can lookup an appropriate comparison function and invoke
sqlite_3_create_collation() as needed.  The fourth parameter to the callback
is the name of the collating sequence in UTF-8.  For sqlite3_collation_need16()
the callback sends the collating sequence name in UTF-16 host byte order.

This page last modified on  2022-01-08 05:02:57 UTC


c2: The generate_series Table-Valued Function
c5: series.html
c4:
1. Overview

The generate_series(START,STOP,STEP) table-valued function is a
loadable extension included in the SQLite source tree, and compiled into
the command-line shell.  The generate_series() table has one visible
result column named "value" holding integer values
and a number of rows determined by the
parameters START, STOP, and STEP.  The first row of the table has
a value of START.  Subsequent rows increment by STEP to a value
not exceeding STOP.

The generate_series() table has additional, hidden columns
named "start", "stop", and "step" whose values are the effective
values of START, STOP and STEP as provided or defaulted.
It also has a rowid, accessible by its usual names.

Omitted parameters take on default values.  STEP defaults to 1.
STOP defaults to 4294967295.  The START parameter is required
as of version 3.37.0 (2021-11-27) and later and an error will
be raised if START is omitted or has a self-referential or otherwise
uncomputable value.  Older versions used a default of 0 for START.
The legacy behavior can be obtained from recent code by compiling
with -DZERO_ARGUMENT_GENERATE_SERIES.

1.1. Equivalent Recursive Common Table Expression

The generate_series table can be simulated for positive step values
using a recursive common table expression.  If the three parameters
are $start, $end, and $step, then the equivalent common table
expression is:

WITH RECURSIVE generate_series(value) AS (
  SELECT $start
  UNION ALL
  SELECT value+$step FROM generate_series
   WHERE value+$step&lt;=$end
) ...


The common table expression works without having to load an
extension.  On the other hand, the extension is easier to program
and faster.

2. Usage Examples

Generate all multiples of 5 less than or equal to 100:

SELECT value FROM generate_series(5,100,5);


Generate the 20 random integer values:

SELECT random() FROM generate_series(1,20);


Find the name of every customer whose account number
   is an even multiple of 100 between 10000 and 20000.

SELECT customer.name
  FROM customer, generate_series(10000,20000,100)
 WHERE customer.id=value;
/* or */
SELECT name FROM customer
 WHERE id IN (SELECT value
                FROM generate_series(10000,20000,200));

This page last modified on  2023-05-01 21:49:55 UTC


c2: Swarmvtab Virtual Table
c5: swarmvtab.html
c4:
1. Overview

The "swarmvtab" virtual table allows the user to query a large number 
of tables (hereafter "component" tables) with similar schemas but distinct
ranges of rowid values as if they were a single database table. The tables may
be (and usually are) located in different databases. Swarmvtab tables are
read-only.

Component tables must not be declared WITHOUT ROWID, and must all have
the same schema, but may have different names within their databases. In
this context, "the same schema" means that:


  All component tables must have the same set of columns, in the same 
      order.
  The types and default collation sequences attached to each column
      must be the same for all component tables.
  All component tables must have the same PRIMARY KEY declaration (if any).


A swarmvtab table has the same schema as each of its component tables.

A swarmvtab virtual table is created as follows:

CREATE VIRTUAL TABLE temp.&lt;name&gt; USING swarmvtab(&lt;sql-statement&gt;);


Swarmvtab virtual tables must be created in the temp schema. Attempting
to create a swarmvtab in the main or an attached database is an error.

The SQL statement supplied as the argument to the CREATE VIRTUAL TABLE
statement is executed when the table is created. It must return either four
or five columns. Each row returned describes one of the component tables. The
first four columns are interpreted, from first to last, as:


   Database URI. A filename or URI that can be used to open the
  database containing the component table.

   Table name. The name of the component table within its database.

   Minimum rowid. The smallest rowid value that the component
  table may contain.

   Maximum rowid. The smallest rowid value that the component
  table may contain.


The interpretation of the final column, if it is present, is 
described here.

For example, say the SQL statement returns the following data when 
executed:


Database URITable nameMinimum rowidMaximum rowid
test.db1 t1 0 10
test.db2 t2 11 20
test.db3 t1 21 30
test.db4 t1 31 40


and the user queries the swarmvtab table for the row with rowid value
25. The swarmvtab table will open database file "test.db3" and read the
data to return from table "t1" (as 25 falls within the range of rowids
assigned to table "t1" in "test.db3").

Swarmvtab efficiently handles range and equality constraints on the
rowid (or other INTEGER PRIMARY KEY) field only. If a query does not 
contain such a constraint, then swarmvtab finds the results by opening
each database in turn and linearly scanning the component table. Which 
generates a correct result, but is often slow.

There must be no overlapping rowid ranges in the rows returned by
the SQL statement. It is an error if there are.

The swarmvtab implementation may open or close databases at any 
point. By default, it attempts to limit the maximum number of 
simultaneously open database files to nine. This is not a hard limit -
it is possible to construct a scenario that will cause swarmvtab to 
exceed it.


2. Compiling and Using Swarmvtab

The code for the swarmvtab virtual table is found in the
ext/misc/unionvtab.c file of the main SQLite source tree. It may be compiled
into an SQLite loadable extension using a command like:

gcc -g -fPIC -shared unionvtab.c -o unionvtab.so


Alternatively, the unionvtab.c file may be compiled into the application. 
In this case, the following function should be invoked to register the
extension with each new database connection:

int sqlite3_unionvtab_init(sqlite3 *db, void*, void*);


 The first argument passed should be the database handle to register the
extension with. The second and third arguments should both be passed 0.

 The source file and entry point are named for "unionvtab" instead of
"swarmvtab". Unionvtab is a separately documented virtual table 
that is bundled with swarmvtab.


3. Advanced Usage

Most users of swarmvtab will only use the features described above. 
This section describes features designed for more esoteric use cases. These
features all involve specifying extra optional parameters following the SQL
statement as part of the CREATE VIRTUAL TABLE command. An optional parameter 
is specified using its name, followed by an "=" character, followed by an
optionally quoted value. Whitespace may separate the name, "=" character 
and value. For example:

CREATE VIRTUAL TABLE temp.sv USING swarmvtab (
  'SELECT ...',                -- the SELECT statement
  maxopen = 20,                -- An optional parameter
  missing='missing_udf'        -- Another optional parameter
);


The following sections describe the supported parameters. Specifying
an unrecognized parameter name is an error.


3.1. SQL Parameters

If a parameter name begins with a ":", then it is assumed to be a
value to bind to the SQL statement before executing it. The value is always
bound as text. It is an error if the specified SQL parameter does not
exist. For example:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT :dir || local_filename, tbl, min, max FROM components",
  :dir = '/home/user/app/databases/'
);


When the above CREATE VIRTUAL TABLE statement is executed, swarmvtab binds
the text value "/home/user/app/databases/" to the :dir parameter of the
SQL statement before executing it.

A single CREATE VIRTUAL TABLE statement may contain any number of SQL
parameters.


3.2. The "maxopen" Parameter

By default, swarmvtab attempts to limit the number of simultaneously
open databases to nine. This parameter allows that limit to be changed.
For example, to create a swarmvtab table that may hold up to 30 databases
open simultaneously:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  maxopen=30
);


Raising the number of open databases may improve performance in some
scenarios.


3.3. The "openclose" Callback

The "openclose" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database, and again just after it closes one. The first
argument passed to the open close function is the filename or URI
identifying the database to be opened or just recently closed (the same
value returned in the leftmost column of the SQL statement provided to
the CREATE VIRTUAL TABLE command). The second argument is integer value
0 when the function is invoked before opening a database, and 1 when it
is invoked after one is closed. For example, if:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf'
);


then before each database containing a component table is opened, 
swarmvtab effectively executes:

SELECT openclose_udf(&lt;database-name&gt;, 0);


After a database is closed, swarmvtab runs the equivalent of:

SELECT openclose_udf(&lt;database-name&gt;, 1);


Any value returned by the openclose function is ignored. If an invocation
made before opening a database returns an error, then the database file is
not opened and the error returned to the user. This is the only scenario
in which swarmvtab will issue an "open" invocation without also eventually
issuing a corresponding "close" call. If there are still databases open,
"close" calls may be issued from within the eventual sqlite3_close() call
on the applications database that deletes the temp schema in which the
swarmvtab table resides.

Errors returned by "close" invocations are always ignored.


3.4. The "missing" Callback

The "missing" parameter allows the user to specify the name of a
application-defined SQL function that will be invoked just before
swarmvtab opens a database if it finds that the required database file
is not present on disk. This provides the application with an opportunity
to retrieve the required database from a remote source before swarmvtab
attempts to open it. The only argument passed to the "missing" function
is the name or URI that identifies the database being opened. Assuming:

CREATE VIRTUAL TABLE temp.x1 USING swarmvtab (
  "SELECT ...",
  openclose = 'openclose_udf',
  missing='missing_udf'
);


then the missing function is invoked as follows:

SELECT missing_udf(&lt;database-name&gt;);


If the missing function returns an error, then the database is not 
opened and the error returned to the user. If an openclose function is
configured, then a "close" invocation is issued at this point to match
the earlier "open". The following pseudo-code illustrates the procedure used
by a swarmvtab instance with both missing and openclose functions configured
when a component database is opened.

SELECT openclose_udf(&lt;database-name&gt;, 0);
if( error ) return error;
if( db does not exist ){
  SELECT missing_udf(&lt;database-name&gt;);
  if( error ){
    SELECT openclose_udf(&lt;database-name&gt;, 1);
    return error;
  }
}
sqlite3_open_v2(&lt;database-name&gt;);
if( error ){
  SELECT openclose_udf(&lt;database-name&gt;, 1);
  return error;
}
// db successfully opened!



3.5. Component table "context" values

 If the SELECT statement specified as part of the CREATE VIRTUAL 
TABLE command returns five columns, then the final column is used
for application context only. Swarmvtab does not use this value at
all, except that it is passed after &lt;database-name&gt; to both
the openclose and missing functions, if specified. In other words,
instead of invoking the functions as described above, if the "context"
column is present swarmvtab instead invokes:

SELECT missing_udf(&lt;database-name&gt;, &lt;context&gt;);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 0);
SELECT openclose_udf(&lt;database-name&gt;, &lt;context&gt;, 1);


as required.
This page last modified on  2022-01-08 05:02:57 UTC


c2: SQLite As An Application File Format
c5: appfileformat.html
c4:
SQLite As An Application File Format


Executive Summary

An SQLite database file with a defined schema
often makes an excellent application file format.
Here are a dozen reasons why this is so:


 Simplified Application Development
 Single-File Documents
 High-Level Query Language
 Accessible Content
 Cross-Platform
 Atomic Transactions
 Incremental And Continuous Updates
 Easily Extensible
 Performance
 Concurrent Use By Multiple Processes
 Multiple Programming Languages
 Better Applications


Each of these points will be described in more detail below,
after first considering more closely the meaning of
"application file format".  See also the short version of this
whitepaper.

What Is An Application File Format?


An "application file format" is the file format
used to persist application state to disk or to exchange
information between programs.
There are thousands of application file formats in use today.
Here are just a few examples:


DOC - Word Perfect and Microsoft Office documents
DWG - AutoCAD drawings
PDF - Portable Document Format from Adobe
XLS - Microsoft Excel Spreadsheet
GIT - Git source code repository
EPUB - The Electronic Publication format used by non-Kindle eBooks
ODT - The Open Document format used by OpenOffice and others
PPT - Microsoft PowerPoint presentations
ODP - The Open Document presentation format used by OpenOffice and others


We make a distinction between a "file format" and an "application format".
A file format is used to store a single object.  So, for example, a GIF or
JPEG file stores a single image, and an XHTML file stores text,
so those are "file formats" and not "application formats".  An EPUB file, 
in contrast, stores both text and images (as contained XHTML and GIF/JPEG
files) and so it is considered an "application format".  This article is
about "application formats".

The boundary between a file format and an application format is fuzzy.
This article calls JPEG a file format, but for an image editor, JPEG 
might be considered the application format.  Much depends on context.
For this article, let us say that a file format stores a single object
and an application format stores many different objects and their relationships
to one another.

Most application formats fit into one of these three categories:


Fully Custom Formats.
Custom formats are specifically designed for a single application.
DOC, DWG, PDF, XLS, and PPT are examples of custom formats.  Custom
formats are usually contained within a single file, for ease of transport.
They are also usually binary, though the DWG format is a notable exception.
Custom file formats require specialized application code
to read and write and are not normally accessible from commonly
available tools such as unix command-line programs and text editors.
In other words, custom formats are usually "opaque blobs".
To access the content of a custom application file format, one needs
a tool specifically engineered to read and/or write that format.

Pile-of-Files Formats.
Sometimes the application state is stored as a hierarchy of
files.  Git is a prime example of this, though the phenomenon occurs
frequently in one-off and bespoke applications.  A pile-of-files format
essentially uses the filesystem as a key/value database, storing small
chunks of information into separate files.  This gives the
advantage of making the content more accessible to common utility
programs such as text editors or "awk" or "grep".  But even if many 
of the files in a pile-of-files format
are easily readable, there are usually some files that have their
own custom format (example: Git "Packfiles") and are hence
"opaque blobs" that are not readable
or writable without specialized tools.  It is also much less convenient
to move a pile-of-files from one place or machine to another, than
it is to move a single file.  And it is hard to make a pile-of-files
document into an email attachment, for example.  Finally, a pile-of-files
format breaks the "document metaphor":
there is no one file that a user can point to
that is "the document".

Wrapped Pile-of-Files Formats.
Some applications use a Pile-of-Files that is then encapsulated into
some kind of single-file container, usually a ZIP archive.  
EPUB, ODT,and ODP are examples of this approach.
An EPUB book is really just a ZIP archive that contains various
XHTML files for the text of book chapters, GIF and JPEG images for
the artwork, and a specialized catalog file that tells the eBook
reader how all the XML and image files fit together.  OpenOffice
documents (ODT and ODP) are also ZIP archives containing XML and
images that represent their content as well as "catalog" files that
show the interrelationships between the component parts.

A wrapped pile-of-files format is a compromise between a full
custom file format and a pure pile-of-files format.
A wrapped pile-of-files format is not an opaque blob in the same sense
as a custom format, since the component parts can still be accessed
using any common ZIP archiver, but the format is not quite as accessible
as a pure pile-of-files format because one does still need the ZIP 
archiver, and one cannot normally use command-line tools like "find"
on the file hierarchy without first un-zipping it.  On the other
hand, a wrapped pile-of-files format does preserve the document
metaphor by putting all content into a single disk file.  And
because it is compressed, the wrapped pile-of-files format tends to
be more compact.

As with custom file formats, and unlike pure pile-of-file formats,
a wrapped pile-of-files format is not as easy to edit, since
usually the entire file must be rewritten in order to change any
component part.


The purpose of this document is to argue in favor of a fourth
new category of application file format: An SQLite database file.

SQLite As The Application File Format


Any application state that can be recorded in a pile-of-files can
also be recorded in an SQLite database with a simple key/value schema
like this:

CREATE TABLE files(filename TEXT PRIMARY KEY, content BLOB);

If the content is compressed, then such an SQLite Archive database is
the same size (&#177;1%)
as an equivalent ZIP archive, and it has the advantage
of being able to update individual "files" without rewriting
the entire document.


But an SQLite database is not limited to a simple key/value structure
like a pile-of-files database.  An SQLite database can have dozens
or hundreds or thousands of different tables, with dozens or
hundreds or thousands of fields per table, each with different datatypes
and constraints and particular meanings, all cross-referencing each other,
appropriately and automatically indexed for rapid retrieval,
and all stored efficiently and compactly in a single disk file.
And all of this structure is succinctly documented for humans
by the SQL schema.

In other words, an SQLite database can do everything that a 
pile-of-files or wrapped pile-of-files format can do, plus much more,
and with greater lucidity.
An SQLite database is a more versatile container than key/value
filesystem or a ZIP archive.  (For a detailed example, see the
OpenOffice case study essay.)

The power of an SQLite database could, in theory, be achieved using
a custom file format.  But any custom file format that is as expressive
as a relational database would likely require an enormous design specification 
and many tens or hundreds of thousands of lines of code to 
implement.  And the end result would be an "opaque blob" that is
inaccessible without specialized tools.


Hence, in comparison to other approaches, the use of
an SQLite database as an application file format has
compelling advantages.  Here are a few of these advantages,
enumerated and expounded:



Simplified Application Development.
No new code is needed for reading or writing the application file.
One has merely to link against the SQLite library, or include the 
single "sqlite3.c" source file with the rest of the
application C code, and SQLite will take care of all of the application
file I/O.  This can reduce application code size by many thousands of
lines, with corresponding saving in development and maintenance costs.

SQLite is one of the
most used software libraries in the world.
There are literally tens of billions of SQLite database files in use 
daily, on smartphones and gadgets and in desktop applications.
SQLite is carefully tested and proven reliable.  It is not
a component that needs much tuning or debugging, allowing developers
to stay focused on application logic.

Single-File Documents.
An SQLite database is contained in a single file, which is easily
copied or moved or attached.  The "document" metaphor is preserved.

SQLite does not have any file naming requirements
and so the application can use any custom file suffix that it wants
to help identify the file as "belonging" to the application.
SQLite database files contain a 4-byte Application ID in
their headers that can be set to an application-defined value
and then used to identify the "type" of the document for utility
programs such as file(1), further
enhancing the document metaphor.


High-Level Query Language.
SQLite is a complete relational database engine, which means that the
application can access content using high-level queries.  Application
developers need not spend time thinking about "how" to retrieve the
information they need from a document.  Developers write SQL that
expresses "what" information they want and let the database engine
to figure out how to best retrieve that content.  This helps developers
operate "heads up" and remain focused on solving the user's problem,
and avoid time spent "heads down" fiddling with low-level file
formatting details.

A pile-of-files format can be viewed as a key/value database.  
A key/value database is better than no database at all.
But without transactions or indices or a high-level query language or
a proper schema,
it is much harder and more error prone to use a key/value database than
a relational database.

Accessible Content.
Information held in an SQLite database file is accessible using 
commonly available open-source command-line tools - tools that 
are installed by default on Mac and Linux systems and that are 
freely available as a self-contained EXE file on Windows.
Unlike custom file formats, application-specific programs are
not required to read or write content in an SQLite database.
An SQLite database file is not an opaque blob.  It is true
that command-line tools such as text editors or "grep" or "awk" are
not useful on an SQLite database, but the SQL query language is a much
more powerful and convenient way for examining the content, so the
inability to use "grep" and "awk" and the like is not seen as a loss.

An SQLite database is a well-defined and well-documented
file format that is in widespread use by literally millions of applications
and is backwards compatible to its inception in 2004 and which promises
to continue to be compatible in decades to come.  The longevity of
SQLite database files is particularly important to bespoke applications,
since it allows the document content to be accessed far in the
future, long after all traces of the original application have been lost.
Data lives longer than code.
SQLite databases are recommended by the US Library of Congress
as a storage format for long-term preservation of digital content.


Cross-Platform.
SQLite database files are portable between 32-bit and 64-bit machines and
between big-endian and little-endian architectures and between any of the
various flavors of Windows and Unix-like operating systems.
The application using an SQLite application file format can store
binary numeric data without having to worry about the byte-order of
integers or floating point numbers.
Text content can be read or written as UTF-8, UTF-16LE, or UTF-16BE and 
SQLite will automatically perform any necessary translations on-the-fly.

Atomic Transactions.
Writes to an SQLite database are atomic.  
They either happen completely
or not at all, even during system crashes or power failures.  So
there is no danger of corrupting a document just because the power happened
to go out at the same instant that a change was being written to disk.

SQLite is transactional, meaning that multiple changes can be grouped
together such that either all or none of them occur, and so that the
changes can be rolled back if a problem is found prior to commit.
This allows an application to make a change incrementally, then run
various sanity and consistency checks on the resulting data prior to
committing the changes to disk.  The
Fossil DVCS 
uses this technique
to verify that no repository history has been lost prior to each change.

Incremental And Continuous Updates.
When writing to an SQLite database file, only those parts of the file that
actually change are written out to disk.  This makes the writing happen faster
and saves wear on SSDs.  This is an enormous advantage over custom
and wrapped pile-of-files formats, both of which usually require a
rewrite of the entire document in order to change a single byte.  
Pure pile-of-files formats can also
do incremental updates to some extent, though the granularity of writes is 
usually larger with pile-of-file formats (a single file) than with SQLite
(a single page).

SQLite also supports continuous update.
Instead of collecting changes in memory and then writing
them to disk only on a File/Save action, changes can be written back to
the disk as they occur.  This avoids loss of work on a system crash or
power failure.  An automated undo/redo stack, managed using triggers,
can be kept in the on-disk database, meaning that undo/redo can occur 
across session boundaries.

Easily Extensible.
As an application grows, new features can be added to an
SQLite application file format simply by adding new tables to the schema
or by adding new columns to existing tables.  Adding columns or tables
does not change the meaning of prior queries, so with a 
modicum of care to ensuring that the meaning of legacy columns and
tables are preserved, backwards compatibility is maintained.

It is possible to extend custom or pile-of-files formats too, of course,
but doing is often much harder.  If indices are added, then all application
code that changes the corresponding tables must be located and modified to
keep those indices up-to-date.  If columns are added, then all application
code that accesses the corresponding table must be located and modified to 
take into account the new columns.

Performance.
In many cases, an SQLite application file format will be 
faster than a pile-of-files format or
a custom format.  In addition to being faster for raw read and
writes, SQLite can often dramatically improves start-up times because 
instead of having to
read and parse the entire document into memory, the application can
do queries to extract only the information needed for the initial screen.
As the application progresses, it only needs to load as much material as
is needed to draw the next screen, and can discard information from
prior screens that is no longer in use.  This helps keep the memory
footprint of the application under control.

A pile-of-files format can be read incrementally just like SQLite.
But many developers are surprised to learn that SQLite can read and 
write smaller BLOBs (less than about 100KB in size) from its database 
faster than those same blobs can be read or written as separate files 
from the filesystem.  (See
35% Faster Than The Filesystem and
Internal Versus External BLOBs for further information.)
There is overhead associated with operating a relational
database engine, however one should not assume that direct file I/O
is faster than SQLite database I/O, as often it is not.

In either case, if performance problems do arise in an SQLite application 
those problems can often be resolved by adding one or two CREATE INDEX
statements to the schema or perhaps running ANALYZE one time
and without having to touch a single line of
application code.  But if a performance problem comes up in a custom or 
pile-of-files format, the fix will often require extensive changes
to application code to add and maintain new indices or to extract 
information using different algorithms.

Concurrent Use By Multiple Processes.
SQLite automatically coordinates concurrent access to the same
document from multiple threads and/or processes.  Two or more
applications can connect and read from the same document at the
same time.  Writes are serialized, but as writes normally only
take milliseconds, applications simply take turns writing.
SQLite automatically ensures that the low-level format of the
document is uncorrupted.  Accomplishing the same with a custom
or pile-of-files format, in contrast, requires extensive support
in the application.  And the application logic needed to support
concurrency is a notorious bug-magnet.

Multiple Programming Languages.
Though SQLite is itself written in ANSI-C, interfaces exist for
just about every other programming language you can think of:
C++, C#, Objective-C, Java, Tcl, Perl, Python, Ruby, Erlang,
JavaScript, and so forth.  So programmers can develop in whatever
language they are most comfortable with and which best matches
the needs of the project.

An SQLite application file format is a great 
choice in cases where there is a collection or "federation" of
separate programs, often written in different languages and by
different development teams.
This comes up commonly in research or laboratory
environments where one team is responsible for data acquisition
and other teams are responsible for various stages of analysis.
Each team can use whatever hardware, operating system,
programming language and development methodology that they are 
most comfortable with, and as long as all programs use an SQLite 
database with a common schema, they can all interoperate.


Better Applications.
If the application file format is an SQLite database, the complete
documentation for that file format consists of the database schema,
with perhaps a few extra words about what each table and column
represents.  The description of a custom file format,
on the other hand, typically runs on for hundreds of 
pages.  A pile-of-files format, while much simpler and easier to
describe than a fully custom format, still tends to be much larger
and more complex than an SQL schema dump, since the names and format
for the individual files must still be described.

This is not a trivial point.  A clear, concise, and easy to understand
file format is a crucial part of any application design.
Fred Brooks, in his all-time best-selling computer science text,
The Mythical Man-Month says:
Representation is the
essence of computer programming....
Show me your flowcharts and conceal your tables, and I shall 
continue to be mystified.  Show me your tables, and I won't usually
need your flowcharts; they'll be obvious.
Rob Pike, in his
Rules of Programming expresses the same idea this way:

Data dominates.  If you've chosen the right data structures
and organized things well, the algorithms will almost always
be self-evident.  Data structures, not algorithms, are central
to programming.
 Linus Torvalds used different words to say
much the same thing on the Git mailing list on 2006-06-27: 

Bad programmers worry about the code.  Good programmers worry
about data structures and their relationships.


The point is this: an SQL database schema almost always does 
a far better job of defining and organizing the tables and 
data structures and their relationships.
And having clear, concise, and well-defined representation
almost always results in an application that performs better,
has fewer problems, and is easier to develop and maintain.


Conclusion


SQLite is not the perfect application file format for every situation.
But in many cases, SQLite is a far better choice than either a custom
file format, a pile-of-files, or a wrapped pile-of-files.
SQLite is a high-level, stable, reliable, cross-platform, widely-deployed,
extensible, performant, accessible, concurrent file format.  It deserves
your consideration as the standard file format on your next application
design.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Why SQLite Uses Bytecode
c5: whybytecode.html
c4:
1. Introduction

Every SQL database engine works in roughly the same way:  It first 
translates the input SQL text into a "prepared statement".  Then it "executes"
the prepared statement to generate a result.

A prepared statement is an object that represents the steps needed
to accomplish the input SQL.  Or, to think of it in another way,
the prepared statement is the SQL statement translated into a form that is
more easily understood by the computer.

In SQLite, a prepared statement is an instance of the
sqlite3_stmt object.  In other systems, the prepared
statement is usually an internal data structure that is not directly visible to
the application programmer.  Developers of other SQL database engines
do not necessarily call these objects "prepared statements".
But such objects exists, whatever they might be called.
This paper will use the term "prepared statement".

There are countless ways of implementing a prepared statement.  This
paper will look at the two most common methods:



Bytecode &rarr;  The input SQL is translated into a virtual machine language
that is then run by a virtual machine interpreter.  This is the technique
used by SQLite.


Tree-Of-Objects &rarr; The input SQL is translated in a tree of objects
that represent the processing to be done.  The SQL is executed by walking this
tree.  This is the technique used by MySQL and PostgreSQL.



There are advantages and disadvantages to each of these representations of
a prepared statement.  The purpose of this paper is to articulate some of those
advantages and disadvantages.

1.1. How To Provide Feedback


This document is written from the perspective of the original author of SQLite.
If you disagree with any of the opinions offered in this document, you are
welcomed to offer corrections and/or contrary views on the
SQLite Forum.  Or you can email the author
directly.

1.2. Definition Of "Bytecode"

The bytecode generated by SQLite might be a little different from what
many readers think of as bytecode.  The bytecode used (for example) by the
Java virtual machine or
by WebAssembly consists almost
entirely of low-level operations, similar to what physical CPUs implement:
basic math operators, comparisons, conditional jumps, and
instructions to move content between different memory locations.  SQLite bytecode
has these kinds of low-level instructions, too.  But SQLite bytecode also contains
some high-level operations that are specific to the needs of a database engine.
Here are just a few examples:



OP_Column &rarr; Extract the value from the N-th column of the
database row that a particular cursor is currently pointing at.
OP_CreateBtree &rarr; Allocate space for a new B-Tree in the
database file.
OP_ParseSchema &rarr; Reread and reparse all or part of the
sqlite_schema table and refresh internal symbol tables accordingly.
OP_SeekGE &rarr; Move a cursor on a particular B-Tree to the first
entry that is greater than or equal to a given key.
OP_Next &rarr; Advance a cursor on a particular B-Tree to the next
entry in the B-Tree and jump, or fall through if there are no more entries in that
B-Tree.


In other words, the "bytecode" used by SQLite is not so much a set of CPU
instructions as it is a list of database primitives that are to be run
in a particular order.

1.3. Definition Of "Abstract Syntax Tree" or "AST"


An "Abstract Syntax Tree" or AST
is a data structure that describes a program or statement in some kind of formal
language.  In our context, the formal language is SQL.  An AST is typically
implemented as a tree of objects where each object represents one small part of
the overall SQL statement.  ASTs emerge naturally from parsers for formal languages.
The usual technique is to use an 
LALR(1) parser.
With such a parser, each terminal
symbol holds metadata that will become a leaf of the AST, and each non-terminal 
symbol holds metadata that will become a sub-branch of the overall AST.
As rules of the grammar are "reduced" by the parser, new nodes of the AST are
allocated and connected to subnodes.
After the parse completes, the start-symbol of the grammar is left holding the
root of the AST.

An AST is a tree of objects.  But an AST is not a suitable form for
a prepared statement.  After being generated, an AST first needs to be
transformed in various ways before it can executed.  Symbols need to be
resolved.  Semantic rules need to be checked.  Optimizations need to be
applied that transform input SQL statement into different forms that
execute more quickly.  Finally, the AST needs to be translated into an
alternative representation that is more amenable to execution.

Some people refer to the tree of objects that is used as the
executable form for MySQL and PostgreSQL as an AST.  This is probably a
misuse of the term "AST", because by the time the tree of objects is
ready to be executed, it has been changed so much that it has little
resemblance to the original SQL text.  The confusion arises in part because
both the final prepared statement object and the original AST are both
trees of objects.  The usual technique is for the original AST that comes
directly out of the parser to be transformed little by little, in multiple
passes, until at the end it is fully converted into an tree of objects
that is no longer strictly an AST but that can be evaluated to
generate a result.  There is not necessarily a clear point during this
process when the tree-of-objects ceases to be an AST and becomes a
prepared statement instead.  And because there is no clear boundary between an
AST and a prepared statement, people often refer to a prepared statement
that is represented as a tree of objects as an "AST", even though that
description is not precise.

1.4. Dataflow Programming

Dataflow programming
is a style of programming in which individual nodes specialize in doing
one small part of the overall computation.  Each node receives inputs from 
other nodes and sends its output to other nodes.  Thus the nodes form a
directed graph that carry inputs into outputs.


A "dataflow program" is perhaps a better description than "AST" for
the tree of objects that an SQL database engine uses as a prepared statement.

2. Advantages To Compiling Into Bytecode

SQLite compiles to bytecode, and the SQLite developers are very happy
with this approach.  Here is why:

2.1. Bytecode Is Easier To Understand

A flat list of opcodes can be easily printed to see exactly
how an SQL statement is being implemented.  This is what happens in SQLite
when you preface an SQL statement with the "EXPLAIN" keyword:  Instead of
actually running the SQL, the result is a listing of the bytecode
that would have been used to implement that SQL.

Bytecode lends itself to this because a bytecode program is easily
represented as a table.  In SQLite bytecode, each instruction
has one opcode and five operands.  Thus a prepared statement can be
rendered as if it were a query against a six-column table.

A tree-of-objects representation is more difficult to publish in
a human-readable form.  The objects that comprise the tree tend to
all be very different, and thus it is tricky to come up with a 
consistent and simple table representation with which to display
the objects.  Any such table representation that you do come up
with would almost certainly have more than six columns, probably many more.
The problem of rendering a tree-of-objects as a table is sufficiently
difficult that nobody does it, as far as I know.  Hence, no
tree-of-objects database engine provides the level
of detail in their "EXPLAIN" output that SQLite provides.

2.2. Bytecode Is Easier To Debug

Bytecode provides a clear separation between front-end parsing and
analysis and back-end evaluation of an SQL statement.  When problems arise
(incorrect answers and/or poor performance), the developers can examine
the bytecode to quickly determine if the source of the trouble is either the
front-end analysis or the back-end data storage section of the product.

In debugging builds of SQLite, the PRAGMA vdbe_trace=ON; command will
cause a trace of the bytecode execution to appear on the console.

2.3. Bytecode Can Be Run Incrementally


SQL statements written in bytecode can be evaluated incrementally.
For example, a statement can be run until it generates just its first
row of output.  The statement then pauses until it is stepped again.
It is not necessary to run the statement to completion before examining
the first row of output.


This is more difficult to achieve in a tree-of-objects design.  When
the prepared statement is a tree-of-objects, execution is normally
accomplished by walking the tree.  To pause the statement in the middle
of a computation means unwinding the stack back up to the caller, all
the while saving enough state to resume evaluation where it last left
off.  This is not impossible to do, but is sufficiently difficult that
I have never seen it actually done.


Most SQL database engines do not really need to do incremental
execution of prepared statements because most SQL database engines
are client/server.  In client/server engines, a single SQL statement is sent
to the server, then the complete reply comes back over the wire
all at once.  Thus each statement runs to completion in a single go.
But SQLite is not client/server.  SQLite is a library
that runs in the same address space and using the same stack as the
application.  Being able to easily and reliably perform incremental
execution of an SQL statement is important to SQLite.

2.4. Bytecode Is Smaller


The bytecode generated by SQLite is usually smaller than the corresponding
AST coming out of the parser.  During initial processing of SQL text
(during the call to sqlite3_prepare() and similar) both the AST and the
bytecode exist in memory at the same time, so more memory is used then.
But that is a transient state.  The AST
is quickly discarded and its memory recycled, even before the call to
sqlite3_prepare() returns, so the resulting prepared statement ends
up consuming less memory in its bytecode representation than it did as an AST.
This is important because calls to sqlite3_prepare() are transient, but
prepared statements are often cached for possible reuse and persist in memory
for a long time.

2.5. Bytecode Is Faster


I believe that a bytecode representation of a prepared statement
runs faster, because fewer decisions need to be made for each step of
the computation.  Emphasis on "believe" in the previous sentence
&rarr; it is difficult to verify
this claim experimentally since nobody has ever put in the multiple years
of effort necessary to generate equivalent bytecode and tree-of-object
representations of a prepared statement to see which one actually runs faster.
We do know that SQLite is very fast, but we
do not have good side-by-side comparisons with other SQL databases since
the other databases spend a lot of time doing client/server message processing,
and it is difficult to untangle the message round-trip overhead from the
actual processing time.

3. Advantages Of Compiling Into A Tree Of Objects

The SQLite developers think that the bytecode approach is
best, at least for the use cases the SQLite tries to fill, but the
tree-of-objects approach to processing SQL does have some advantages over 
bytecode. There are always tradeoffs.

3.1. Query Planning Decisions Can Be Deferred Until Runtime


When a prepared statement is bytecode, once the bytecode has been generated,
the algorithm is fixed and cannot be subsequently changed without completely
rewriting the bytecode.
This is not the case with a tree-of-objects prepared
statement.  A tree-of-objects is easier to modify on-the-fly.  The query
plan is mutable and can be tweaked as it is running, based on the progress
of the query.  Thus a query can be dynamically self-tuning.

3.2. Dataflow Programs Are Easy To Parallelize

In a dataflow program, each processing node can be assigned to a
different thread.  There needs to be some kind of threadsafe queuing
mechanism for transferring intermediate results from one node to the
next.  But no synchronization primitives are typically needed within
each node of the program.  Node schedule is trivial:  A node becomes
eligible to run when it has data available and there is space in its
output queue.

This is an important consideration for database engines that are
designed to run large analytic queries 
(OLAP)
on large multi-core servers.
The primary focus of SQLite is transaction processing 
(OLTP) 
on the internet-of-things, so there is less need to
represent prepared statements as dataflow programs in SQLite.
This page last modified on  2024-05-09 17:38:03 UTC


c2: Deterministic SQL Functions
c5: deterministic.html
c4:
1. Overview


SQL functions in SQLite can be either "deterministic" or "non-deterministic".


A deterministic function always gives the same answer when it has
the same inputs.  Most built-in SQL functions in SQLite are
deterministic.  For example, the abs(X) function always returns
the same answer as long as its input X is the same.


Non-deterministic functions might give different answers on each
invocation, even if the arguments are always the same.  The following
are examples of non-deterministic functions:


 random()
 changes()
 last_insert_rowid()
 sqlite3_version()



The random() function is obviously non-deterministic because it gives
a different answer every time it is invoked.  The answers from changes()
and last_insert_rowid() depend on prior SQL statements, and so they
are also non-deterministic. The
sqlite3_version() function is mostly constant, but it can change when
SQLite is upgraded, and so even though it always returns the same answer
for any particular session, because it can change answers across sessions
it is still considered non-deterministic.


2. Restrictions on the use of non-deterministic functions


There are some contexts in SQLite that do not allow the use of
non-deterministic functions:


In the expression of a CHECK constraint.
In the WHERE clause of a partial index.
In an expression used as part of an expression index.
In the expression of a generated column.



In the cases above, the values returned by the function affects the
information stored in the database file.  The values of functions
in CHECK constraints determines which entries are valid for a table,
and functions in the WHERE clause of a partial index or in an index on
an expression compute values stored in the index b-tree.
If any of these functions later returns a different
value, then the database might no longer be well-formed.  
Hence, to avoid database corruption,
only deterministic functions can be used in the contexts
above.



3. Special-case Processing For Date/Time Functions


The built-in date and time functions of SQLite are a special case.
These functions are usually considered deterministic.  However, if
these functions use the string "now" as the date, or if they use
the localtime modifier or the utc modifier, then they are
considered non-deterministic.  Because the function inputs are
not necessarily known until run-time, the date/time functions will
throw an exception if they encounter any of the non-deterministic
features in a context where only deterministic functions are allowed.


Prior to SQLite 3.20.0 (2017-08-01) all date/time functions were
always considered non-deterministic.  The ability for date/time functions
to be deterministic sometimes and non-deterministic at other times,
depending on their arguments, was added for the 3.20.0 release.

3.1. Bug fix in version 3.35.2


When the enhancement was made to SQLite 3.20.0 such that date/time
functions would be considered deterministic as they do not depend
on the current time, one case was overlooked:
Many of the date/time functions can be called
with no arguments at all.  These no-argument date/time functions
behave as if they had a single "'now'" argument.
Thus "datetime()" and
"datetime('now')" both yield the current date and time.
However, only the second form was recognized as non-deterministic.
This meant that developers could sneak the non-deterministic
"datetime()" form into CHECK constraints, index
expressions, generated column expressions, and similar places
where non-deterministic functions make no sense.
This oversight was fixed in version 3.35.2 (2021-03-17).
However, there may be legacy databases in circulation that were created
by SQLite version 3.20.0 through 3.35.1 that have non-deterministic
date/time functions in their schemas.

4. Application-defined deterministic functions


By default, application-defined SQL functions are considered to
be non-deterministic.  However, if the 4th parameter to
sqlite3_create_function_v2() is OR-ed with 
SQLITE_DETERMINISTIC, then SQLite will treat that function as if it
were deterministic.


Note that if a non-deterministic function is tagged with
SQLITE_DETERMINISTIC and if that function ends up being used in
the WHERE clause of a partial index or in an
expression index, then when the function begins to return different
answers, the associated index may become corrupt.  If an SQL function
is nearly deterministic (which is to say, if it only rarely changes,
like sqlite_version()) and it is used in an index that becomes
corrupt, the corruption can be fixed by running REINDEX.


The interfaces necessary to construct a function that is sometimes
deterministic and sometimes non-deterministic depending on their
inputs, such as the built-in date/time functions, are not published.
Generic application-defined SQL functions must
be always deterministic or always non-deterministic.
This page last modified on  2022-01-08 05:02:57 UTC


c2: The Geopoly Interface To The SQLite R*Tree Module
c5: geopoly.html
c4:
1. Overview


The Geopoly module is an alternative interface to the R-Tree extension that uses
the GeoJSON notation
(RFC-7946) to describe two-dimensional
polygons.  Geopoly includes functions for detecting when one polygon is
contained within or overlaps with another, for computing the
area enclosed by a polygon, for doing linear transformations of polygons,
for rendering polygons as
SVG, and other
similar operations.


The source code for Geopoly is included in the amalgamation.  However,
depending on configuration options and the a particular version of SQLite
you are using, the Geopoly extension may or may not be enabled by default.
To ensure that Geopoly is enabled for your build, 
add the -DSQLITE_ENABLE_GEOPOLY=1 compile-time option.


Geopoly operates on "simple" polygons - that is, polygons for which
the boundary does not intersect itself.  Geopoly thus extends the capabilities
of the R-Tree extension which can only deal with rectangular areas.
On the other hand, the R-Tree extension is
able to handle between 1 and 5 coordinate dimensions, whereas Geopoly is restricted
to 2-dimensional shapes only.


Each polygon in the Geopoly module can be associated with an arbitrary
number of auxiliary data fields.

1.1. GeoJSON

The GeoJSON standard is syntax for
exchanging geospatial information using JSON.  GeoJSON is a rich standard
that can describe nearly any kind of geospatial content.

The Geopoly module only understands
a small subset of GeoJSON, but a critical subset.  
In particular, GeoJSON understands
the JSON array of vertexes that describes a simple polygon.

A polygon is defined by its vertexes.
Each vertex is a JSON array of two numeric values which are the
X and Y coordinates of the vertex.
A polygon is a JSON array of at least four of these vertexes, 
and hence is an array of arrays.
The first and last vertex in the array must be the same.
The polygon follows the right-hand rule:  When tracing a line from
one vertex to the next, the area to the right of the line is outside
of the polygon and the area to the left is inside the polygon.
In other words, the net rotation of the vertexes is counter-clockwise.


For example, the following JSON describes an isosceles triangle, sitting
on the X axis and with an area of 0.5:

&#91;&#91;0,0],&#91;1,0],&#91;0.5,1],&#91;0,0]]



A triangle has three vertexes, but the GeoJSON description of the triangle
has 4 vertexes because the first and last vertex are duplicates.

1.2. Binary storage format


Internally, Geopoly stores polygons in a binary format - an SQL BLOB.
Details of the binary format are given below.
All of the Geopoly interfaces are able to accept polygons in either the
GeoJSON format or in the binary format.

2. Using The Geopoly Extension


A geopoly table is created as follows:

CREATE VIRTUAL TABLE newtab USING geopoly(a,b,c);



The statement above creates a new geopoly table named "newtab".
Every geopoly table contains a built-in integer "rowid" column
and a "_shape" column that contains
the polygon associated with that row of the table.
The example above also defines three auxiliary data columns 
named "a", "b", and "c" that can store whatever additional
information the application needs to associate
with each polygon.  If there is no need to store auxiliary
information, the list of auxiliary columns can be omitted.


Store new polygons in the table using ordinary INSERT statements:

INSERT INTO newtab(_shape) VALUES('&#91;&#91;0,0],&#91;1,0],&#91;0.5,1],&#91;0,0]]');



UPDATE and DELETE statements work similarly.

2.1. Queries


To query the geopoly table using an indexed geospatial search, 
use one of the functions geopoly_overlap()
or geopoly_within() as a boolean function in the WHERE clause,
with the "_shape" column as the first argument to the function.
For example:

SELECT * FROM newtab WHERE geopoly_overlap(_shape, $query_polygon);



The previous example will return every row for which the _shape
overlaps the polygon in the $query_polygon parameter.  The
geopoly_within() function works similarly, but only returns rows for
which the _shape is completely contained within $query_polygon.


Queries (and also DELETE and UPDATE statements) in which the WHERE
clause contains a bare geopoly_overlap() or geopoly_within() function
make use of the underlying R*Tree data structures for a fast lookup that
only has to examine a subset of the rows in the table.  The number of
rows examines depends, of course, on the size of the $query_polygon.
Large $query_polygons will normally need to look at more rows than small
ones.


Queries against the rowid of a geopoly table are also very quick, even
for tables with a vast number of rows.
However, none of the auxiliary data columns are indexes, and so queries
against the auxiliary data columns will involve a full table scan.

3. Special Functions


The geopoly module defines several new SQL functions that are useful for
dealing with polygons.  All polygon arguments to these functions can be
either the GeoJSON format or the internal binary format.



3.1. The geopoly_overlap(P1,P2) Function


If P1 and P2 are both polygons, then the geopoly_overlap(P1,P2) function returns
a non-zero integer if there is any overlap between P1 and P2, or it returns
zero if P1 and P2 completely disjoint.
If either P1 or P2 is not a polygon, this routine returns NULL.


The geopoly_overlap(P1,P2) function is special in that the geopoly virtual
table knows how to use R*Tree indexes to optimize queries in which the 
WHERE clause uses geopoly_overlap() as a boolean function.  Only the
geopoly_overlap(P1,P2) and geopoly_within(P1,P2) functions have this
capability.



3.2. The geopoly_within(P1,P2) Function


If P1 and P2 are both polygons, then the geopoly_within(P1,P2) function returns
a non-zero integer if P1 is completely contained within P2, or it returns zero
if any part of P1 is outside of P2.  If P1 and P2 are the same polygon, this routine
returns non-zero.
If either P1 or P2 is not a polygon, this routine returns NULL.


The geopoly_within(P1,P2) function is special in that the geopoly virtual
table knows how to use R*Tree indexes to optimize queries in which the 
WHERE clause uses geopoly_within() as a boolean function.  Only the
geopoly_within(P1,P2) and geopoly_overlap(P1,P2) functions have this
capability.



3.3. The geopoly_area(P) Function


If P is a polygon, then geopoly_area(P) returns the area enclosed by
that polygon.  If P is not a polygon, geopoly_area(P) returns NULL.



3.4. The geopoly_blob(P) Function


If P is a polygon, then geopoly_blob(P) returns the binary encoding
of that polygon as a BLOB.
If P is not a polygon, geopoly_blob(P) returns NULL.



3.5. The geopoly_json(P) Function


If P is a polygon, then geopoly_json(P) returns the GeoJSON representation
of that polygon as a TEXT string.
If P is not a polygon, geopoly_json(P) returns NULL.



3.6. The geopoly_svg(P,...) Function


If P is a polygon, then geopoly_svg(P,...) returns a text string which is a
Scalable Vector Graphics (SVG)
representation of that polygon.  If there is more one argument, then second
and subsequent arguments are added as attributes to each SVG glyph.  For example:

SELECT geopoly_svg($polygon,'class="poly"','style="fill:blue;"');



If P is not a polygon, geopoly_svg(P,...) returns NULL.


Note that geopoly uses a traditional right-handed cartesian coordinate system
with the origin at the lower left, whereas SVG uses a left-handed coordinate
system with the origin at the upper left.  The geopoly_svg() routine makes no
attempt to transform the coordinate system, so the displayed images are shown
in mirror image and rotated.  If that is undesirable, the geopoly_xform() routine
can be used to transform the output from cartesian to SVG coordinates prior to
passing the polygons into geopoly_svg().



3.7. The geopoly_bbox(P) and geopoly_group_bbox(P) Functions


If P is a polygon, then geopoly_bbox(P) returns a new polygon that is
the smallest (axis-aligned) rectangle completely enclosing P.
If P is not a polygon, geopoly_bbox(P) returns NULL.


The geopoly_group_bbox(P) function is an aggregate version of geopoly_bbox(P).
The geopoly_group_bbox(P) function returns the smallest rectangle that will
enclose all P values seen during aggregation.



3.8. The geopoly_contains_point(P,X,Y) Function


If P is a polygon, then geopoly_contains_point(P,X,Y) returns a 
non-zero integer if and only
if the coordinate X,Y is inside or on the boundary of the polygon P.
If P is not a polygon, geopoly_contains_point(P,X,Y) returns NULL.



3.9. The geopoly_xform(P,A,B,C,D,E,F) Function


The geopoly_xform(P,A,B,C,D,E,F) function returns a new polygon that is an
affine transformation of the polygon P and where the transformation
is defined by values A,B,C,D,E,F. If P is not a valid polygon, this
routine returns NULL.


The transformation converts each vertex of the polygon according to the
following formula:

x1 = A*x0 + B*y0 + E
y1 = C*x0 + D*y0 + F



So, for example, to move a polygon by some amount DX, DY without changing
its shape, use:

geopoly_xform($polygon, 1, 0, 0, 1, $DX, $DY)



To rotate a polygon by R radians around the point 0, 0:

geopoly_xform($polygon, cos($R), sin($R), -sin($R), cos($R), 0, 0)



Note that a transformation that flips the polygon might cause the
order of vertexes to be reversed.  In other words, the transformation
might cause the vertexes to circulate in clockwise order instead of
counter-clockwise.  This can be corrected by sending the result
through the geopoly_ccw() function after transformation.




3.10. The geopoly_regular(X,Y,R,N) Function


The geopoly_regular(X,Y,R,N) function returns a convex, simple, regular,
equilateral, equiangular polygon with N sides, centered at X,Y, and with
a circumradius of R.  Or, if R is negative or if N is less than 3, the
function returns NULL.  The N value is capped at 1000 so that the routine
will never render a polygon with more than 1000 sides even if the N value
is larger than 1000.


As an example, the following graphic:


<svg width="600" height="300" style="border:1px solid black">
<polyline points="140.003,100 80.0019,134.644 80.0019,65.3565 140.003,100" style="fill:none;stroke:red;stroke-width:2"></polyline> <text x="100" y="106" alignment-baseline="central" text-anchor="middle">3</text>
<polyline points="240.003,100 200,140.003 159.997,100 200,59.9973 240.003,100" style="fill:none;stroke:orange;stroke-width:2"></polyline> <text x="200" y="106" alignment-baseline="central" text-anchor="middle">4</text>
<polyline points="340.003,100 312.358,138.042 267.637,123.511 267.637,76.4893 312.358,61.9583 340.003,100" style="fill:none;stroke:green;stroke-width:2"></polyline> <text x="300" y="106" alignment-baseline="central" text-anchor="middle">5</text>
<polyline points="440.003,100 419.998,134.644 380.002,134.644 359.997,100 380.002,65.3565 419.998,65.3565 440.003,100" style="fill:none;stroke:blue;stroke-width:2"></polyline> <text x="400" y="106" alignment-baseline="central" text-anchor="middle">6</text>
<polyline points="540.003,100 524.94,131.276 491.101,138.995 463.959,117.353 463.959,82.6471 491.101,61.005 524.94,68.7243 540.003,100" style="fill:none;stroke:purple;stroke-width:2"></polyline> <text x="500" y="106" alignment-baseline="central" text-anchor="middle">7</text>
<polyline points="140.003,200 128.286,228.286 100,240.003 71.7143,228.286 59.9973,200 71.7143,171.714 100,159.997 128.286,171.714 140.003,200" style="fill:none;stroke:red;stroke-width:2"></polyline> <text x="100" y="206" alignment-baseline="central" text-anchor="middle">8</text>
<polyline points="240.003,200 232.363,223.511 212.358,238.042 187.642,238.042 167.637,223.511 159.997,200 167.637,176.489 187.642,161.958 212.358,161.958 232.363,176.489 240.003,200" style="fill:none;stroke:orange;stroke-width:2"></polyline> <text x="200" y="206" alignment-baseline="central" text-anchor="middle">10</text>
<polyline points="340.003,200 334.644,219.998 319.998,234.644 300,240.003 280.002,234.644 265.356,219.998 259.997,200 265.356,180.002 280.002,165.356 300,159.997 319.998,165.356 334.644,180.002 340.003,200" style="fill:none;stroke:green;stroke-width:2"></polyline> <text x="300" y="206" alignment-baseline="central" text-anchor="middle">12</text>
<polyline points="440.003,200 436.956,215.305 428.286,228.286 415.305,236.956 400,240.003 384.695,236.956 371.714,228.286 363.044,215.305 359.997,200 363.044,184.695 371.714,171.714 384.695,163.044 400,159.997 415.305,163.044 428.286,171.714 436.956,184.695 440.003,200" style="fill:none;stroke:blue;stroke-width:2"></polyline> <text x="400" y="206" alignment-baseline="central" text-anchor="middle">16</text>
<polyline points="540.003,200 538.042,212.358 532.363,223.511 523.511,232.363 512.358,238.042 500,240.003 487.642,238.042 476.489,232.363 467.637,223.511 461.958,212.358 459.997,200 461.958,187.642 467.637,176.489 476.489,167.637 487.642,161.958 500,159.997 512.358,161.958 523.511,167.637 532.363,176.489 538.042,187.642 540.003,200" style="fill:none;stroke:purple;stroke-width:2"></polyline> <text x="500" y="206" alignment-baseline="central" text-anchor="middle">20</text>
</svg>


Was generated by this script:

SELECT '&lt;svg width="600" height="300">';
WITH t1(x,y,n,color) AS (VALUES
   (100,100,3,'red'),
   (200,100,4,'orange'),
   (300,100,5,'green'),
   (400,100,6,'blue'),
   (500,100,7,'purple'),
   (100,200,8,'red'),
   (200,200,10,'orange'),
   (300,200,12,'green'),
   (400,200,16,'blue'),
   (500,200,20,'purple')
)
SELECT
   geopoly_svg(geopoly_regular(x,y,40,n),
        printf('style="fill:none;stroke:%s;stroke-width:2"',color))
   || printf(' &lt;text x="%d" y="%d" alignment-baseline="central" text-anchor="middle">%d&lt;/text>',x,y+6,n)
  FROM t1;
SELECT '&lt;/svg>';




3.11. The geopoly_ccw(J) Function

The geopoly_ccw(J) function returns the polygon J with counter-clockwise (CCW) rotation.


RFC-7946 requires that polygons use CCW rotation.
But the spec also observes that many legacy GeoJSON files do not following the spec and
contain polygons with clockwise (CW) rotation.  The geopoly_ccw() function is useful for
applications that are reading legacy GeoJSON scripts.  If the input to geopoly_ccw() is
a correctly-formatted polygon, then no changes are made.  However, if the circulation of
the input polygon is backwards, then geopoly_ccw() reverses the circulation order so that
it conforms to the spec and so that it will work correctly with the Geopoly module.



4. Implementation Details

The geopoly module is an extension to the R-Tree extension.  Geopoly
uses the same underlying logic and shadow tables as the R-Tree extension.
Geopoly merely presents a different interface, and provides some extra logic
to compute polygon decoding, overlap, and containment.

4.1. Binary Encoding of Polygons


Geopoly stores all polygons internally using a binary format.  A binary
polygon consists of a 4-byte header following by an array of coordinate
pairs in which each dimension of each coordinate is a 32-bit floating point
number.


The first byte of the header is a flag byte.  The least significant bit
of the flag byte determines whether the coordinate pairs that follow the
header are stored big-endian or little-endian.  A value of 0 for the least
significant bit means big-endian and a value of 1 means little endian.
Other bits of the first byte in the header are reserved for future expansion.


The next three bytes in the header record the number of vertexes in the polygon
as a big-endian integer.  Thus there is an upper bound of about 16 million
vertexes per polygon.


Following the header is the array of coordinate pairs.  Each coordinate is
a 32-bit floating point number.  The use of 32-bit floating point values for
coordinates means that any point on the earth's surface can be mapped with
a resolution of approximately 2.5 meters.  Higher resolutions are of course
possible if the map is restricted to a single continent or country.
Note that the resolution of coordinates in the geopoly module is similar
in magnitude to daily movement of points on the earth's surface due to
tidal forces.


The list of coordinates in the binary format contains no redundancy.  
The last coordinate is not a repeat of the first as it is with GeoJSON.  
Hence, there is always one fewer coordinate pair in the binary representation of
a polygon compared to the GeoJSON representation.

4.2. Shadow Tables


The geopoly module is built on top of the R-Tree extension and uses the
same underlying shadow tables and algorithms.  For indexing purposes, each
polygon is represented in the shadow tables as a rectangular bounding box.
The underlying R-Tree implementation uses bounding boxes to limit the search
space.  Then the geoploy_overlap() and/or geopoly_within() routines further
refine the search to the exact answer.
This page last modified on  2023-12-05 14:43:20 UTC


c2: How To Download Canonical SQLite Source Code
c5: getthecode.html
c4:
1. Introduction

Most programmers compile SQLite into their applications using
the amalgamation.  The amalgamation is C-code but it is not
"source code".  The amalgamation is generated from source code
by scripts.

This document describes how to obtain the canonical source code
for SQLite - the raw source files from which the amalgamation is
built.  See the How To Compile SQLite page for additional information
on what to do with the canonical source code once it is obtained.

2. Direct Downloads

Snapshots of official releases of SQLite source code can often
be obtained directly from the download page of the SQLite website.
Even if the specific version desired is not listed on the download page,
the naming conventions are fairly clear and so programmers can often
guess the name of an historical release and download it that way.

3. Obtaining Code Directly From the Version Control System

For any historical version of SQLite, the source tree can be obtained
from the Fossil version control system,
either downloading a tarball or ZIP archive for a specific version, or
by cloning the entire project history.

SQLite sources are maintained on three geographically dispersed
servers:


https://www.sqlite.org/src (Dallas)
https://www2.sqlite.org/src (Newark)
https://www3.sqlite.org/src (San Francisco)


The documentation is maintained in separate source repositories on
those same servers:


https://www.sqlite.org/docsrc (Dallas)
https://www2.sqlite.org/docsrc (Newark)
https://www3.sqlite.org/docsrc (San Francisco)


To download a specific historical version, first locate the specific
version desired by visiting the timeline page on one of these servers
(for example: https://www.sqlite.org/src/timeline).  If
you know the approximate date of the version you want to download, you
can add a query parameter like "c=YYYY-MM-DD" to the "timeline" URL to
see a timeline centered on that date.  For example, to see all the check-ins
that occurred around August 26, 2013, visit
https://www.sqlite.org/src/timeline?c=2013-08-26.
If you are looking for an official release, visit the
chronology page, click on the date to the left of the release
you are looking for, and that will take you immediately to the
check-in corresponding to the release.

Once you locate a specific version, click on the hyperlink for that
version to see the "Check-in Information Page".
Then click on either the "Tarball" link or the
"ZIP archive" link to download the complete source tree.

4. Verifying That The Code Is Unmodified


The "manifest" file at the root directory of the source tree
contains either a SHA3-256 hash or a SHA1 hash for every source file
in the repository. The name of the version of the entire source tree
is just the SHA3-256 hash of the "manifest" file itself, possibly with
the last line of that file omitted if the last line begins with 
"# Remove this line". The "manifest.uuid" file should contain
the SHA3-256 hash of the "manifest" file. If all of the above hash 
comparisons are correct, then you can be confident that your source
tree is authentic and unadulterated. Details on the format of
manifest files are available on the 

Fossil website.


The process of checking source code authenticity is automated by the makefile:

make verify-source



Or on windows:

nmake /f Makefile.msc verify-source



Using the makefile to verify source integrity is good for detecting
accidental changes to the source tree, but malicious changes could
be hidden by also modifying the makefiles.



5. Cloning The Complete Development History

To clone the entire history of SQLite, first go to the
https://www.fossil-scm.org/download.html page and grab a precompiled binary
for the Fossil version control program.  Or get the source code on the
same page and compile it yourself.

As of 2017-03-12, you must use Fossil version
2.0 or later for the following instructions to work.  
The SQLite repository started using
artifacts named using SHA3 hashes instead of SHA1 hashes on that date,
and Fossil 2.0 or later is needed in order to understand the new SHA3
hashes.  To find out what version of Fossil you are running, 
type "fossil -v".

Fossil is a completely stand-alone
program, so install it simply by putting the "fossil" or "fossil.exe"
executable someplace on your $PATH or %PATH%.  After you have Fossil
installed, do this:

fossil clone https://www.sqlite.org/src sqlite.fossil


The command above
will make a copy of the complete development history of
SQLite into the "sqlite.fossil" file on your computer.  Making this copy
takes about a minute and uses about 32 megabytes of transfer.  After
making the copy, "open" the repository by typing:

fossil open sqlite.fossil


This second command will "checkout" the latest check-in from the SQLite
source tree into your current directory.  Subsequently, you can easily switch
to a different version by typing:

fossil update VERSION


Where VERSION can be a branch name (like "trunk" or "session") to get the
latest check-in on a specific branch, or VERSION can be a SHA1 hash or a
prefix of a SHA1 hash for a specific check-in, or VERSION can be a tag
such as "version-3.8.8".  Every time you run "fossil update" it will
automatically reach out to the original repository at
https://www.sqlite.org/src to obtain new check-ins that might have been
made by others since your previous update.


c2: The SQLITE_MEMSTAT Virtual Table
c5: memstat.html
c4:
1. Overview


The SQLITE_MEMSTAT extension implements an eponymous-only virtual table that
provides SQL access to the sqlite3_status64() and
sqlite3_db_status() interfaces.



The SQLITE_STMT extension can also be loaded at run-time
by compiling the extension into a shared library or DLL using the source
code at https://sqlite.org/src/file/ext/misc/memstat.c and following the
instructions for how to compile loadable extensions.


2. Usage


The SQLITE_MEMSTAT virtual table is a read-only table that can be
queried to determine performance characteristics (primarily the
amount of memory being used) of the current instance of SQLite.
The SQLITE_MEMSTATE table is essentially a wrapper around the
C-language APIs sqlite3_status64() and sqlite3_db_status().
If the 
memstat.c source
file is compiled with the -DSQLITE_ENABLE_ZIPVFS option, then SQLITE_MEMSTAT
will also do some file-control calls to extract
memory usage information about the 
ZIPVFS subsystem, 
if that subsystem as been licensed, installed, and is in use.


The SQLITE_MEMSTAT table appears to have the following schema:

CREATE TABLE sqlite_memstat(
  name TEXT,
  schema TEXT,
  value INT,
  hiwtr INT
);



Each row of the SQLITE_MEMSTAT table corresponds to a single call to
one of the sqlite3_status64() or sqlite3_db_status() interfaces.
The NAME column of the row identifies which "verb" was passed to those
interfaces.  For example, if sqlite3_status64() is invoked with
SQLITE_STATUS_MEMORY_USED, then the NAME column is 'MEMORY_USED'.
Or if sqlite3_db_status() is invoked with SQLITE_DBSTATUS_CACHE_USED,
then the NAME column is "DB_CACHE_USED".


The SCHEMA column is NULL, except for cases when the sqlite3_file_control()
interface is used to interrogate the ZIPVFS backend.  As this only happens
when the memstat.c module is compiled with -DSQLITE_ENABLE_ZIPVFS and when
ZIPVFS is in use, 
SCHEMA is usually NULL.


The VALUE and HIWTR columns report the current value of the measure and
its "high-water mark".  The high-water mark is the highest value ever seen
for the measurement, at least since the last reset.  The SQLITE_MEMSTAT
virtual table does not provide a mechanism for resetting the high-water mark.


Depending on which parameter is being interrogated, one of the VALUE
or HIWTR mark measurements might be undefined.  For example, only the
high-water mark is meaningful for SQLITE_STATUS_MALLOC_SIZE, and
only the current value is meaningful for SQLITE_DBSTATUS_CACHE_USED.
For rows where one or the other of VALUE or HIWTR is not meaningful,
that value is returned as NULL.
the 

interfaces, with the initial
This page last modified on  2022-01-08 05:02:57 UTC


c2: Full-Featured SQL
c5: fullsql.html
c4:
Do not be misled by the "Lite" in the name.  SQLite has a full-featured
SQL implementation, including:


Tables, indexes,
    triggers, and views
    in unlimited quantity
Up to 32K columns in a table and unlimited rows
Multi-column indexes
Indexes can use DESC and COLLATE
Partial indexes
Indexes On Expressions
Clustered indexes
Covering indexes
CHECK, UNIQUE, NOT NULL, and FOREIGN KEY constraints.
ACID transactions using BEGIN, COMMIT, and ROLLBACK
Nested transactions using SAVEPOINT, RELEASE, and 
    ROLLBACK TO
Subqueries, including correlated subqueries
Up to 64-way joins
LEFT, RIGHT, and FULL OUTER JOINs
DISTINCT, ORDER BY, GROUP BY, HAVING, LIMIT, and OFFSET
UNION, UNION ALL, INTERSECT, and EXCEPT
A rich library of standard SQL functions
Aggregate functions including DISTINCT aggregates
Window functions
UPDATE, DELETE, and INSERT (of course)
Common table expressions including
    recursive common table expressions
Row values
UPSERT
An advanced query planner
Full-text search
R-tree indexes
JSON support
The IS operator
Table-valued functions
REPLACE INTO
VACUUM
REINDEX
The GLOB operator
Hexadecimal integer literals
The ON CONFLICT clause
The INDEXED BY clause
Virtual tables
Multiple databases on the same database connection using
    ATTACH DATABASE
The ability to add application-defined SQL functions, including
    aggregate and table-valued functions.
Application-defined collating functions



There are many more features not listed above.
SQLite may be small in size and have "Lite" in its name, but it is
not lacking in capability.


c2: Change in Default Page Size in SQLite Version 3.12.0
c5: pgszchng2016.html
c4:
The Default Page Size Change of SQLite 3.12.0

1.0 Introduction


An SQLite database file consists of one or more "pages".
For a single database file, all pages are the same size, though
for different database files, the page size can any power of
two between 512 and 65536, inclusive.



Since the SQLite database file format was designed (in 2003) the
default page size for new databases has been 1024 bytes.  
This was a reasonable choice in 2003.  But on modern hardware, 
a 4096 byte page is a faster and better choice.
So, beginning with SQLite version 3.12.0 (2016-03-29)) the default 
page size for new database files has been increased to 4096 bytes.



The upper bound on the database cache size has 
traditionally defaulted to 2000 pages.  SQLite version 3.12.0 also
changes this default setting to be "-2000" which means 2000*1024
bytes, regardless of page size.  So, the upper bound on the amount
of memory used for the page cache is unchanged.


2.0 Not a Compatibility Break


These changes in the default behavior of SQLite are not
a compatibility break.  All legacy database files continue to be
readable and writable by newer versions of SQLite, and all newly
created database files continue to be readable and writable by
legacy versions of the SQLite library.  The only thing that is changing
is some default settings.  This should result in a performance
increase for many applications.



Though most application should not notice any change (except that
they run a little faster), if problems arise then the
legacy behavior can be restored at compile-time by using the
following options to the C-compiler:


-DSQLITE_DEFAULT_PAGE_SIZE=1024 
-DSQLITE_DEFAULT_CACHE_SIZE=2000



The page size and cache size can also be set or changed at run-time using the
page_size pragma and cache_size pragma, respectively.

3.0 Possible Negative Consequences Of This Change


The minimum size of an SQLite database is one page for each table and
each index.  With a larger page size, the size of an empty database
for a given schema will grow by a factor of four, therefore.  However,
once the database begins to fill with content the size of the older
1024-byte page databases and the newer 4096-byte page databases will
quickly converge.  Due to relaxed bin-packing constraints, the 
4096-byte page size might actually result in a smaller file, once
substantial content is added.

This page last modified on  2022-01-08 05:02:57 UTC


c2: Zero-Configuration
c5: zeroconf.html
c4:
SQLite Is A Zero-Configuration Database


SQLite does not need to be "installed" before it is used. 
There is no "setup" procedure.  There is no
server process that needs to be started, stopped, or configured.
There is
no need for an administrator to create a new database instance or assign
access permissions to users.
SQLite uses no configuration files.
Nothing needs to be done to tell the system that SQLite is running.
No actions are required to recover after a system crash or power failure.
There is nothing to troubleshoot.




SQLite just works.



Other database engines may run great once you get them going.
But doing the initial installation and configuration can often
be intimidating.

This page last modified on  2022-01-08 05:02:57 UTC


c2: Rowid Tables
c5: rowidtable.html
c4:
Rowid Tables

1.0 Definition

A "rowid table" is any table in an SQLite schema that

is not a virtual table, and
is not a WITHOUT ROWID table.

Most tables in a typical SQLite database schema are rowid tables.

Rowid tables are distinguished by the fact that they all have
a unique, non-NULL, signed 64-bit integer rowid that is used as
the access key for the data in the underlying B-tree storage engine.

2.0 Quirks



The PRIMARY KEY of a rowid table (if there is one) is usually not the
true primary key for the table, in the sense that it is not the unique
key used by the underlying B-tree storage engine.  The exception to
this rule is when the rowid table declares an INTEGER PRIMARY KEY.
In the exception, the INTEGER PRIMARY KEY becomes an alias for the 
rowid.


The true primary key for a rowid table (the value that is used as the
key to look up rows in the underlying B-tree storage engine)
is the rowid.


The PRIMARY KEY constraint for a rowid table (as long as it is not
the true primary key or INTEGER PRIMARY KEY) is really the same thing
as a UNIQUE constraint.  Because it is not a true primary key,
columns of the PRIMARY KEY are allowed to be NULL, in violation of
all SQL standards.


The rowid of a rowid table can be accessed (or changed) by reading or
writing to any of the "rowid" or "oid" or "_rowid_" columns.  Except,
if there is a declared columns in the table that use those
special names, then those names refer to the declared columns, not to
the underlying rowid.


Access to records via rowid is highly optimized and very fast.


If the rowid is not aliased by INTEGER PRIMARY KEY then it is not
persistent and might change.  In particular the VACUUM command will
change rowids for tables that do not declare an INTEGER PRIMARY KEY.
Therefore, applications should not normally access the rowid directly,
but instead use an INTEGER PRIMARY KEY.


In the underlying file format, each rowid is stored as a
variable-length integer.  That means that small non-negative
rowid values take up less disk space than large or negative
rowid values.


All of the complications above (and others not mentioned here)
arise from the need to preserve backwards
compatibility for the hundreds of billions of SQLite database files in
circulation.  In a perfect world, there would be no such thing as a "rowid"
and all tables would following the standard semantics implemented as
WITHOUT ROWID tables, only without the extra "WITHOUT ROWID" keywords.
Unfortunately, life is messy.  The designer of SQLite offers his
sincere apology for the current mess.

This page last modified on  2022-01-08 05:02:57 UTC


c2: NULL Handling in SQLite
c5: nulls.html
c4:
NULL Handling in SQLite Versus Other Database Engines


The goal is
to make SQLite handle NULLs in a standards-compliant way.
But the descriptions in the SQL standards on how to handle
NULLs seem ambiguous. 
It is not clear from the standards documents exactly how NULLs should
be handled in all circumstances.



So instead of going by the standards documents, various popular
SQL engines were tested to see how they handle NULLs.  The idea
was to make SQLite work like all the other engines.
An SQL test script was developed and run by volunteers on various
SQL RDBMSes and the results of those tests were used to deduce
how each engine processed NULL values.
The original tests were run in May of 2002.
A copy of the test script is found at the end of this document.



SQLite was originally coded in such a way that the answer to
all questions in the chart below would be "Yes".  But the
experiments run on other SQL engines showed that none of them
worked this way.  So SQLite was modified to work the same as
Oracle, PostgreSQL, and DB2.  This involved making NULLs
indistinct for the purposes of the SELECT DISTINCT statement and
for the UNION operator in a SELECT.  NULLs are still distinct
in a UNIQUE column.  This seems somewhat arbitrary, but the desire
to be compatible with other engines outweighed that objection.



It is possible to make SQLite treat NULLs as distinct for the
purposes of the SELECT DISTINCT and UNION.  To do so, one should
change the value of the NULL_ALWAYS_DISTINCT #define in the
sqliteInt.h source file and recompile.




Update 2003-07-13:
Since this document was originally written some of the database engines
tested have been updated and users have been kind enough to send in
corrections to the chart below.  The original data showed a wide variety
of behaviors, but over time the range of behaviors has converged toward
the PostgreSQL/Oracle model.  The only significant difference 
is that Informix and MS-SQL both treat NULLs as
indistinct in a UNIQUE column.



The fact that NULLs are distinct for UNIQUE columns but are indistinct for
SELECT DISTINCT and UNION continues to be puzzling.  It seems that NULLs
should be either distinct everywhere or nowhere.  And the SQL standards
documents suggest that NULLs should be distinct everywhere.  Yet as of
this writing, no SQL engine tested treats NULLs as distinct in a SELECT
DISTINCT statement or in a UNION.





The following table shows the results of the NULL handling experiments.



&nbsp;&nbsp;
SQLite
PostgreSQL
Oracle
Informix
DB2
MS-SQL
OCELOT


Adding anything to null gives null
Yes
Yes
Yes
Yes
Yes
Yes
Yes

Multiplying null by zero gives null
Yes
Yes
Yes
Yes
Yes
Yes
Yes

nulls are distinct in a UNIQUE column
Yes
Yes
Yes
No
(Note 4)
No
Yes

nulls are distinct in SELECT DISTINCT
No
No
No
No
No
No
No

nulls are distinct in a UNION
No
No
No
No
No
No
No

"CASE WHEN null THEN 1 ELSE 0 END" is 0?
Yes
Yes
Yes
Yes
Yes
Yes
Yes

"null OR true" is true
Yes
Yes
Yes
Yes
Yes
Yes
Yes

"not (null AND false)" is true
Yes
Yes
Yes
Yes
Yes
Yes
Yes




&nbsp;&nbsp;
MySQL3.23.41
MySQL4.0.16
Firebird
SQLAnywhere
BorlandInterbase


Adding anything to null gives null
Yes
Yes
Yes
Yes
Yes

Multiplying null by zero gives null
Yes
Yes
Yes
Yes
Yes

nulls are distinct in a UNIQUE column
Yes
Yes
Yes
(Note 4)
(Note 4)

nulls are distinct in SELECT DISTINCT
No
No
No (Note 1)
No
No

nulls are distinct in a UNION
(Note 3)
No
No (Note 1)
No
No

"CASE WHEN null THEN 1 ELSE 0 END" is 0?
Yes
Yes
Yes
Yes
(Note 5)

"null OR true" is true
Yes
Yes
Yes
Yes
Yes

"not (null AND false)" is true
No
Yes
Yes
Yes
Yes





Notes:&nbsp;&nbsp;
1.&nbsp;
Older versions of firebird omit all NULLs from SELECT DISTINCT
and from UNION.

2.&nbsp;
Test data unavailable.

3.&nbsp;
MySQL version 3.23.41 does not support UNION.

4.&nbsp;
DB2, SQL Anywhere, and Borland Interbase 
do not allow NULLs in a UNIQUE column.

5.&nbsp;
Borland Interbase does not support CASE expressions.




&nbsp;

The following script was used to gather information for the table
above.



-- I have about decided that SQL's treatment of NULLs is capricious and cannot be
-- deduced by logic.  It must be discovered by experiment.  To that end, I have 
-- prepared the following script to test how various SQL databases deal with NULL.
-- My aim is to use the information gathered from this script to make SQLite as
-- much like other databases as possible.
--
-- If you could please run this script in your database engine and mail the results
-- to me at drh@hwaci.com, that will be a big help.  Please be sure to identify the
-- database engine you use for this test.  Thanks.
--
-- If you have to change anything to get this script to run with your database
-- engine, please send your revised script together with your results.
--

-- Create a test table with data
create table t1(a int, b int, c int);
insert into t1 values(1,0,0);
insert into t1 values(2,0,1);
insert into t1 values(3,1,0);
insert into t1 values(4,1,1);
insert into t1 values(5,null,0);
insert into t1 values(6,null,1);
insert into t1 values(7,null,null);

-- Check to see what CASE does with NULLs in its test expressions
select a, case when b0 then 1 else 0 end from t1;
select a+10, case when not b0 then 1 else 0 end from t1;
select a+20, case when b0 and c0 then 1 else 0 end from t1;
select a+30, case when not (b0 and c0) then 1 else 0 end from t1;
select a+40, case when b0 or c0 then 1 else 0 end from t1;
select a+50, case when not (b0 or c0) then 1 else 0 end from t1;
select a+60, case b when c then 1 else 0 end from t1;
select a+70, case c when b then 1 else 0 end from t1;

-- What happens when you multiply a NULL by zero?
select a+80, b*0 from t1;
select a+90, b*c from t1;

-- What happens to NULL for other operators?
select a+100, b+c from t1;

-- Test the treatment of aggregate operators
select count(*), count(b), sum(b), avg(b), min(b), max(b) from t1;

-- Check the behavior of NULLs in WHERE clauses
select a+110 from t1 where b10;
select a+130 from t1 where b
This page last modified on  2022-04-18 02:55:50 UTC


c2: The SQLite R*Tree Module
c5: rtree.html
c4:
1. Overview


An R-Tree is a special
index that is designed for doing range queries.  R-Trees are most commonly
used in geospatial systems where each entry is a rectangle with minimum and
maximum X and Y coordinates.  Given a query rectangle, an R-Tree is able
to quickly find all entries that are contained within the query rectangle
or which overlap the query rectangle.  This idea is easily extended to
three dimensions for use in CAD systems.  R-Trees also find use in time-domain
range look-ups.  For example, suppose a database records the starting and
ending times for a large number of events.  A R-Tree is able to quickly
find all events that were active at any time during a given
time interval, or all events that started during a particular time interval,
or all events that both started and ended within a given time interval.
And so forth.



The R-Tree concept originated with
Toni Guttman:
R-Trees: A Dynamic Index Structure for Spatial Searching,
Proc. 1984 ACM SIGMOD International Conference on Management of Data,
pp. 47-57.
The implementation found in SQLite is a refinement of Guttman's original
idea, commonly called "R*Trees", that was described by
Norbert Beckmann, Hans-Peter Kriegel, Ralf Schneider, Bernhard Seeger:
The R*-Tree: An Efficient and Robust Access Method for Points
and Rectangles. SIGMOD Conference 1990: 322-331.


2. Compiling The R*Tree Module


The source code to the SQLite R*Tree module is included as part
of the amalgamation.  However, depending on configuration options
and the particular version of SQLite you are using, it may or may not
be enabled by default.  To ensure that the R*Tree module is enabled,
simply compile with the SQLITE_ENABLE_RTREE 
C-preprocessor macro defined.  With many compilers, this is accomplished
by adding the option "-DSQLITE_ENABLE_RTREE=1" to the compiler
command-line.


3. Using the R*Tree Module


The SQLite R*Tree module is implemented as a
virtual table.  Each R*Tree index is a
virtual table with an odd number of columns between 3 and 11.
The first column is always a 64-bit signed integer primary key.
The other columns are pairs, one pair per dimension, containing the
minimum and maximum values for that dimension, respectively.
A 1-dimensional R*Tree thus has 3 columns.
A 2-dimensional R*Tree has 5 columns.
A 3-dimensional R*Tree has 7 columns.
A 4-dimensional R*Tree has 9 columns.
And a 5-dimensional R*Tree has 11 columns.  The SQLite R*Tree implementation
does not support R*Trees wider than 5 dimensions.



The first column of an SQLite R*Tree is similar to an integer primary
key column of a normal SQLite table. It may only store a 64-bit signed
integer value. Inserting a NULL value into this column causes SQLite
to automatically generate a new unique primary key value. If an attempt
is made to insert any other non-integer value into this column,
the r-tree module silently converts it to an integer before writing it
into the database.

The min/max-value pair columns are stored as 32-bit floating point values for
"rtree" virtual tables or as 32-bit signed integers in "rtree_i32" virtual
tables.  Unlike regular SQLite tables which can store data in a variety of
datatypes and formats, the R*Tree rigidly enforce these storage types.
If any other type of value is inserted into such a column, the r-tree
module silently converts it to the required type before writing the
new record to the database.

3.1. Creating An R*Tree Index


A new R*Tree index is created as follows:


CREATE VIRTUAL TABLE &lt;name&gt; USING rtree(&lt;column-names&gt;);



The &lt;name&gt; is the name your application chooses for the
R*Tree index and &lt;column-names&gt; is a comma separated list
of between 3 and 11 columns.
The virtual &lt;name&gt; table creates three shadow tables to actually
store its content.  The names of these shadow tables are:


&lt;name&gt;_node
&lt;name&gt;_rowid
&lt;name&gt;_parent




The shadow tables are ordinary SQLite data tables.  You can query them
directly if you like, though this unlikely to reveal anything particularly
useful.
And you can UPDATE, DELETE, INSERT or even DROP
the shadow tables, though doing so will corrupt your R*Tree index.
So it is best to simply ignore the shadow tables.  Recognize that they
hold your R*Tree index information and let it go as that.



As an example, consider creating a two-dimensional R*Tree index for use in
spatial queries:


CREATE VIRTUAL TABLE demo_index USING rtree(
   id,              -- Integer primary key
   minX, maxX,      -- Minimum and maximum X coordinate
   minY, maxY       -- Minimum and maximum Y coordinate
);



3.1.1. Column naming details


In the argments to "rtree" in the CREATE VIRTUAL TABLE statement, the
names of the columns are taken from the first token of each argument.
All subsequent tokens within each argument are silently ignored.
This means, for example, that if you try to give a column a
type affinity or add a constraint such as UNIQUE or NOT NULL or DEFAULT to
a column, those extra tokens are accepted as valid, but they do not change
the behavior of the rtree.
In an RTREE virtual table, the first column always has a
type affinity of INTEGER and all other data columns have a
type affinity of REAL.
In an RTREE_I32 virtual table, all columns have type affinity of INTEGER.


Recommended practice is to omit any extra tokens in the rtree specification.
Let each argument to "rtree" be a single ordinary label that is the name of
the corresponding column, and omit all other tokens from the argument list.

3.2. Populating An R*Tree Index


The usual INSERT, UPDATE, and DELETE commands work on an R*Tree
index just like on regular tables.  So to insert some data into our sample
R*Tree index, we can do something like this:


INSERT INTO demo_index VALUES
  (28215, -80.781227, -80.604706, 35.208813, 35.297367),
  (28216, -80.957283, -80.840599, 35.235920, 35.367825),
  (28217, -80.960869, -80.869431, 35.133682, 35.208233),
  (28226, -80.878983, -80.778275, 35.060287, 35.154446),
  (28227, -80.745544, -80.555382, 35.130215, 35.236916),
  (28244, -80.844208, -80.841988, 35.223728, 35.225471),
  (28262, -80.809074, -80.682938, 35.276207, 35.377747),
  (28269, -80.851471, -80.735718, 35.272560, 35.407925),
  (28270, -80.794983, -80.728966, 35.059872, 35.161823),
  (28273, -80.994766, -80.875259, 35.074734, 35.172836),
  (28277, -80.876793, -80.767586, 35.001709, 35.101063),
  (28278, -81.058029, -80.956375, 35.044701, 35.223812),
  (28280, -80.844208, -80.841972, 35.225468, 35.227203),
  (28282, -80.846382, -80.844193, 35.223972, 35.225655);




The entries above are bounding boxes (longitude and latitude) for 14
zipcodes near Charlotte, NC.  A real database would have many thousands,
millions, or billions of such entries, but this small 14-row sample will
be sufficient to illustrate the ideas.


3.3. Querying An R*Tree Index


Any valid query will work against an R*Tree index.  The R*Tree
implementation just makes some kinds of queries especially
efficient.  Queries against the primary key are efficient:


SELECT * FROM demo_index WHERE id=28269;




Of course, an ordinary SQLite table will also do a query against its
integer primary key efficiently, so the previous is not important.
The big reason for using an R*Tree is so that
you can efficiently do range queries against the coordinate
ranges.  For example, the main office of the SQLite project is
located at 35.37785, -80.77470.
To find which zipcodes might service that office, one could right:


SELECT id FROM demo_index
 WHERE minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;




The query above will quickly locate all zipcodes that contain
the SQLite main office in their bounding box, even if the
R*Tree contains many entries.  The previous is an example
of a "contained-within" query.  The R*Tree also supports "overlapping"
queries.  For example, to find all zipcode bounding boxes that overlap
with the 28269 zipcode:


SELECT A.id FROM demo_index AS A, demo_index AS B
 WHERE A.maxX&gt;=B.minX AND A.minX&lt;=B.maxX
   AND A.maxY&gt;=B.minY AND A.minY&lt;=B.maxY
   AND B.id=28269;




This second query will find both 28269 entry (since every bounding box
overlaps with itself) and also other zipcode that is close enough to
28269 that their bounding boxes overlap.



Note that it is not necessary for all coordinates in an R*Tree index
to be constrained in order for the index search to be efficient.
One might, for example, want to query all objects that overlap with
the 35th parallel:


SELECT id FROM demo_index
 WHERE maxY&gt;=35.0  AND minY&lt;=35.0;




But, generally speaking, the more constraints that the R*Tree module
has to work with, and the smaller the bounding box, the faster the
results will come back.


3.4. Roundoff Error


By default, coordinates are stored in an R*Tree using 32-bit floating
point values.  When a coordinate cannot be exactly represented by a
32-bit floating point number, the lower-bound coordinates are rounded down
and the upper-bound coordinates are rounded up.  Thus, bounding boxes might
be slightly larger than specified, but will never be any smaller.  This
is exactly what is desired for doing the more common "overlapping" queries
where the application wants to find every entry in the R*Tree that overlaps
a query bounding box.  Rounding the entry bounding boxes outward might cause a
few extra entries to appears in an overlapping query if the edge of the
entry bounding box corresponds to an edge of the query bounding box.  But
the overlapping query will never miss a valid table entry.

However, for a "contained-within" style query, rounding the bounding
boxes outward might cause some entries to be excluded from the result set
if the edge of the entry bounding box corresponds to the edge of the query
bounding box.  To guard against this, applications should expand their
contained-within query boxes slightly (by 0.000012%) by rounding down the
lower coordinates and rounding up the top coordinates, in each dimension.



3.5. Reading And Writing At The Same Time


It is the nature of the Guttman R-Tree algorithm that any write might
radically restructure the tree, and in the process change the scan order
of the nodes.  For this reason, it is not generally possible to modify
the R-Tree in the middle of a query of the R-Tree.  Attempts to do so
will fail with a SQLITE_LOCKED "database table is locked" error.


So, for example, suppose an application runs one query against an R-Tree like
this:

SELECT id FROM demo_index
 WHERE maxY&gt;=35.0  AND minY&lt;=35.0;



Then for each "id" value returned, suppose the application creates an
UPDATE statement like the following and binds the "id" value returned against
the "?1" parameter:

UPDATE demo_index SET maxY=maxY+0.5 WHERE id=?1;



Then the UPDATE might fail with an SQLITE_LOCKED error.  The reason is that
the initial query has not run to completion.  It is remembering its place
in the middle of a scan of the R-Tree.  So an update to the R-Tree cannot
be tolerated as this would disrupt the scan.


This is a limitation of the R-Tree extension only.  Ordinary tables in
SQLite are able to read and write at the same time.  Other virtual tables
might (or might not) also that capability.  And R-Tree can appear to read
and write at the same time in some circumstances, if it can figure out how
to reliably run the query to completion before starting the update.  But
you shouldn't count on that for every query.  Generally speaking, it is
best to avoid running queries and updates to the same R-Tree at the same
time.


If you really need to update an R-Tree based on complex queries against
the same R-Tree, it is best to run the complex queries first and store
the results in a temporary table, then update the R-Tree based on the values
stored in the temporary table.

4. Using R*Trees Effectively


For SQLite versions prior to 3.24.0 (2018-06-04),
the only information that an R*Tree index stores about an object is
its integer ID and its bounding box.  Additional information needs to
be stored in separate tables and related to the R*Tree index using
the primary key.  For the example above, one might create an auxiliary
table as follows:


CREATE TABLE demo_data(
  id INTEGER PRIMARY KEY,  -- primary key
  objname TEXT,            -- name of the object
  objtype TEXT,            -- object type
  boundary BLOB            -- detailed boundary of object
);



In this example, the demo_data.boundary field is intended to hold some
kind of binary representation of the precise boundaries of the object.
The R*Tree index only holds an axis-aligned rectangular boundary for the
object.  The R*Tree boundary is just an approximation of the true object
boundary.  So what typically happens is that the R*Tree index is used to
narrow a search down to a list of candidate objects and then more detailed
and expensive computations are done on each candidate to find if the
candidate truly meets the search criteria.



Key Point:
An R*Tree index does not normally provide the exact answer but merely
reduces the set of potential answers from millions to dozens.



Suppose the demo_data.boundary field holds some proprietary data description
of a complex two-dimensional boundary for a zipcode and suppose that the
application has used the sqlite3_create_function() interface to
created an application-defined function "contained_in(boundary,lat,long)"
that accepts the demo_data.boundary object and a latitute and longitude
and returns return true or false if the lat/long is contained within
the boundary.
One may assume that "contained_in()" is a relatively slow
functions that we do not want to invoke too frequently.
Then an efficient way to find the specific ZIP code for the main
SQLite office would be to run a query like this:



SELECT objname FROM demo_data, demo_index
 WHERE demo_data.id=demo_index.id
   AND contained_in(demo_data.boundary, 35.37785, -80.77470)
   AND minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;


Notice how the query above works:  The R*Tree index runs in the outer
loop to find entries that contain the SQLite main office in their
boundary box.
For each row found, SQLite looks up
the corresponding entry in the demo_data table.  It then uses the boundary
field from the demo_data table as a parameter to the contained_in()
function and if that function returns true, then we know the sought after
coordinate is in that ZIP code boundary.

One would get the same answer without the use of the R*Tree index
using the following simpler query:

SELECT objname FROM demo_data
 WHERE contained_in(demo_data.boundary, 35.37785, -80.77470);


The problem with this latter query is that it must apply the
contained_in() function to all entries in the demo_data table.
The use of the R*Tree in the penultimate query reduces the number of
calls to contained_in() function to a small subset of the entire table.
The R*Tree index did not find the exact answer itself, it merely
limited the search space.



4.1. Auxiliary Columns


Beginning with SQLite version 3.24.0 (2018-06-04), r-tree tables
can have auxiliary columns that store arbitrary data.
Auxiliary columns can be used in place of
secondary tables such as "demo_data".


Auxiliary columns are marked with a "+" symbol before the column name.
Auxiliary columns must come after all of the coordinate boundary columns.
An RTREE table can have no more than 100 columns total.  In other words,
the count of columns including the integer primary key column,
the coordinate boundary columns, and all auxiliary columns must be 100 or less.
The following example shows an r-tree table with auxiliary columns that
is equivalent to the two tables "demo_index" and "demo_data" above:

CREATE VIRTUAL TABLE demo_index2 USING rtree(
   id,              -- Integer primary key
   minX, maxX,      -- Minimum and maximum X coordinate
   minY, maxY,      -- Minimum and maximum Y coordinate
   +objname TEXT,   -- name of the object
   +objtype TEXT,   -- object type
   +boundary BLOB   -- detailed boundary of object
);



By combining location data and related information into the same
table, auxiliary columns can provide a cleaner model
and reduce the need to joins.
For example, the earlier
join between demo_index and demo_data can now
be written as a simple query, like this:

SELECT objname FROM demo_index2
 WHERE contained_in(boundary, 35.37785, -80.77470)
   AND minX&lt;=-80.77470 AND maxX&gt;=-80.77470
   AND minY&lt;=35.37785  AND maxY&gt;=35.37785;


4.1.1. Limitations


For auxiliary columns, only the name of the column matters.
The type affinity is ignored.
Constraints such as NOT NULL, UNIQUE, REFERENCES, or CHECK
are also ignored.  However, future versions
of SQLite might start paying attention to the type affinity and
constraints, so users of auxiliary columns are advised to leave
both blank, to avoid future compatibility problems.



5. Integer-Valued R-Trees


The default virtual table ("rtree") stores coordinates as
single-precision (4-byte) floating point numbers.  If integer coordinates
are desired, declare the table using "rtree_i32" instead:

CREATE VIRTUAL TABLE intrtree USING rtree_i32(id,x0,x1,y0,y1,z0,z1);



An rtree_i32 stores coordinates as 32-bit signed integers.
Even though it stores values using integer, the rtree_i32 virtual
table still uses floating point computations internally as part of
the r-tree algorithm.



6. Custom R-Tree Queries

By using standard SQL expressions in the WHERE clause of a SELECT query,
a programmer can query for all R*Tree entries that
intersect with or are contained within a particular bounding-box.
Custom R*Tree queries, using the MATCH
operator in the WHERE clause of a SELECT, allow the programmer to query for
the set of R*Tree entries that intersect any arbitrary region or shape, not
just a box.  This capability is useful, for example, in computing the
subset of objects in the R*Tree that are visible from a camera positioned
in 3-D space.

Regions for custom R*Tree queries are defined by R*Tree geometry callbacks
implemented by the application and registered with SQLite via a call to one
of the following two APIs:

int sqlite3_rtree_query_callback(
  sqlite3 *db,
  const char *zQueryFunc,
  int (*xQueryFunc)(sqlite3_rtree_query_info*),
  void *pContext,
  void (*xDestructor)(void*)
);
int sqlite3_rtree_geometry_callback(
  sqlite3 *db,
  const char *zGeom,
  int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes),
  void *pContext
);


The sqlite3_rtree_query_callback() became available with SQLite
version 3.8.5 (2014-06-04) and is the preferred interface.
The sqlite3_rtree_geometry_callback() is an older and less flexible
interface that is supported for backwards compatibility.

A call to one of the above APIs creates a new SQL function named by the
second parameter (zQueryFunc or zGeom).  When that SQL function appears
on the right-hand side of the MATCH operator and the left-hand side of the
MATCH operator is any column in the R*Tree virtual table, then the callback
defined by the third argument (xQueryFunc or xGeom) is invoked to determine
if a particular object or subtree overlaps the desired region.

For example, a query like the following might be used to find all
R*Tree entries that overlap with a circle centered a 45.3,22.9 with a
radius of 5.0:

SELECT id FROM demo_index WHERE id MATCH circle(45.3, 22.9, 5.0)


The SQL syntax for custom queries is the same regardless of which
interface, sqlite3_rtree_geometry_callback() or sqlite3_rtree_query_callback(),
is used to register the SQL function.  However, the newer query-style
callbacks give the application greater control over how the query proceeds.

6.1. The Legacy xGeom Callback

The legacy xGeom callback is invoked with four arguments.  The first
argument is a pointer to an sqlite3_rtree_geometry structure which provides
information about how the SQL function was invoked.  The second argument
is the number of coordinates in each r-tree entry, and is always the same
for any given R*Tree.  The number of coordinates is 2 for a 1-dimensional R*Tree,
4 for a 2-dimensional R*Tree, 6 for a 3-dimensional R*Tree, and so forth.
The third argument, aCoord[], is an array of nCoord coordinates that defines
a bounding box to be tested.  The last argument is a pointer into which
the callback result should be written.  The result is zero
if the bounding-box defined by aCoord[] is completely outside
the region defined by the xGeom callback and the result is non-zero if
the bounding-box is inside or overlaps with the xGeom region.  The xGeom
callback should normally return SQLITE_OK.  If xGeom returns anything other
than SQLITE_OK, then the r-tree query will abort with an error.

The sqlite3_rtree_geometry structure that the first argument to the
xGeom callback points to has a structure shown below.  The exact same
sqlite3_rtree_geometry
structure is used for every callback for same MATCH operator in the same
query.  The contents of the sqlite3_rtree_geometry
structure are initialized by SQLite but are
not subsequently modified.  The callback is free to make changes to the
pUser and xDelUser elements of the structure if desired.

typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry;
struct sqlite3_rtree_geometry {
  void *pContext;                 /* Copy of pContext passed to s_r_g_c() */
  int nParam;                     /* Size of array aParam */
  double *aParam;                 /* Parameters passed to SQL geom function */
  void *pUser;                    /* Callback implementation user data */
  void (*xDelUser)(void *);       /* Called by SQLite to clean up pUser */
};


The pContext member of the sqlite3_rtree_geometry
structure is always set to a copy of the pContext
argument passed to sqlite3_rtree_geometry_callback() when the
callback is registered. The aParam[] array (size nParam) contains the parameter
values passed to the SQL function on the right-hand side of the MATCH operator.
In the example "circle" query above, nParam would be set to 3 and the aParam[]
array would contain the three values 45.3, 22.9 and 5.0.

The pUser and xDelUser members of the sqlite3_rtree_geometry structure are
initially set to NULL. The pUser variable may be set by the callback
implementation to any arbitrary value that may be useful to subsequent
invocations of the callback within the same query (for example, a
pointer to a complicated data structure used to test for region intersection).
If the xDelUser variable is set to a non-NULL value, then after the
query has finished running SQLite automatically invokes it with the
value of the pUser variable as the only argument. In other words, xDelUser
may be set to a destructor function for the pUser value.

The xGeom callback always does a depth-first search of the r-tree.



6.2. The New xQueryFunc Callback

The newer xQueryFunc callback receives more information from the r-tree
query engine on each call, and it sends more information back to the query engine
before it returns.
To help keep the interface manageable, the xQueryFunc callback sends and receives
information from the query engine as fields in the
sqlite3_rtree_query_info structure:

struct sqlite3_rtree_query_info {
  void *pContext;                   /* pContext from when function registered */
  int nParam;                       /* Number of function parameters */
  sqlite3_rtree_dbl *aParam;        /* value of function parameters */
  void *pUser;                      /* callback can use this, if desired */
  void (*xDelUser)(void*);          /* function to free pUser */
  sqlite3_rtree_dbl *aCoord;        /* Coordinates of node or entry to check */
  unsigned int *anQueue;            /* Number of pending entries in the queue */
  int nCoord;                       /* Number of coordinates */
  int iLevel;                       /* Level of current node or entry */
  int mxLevel;                      /* The largest iLevel value in the tree */
  sqlite3_int64 iRowid;             /* Rowid for current entry */
  sqlite3_rtree_dbl rParentScore;   /* Score of parent node */
  int eParentWithin;                /* Visibility of parent node */
  int eWithin;                      /* OUT: Visiblity */
  sqlite3_rtree_dbl rScore;         /* OUT: Write the score here */
  /* The following fields are only available in 3.8.11 and later */
  sqlite3_value **apSqlParam;       /* Original SQL values of parameters */
};


The first five fields of the sqlite3_rtree_query_info structure are identical
to the sqlite3_rtree_geometry structure, and have exactly the same meaning.
The sqlite3_rtree_query_info structure also contains nCoord and aCoord fields
which have the same meaning as the parameter of the same name in the xGeom callback.

The xQueryFunc must set the eWithin field of sqlite3_rtree_query_info to
one of the values NOT_WITHIN, PARTLY_WITHIN, or FULLY_WITHIN depending on whether
or not the bounding box defined by aCoord[] is completely outside the region,
overlaps the region, or is completely inside the region, respectively.  In
addition, the xQueryFunc must set the rScore field to a non-negative value that
indicates the order in which subtrees and entries of the query should be analyzed
and returned.  Smaller scores are processed first.

As its name implies, an R*Tree is organized as a tree.  Each node of the
tree is a bounding box.  The root of the tree is a bounding box that encapsulates
all elements of the tree.  Beneath the root are a number of subtrees (typically
20 or more) each with their own smaller bounding boxes and each containing some
subset of the R*Tree entries.  The subtrees may have sub-subtrees, and so forth
until finally one reaches the leaves of the tree which are the actual R*Tree
entries.

An R*Tree query is initialized by making the root node the only entry
in a priority queue sorted by rScore.
The query proceeds by extracting the entry from the priority queue that has
the lowest score.  If that entry is a leaf (meaning that it is an actual
R*Tree entry and not a subtree) then that entry
is returned as one row of the query result.
If the extracted priority queue entry is a node (a subtree),
then the next child of that node is passed to the xQueryFunc callback.
If the node has more children then it is returned to the priority queue.
Otherwise it is discarded. Those subelements for which the xQueryFunc
callback sets eWithin to PARTLY_WITHIN or FULLY_WITHIN are added to the
priority queue using the score supplied by the callback.  Subelements that
return NOT_WITHIN are discarded.  The query runs until the priority queue is
empty.

Every leaf entry and node (subtree) within the R*Tree has an integer "level".
The leaves have a level of 0.  The first containing subtree of the leaves has
a level of 1.  The root of the R*Tree has the largest level value.  The
mxLevel entry in the sqlite3_rtree_query_info structure is the level value for
the root of the R*Tree.  The iLevel entry in sqlite3_rtree_query_info gives the
level for the object being interrogated.

Most R*Tree queries use a depth-first search.  This is accomplished by setting
the rScore equal to iLevel.  A depth-first search is usually preferred since it
minimizes the number of elements in the priority queue, which reduces memory
requirements and speeds processing.  However, some application may prefer a
breadth-first search, which can be accomplished by setting rScore to mxLevel-iLevel.
By creating more complex formulas for rScore, applications can exercise
detailed control over the order in which subtree are searched and leaf
R*Tree entries are returned.  For example, in an application with many
millions of R*Tree entries, the rScore might be arranged so that the
largest or most significant entries are returned first, allowing the
application to display the most important information quickly, and
filling in smaller and less important details as they become available.

Other information fields of the sqlite3_rtree_query_info structure are
available for use by the xQueryFunc callback, if desired.  The iRowid field
is the rowid (the first of the 3 to 11 columns in the R*Tree) for the element
being considered.  iRowid is only valid for leaves.  The eParentWithin and
rParentScore values are copies of the eWithin and rScore values from the
containing subtree of the current row.  The anQueue field is an array
of mxLevel+1 unsigned integers that tell the current number of elements in
the priority queue at each level.

6.3. Additional Considerations for Custom Queries


The MATCH operator of a custom R*Tree query function must be a top-level
AND-connected term of the WHERE clause, or else it will not be usable
by the R*Tree query optimizer and the query will not be runnable.
If the MATCH operator is connected to other terms of the WHERE clause
via an OR operator, for example, the query will fail with an error.


Two or more MATCH operators are allowed in the same WHERE clause, as long
as they are connected by AND operators.  However,
the R*Tree query engine only contains a single priority queue.  The priority
assigned to each node in the search is the lowest priority returned by any
of the MATCH operators.

7. Implementation Details


The following sections describe some low-level details of the R*Tree implementation,
that might be useful for trouble-shooting or performance analysis.



7.1. Shadow Tables

The content of an R*Tree index is actually stored in three ordinary
SQLite tables with names derived from the name of the R*Tree.  These
three tables are called "shadow tables".  This is their schema:

CREATE TABLE %_node(nodeno INTEGER PRIMARY KEY, data)
CREATE TABLE %_parent(nodeno INTEGER PRIMARY KEY, parentnode)
CREATE TABLE %_rowid(rowid INTEGER PRIMARY KEY, nodeno)


The "%" in the name of each shadow table is replaced by the name of the
R*Tree virtual table.  So, if the name of the R*Tree table is "xyz" then
the three shadow tables would be "xyz_node", "xyz_parent", and "xyz_rowid".

There is one entry in the %_node table for each R*Tree node.  An
R*Tree node consists of one or more entries that are proximate to one another.
The nodes of an R*Tree for a tree.  All nodes other than the root have an
entry in the %_parent shadow table that identifies the parent node.
Each entry in an R*Tree has a rowid.  The %_rowid shadow table maps entry
rowids to the node that contains that entry.

Extra columns appended to the %_rowid table hold the
content of auxiliary columns.  The names of these extra
%_rowid columns are probably not the same as the
actual auxiliary column names.



7.2. Integrity Check using the rtreecheck() SQL function

The scalar SQL function rtreecheck(R) or rtreecheck(S,R) runs an
integrity check on the rtree table named R contained within database S.
The function returns a human-language description of any problems found,
or the string 'ok' if everything is ok.  Running rtreecheck() on an R*Tree
virtual table is similar to running PRAGMA integrity_check on a
database.

Example: To verify that an R*Tree named "demo_index" is well-formed
and internally consistent, run:

SELECT rtreecheck('demo_index');


The rtreecheck() function performs the following checks:



For each cell in the r-tree structure (%_node table), that:

 for each dimension, (coord1 &lt;= coord2).
 unless the cell is on the root node, that the cell is bounded
        by the parent cell on the parent node.
 for leaf nodes, that there is an entry in the %_rowid
        table corresponding to the cell's rowid value that
        points to the correct node.
 for cells on non-leaf nodes, that there is an entry in the
        %_parent table mapping from the cell's child node to the
        node that it resides on.


That there are the same number of entries in the %_rowid table
as there are leaf cells in the r-tree structure, and that there
is a leaf cell that corresponds to each entry in the %_rowid table.

That there are the same number of entries in the %_parent table
as there are non-leaf cells in the r-tree structure, and that
there is a non-leaf cell that corresponds to each entry in the
%_parent table.

This page last modified on  2023-02-20 00:00:42 UTC


c2: The Session Extension
c5: sessionintro.html
c4:
1. Introduction

The session extension provide a mechanism for conviently recording
changes to some or all of certain tables in an SQLite database, and
packaging those changes into a "changeset" or "patchset" file that can
later be used to apply the same set of changes to another database with
the same schema and compatible starting data.  A "changeset" may
also be inverted and used to "undo" a session.

This document is an introduction to the session extension.
The details of the interface are in the separate
Session Extension C-language Interface document.

1.1. Typical Use Case

Suppose SQLite is used as the application file format for a
particular design application.  Two users, Alice and Bob, each start
with a baseline design that is about a gigabyte in size.  They work
all day, in parallel, each making their own customizations and tweaks
to the design.  At the end of the day, they would like to merge their
changes together into a single unified design.

The session extension facilitates this by recording all changes to
both Alice's and Bob's databases and writing those changes into
changeset or patchset files.  At the end of the day, Alice can send her
changeset to Bob and Bob can "apply" it to his database.  The result (assuming
there are no conflicts) is that Bob's database then contains both his
changes and Alice's changes.  Likewise, Bob can send a changeset of
his work over to Alice and she can apply his changes to her database.

In other words, the session extension provides a facility for
SQLite database files that is similar to the unix
patch utility program,
or to the "merge" capabilities of version control systems such
as Fossil, Git,
or Mercurial.

1.2. Obtaining the Session Extension

 Since version 3.13.0 (2016-05-18),
the session extension has been included in the SQLite
amalgamation source distribution. By default, the session extension is
disabled. To enable it, build with the following compiler switches:

-DSQLITE_ENABLE_SESSION -DSQLITE_ENABLE_PREUPDATE_HOOK


 Or, if using the autoconf build system,
pass the --enable-session option to the configure script.

1.3. Limitations



 Prior to SQLite version 3.17.0, the session extension only worked with
        rowid tables, not WITHOUT ROWID tables. As of 3.17.0, both
        rowid and WITHOUT ROWID tables are supported. However, extra steps are
        needed to record primary keys for WITHOUT ROWID table changes.

 There is no support for virtual tables. Changes to virtual tables are
        not captured.

 The session extension only works with tables that have a declared
        PRIMARY KEY. The PRIMARY KEY of a table may be an INTEGER PRIMARY KEY
        (rowid alias) or an external PRIMARY KEY.

 SQLite allows NULL values to be stored in
        PRIMARY KEY columns. However, the session extension ignores all
        such rows. No changes affecting rows with one or more NULL values
        in PRIMARY KEY columns are recorded by the sessions module.


2. Concepts



2.1. Changesets and Patchsets
 The sessions module revolves around creating and manipulating
changesets. A changeset is a blob of data that encodes a series of
changes to a database. Each change in a changeset is one of the
following:


   An INSERT. An INSERT change contains a single row to add to
       a database table. The payload of the INSERT change consists of the
       values for each field of the new row.

   A DELETE. A DELETE change represents a row, identified by
       its primary key values, to remove from a database table. The payload
       of a DELETE change consists of the values for all fields of the
       deleted row.

   An UPDATE. An UPDATE change represents the modification of
       one or more non-PRIMARY KEY fields of a single row within a database
       table, identified by its PRIMARY KEY fields. The payload for an UPDATE
       change consists of:
   
      The PRIMARY KEY values identifying the modified row,
      The new values for each modified field of the row, and
      The original values for each modified field of the row.
   
        An UPDATE change does not contain any information regarding
       non-PRIMARY KEY fields that are not modified by the change. It is not
       possible for an UPDATE change to specify modifications to PRIMARY
       KEY fields.


 A single changeset may contain changes that apply to more than one
database table. For each table that the changeset includes at least one change
for, it also encodes the following data:


   The name of the database table,
   The number of columns the table has, and
   Which of those columns are PRIMARY KEY columns.


 Changesets may only be applied to databases that contain tables
matching the above three criteria as stored in the changeset.

 A patchset is similar to a changeset. It is slightly more compact than
a changeset, but provides more limited conflict detection and resolution
options (see the next section for details). The differences between a
patchset and a changeset are that:


   For a DELETE change, the payload consists of the PRIMARY KEY
          fields only. The original values of other fields are not stored as
          part of a patchset.

   For an UPDATE change, the payload consists of the PRIMARY KEY
          fields and the new values of modified fields only. The original
          values of modified fields are not stored as part of a patchset.


2.2. Conflicts

 When a changeset or patchset is applied to a database, an attempt is
made to insert a new row for each INSERT change, remove a row for each
DELETE change and modify a row for each UPDATE change. If the target
database is in the same state as the original database that the changeset
was recorded on, this is a simple matter. However, if the contents of the
target database is not in exactly this state, conflicts can occur when
applying the changeset or patchset.

When processing an INSERT change, the following conflicts can
occur:


   The target database may already contain a row with the same PRIMARY
       KEY values as specified by the INSERT change.

   Some other database constraint, for example a UNIQUE or CHECK
       constraint, may be violated when the new row is inserted.


When processing a DELETE change, the following conflicts may be
detected:


   The target database may contain no row with the specified PRIMARY
       KEY values to delete.

   The target database may contain a row with the specified PRIMARY
       KEY values, but the other fields may contain values that do not
       match those stored as part of the changeset. This type of conflict
       is not detected when using a patchset.


When processing an UPDATE change, the following conflicts may be
detected:


   The target database may contain no row with the specified PRIMARY
       KEY values to modify.

   The target database may contain a row with the specified PRIMARY
       KEY values, but the current values of the fields that will be modified
       by the change may not match the original values stored within the
       changeset. This type of conflict is not detected when using a patchset.

   Some other database constraint, for example a UNIQUE or CHECK
       constraint, may be violated when the row is updated.


 Depending on the type of conflict, a sessions application has a variety
of configurable options for dealing with conflicts, ranging from omitting the
conflicting change, aborting the entire changeset application or applying
the change despite the conflict. For details, refer to the documentation for
the sqlite3changeset_apply() API.

2.3. Changeset Construction

 After a session object has been configured, it begins monitoring for
changes to its configured tables. However, it does not record an entire
change each time a row within the database is modified. Instead, it records
just the PRIMARY KEY fields for each inserted row, and just the PRIMARY KEY
and all original row values for any updated or deleted rows. If a row is
modified more than once by a single session, no new information is recorded.

 The other information required to create a changeset or patchset is
read from the database file when sqlite3session_changeset() or
sqlite3session_patchset() is called. Specifically,


   For each primary key recorded as a result of an INSERT operation,
       the sessions module checks if there is a row with a matching primary
       key still in the table. If so, an INSERT change is added to the
       changeset.

   For each primary key recorded as a result of an UPDATE or DELETE
       operation, the sessions module also checks for a row with a matching
       primary key within the table. If one can be found, but one or more
       of the non-PRIMARY KEY fields does not match the original recorded
       value, an UPDATE is added to the changeset. Or, if there is no row
       at all with the specified primary key, a DELETE is added to the
       changeset. If the row does exist but none of the non-PRIMARY KEY
       fields have been modified, no change is added to the changeset.


 One implication of the above is that if a change is made and then
unmade within a single session (for example if a row is inserted and then
deleted again), the sessions module does not report any change at all. Or
if a row is updated multiple times within the same session, all updates
are coalesced into a single update within any changeset or patchset blob.

3. Using The Session Extension

 This section provides examples that demonstrate how to use the sessions
    extension.

3.1. Capturing a Changeset

 The example code below demonstrates the steps involved in capturing a
changeset while executing SQL commands. In summary:


   A session object (type sqlite3_session*) is created by making a
          call to the sqlite3session_create() API function.

       A single session object monitors changes made to a single database
          (i.e. "main", "temp" or an attached database) via a single
          sqlite3* database handle.

   The session object is configured with a set of tables to monitor
          changes on.

        By default a session object does not monitor changes on any
           database table. Before it does so it must be configured. There
           are three ways to configure the set of tables to monitor changes
           on:
       
          By explicitly specifying tables using one call to
              sqlite3session_attach() for each table, or

          By specifying that all tables in the database should be monitored
              for changes using a call to sqlite3session_attach() with a
              NULL argument, or

          By configuring a callback to be invoked the first time each table
              is written to that indicates to the session module whether or
              not changes on the table should be monitored.
       
         The example code below uses the second of the methods enumerated
            above - it monitors for changes on all database tables.

    Changes are made to the database by executing SQL statements. The
           session object records these changes.

    A changeset blob is extracted from the session object using a call
           to sqlite3session_changeset() (or, if using patchsets, a call to
           the sqlite3session_patchset() function).

    The session object is deleted using a call to the
           sqlite3session_delete() API function.

        It is not necessary to delete a session object after extracting
           a changeset or patchset from it. It can be left attached to the
           database handle and will continue monitoring for changes on the
           configured tables as before. However, if
           sqlite3session_changeset() or sqlite3session_patchset() is
           called a second time on a session object, the changeset or patchset
           will contain all changes that have taken place on the connection
           since the session was created. In other words,
           a session object is not reset or
           zeroed by a call to sqlite3session_changeset() or
           sqlite3session_patchset().


/*
** Argument zSql points to a buffer containing an SQL script to execute
** against the database handle passed as the first argument. As well as
** executing the SQL script, this function collects a changeset recording
** all changes made to the "main" database file. Assuming no error occurs,
** output variables (*ppChangeset) and (*pnChangeset) are set to point
** to a buffer containing the changeset and the size of the changeset in
** bytes before returning SQLITE_OK. In this case it is the responsibility
** of the caller to eventually free the changeset blob by passing it to
** the sqlite3_free function.
**
** Or, if an error does occur, return an SQLite error code. The final
** value of (*pChangeset) and (*pnChangeset) are undefined in this case.
*/
int sql_exec_changeset(
  sqlite3 *db,                  /* Database handle */
  const char *zSql,             /* SQL script to execute */
  int *pnChangeset,             /* OUT: Size of changeset blob in bytes */
  void **ppChangeset            /* OUT: Pointer to changeset blob */
){
  sqlite3_session *pSession = 0;
  int rc;

  /* Create a new session object */
  rc = sqlite3session_create(db, "main", &pSession);

  /* Configure the session object to record changes to all tables */
  if( rc==SQLITE_OK ) rc = sqlite3session_attach(pSession, NULL);

  /* Execute the SQL script */
  if( rc==SQLITE_OK ) rc = sqlite3_exec(db, zSql, 0, 0, 0);

  /* Collect the changeset */
  if( rc==SQLITE_OK ){
    rc = sqlite3session_changeset(pSession, pnChangeset, ppChangeset);
  }

  /* Delete the session object */
  sqlite3session_delete(pSession);

  return rc;
}


3.2. Applying a Changeset to a Database

 Applying a changeset to a database is simpler than capturing a changeset.
Usually, a single call to sqlite3changeset_apply(), as depicted in the
example code below, suffices.

 In cases where it is complicated, the complications in applying a
changeset lie in conflict resolution. Refer to the API documentation linked
above for details.

  /*
** Conflict handler callback used by apply_changeset(). See below.
*/
static int xConflict(void *pCtx, int eConflict, sqlite3_changeset_iter *pIter){
  int ret = (int)pCtx;
  return ret;
}

/*
** Apply the changeset contained in blob pChangeset, size nChangeset bytes,
** to the main database of the database handle passed as the first argument.
** Return SQLITE_OK if successful, or an SQLite error code if an error
** occurs.
**
** If parameter bIgnoreConflicts is true, then any conflicting changes
** within the changeset are simply ignored. Or, if bIgnoreConflicts is
** false, then this call fails with an SQLITE_ABORT error if a changeset
** conflict is encountered.
*/
int apply_changeset(
  sqlite3 *db,                  /* Database handle */
  int bIgnoreConflicts,         /* True to ignore conflicting changes */
  int nChangeset,               /* Size of changeset in bytes */
  void *pChangeset              /* Pointer to changeset blob */
){
  return sqlite3changeset_apply(
      db,
      nChangeset, pChangeset,
      0, xConflict,
      (void*)bIgnoreConflicts
  );
}


3.3. Inspecting the Contents of a Changeset

 The example code below demonstrates the techniques used to iterate
through and extract the data related to all changes in a changeset. To
summarize:


   The sqlite3changeset_start() API is called to create and
          initialize an iterator to iterate through the contents of a
          changeset. Initially, the iterator points to no element at all.

   The first call to sqlite3changeset_next() on the iterator moves
          it to point to the first change  in the changeset (or to EOF, if
          the changeset is completely empty). sqlite3changeset_next() returns
          SQLITE_ROW if it moves the iterator to point to a valid entry,
          SQLITE_DONE if it moves the iterator to EOF, or an SQLite error
          code if an error occurs.

   If the iterator points to a valid entry, the sqlite3changeset_op()
          API may be used to determine the type of change (INSERT, UPDATE or
          DELETE) that the iterator points to. Additionally, the same API
          can be used to obtain the name of the table the change applies to
          and its expected number of columns and primary key columns.

   If the iterator points to a valid INSERT or UPDATE entry, the
          sqlite3changeset_new() API may be used to obtain the new.* values
          within the change payload.

   If the iterator points to a valid DELETE or UPDATE entry, the
          sqlite3changeset_old() API may be used to obtain the old.* values
          within the change payload.

   An iterator is deleted using a call to the
          sqlite3changeset_finalize() API. If an error occured while
          iterating, an SQLite error code is returned (even if the same error
          code has already been returned by sqlite3changeset_next()). Or,
          if no error has occurred, SQLITE_OK is returned.


  /*
** Print the contents of the changeset to stdout.
*/
static int print_changeset(void *pChangeset, int nChangeset){
  int rc;
  sqlite3_changeset_iter *pIter = 0;

  /* Create an iterator to iterate through the changeset */
  rc = sqlite3changeset_start(&pIter, nChangeset, pChangeset);
  if( rc!=SQLITE_OK ) return rc;

  /* This loop runs once for each change in the changeset */
  while( SQLITE_ROW==sqlite3changeset_next(pIter) ){
    const char *zTab;           /* Table change applies to */
    int nCol;                   /* Number of columns in table zTab */
    int op;                     /* SQLITE_INSERT, UPDATE or DELETE */
    sqlite3_value *pVal;

    /* Print the type of operation and the table it is on */
    rc = sqlite3changeset_op(pIter, &zTab, &nCol, &op, 0);
    if( rc!=SQLITE_OK ) goto exit_print_changeset;
    printf("%s on table %s\n",
      op==SQLITE_INSERT?"INSERT" : op==SQLITE_UPDATE?"UPDATE" : "DELETE",
      zTab
    );

    /* If this is an UPDATE or DELETE, print the old.* values */
    if( op==SQLITE_UPDATE || op==SQLITE_DELETE ){
      printf("Old values:");
      for(i=0; i&lt;nCol; i++){
        rc = sqlite3changeset_old(pIter, i, &pVal);
        if( rc!=SQLITE_OK ) goto exit_print_changeset;
        printf(" %s", pVal ? sqlite3_value_text(pVal) : "-");
      }
      printf("\n");
    }

    /* If this is an UPDATE or INSERT, print the new.* values */
    if( op==SQLITE_UPDATE || op==SQLITE_INSERT ){
      printf("New values:");
      for(i=0; i&lt;nCol; i++){
        rc = sqlite3changeset_new(pIter, i, &pVal);
        if( rc!=SQLITE_OK ) goto exit_print_changeset;
        printf(" %s", pVal ? sqlite3_value_text(pVal) : "-");
      }
      printf("\n");
    }
  }

  /* Clean up the changeset and return an error code (or SQLITE_OK) */
 exit_print_changeset:
  rc2 = sqlite3changeset_finalize(pIter);
  if( rc==SQLITE_OK ) rc = rc2;
  return rc;
}


4. Extended Functionality

 Most applications will only use the session module functionality described
in the previous section. However, the following additional functionality is
available for the use and manipulation of changeset and patchset blobs:


   Two or more changeset/patchsets may be combined using the
       sqlite3changeset_concat() or sqlite3_changegroup interfaces.

   A changeset may be "inverted" using the sqlite3changeset_invert()
       API function. An inverted changeset undoes the changes made by the
       original. If changeset C+ is the inverse of changeset C, then
       applying C and then C+ to a database should leave
       the database unchanged.
This page last modified on  2024-06-01 14:57:22 UTC


c2: Window Functions
c5: windowfunctions.html
c4:
1. Introduction to Window Functions

A window function is an SQL function where the input
values are taken from
a "window" of one or more rows in the results set of a SELECT statement.

Window functions are distinguished from scalar functions and
aggregate functions by the
presence of an OVER clause. If a function has an OVER clause,
then it is a window function. If it lacks an OVER clause, then it is an
ordinary aggregate or scalar function. Window functions might also
have a FILTER clause in between the function and the OVER clause.

The syntax for a window function is like this:


window-function-invocation:
hide
 
 
expr:
show
 
 
function-arguments:
show
 
 
ordering-term:
show
 
 


literal-value:
show
 
 

over-clause:
show
 
 
frame-spec:
hide
 
 

ordering-term:
show
 
 


raise-function:
show
 
 

select-stmt:
show
 
 
common-table-expression:
show
 
 

compound-operator:
show
 
 

join-clause:
show
 
 
join-constraint:
show
 
 

join-operator:
show
 
 


ordering-term:
show
 
 

result-column:
show
 
 

table-or-subquery:
show
 
 


type-name:
show
 
 
signed-number:
show
 
 



filter-clause:
hide
 
 

window-defn:
hide
 
 
frame-spec:
hide
 
 

ordering-term:
show
 
 






Unlike ordinary functions, window functions
cannot use the DISTINCT keyword.
Also, Window functions may only appear in the result set and in the
ORDER BY clause of a SELECT statement.

Window functions come in two varieties:
aggregate window functions and
built-in window functions.  Every aggregate window function
can also work as an ordinary aggregate function, simply by omitting
the OVER and FILTER clauses.  Furthermore, all of the built-in
aggregate functions of SQLite can be used as an
aggregate window function by adding an appropriate OVER clause.
Applications can register new aggregate window functions using
the sqlite3_create_window_function() interface.
The built-in window functions, however, require special-case
handling in the query planner and hence new window functions
that exhibit the exceptional properties found in the built-in
window functions cannot be added by the application.

Here is an example using the built-in row_number()
window function:

CREATE TABLE t0(x INTEGER PRIMARY KEY, y TEXT);
INSERT INTO t0 VALUES (1, 'aaa'), (2, 'ccc'), (3, 'bbb');

-- The following SELECT statement returns:
-- 
--   x | y | row_number
-----------------------
--   1 | aaa | 1         
--   2 | ccc | 3         
--   3 | bbb | 2         
-- 
SELECT x, y, row_number() OVER (ORDER BY y) AS row_number FROM t0 ORDER BY x;



The row_number() window function
assigns consecutive integers to each
row in order of the "ORDER BY" clause within the
window-defn (in this case "ORDER BY y"). Note that
this does not affect the order in which results are returned from
the overall query.  The order of the final output is
still governed by the ORDER BY clause attached to the SELECT
statement (in this case "ORDER BY x").

Named window-defn clauses may also be added to a SELECT
statement using a WINDOW clause and then referred to by name within window
function invocations. For example, the following SELECT statement contains
two named window-defs clauses, "win1" and "win2":

SELECT x, y, row_number() OVER win1, rank() OVER win2
FROM t0
WINDOW win1 AS (ORDER BY y RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW),
       win2 AS (PARTITION BY y ORDER BY x)
ORDER BY x;


The WINDOW clause, when one is present, comes after any HAVING clause and
before any ORDER BY.



2. Aggregate Window Functions

 The examples in this section all assume that the database is populated as
follows:

CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
INSERT INTO t1 VALUES   (1, 'A', 'one'  ),
                        (2, 'B', 'two'  ),
                        (3, 'C', 'three'),
                        (4, 'D', 'one'  ),
                        (5, 'E', 'two'  ),
                        (6, 'F', 'three'),
                        (7, 'G', 'one'  );


 An aggregate window function is similar to an
ordinary aggregate function, except
adding it to a query does not change the number of rows returned. Instead,
for each row the result of the aggregate window function is as if the
corresponding aggregate were run over all rows in the "window frame"
specified by the OVER clause.


-- The following SELECT statement returns:
-- 
--   a | b | group_concat
-------------------------
--   1 | A | A.B         
--   2 | B | A.B.C       
--   3 | C | B.C.D       
--   4 | D | C.D.E       
--   5 | E | D.E.F       
--   6 | F | E.F.G       
--   7 | G | F.G         
-- 
SELECT a, b, group_concat(b, '.') OVER (
  ORDER BY a ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS group_concat FROM t1;


 In the example above, the window frame consists of all rows between the
previous row ("1 PRECEDING") and the following row ("1 FOLLOWING"), inclusive,
where rows are sorted according to the ORDER BY clause in the
window-defn (in this case "ORDER BY a").
For example, the frame for the row with (a=3) consists of rows (2, 'B', 'two'),
(3, 'C', 'three') and (4, 'D', 'one'). The result of group_concat(b, '.')
for that row is therefore 'B.C.D'.

 All of SQLite's aggregate functions may
be used as aggregate window functions. It is also possible to
create user-defined aggregate window functions.




2.1. The PARTITION BY Clause

 For the purpose of computing window functions, the result set
of a query is divided into one or more "partitions".  A partition consists
of all rows that have the same value for all terms of the PARTITION BY clause
in the window-defn.  If there is no PARTITION BY clause,
then the entire result set of the query is a single partition.
Window-function processing is performed separately for each partition.

 For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   one   | 4 | D | D.G         
--   one   | 7 | G | G           
--   three | 3 | C | C.F         
--   three | 6 | F | F           
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;


 In the query above, the "PARTITION BY c" clause breaks the
result set up into three partitions. The first partition has
three rows with c=='one'.  The second partition has two rows with
c=='three' and the third partition has two rows with c=='two'.

 In the example above, all the rows for each partition are
grouped together in the final output.  This is because the PARTITION BY
clause is a prefix of the ORDER BY clause on the overall query.
But that does not have
to be the case.  A partition can be composed of rows scattered
about haphazardly within the result set.  For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G       
--   two   | 2 | B | B.E         
--   three | 3 | C | C.F         
--   one   | 4 | D | D.G         
--   two   | 5 | E | E           
--   three | 6 | F | F           
--   one   | 7 | G | G           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  PARTITION BY c ORDER BY a RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY a;





2.2. Frame Specifications

 The frame-spec determines which output rows are
read by an aggregate window function.  The
frame-spec consists of four parts:


   A frame type - either ROWS, RANGE or GROUPS,
   A starting frame boundary,
   An ending frame boundary,
   An EXCLUDE clause.


 Here are the syntax details:

frame-spec:
hide
 
 
expr:
show
 
 
filter-clause:
show
 
 

function-arguments:
show
 
 
ordering-term:
show
 
 


literal-value:
show
 
 

over-clause:
show
 
 
ordering-term:
show
 
 


raise-function:
show
 
 

select-stmt:
show
 
 
common-table-expression:
show
 
 

compound-operator:
show
 
 

join-clause:
show
 
 
join-constraint:
show
 
 

join-operator:
show
 
 


ordering-term:
show
 
 

result-column:
show
 
 

table-or-subquery:
show
 
 

window-defn:
show
 
 


type-name:
show
 
 
signed-number:
show
 
 







The ending frame boundary can be omitted (if the
BETWEEN and&nbsp;AND keywords that surround the starting frame boundary
are also omitted),
in which case the ending frame boundary defaults to CURRENT ROW.

 If the frame type is RANGE or GROUPS, then rows with the same values for
all ORDER BY expressions are considered "peers". Or, if there are no ORDER BY
terms, all rows are peers. Peers are always within the same frame.

The default frame-spec is:

RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS


The default means that aggregate window functions read all
rows from the beginning of the partition up to and including the
current row and its peers. This implies that rows that have the same values for
all ORDER BY expressions will also have the same value for the result of the
window function (as the window frame is the same). For example:

-- The following SELECT statement returns:
-- 
--   a | b | c | group_concat
-----------------------------
--   1 | A | one   | A.D.G       
--   2 | B | two   | A.D.G.C.F.B.E
--   3 | C | three | A.D.G.C.F   
--   4 | D | one   | A.D.G       
--   5 | E | two   | A.D.G.C.F.B.E
--   6 | F | three | A.D.G.C.F   
--   7 | G | one   | A.D.G       
-- 
SELECT a, b, c,
       group_concat(b, '.') OVER (ORDER BY c) AS group_concat
FROM t1 ORDER BY a;




2.2.1. Frame Type


There are three frame types:  ROWS, GROUPS, and RANGE.
The frame type determines how the starting and ending boundaries
of the frame are measured.


ROWS:
The ROWS frame type means that the starting and ending boundaries
for the frame are determined by counting individual rows relative
to the current row.



GROUPS:
The GROUPS frame type means that the starting and ending boundaries
are determine by counting "groups" relative to the current group.
A "group" is a set of rows that all have equivalent values for all
all terms of the window ORDER BY clause.  ("Equivalent" means that
the IS operator is true when comparing the two values.)
In other words, a group consists of all peers of a row.



RANGE:
The RANGE frame type requires that the ORDER BY clause of the
window have exactly one term.  Call that term "X".  With the
RANGE frame type, the elements of the frame are determined by
computing the value of expression X for all rows in the partition
and framing those rows for which the value of X is within a certain
range of the value of X for the current row.  See the description
in the "&lt;expr&gt; PRECEDING" boundary
specification below for details.


The ROWS and GROUPS frame types are similar in that they
both determine the extent of a frame by counting relative to
the current row.  The difference is that ROWS counts individual
rows and GROUPS counts peer groups.
The RANGE frame type is different.
The RANGE frame type determines the extent of a frame by
looking for expression values that are within some band of
values relative to the current row.



2.2.2. Frame Boundaries

 There are five ways to describe starting and ending frame boundaries:


 UNBOUNDED PRECEDING
     The frame boundary is the first
     row in the partition.

 &lt;expr&gt; PRECEDING
        &lt;expr&gt; must be a non-negative constant numeric expression.
        The boundary is a row that is &lt;expr&gt; "units" prior to
        the current row.  The meaning of "units" here depends on the
        frame type:
        
         ROWS &rarr;
        The frame boundary is the row that is &lt;expr&gt;
        rows before the current row, or the first row of the
        partition if there are fewer than &lt;expr&gt; rows
        before the current row.  &lt;expr&gt; must be an integer.
         GROUPS &rarr;
        A "group" is a set of peer rows - rows that all have
        the same values for every term in the ORDER BY clause.
        The frame boundary is the group that is &lt;expr&gt;
        groups before the group containing the current row, or the
        first group of the partition if there are fewer
        than &lt;expr&gt; groups before the current row.
        For the starting boundary of a frame, the first
        row of the group is used and for the ending boundary
        of a frame, the last row of the group is used.
        &lt;expr&gt; must be an integer.
        

         RANGE &rarr;
        For this form, the ORDER BY clause of the
        window-defn must have a single
        term.  Call that ORDER BY term "X".  Let
        Xi be the value of the X
        expression for the i-th row in the partition and let
        Xc be the value of X for the
        current row.  Informally, a RANGE bound is the first row
        for which Xi is within
        the &lt;expr&gt; of Xc.
        More precisely:
        
         If either Xi or
             Xc are non-numeric, then
             the boundary is the first row for which the expression
             "Xi IS Xc"
             is true.
         Else if the ORDER BY is ASC then the boundary
             is the first row for which
             Xi&gt;=Xc-&lt;expr&gt;.
         Else if the ORDER BY is DESC then the boundary
        is the first row for which
        Xi&lt;=Xc+&lt;expr&gt;.
        
        For this form, the &lt;expr&gt; does not have to be an
        integer. It can evaluate to a real number as long as
        it is constant and non-negative.
        
    The boundary description "0 PRECEDING" always means the same
    thing as "CURRENT ROW".

CURRENT ROW
        The current row. For RANGE and GROUPS frame types,
        peers of the current row are also included in the frame,
        unless specifically excluded by the EXCLUDE clause.
        This is true regardless of whether CURRENT ROW is used
        as the starting or ending frame boundary.

&lt;expr&gt; FOLLOWING
        This is the same as "&lt;expr&gt; PRECEDING" except that
        the boundary is &lt;expr&gt; units after the current
        rather than before the current row.

 UNBOUNDED FOLLOWING
   The frame boundary is the last
   row in the partition.



 The ending frame boundary may not take a form that appears higher in
the above list than the starting frame boundary.

 In the following example, the window frame for each row consists of all
rows from the current row to the end of the set, where rows are sorted
according to "ORDER BY a".

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A.D.G.C.F.B.E
--   one   | 4 | D | D.G.C.F.B.E 
--   one   | 7 | G | G.C.F.B.E   
--   three | 3 | C | C.F.B.E     
--   three | 6 | F | F.B.E       
--   two   | 2 | B | B.E         
--   two   | 5 | E | E           
-- 
SELECT c, a, b, group_concat(b, '.') OVER (
  ORDER BY c, a ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
) AS group_concat
FROM t1 ORDER BY c, a;




2.2.3. The EXCLUDE Clause

 The optional EXCLUDE clause may take any of the following four forms:


    EXCLUDE NO OTHERS: This is the default. In this case no
  rows are excluded from the window frame as defined by its starting and ending
  frame boundaries.

    EXCLUDE CURRENT ROW: In this case the current row is
  excluded from the window frame.  Peers of the current row remain in
  the frame for the GROUPS and RANGE frame types.

    EXCLUDE GROUP: In this case the current row and all other
  rows that are peers of the current row are excluded from the frame. When
  processing an EXCLUDE clause, all rows with the same ORDER BY values, or all
  rows in the partition if there is no ORDER BY clause, are considered peers,
  even if the frame type is ROWS.

    EXCLUDE TIES: In this case the current row is part of the
  frame, but peers of the current row are excluded.


 The following example demonstrates the effect of the various
forms of the EXCLUDE clause:

-- The following SELECT statement returns:
-- 
--   c    | a | b | no_others     | current_row | grp       | ties
--  one   | 1 | A | A.D.G         | D.G         |           | A
--  one   | 4 | D | A.D.G         | A.G         |           | D
--  one   | 7 | G | A.D.G         | A.D         |           | G
--  three | 3 | C | A.D.G.C.F     | A.D.G.F     | A.D.G     | A.D.G.C
--  three | 6 | F | A.D.G.C.F     | A.D.G.C     | A.D.G     | A.D.G.F
--  two   | 2 | B | A.D.G.C.F.B.E | A.D.G.C.F.E | A.D.G.C.F | A.D.G.C.F.B
--  two   | 5 | E | A.D.G.C.F.B.E | A.D.G.C.F.B | A.D.G.C.F | A.D.G.C.F.E
-- 
SELECT c, a, b,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS
  ) AS no_others,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE CURRENT ROW
  ) AS current_row,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE GROUP
  ) AS grp,
  group_concat(b, '.') OVER (
    ORDER BY c GROUPS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE TIES
  ) AS ties
FROM t1 ORDER BY c, a;


2.3. The FILTER Clause

filter-clause:
hide
 
 
expr:
show
 
 
function-arguments:
show
 
 
ordering-term:
show
 
 


literal-value:
show
 
 

over-clause:
show
 
 
frame-spec:
show
 
 

ordering-term:
show
 
 


raise-function:
show
 
 

select-stmt:
show
 
 
common-table-expression:
show
 
 

compound-operator:
show
 
 

join-clause:
show
 
 
join-constraint:
show
 
 

join-operator:
show
 
 


ordering-term:
show
 
 

result-column:
show
 
 

table-or-subquery:
show
 
 

window-defn:
show
 
 
frame-spec:
show
 
 



type-name:
show
 
 
signed-number:
show
 
 






If a FILTER clause is provided, then only rows for which the expr is
true are included in the window frame. The aggregate window still returns a
value for every row, but those for which the FILTER expression evaluates to
other than true are not included in the window frame for any row. For example:

-- The following SELECT statement returns:
-- 
--   c     | a | b | group_concat
---------------------------------
--   one   | 1 | A | A           
--   two   | 2 | B | A           
--   three | 3 | C | A.C         
--   one   | 4 | D | A.C.D       
--   two   | 5 | E | A.C.D       
--   three | 6 | F | A.C.D.F     
--   one   | 7 | G | A.C.D.F.G   
-- 
SELECT c, a, b, group_concat(b, '.') FILTER (WHERE c!='two') OVER (
  ORDER BY a
) AS group_concat
FROM t1 ORDER BY a;





3. Built-in Window Functions

 As well as aggregate window functions, SQLite features a set of built-in
window functions based on

those supported by PostgreSQL.

 Built-in window functions honor any PARTITION BY clause in the same way
as aggregate window functions - each selected row is assigned to a partition
and each partition is processed separately. The ways in which any ORDER BY
clause affects each built-in window function is described below. Some of
the window functions (rank(), dense_rank(), percent_rank() and ntile()) use
the concept of "peer groups" (rows within the same partition that have the
same values for all ORDER BY expressions). In these cases, it does not matter
whether the frame-spec specifies ROWS, GROUPS, or RANGE.
For the purposes of built-in window function processing, rows with the same values
for all ORDER BY expressions are considered peers regardless of the frame type.

 Most built-in window functions ignore the
frame-spec, the exceptions being first_value(),
last_value() and nth_value(). It is a syntax error to specify a FILTER
clause as part of a built-in window function invocation.



 SQLite supports the following 11 built-in window functions:


  row_number()
   The number of the row within the current partition. Rows are
       numbered starting from 1 in the order defined by the ORDER BY clause in
       the window definition, or in arbitrary order otherwise.
  rank()
   The row_number() of the first peer in each group - the rank of the
       current row with gaps. If there is no ORDER BY clause, then all rows
       are considered peers and this function always returns 1.
  dense_rank()
   The number of the current row's peer group within its partition - the
       rank of the current row without gaps. Rows are numbered starting
       from 1 in the order defined by the ORDER BY clause in the window
       definition. If there is no ORDER BY clause, then all rows are
       considered peers and this function always returns 1.
  percent_rank()
   Despite the name, this function always returns a value between 0.0
       and 1.0 equal to (rank - 1)/(partition-rows - 1), where
       rank is the value returned by built-in window function rank()
       and partition-rows is the total number of rows in the
       partition. If the partition contains only one row, this function
       returns 0.0.
  cume_dist()
   The cumulative distribution. Calculated as
       row-number/partition-rows, where row-number is
       the value returned by row_number() for the last peer in the group
       and partition-rows the number of rows in the partition.
  ntile(N)
   Argument N is handled as an integer. This function divides the
       partition into N groups as evenly as possible and assigns an integer
       between 1 and N to each group, in the order defined by the ORDER
       BY clause, or in arbitrary order otherwise. If necessary, larger groups
       occur first. This function returns the integer value assigned to the
       group that the current row is a part of.

  lag(expr)lag(expr, offset)lag(expr, offset, default)
   The first form of the lag() function returns the result of evaluating
       expression expr against the previous row in the partition. Or, if
       there is no previous row (because the current row is the first), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows before the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows before the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.

  lead(expr)lead(expr, offset)lead(expr, offset, default)
   The first form of the lead() function returns the result of evaluating
       expression expr against the next row in the partition. Or, if
       there is no next row (because the current row is the last), NULL.

       If the offset argument is provided, then it must be a
       non-negative integer. In this case the value returned is the result
       of evaluating expr against the row offset rows after the
       current row within the partition. If offset is 0, then
       expr is evaluated against the current row. If there is no row
       offset rows after the current row, NULL is returned.

       If default is also provided, then it is returned instead of
      NULL if the row identified by offset does not exist.
  first_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the first row in the window frame
       for each row.
  last_value(expr)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the last row in the window frame
       for each row.
  nth_value(expr, N)
   This built-in window function calculates the window frame for each
       row in the same way as an aggregate window function. It returns the
       value of expr evaluated against the row N of the window
       frame. Rows are numbered within the window frame starting from 1 in
       the order defined by the ORDER BY clause if one is present, or in
       arbitrary order otherwise. If there is no Nth row in the
       partition, then NULL is returned.
  


The examples in this section use the
previously defined T1 table
as well as the following T2 table:

CREATE TABLE t2(a, b);
INSERT INTO t2 VALUES('a', 'one'),
                     ('a', 'two'),
                     ('a', 'three'),
                     ('b', 'four'),
                     ('c', 'five'),
                     ('c', 'six');


The following example illustrates the behaviour of the five ranking
functions - row_number(), rank(), dense_rank(), percent_rank() and
cume_dist().

-- The following SELECT statement returns:
-- 
--   a | row_number | rank | dense_rank | percent_rank | cume_dist
------------------------------------------------------------------
--   a |          1 |    1 |          1 |          0.0 |       0.5
--   a |          2 |    1 |          1 |          0.0 |       0.5
--   a |          3 |    1 |          1 |          0.0 |       0.5
--   b |          4 |    4 |          2 |          0.6 |       0.66
--   c |          5 |    5 |          3 |          0.8 |       1.0
--   c |          6 |    5 |          3 |          0.8 |       1.0
-- 
SELECT a                        AS a,
       row_number() OVER win    AS row_number,
       rank() OVER win          AS rank,
       dense_rank() OVER win    AS dense_rank,
       percent_rank() OVER win  AS percent_rank,
       cume_dist() OVER win     AS cume_dist
FROM t2
WINDOW win AS (ORDER BY a);


The example below uses ntile() to divide the six rows into two groups (the
ntile(2) call) and into four groups (the ntile(4) call). For ntile(2), there
are three rows assigned to each group. For ntile(4), there are two groups of
two and two groups of one. The larger groups of two appear first.

-- The following SELECT statement returns:
-- 
--   a | b     | ntile_2 | ntile_4
----------------------------------
--   a | one   |       1 |       1
--   a | two   |       1 |       1
--   a | three |       1 |       2
--   b | four  |       2 |       2
--   c | five  |       2 |       3
--   c | six   |       2 |       4
-- 
SELECT a                        AS a,
       b                        AS b,
       ntile(2) OVER win        AS ntile_2,
       ntile(4) OVER win        AS ntile_4
FROM t2
WINDOW win AS (ORDER BY a);


 The next example demonstrates lag(), lead(), first_value(), last_value()
and nth_value(). The frame-spec is ignored by
both lag() and lead(), but respected by first_value(), last_value()
and nth_value().

-- The following SELECT statement returns:
-- 
--   b | lead | lag  | first_value | last_value | nth_value_3
-------------------------------------------------------------
--   A | C    | NULL | A           | A          | NULL       
--   B | D    | A    | A           | B          | NULL       
--   C | E    | B    | A           | C          | C          
--   D | F    | C    | A           | D          | C          
--   E | G    | D    | A           | E          | C          
--   F | n/a  | E    | A           | F          | C          
--   G | n/a  | F    | A           | G          | C          
-- 
SELECT b                          AS b,
       lead(b, 2, 'n/a') OVER win AS lead,
       lag(b) OVER win            AS lag,
       first_value(b) OVER win    AS first_value,
       last_value(b) OVER win     AS last_value,
       nth_value(b, 3) OVER win   AS nth_value_3
FROM t1
WINDOW win AS (ORDER BY b ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)





4. Window Chaining


Window chaining is a shorthand that allows one window to be defined in terms
of another. Specifically, the shorthand allows the new window to implicitly
copy the PARTITION BY and optionally ORDER BY clauses of the base window. For
example, in the following:

SELECT group_concat(b, '.') OVER (
  win ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)
FROM t1
WINDOW win AS (PARTITION BY a ORDER BY c)



the window used by the group_concat() function is equivalent
to "PARTITION BY a ORDER BY c ROWS BETWEEN UNBOUNDED PRECEDING
AND CURRENT ROW". In order to use window
chaining, all of the following must be true:


  The new window definition must not include a PARTITION BY clause. The
       PARTITION BY clause, if there is one, must be supplied by the base
       window specification.

  If the base window has an ORDER BY clause, it is copied into the new
       window. In this case the new window must not specify an ORDER BY clause.
       If the base window has no ORDER BY clause, one may be specified as part
       of the new window definition.

  The base window may not specify a frame specification.  The frame
      specification can only be given in the new window specification.


The two fragments of SQL below are similar, but not entirely equivalent, as
the latter will fail if the definition of window "win" contains a frame
specification.

SELECT group_concat(b, '.') OVER win ...
SELECT group_concat(b, '.') OVER (win) ...




5. User-Defined Aggregate Window Functions

 User-defined aggregate window functions may be created using the
sqlite3_create_window_function() API. Implementing an aggregate window
function is very similar to an ordinary aggregate function. Any user-defined
aggregate window function may also be used as an ordinary aggregate. To
implement a user-defined aggregate window function the application must
supply four callback functions:


Callback Description
xStep 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to add a row to the current
      window. The function arguments, if any, corresponding to the row being
      added are passed to the implementation of xStep.

xFinal 
      This method is required by both window aggregate and legacy aggregate
      function implementations. It is invoked to return the current value
      of the aggregate (determined by the contents of the current window),
      and to free any resources allocated by earlier calls to xStep.

xValue 
      This method is  only required for window aggregate functions.  The presence
      of this method is what distinguishes a window aggregate function from a
      legacy  aggregate function.  This method is invoked to return the current
      value of the aggregate. Unlike xFinal, the implementation should not
      delete any context.

xInverse 
      This method is only required for window aggregate functions, not legacy
      aggregate function implementations. It is invoked to remove the oldest
      presently aggregated result of xStep from the current window.
      The function arguments, if any, are those
      passed to xStep for the row being removed.


 The C code below implements a simple window aggregate function named
sumint(). This works in the same way as the built-in sum() function, except
that it throws an exception if passed an argument that is not an integer
value.


/*
** xStep for sumint().
**
** Add the value of the argument to the aggregate context (an integer).
*/
static void sumintStep(
  sqlite3_context *ctx,
  int nArg,
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;

  assert( nArg==1 );
  if( sqlite3_value_type(apArg[0])!=SQLITE_INTEGER ){
    sqlite3_result_error(ctx, "invalid argument", -1);
    return;
  }
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  if( pInt ){
    *pInt += sqlite3_value_int64(apArg[0]);
  }
}

/*
** xInverse for sumint().
**
** This does the opposite of xStep() - subtracts the value of the argument
** from the current context value. The error checking can be omitted from
** this function, as it is only ever called after xStep() (so the aggregate
** context has already been allocated) and with a value that has already
** been passed to xStep() without error (so it must be an integer).
*/
static void sumintInverse(
  sqlite3_context *ctx,
  int nArg,
  sqlite3_value *apArg[]
){
  sqlite3_int64 *pInt;
  assert( sqlite3_value_type(apArg[0])==SQLITE_INTEGER );
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, sizeof(sqlite3_int64));
  *pInt -= sqlite3_value_int64(apArg[0]);
}

/*
** xFinal for sumint().
**
** Return the current value of the aggregate window function. Because
** this implementation does not allocate any resources beyond the buffer
** returned by sqlite3_aggregate_context, which is automatically freed
** by the system, there are no resources to free. And so this method is
** identical to xValue().
*/
static void sumintFinal(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** xValue for sumint().
**
** Return the current value of the aggregate window function.
*/
static void sumintValue(sqlite3_context *ctx){
  sqlite3_int64 res = 0;
  sqlite3_int64 *pInt;
  pInt = (sqlite3_int64*)sqlite3_aggregate_context(ctx, 0);
  if( pInt ) res = *pInt;
  sqlite3_result_int64(ctx, res);
}

/*
** Register sumint() window aggregate with database handle db.
*/
int register_sumint(sqlite3 *db){
  return sqlite3_create_window_function(db, "sumint", 1, SQLITE_UTF8, 0,
      sumintStep, sumintFinal, sumintValue, sumintInverse, 0
  );
}




 The following example uses the sumint() function implemented by the above
C code. For each row, the window consists of the preceding row (if any), the current row and the following row (again, if any):

CREATE TABLE t3(x, y);
INSERT INTO t3 VALUES('a', 4),
                     ('b', 5),
                     ('c', 3),
                     ('d', 8),
                     ('e', 1);

-- Assuming the database is populated using the above script, the 
-- following SELECT statement returns:
-- 
--   x | sum_y
--------------
--   a | 9    
--   b | 12   
--   c | 16   
--   d | 12   
--   e | 9    
-- 
SELECT x, sumint(y) OVER (
  ORDER BY x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
) AS sum_y
FROM t3 ORDER BY x;


In processing the query above, SQLite invokes the sumint callbacks as
follows:



   xStep(4) - add "4" to the current window.
   xStep(5) - add "5" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='a'). The window currently consists of values 4 and 5,
       and so the result is 9.
   xStep(3) - add "3" to the current window.
   xValue() - invoke xValue() to obtain the value of sumint() for
       the row with (x='b'). The window currently consists of values 4, 5 and
       3, and so the result is 12.
   xInverse(4) - remove "4" from the window.
   xStep(8) - add "8" to the current window. The window now consists
       of values 5, 3 and 8.
   xValue() - invoked to obtain the value for the row with (x='c').
       In this case, 16.
   xInverse(5) - remove value "5" from the window.
   xStep(1) - add value "1" to the window.
   xValue() - invoked to obtain the value for row (x='d').
   xInverse(3) - remove value "3" from the window. The window now
       contains values 8 and 1 only.
   xFinal() - invoked to reclaim any allocated resources and to
                         obtain the value for row (x='e'). 9.     .


If the user were to abandon query execution by calling sqlite3_reset() or
sqlite3_finalize() on the statement handle before SQLite has called xFinal(),
then xFinal() is called automatically from within the sqlite3_reset() or
sqlite3_finalize() call in order to reclaim any allocated resources, even
though the value is not required. In this case any error returned by the xFinal
implementation is silently discarded.

6. History

Window function support was first added to SQLite with release
version 3.25.0 (2018-09-15). The SQLite developers used
the PostgreSQL window function
documentation as their primary reference for how window functions
ought to behave.  Many test cases have been run against PostgreSQL
to ensure that window functions operate the same way in both
SQLite and PostgreSQL.

In SQLite version 3.28.0 (2019-04-16),
windows function support was extended to include the EXCLUDE clause,
GROUPS frame types, window chaining, and support for
"&lt;expr&gt; PRECEDING" and "&lt;expr&gt; FOLLOWING" boundaries
in RANGE frames.
This page last modified on  2024-04-16 17:22:18 UTC


c2: SQLite Older News
c5: oldnews.html
c4:
2015-11-02 - Release 3.9.2
SQLite version 3.9.2 is a patch release fixing two obscure bugs.
(Details: (1),
(2)).
Upgrade only if you are having problems.
2015-10-16 - Release 3.9.1
SQLite version 3.9.1 is a
small patch
to version 3.9.0 that includes
a few simple build script and #ifdef tweaks to make the code easier to
compile on a wider variety of platform.  There are no functional changes,
except for a single
minor bug-fix in
the json1 extension to stop it from recognizing form-feed
(ASCII 0x0c) as a whitespace character, in conformance with
RFC7159.
2015-10-14 - Release 3.9.0
SQLite version 3.9.0 is a regularly schedule maintenance release.
Key changes include:

Begin using semantic versioning.
JSON SQL functions
The FTS5 full-text search engine
Support for indexes on expressions
Support for table-valued functions

See the change log for a long and more complete list
of changes.
2015-07-29 - Release 3.8.11.1
SQLite version 3.8.11.1 is a patch release that fixes two arcane
   issues that were reported shortly after 3.8.11 was released.  Upgrade
   from 3.8.11 only in the unlikely event that one of these obscure
   issues affect your code.
2015-07-27 - Release 3.8.11
SQLite version 3.8.11 is a regularly scheduled maintenance release.
   See the change log for details.
2015-05-20 - Release 3.8.10.2
Yikes!  Index corruption after a sequence of valid SQL statements!
It has been many years since anything like
   this bug has snuck into
   an official SQLite release.  But for the pasts seven months
   (version 3.8.7 through version 3.8.10.1)
   if you do an INSERT into a carefully
   crafted schema in which there are two nested triggers that convert
   an index key value from TEXT to INTEGER and then back
   to TEXT again, the INTEGER value might get inserted as the index
   key instead of the correct TEXT, resulting in index corruption.
   This patch release adds a single line of code to fix the problem.
If you do actually encounter this problem, running REINDEX on the
   damaged indexes will clear it.
2015-05-09 - Release 3.8.10.1
The 3.8.10 release did not add the new SQLITE_ENABLE_DBSTAT_VTAB
   compile-time option to the sqlite3_compileoption_used() interface.
   This patch release fixes that omission.  And while we are at it,
   the associated dbstat virtual table was enhanced slightly and a
   harmless compiler warning was fixed.

   There is no reason to upgrade from version 3.8.10 unless you are
   using the new SQLITE_ENABLE_DBSTAT_VTAB compile-time option.
2015-05-07 - Release 3.8.10
SQLite version 3.8.10 is a regularly scheduled maintenance release.
   This release features performance improvements, fixes to several
   arcane bugs found by the AFL fuzzer, the new "sqldiff.exe" command-line
   utility, improvements to the documentation, and other enhancements.
   See the release notes for
   additional information.
2015-04-08 - Release 3.8.9
SQLite version 3.8.9 is a regularly scheduled maintenance release.
   New features in this release include the
   PRAGMA index_xinfo command, the sqlite3_status64() interface,
   and the ".dbinfo" command of the command-line shell.
   See the release notes for
   additional information.
2015-02-25 - Release 3.8.8.3
The 3.8.8.3 patch release fixes an obscure problem in the SQLite code
   generator that can cause incorrect results when the qualifying expression
   of a partial index is used inside the ON clause of a LEFT JOIN.
   This problem has been in the code since support for partial indexes
   was first added in version 3.8.0.  However, it is difficult to imagine
   a valid reason to every put the qualifying constraint inside the ON
   clause of a LEFT JOIN, and so this issue has never come up before.

Any applications that is vulnerable to this bug would have encountered
   problems already.  Hence, upgrading from the previous release is optional.
2015-01-30 - Release 3.8.8.2
The 3.8.8.2 patch release fixes a single minor problem:  It ensures
   that the sqlite3_wal_checkpoint(TRUNCATE) operation will always truncate
   the write-ahead log even if log had already been reset and contained
   no new content.  It is unclear if this is a bug fix or a new feature.

Something like this would normally go into the next regularly scheduled
   release, but a prominent SQLite user needed the change in a hurry so
   we were happy to rush it out via this patch.

There is no reason to upgrade unless you actually need the enhanced
   behavior of sqlite3_wal_checkpoint(TRUNCATE).
2015-01-20 - Release 3.8.8.1
Within hours of releasing version 3.8.8, a bug was reported against
   the 10-month-old 3.8.4 release.  As that bug exists in all subsequent
   releases, the decision was made to issue a small patch to the
   3.8.8 before it came into widespread use.

See ticket
   f97c4637102a3ae72b7911
   for a description of the bug.

The changes between versions 3.8.8 and 3.8.8.1 are minimal.
2015-01-16 - Release 3.8.8
SQLite version 3.8.8 is a regularly schedule maintenance release of
   SQLite.

There are no dramatic new features or performance enhancements in this
   release, merely incremental improvements.  Most of the performance gain
   in this release comes from refactoring the B-Tree rebalancing logic to
   avoid unnecessary memcpy() operations.  New features include the
   PRAGMA data_version statement and the ability to accept a
   VALUES clause with no arbitrary limit on the number of rows.
   Several obscure bugs have been fixed, including some multithreading
   races and a work-around for a compiler bug on some Macs.

See the change log for a longer list of
   enhancements and bug fixes.
2014-12-09 - Release 3.8.7.4
SQLite version 3.8.7.4 an unscheduled bug-fix release.  Changes from
   the previous release and from version 3.8.7 are minimal.

This release fixes adds in a mutex that is required by the changes of
   the 3.8.7.3 patch but was accidentally omitted.  The mutex was not required
   by any of the internal SQLite tests, but Firefox crashes without it.
   Test cases have been added to ensure that mutex is never again missed.
2014-12-06 - Release 3.8.7.3
SQLite version 3.8.7.3 an unscheduled bug-fix release.  Changes from
   the previous release and from version 3.8.7 are minimal.

This release fixes two obscure bugs that can result in incorrect
   query results and/or application crashes, but not (as far as we can
   tell) security vulnerabilities.  Both bugs have been latent in the
   code across multiple prior releases and have never before been encountered,
   so they are unlikely to cause problems.  Nevertheless
   it seems prudent to publish fixes for them both.  See the
   change log for details.
2014-11-19 - Release 3.8.7.2
SQLite version 3.8.7.2 is a patch and bug-fix release.  Changes from
   the previous release are minimal.

The primary reason for this release is to enhance the ROLLBACK command
   so that it allows running queries on the same database connection to
   continue running as long as the ROLLBACK does not change the schema.
   In all previous versions of SQLite, a ROLLBACK would cause pending
   queries to stop immediately and return SQLITE_ABORT or
   SQLITE_ABORT_ROLLBACK. Pending queries still abort if the ROLLBACK
   changes the database schema, but as of this patch release, the queries
   are allowed to continue running if the schema is unmodified.

In addition to the ROLLBACK enhancement, this patch release also
   includes fixes for three obscure bugs.  See the
   change log for details.
2014-10-30 - Release 3.8.7.1
SQLite version 3.8.7.1 is a bug-fix release.

The primary reason for this bug-fix release is to address a problem with
   updating the value of fields at the end of a table that were added
   using ALTER TABLE ADD COLUMN.  This problem
   1 first appeared in the
   3.8.7 release.

Another minor annoyance in the 3.8.7 release was the fact that the
   Android build tried to use the strchrnul() function from the standard
   C library but that function is not available on Android.  Android builds
   had to add -DHAVE_STRCHRNUL=0 to work around the problem.  This patch
   fixes that so that Android builds should now work without any changes.

The operation of PRAGMA journal_mode=TRUNCATE has been enhanced so that
   it invokes fsync() after truncating the journal file when
   PRAGMA synchronous=FULL.  This helps to preserve transaction durability
   in the case of a power loss occurring shortly after commit.

Finally, a couple of long-standing and obscure problems associated with run
   UPDATE and DELETE on VIEWs were fixed.

The changes from 3.8.7 are minimal.
2014-10-17 - Release 3.8.7
SQLite version 3.8.7 is a regularly scheduled maintenance release.
   Upgrading from all prior versions is recommended.

Most of the changes from the previous release have been micro-optimizations
   designed to help SQLite run a little faster.  Each individual optimization
   has an unmeasurably small performance impact.  But the improvements add up.
   Measured on a well-defined workload (which the SQLite developers use
   as a proxy for a typical application workload) using cachegrind on Linux
   and compiled with gcc 4.8.1 and -Os on x64 linux, the current release
   does over 20% more work for the same number of CPU cycles compared to the
   previous release.  Cachegrind is not a real CPU, and the workload
   used for measurement is only a proxy.  So your performance may vary.
   We expect to see about half the measured and reported improvement in
   real-world applications. 10% is less than 20% but it is still pretty
   good, we think.

This release includes a new set of C-language interfaces that have
   unsigned 64-bit instead of signed 32-bit length parameters.  The new
   APIs do not provide any new capabilities.  But they do make it easier
   to write applications that are more resistant to integer overflow
   vulnerabilities.

This release also includes a new sorter that is able to use multiple
   threads to help with large sort operations.  (Sort operations are
   sometimes required to implement ORDER BY and/or GROUP BY clauses and
   are almost always required for CREATE INDEX.)  The multi-threads sorter
   is turned off by default and must be enabled using the
   PRAGMA threads SQL command.  Note that the multi-threaded sorter
   provides faster real-time performance for large sorts, but it also
   uses more CPU cycles and more energy.
2014-08-15 - Release 3.8.6
SQLite version 3.8.6 is a regularly scheduled maintenance release.
   Upgrading from all previous versions is recommended.

This release contains the usual assortment of obscure bug fixes.
   One bug, however, deserves special attention.
   A problem appeared in the CREATE INDEX command beginning with
   version 3.8.2 (2013-12-06) that allowed, under some circumstances,
   a UNIQUE index to be created on a column that was not unique.  Once
   the index was created, no new non-unique entries could be inserted, but
   preexisting non-unique entries would remain.  See ticket
   9a6daf340df99ba93c
   for further information.  In addition to fixing this bug, the
   PRAGMA integrity_check command has been enhanced to detect
   non-uniqueness in UNIQUE indices, so that if this bug did introduce
   any problems in databases, those problems can be easily detected.

Other noteworthy changes include the addition of support for
   hexadecimal integers (ex: 0x1234), and performance enhancements
   to the IN operator which, according to
   mailing list reports,
   help some queries run up to five times faster.

Version 3.8.6 uses 25% fewer CPU cycles than version 3.8.0 from
   approximately one year ago, according to valgrind
   and the
   test/speedtest1.c
   test program.
   On the other hand,
   the compiled binary for version 3.8.6 is about 5% larger than 3.8.0.
   The size increase is
   due in part to the addition of new features such as WITHOUT ROWID
   tables and common table expressions.
2014-06-04 - Release 3.8.5
SQLite version 3.8.5 is a regularly scheduled maintenance release.
   Upgrading from the previous version is recommended.

Version 3.8.5 fixes more than a dozen obscure bugs.  None of these
   bugs should be a problem for existing applications.  Nor do any of
   the bugs represent a security vulnerability.  Nevertheless, upgrading
   is recommended to prevent future problems.

In addition to bug fixes, the 3.8.5 release adds improvements to the
   query planner, especially regarding sorting using indices and handling
   OR terms
   in the WHERE clause for WITHOUT ROWID tables.  The ".system" and
   ".once" dot-commands were added to the command-line interface.  And
   there were enhancements to the FTS4 and RTREE virtual tables.  See
   the change log for details.
2014-04-03 - Release 3.8.4.3
The optimizations added in version 3.8.4 caused some queries that involve
   subqueries in the FROM clause, DISTINCT, and ORDER BY clauses, to give an incorrect
   result.  See
   ticket 98825a79ce145
   for details.
   This release adds a
   one-character change
   to a single line of code to fix the problem.
2014-03-26 - Release 3.8.4.2
The code changes that resulted in the performance improvements
   in version 3.8.4 missed a single buffer overflow test, which could
   result in a read past the end of a buffer while searching a database
   that is corrupted in a particular way.  Version 3.8.4.2 fixes that
   problem using a
   one-line patch.

We are not aware of any problems in version 3.8.4
   when working with well-formed database files.  The problem fixed by this
   release only comes up when reading corrupt database files.
2014-03-11 - Release 3.8.4.1
SQLite version 3.8.4.1 is a patch against version 3.8.4 that fixes
   two minor issues:

 Work around a C-preprocessor macro conflict that causes compilation
     problems for some configurations of Visual Studio.
 Adjust the cost computation for the skip-scan optimization for
     improved performance.

Both of these issues came to light within minutes of tagging the previous
release.  Neither issue is serious but they can be annoying.  Hence, the
decision was made to do a quick patch release to address both issues.
2014-03-10 - Release 3.8.4
SQLite version 3.8.4 is a maintenance release featuring performance
   enhancements and fixes for a number of obscure bugs.
   There are no significant new features in SQLite version 3.8.4.
   However, the number of CPU cycles (measured by valgrind) needed to
   do many common operations has be reduced by about 12% relative to the
   previous release, and by about 25% relative to version 3.7.16
   from approximately one year ago.

Version 3.8.4 of SQLite fixes several corner-case bugs that were
   found since the previous release.  These bugs were unlikely to appear
   in practice, and none represent a security vulnerability.
   Nevertheless, developers are encouraged to upgrade from all prior releases.
2014-02-11 - Release 3.8.3.1
SQLite version 3.8.3.1 fixes a bug present in versions 3.8.1,
   3.8.2 and 3.8.3 that can cause queries to omit valid output rows.
   Upgrading from those versions is recommended.

The problem only comes up if SQLite is compiled with either the
   SQLITE_ENABLE_STAT3 or SQLITE_ENABLE_STAT4 compile-time options.
   In that case, if a query has a WHERE clause that contains expressions
   like this:
   
WHERE (expr1 OR expr2 OR ... OR exprN) AND column IS NOT NULL
   
   Where all of expr1 through exprN are suitable for use by indexes,
   then during query planning SQLite might mistakenly converted
   the "column IS NOT NULL" term into "column>NULL".  But the latter
   term is never true, and so the query would return no rows.

The trouble ticket for this bug is
   &#91;4c86b126f2&#93;.
   It is recommended that all users upgrade to avoid this problem.
2014-02-03 - Release 3.8.3
  SQLite version 3.8.3 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional.

  The most visible change in version 3.8.3 is the addition of
  support for common table expressions.  It is now possible to write a
  single SELECT statement that will query a tree or graph, using either
  a depth-first or a breadth-first search.  A single SQLite query will
  even solve Sudoku puzzles or compute the Mandelbrot set.  As part
  of this change, SQLite now accepts a VALUES clause anyplace that
  a SELECT statement is valid.

  This release also includes many small performance enhancements which
  should give a small speed boost to legacy applications.  And there are
  other minor enhancements such as the addition of the printf() SQL
  function.  See the change log for details.
2013-12-06 - Release 3.8.2
  SQLite version 3.8.2 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional.

  Version 3.8.2 adds support for WITHOUT ROWID tables.  This is a
  significant extension to SQLite.  Database files that contain WITHOUT ROWID
  tables are not readable or writable by prior versions of SQLite, however
  databases that do not use WITHOUT ROWID tables are fully backwards
  and forwards compatible.

  The 3.8.2 release contains a potentially incompatible change.  In
  all prior versions of SQLite, a cast from a very large positive
  floating point number into an integer resulted in the most negative integer.
  In other words, CAST(+99.9e99 to INT) would yield -9223372036854775808.
  This behavior came about because it is what x86/x64 hardware does
  for the equivalent cast in the C language.  But the behavior is
  bizarre.  And so it has been changed effective with this release so that
  a cast from a floating point number into an integer returns the integer
  between the floating point value and zero that is closest to the floating
  point value.  Hence, CAST(+99.9e99 to INT) now returns +9223372036854775807.
  Since routines like sqlite3_column_int64() do an implicit cast if the
  value being accessed is really a floating point number, they are also
  affected by this change.

  Besides the two changes mentioned above, the 3.8.2 release also
  includes a number of performance enhancements.  The
  skip-scan optimization is now available for databases that have been
  processed by ANALYZE.  Constant SQL functions are now factored out of
  inner loops, which can result in a significant speedup for queries that
  contain WHERE clause terms like "date>datetime('now','-2 days')".  And
  various high-runner internal routines have been refactored for reduced
  CPU load.
2013-10-17 - Release 3.8.1
  SQLite version 3.8.1 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional, though you should upgrade
  if you are using partial indices as there was a
  bug related to partial
  indices in the previous release that could result in an incorrect answer
  for count(*) queries.

  The next generation query planner that was premiered in the previous
  release continues to work well.
  The new query planner has been tweaked slightly
  in the current release to help it make better decisions in some
  cases, but is largely unchanged.  Two new SQL functions, likelihood() and
  unlikely(), have been added to allow developers to give hints to the
  query planner without forcing the query planner into a particular decision.

  Version 3.8.1 is the first SQLite release to take into account the
  estimated size of table and index rows when choosing a query plan.
  Row size estimates are based on the declared datatypes of columns.
  For example, a column of type VARCHAR(1000) is assumed
  to use much more space than a column of type INT.  The datatype-based
  row size estimate can be
  overridden by appending a term of the form "sz=NNN" (where NNN is the
  average row size in bytes) to the end of the sqlite_stat1.stat
  record for a table or index.  Currently, row sizes are only used to help the
  query planner choose between a table or one of its indices when doing a
  table scan or a count(*) operation, though future releases are likely
  to use the estimated row size in other contexts as well.  The new
  PRAGMA stats statement can be used to view row size estimates.

  Version 3.8.1 adds the SQLITE_ENABLE_STAT4 compile-time option.
  STAT4 is very similar to STAT3 in that it uses samples from indices to
  try to guess how many rows of the index will be satisfy by WHERE clause
  constraints.  The difference is that STAT4 samples all columns of the
  index whereas the older STAT3 only sampled the left-most column.  Users
  of STAT3 are encouraged to upgrade to STAT4.  Application developers should
  use STAT3 and STAT4 with caution since both options, by design, violate
  the query planner stability guarantee, making it more difficult to ensure
  uniform performance is widely-deployed and mass-produced embedded
  applications.
2013-09-03 - Release 3.8.0.2
  SQLite version 3.8.0.2 contains a one-line fix to a bug in the
  new optimization that tries to omit unused LEFT JOINs from a query.
2013-08-29 - Release 3.8.0.1
  SQLite version 3.8.0.1 fixes some obscure bugs that were uncovered by
  users in the 3.8.0 release.  Changes from 3.8.0 are minimal.
2013-08-26 - Release 3.8.0
  Do not fear the zero!

  SQLite version 3.8.0 might easily have been called "3.7.18" instead.
  However, this release features the cutover of the
  next generation query planner or NGQP, and there is a small chance of
  breaking legacy programs that
  rely on undefined behavior in previous SQLite releases, and so the
  minor version number was incremented for that reason.
  But the risks are low and there is a query planner checklist is
  available to application developers to aid in avoiding problems.

  SQLite version 3.8.0 is actually one of the most heavily tested
  SQLite releases ever.  Thousands and thousands of beta copies have
  be downloaded, and presumably tested, and there have been no problem
  reports.

  In addition to the next generation query planner, the 3.8.0 release
  adds support for partial indices, as well as several other new features.
  See the change log for further detail.
2013-05-20 - Release 3.7.17
  SQLite version 3.7.17 is a regularly schedule maintenance release.
  Visit the change log for a full explanation of the
  changes in this release.

  There are many bug fixes in version 3.7.17.  But this does not indicate
  that 3.7.16 was a problematic release.  All of the bugs in 3.7.17 are
  obscure and are unlikely to impact any particular application.  And most
  of the bugs that are fixed in 3.7.17 predate 3.7.16 and have been in
  the code for years without ever before being noticed.
  Nevertheless, due to the large number of fixes,
  all users are encouraged to upgrade when possible.
2013-04-12 - Release 3.7.16.2
  SQLite version 3.7.16.2 fixes a long-standing flaw in the Windows
  OS interface that
  can result in database corruption under a rare race condition.
  See https://www.sqlite.org/src/info/7ff3120e4f for a full description
  of the problem.

  As far as we know, this bug has never been seen in the wild.  The
  problem was discovered by the SQLite developers while writing stress tests
  for a separate component of SQLite.  Those stress tests have not yet
  found any problems with the component they were intended to verify, but
  they did find the bug which is the subject of this patch release.

  Other than updates to version numbers, the only difference between this
  release and 3.7.16.1 is a two-character change in a single identifier,
  which is contained in the windows-specific OS interface logic.  There
  are no changes in this release (other than version numbers) for platforms
  other than Windows.
2013-03-29 - Release 3.7.16.1
  SQLite version 3.7.16.1 is a bug fix release that fixes a few problems
  that were present in the previous releases.

  The primary motivation for version 3.7.16.1 is to fix a bug in the
  query optimizer that was introduced as part of version 3.7.15.  The
  query optimizer was being a little overzealous in optimizing out some
  ORDER BY clauses, which resulted in sorting being omitted on occasions
  where sorting is required to get the correct answer.  See
  ticket a179fe7465 for
  details.

  In addition to the ORDER BY fix, several other patches to fix obscure
  (and mostly harmless) bugs and to fix spelling errors in source code
  comments are also included in this release.
2013-03-18 - Release 3.7.16
  SQLite version 3.7.16 is a regularly scheduled release of SQLite.
  This release contains several language enhancements and improvements
  to the query optimizer.  A list of the major enhancements and optimizations
  can be see on the change log.

  There was one important bug fix
  (see Ticket fc7bd6358f)
  that addresses an incorrect query result that could have occurred in
  a three-way join where the join constraints compared INTEGER columns
  to TEXT columns.  This issue had been in the code for time out of mind
  and had never before been reported, so we surmise that it is very obscure.
  Nevertheless, all users are advised to upgrade to avoid any future problems
  associated with this issue.
2013-01-09 - Release 3.7.15.2
  SQLite version 3.7.15.2 is a patch release that fixes a single bug
  that was introduced in version version 3.7.15.  The fix is a 4-character
  edit to a single line of code.  Other than this 4-character change and
  the update of the version number, nothing has changed from
  version 3.7.15.1.
2012-12-19 - Release 3.7.15.1
  SQLite version 3.7.15.1 is a patch release that fixes a single bug
  that was introduced in version version 3.7.15.  The fix involved changing
  two lines of code and adding a single assert().  This release also includes
  some new test cases to prevent a regression of the bug, and the version
  number is increased, of course.  But otherwise, nothing has changed from
  version 3.7.15.
2012-12-12 - Release 3.7.15
  SQLite version 3.7.15 is a regularly schedule release of SQLite.  This
  release contains several improvements to the query planner and optimizer
  and one important bug fix.  This is the first release to officially
  support Windows 8 Phone.

  The important bug fix is a problem that can lead to segfaults when using
  shared cache mode on a schema that contains a COLLATE operator within
  a CHECK constraint or within a view.  Collating functions are associated
  with individual database connections.  But a pointer to the collating function
  was also being cached within expressions.  If an expression was part of the
  schema and contained a cached collating function, it would point to the
  collating function in the database connection that originally parsed the
  schema.  If that database connection closed while other database
  connections using the same shared cache continued to operate, they other
  database connections would try to use the deallocated collating function
  in the database connection that closed.  The fix in version 3.7.15 was to
  not cache collating function pointers in the expression structure but
  instead look them up each time a new statement is prepared.

  This release also contains some important enhancements to the query planner
  which should (we hope) make some queries run faster.  The enhancements
  include:

  When doing a full-table scan, try to use an index instead of
  the original table, under the theory that indices contain less information
  and are thus smaller and hence require less disk I/O to scan.

  Enhance the IN operator to allow it to make use of
  indices that have numeric affinity.

  Do a better job of recognizing when an ORDER BY clause can be
  implemented using indices - especially in cases where the ORDER BY clause
  contains terms from two or more tables in a join.
  
2012-10-04 - Release 3.7.14.1
  SQLite version 3.7.14.1 is a patch release.  Changes from the baseline
  version 3.7.14 are minimal and are restricted to fixing three bugs.

  One of the fixed bugs is a long-standing issue with the TCL interface.
  Another is an external compiler bug that SQLite merely works around and
  that only comes up if you are using the VisualStudio-2012 compiler to
  generate WinRT applications on ARM with optimizations enabled.  The
  third problem is an SQLite core bug, introduced in version 3.7.14, that
  can cause a segfault if a query contains a LEFT JOIN that contains an OR
  in the ON clause.
2012-09-03 - Release 3.7.14
  SQLite version 3.7.14 is a regularly scheduled maintenance release
  of SQLite.  The previous release continues to work well.  Upgrading
  is optional.

  Version 3.7.14 drops native support for OS/2.  We are not aware of any
  active projects that were using SQLite on OS/2 and since the SQLite
  developers had no way of testing on OS/2 it seemed like it was time
  to simply remove the OS/2 code from the SQLite tree.  If there are
  OS/2 projects out there that still need SQLite support, they can
  continue to maintain their own private VFS which can be linked to
  SQLite at start-time using the sqlite3_vfs_register() interface.

  The sqlite3_close_v2() interface has been added.  The sqlite3_close_v2()
  interface differs from sqlite3_close() in that it is designed to work
  better for host language that use a garbage collector.  With the older
  sqlite3_close() interface, the associated prepared statements and
  sqlite3_backup objects must be destroyed before the database connection.
  With the newer sqlite3_close_v2() interface, the objects can be destroyed
  in any order.

  This release also includes performance improvements to the sort algorithm
  that is used to implement ORDER BY and CREATE INDEX.  And the query planner
  has been enhanced to better use covering indices on queries that use OR
  terms in the WHERE clause.
2012-06-11 - Release 3.7.13
  SQLite version 3.7.13 adds support for WinRT and metro style
  applications for Microsoft Windows 8.  The 3.7.13 release is
  coming sooner than is usual after the previous release in order to get
  this new capability into the hands of developers.  To use SQLite in
  a metro style application, compile with the -DSQLITE_OS_WINRT flag.
  Because of the increased application security and safety requirements
  of WinRT, all database
  filenames should be full pathnames.  Note that SQLite is not capable
  of accessing databases outside the installation directory and application
  data directory.  This restriction is another security and safety feature
  of WinRT. Apart from these restrictions, SQLite should work exactly
  the same on WinRT as it does on every other system.

  Also in this release: when a database is opened using URI filenames
  and the mode=memory query parameter
  then the database is an in-memory database, just as if it had
  been named ":memory:".  But, if shared cache mode is enabled, then
  all other database connections that specify the same URI filename
  will connect to the same in-memory database.  This allows two or more
  database connections (in the same process) to share the same in-memory
  database.

  This release also includes some corner-case performance optimizations
  that are obscure yet significant to an important subset of SQLite users.
  Getting these performance optimizations into circulation quickly is
  yet another reason for making this release so soon following the previous.

  The next release of SQLite is scheduled to occur after the usual
  2 or 3 month interval.
2012-05-22 - Patch Release 3.7.12.1
  SQLite version 3.7.12.1 is a patch release for version 3.7.12 that
  fixes a bug that was
  introduced in version 3.7.12 and that can
  cause a segfault for certain obscure nested aggregate queries.
  There are very few changes in 3.7.12.1, and upgrading is only needed for
  applications that do nested aggregate queries.
2012-05-14 - Version 3.7.12
  SQLite version 3.7.12 is a regularly scheduled maintenance release.
  This release contains several new optimizations and bug fixes and upgrading
  is recommended.  See the change summary for details.
2012-03-20 - Version 3.7.11
  SQLite version 3.7.11 is a regularly scheduled maintenance release
  which was rushed out early due to a
  bug in the query optimizer
  introduced in the previous release.  The bug is obscure - it changes
  a LEFT JOIN into an INNER JOIN in some cases when there is a 3-way join
  and OR terms in the WHERE clause.  But it was considered serious enough to
  rush out a fix.  Apart from this one problem, SQLite version 3.7.10 has
  not given any trouble.  Upgrading to version 3.7.11 from versions
  3.7.6.3, 3.7.7, 3.7.7.1, 3.7.8, or 3.7.9 is
  optional.  Upgrading from other releases, including the previous release
  3.7.10, is recommended.

  Other enhancements found in this release are enumerated in the
  change log.
2012-01-16 - Version 3.7.10
  SQLite version 3.7.10 is a regularly scheduled maintenance release.
  Upgrading from version 3.7.6.3, 3.7.7, 3.7.7.1, 3.7.8, or 3.7.9 is
  optional.  Upgrading from other releases is recommended.

  The SQLITE_CONFIG_PCACHE mechanism has been replaced with
  SQLITE_CONFIG_PCACHE2.  If you do not know what this mechanism
  is (it is an extreme corner-case and is seldom used) then this
  change will not effect you in the least.

  The default schema format number for new database files has changed
  from 1 to 4.  SQLite has been able to generate and read database files
  using schema format 4 for six years.  But up unto now, the default
  schema format has been 1 so that older versions of SQLite could read
  and write databases generated by newer versions of SQLite.  But those
  older versions of SQLite have become so scarce now that it seems
  reasonable to make the new format the default.

  SQLite is changing some of the assumptions it makes above the behavior
  of disk drives and flash memory devices during a sudden power loss.
  This change is completely transparent to applications.
  Read about the powersafe overwrite property for additional information.

  Lots of new interfaces have been added in this release:
  
   sqlite3_db_release_memory()
   PRAGMA shrink_memory
   sqlite3_db_filename()
   sqlite3_stmt_busy()
   sqlite3_uri_boolean()
   sqlite3_uri_int64()
  

  The PRAGMA cache_size statement has been enhanced.  Formerly, you would
  use this statement to tell SQLite how many pages of the database files it
  should hold in its cache at once.  The total memory requirement would
  depend on the database page size. Now, if you give PRAGMA cache_size
  a negative value -N, it will allocate roughly N
  kibibytes of memory to cache,
  divided up according to page size.  This enhancement allows programs to
  more easily control their memory usage.

  There have been several obscure bug fixes.  One noteworthy bug,
  ticket ff5be73dee,
  could in theory result in a corrupt database file if a power loss
  occurred at just the wrong moment on an unusually cantankerous disk
  drive.  But that is mostly a theoretical concern and is very unlikely
  to happen in practice.  The bug was found during laboratory testing
  and has never been observed to occur in the wild.
2011-11-01 - Version 3.7.9
  SQLite version 3.7.9 is a regularly scheduled maintenance release.
  Upgrading from version 3.7.6.3, 3.7.7, 3.7.7.1, and 3.7.8 is optional.
  Upgrading from other versions is recommended.

  The SQLITE_ENABLE_STAT2 compile-time option is now a no-op.  The enhanced
  query-planner functionality formerly available using SQLITE_ENABLE_STAT2
  is now available through SQLITE_ENABLE_STAT3.  The enhanced query planning
  is still disabled by default.  However, future releases of SQLite might
  convert STAT3 from an enable-option to a disable-option so that it is
  available by default and is only omitted upon request.

  The FTS4 full-text search engine has been enhanced such that tokens in
  the search string that begin with "&#94;" must be the first token in their
  respective columns in order to match.   Formerly, "&#94;" characters in the
  search string were simply ignored.  Hence, if a legacy application was
  including "&#94;" characters in FTS4 search strings, thinking that they would
  always be ignored, then those legacy applications might break with this
  update.  The fix is simply remove the "&#94;" characters from the search
  string.

  See the change summary for additional changes associated
  with this release.
2011-September-19 - Version 3.7.8
  SQLite version 3.7.8 is a quarterly maintenance release.  Upgrading from
  versions 3.7.6.3, 3.7.7, or 3.7.7.1 is optional.  Upgrading from other
  versions is recommended.

  This release features a new "external merge sort" algorithm used to
  implement ORDER BY and GROUP BY and also to presort the content of an
  index for CREATE INDEX.  The new algorithm does approximately the same
  number of comparisons and I/Os as before, but the I/Os are much more
  sequential and so runtimes are greatly reduced when the size of the
  set being sorted is larger than the filesystem cache.  The performance
  improvement can be dramatic - orders of magnitude faster
  for large CREATE INDEX commands.  On the other hand,
  the code is slightly slower (1% or 2%)
  for a small CREATE INDEX.  Since CREATE INDEX is not an
  operation that commonly occurs on a speed-critical path, we feel that
  this tradeoff is a good one.  The slight slowdown for small CREATE INDEX
  statements might be recovered in a future release.  ORDER BY and GROUP BY
  operations should now be faster for all cases, large and small.

  The query planner has been enhanced to do a better job of handling
  the DISTINCT keyword on SELECT statements.

  There has been a lot of work on the default VFSes.  The Unix VFS has
  been enhanced to include more overrideable system calls - a feature requested
  by Chromium to make it easier to build SQLite into a sandbox.  The
  windows VFS has been enhanced to be more resistant to interference from
  anti-virus software.

  Every version of SQLite is better tested than the previous, and 3.7.8
  is no exception to this rule.  Version 3.7.8 has been used internally by
  the SQLite team for mission critical functions and has performed flawlessly.
  And, of course, it passes our rigorous testing procedures with no
  problems detected.  Version 3.7.8 is recommended for all new development.
2011-06-28 - Version 3.7.7.1
  SQLite version 3.7.7.1 adds a one-line bug fix to 3.7.7 to fix
  a problem
  causing PRAGMA case_sensitive_like statements compiled using the legacy
  sqlite3_prepare() interface to fail with an SQLITE_SCHEMA error. Because
  sqlite3_exec() uses sqlite3_prepare() internally, the problem also affects
  sqlite3_exec().

  Upgrading from 3.7.7 is only required for applications that use "PRAGMA
  case_sensitive_like" and the sqlite3_prepare() (or sqlite3_exec()) interface.
2011-06-24 - Version 3.7.7
  SQLite version 3.7.7 is a regularly scheduled bi-monthly maintenance
  release.  Upgrading from version 3.7.6.3 is optional.  Upgrading from all
  prior releases is recommended.

  This release adds support for naming database files using URI filenames.
  URI filenames are disabled by default (for backwards compatibility) but
  applications are encouraged to enable them since incompatibilities are
  likely to be exceedingly rare and the feature is useful.  See the
  URI filename documentation for details.

  Most of the other enhancements in this release involve
  virtual tables.  The virtual table interface has been enhanced to
  support SAVEPOINT and ON CONFLICT clause processing, and the built-in
  RTREE and FTS3/FTS4 have been augmented to take advantage of
  the new capability.  This means, for example, that it is now possible
  to use the REPLACE command on FTS3/FTS4 and RTREE tables.

  The FTS4 full-text index extension has been enhanced to support
  the FTS4 prefix option and the FTS4 order option.  These two enhancements
  are provided in support of search-as-you-type interfaces where search
  results begin to appear after the first keystroke in the "search" box
  and are refined with each subsequent keystroke.  The way this is done is
  to do a separate full-text search after each key stroke, and add the
  "*" wildcard at the end of the word currently being typed.  So, for
  example, if the text typed so far is "fast da" and the next character
  typed is "t", then the application does a full-text search of the
  pattern "fast dat*" and displays the results.  Such capability has
  always existed.  What is new is that the FTS4 prefix option allows
  the search to be very fast (a matter of milliseconds) even for difficult
  cases such as "t*" or "th*".

  There has been a fair amount of work done on the FTS4 module for this
  release.  But the core SQLite code has changed little and the previous
  release has not given any problems, so we expect this to be a very
  stable release.
2011-05-19 - Version 3.7.6.3
  SQLite version 3.7.6.3 is a patch release that fixes a
  single bug
  associated with WAL mode.  The bug has been in SQLite ever since WAL
  was added, but the problem is very obscure and so nobody has noticed
  before now.  Nevertheless, all users are encouraged to upgrade to
  version 3.7.6.3 or later.

  The bug is this:
  If the cache_size is set very small (less than 10) and SQLite comes
  under memory pressure and if a multi-statement transaction is started
  in which the last statement prior to COMMIT is a SELECT statement and if
  a checkpoint occurs right after the transaction commit, then
  it might happen that the transaction will be silently rolled back instead
  of being committed.

  The default setting for cache_size is 2000. So in most situations, this
  bug will never appear.  But sometimes programmers set cache_size to
  very small values on gadgets and other low-memory devices in order to
  save memory space.  Such applications are vulnerable.
  Note that this bug does not cause database corruption.  It is
  as if ROLLBACK were being run instead of COMMIT in some cases.

  Bug Details

  Transactions commit in WAL mode by adding a record onto the end of
  the WAL (the write-ahead log) that contains a "commit" flag.  So to
  commit a transaction, SQLite takes all the pages that have changed
  during that transaction, appends them to the WAL, and sets the commit
  flag on the last page.  Now, if SQLite comes under memory pressure, it
  might try to free up memory space by writing changed pages to the WAL
  prior to the commit.  We call this "spilling" the cache to WAL.  There
  is nothing wrong with spilling cache to WAL.  But if the
  memory pressure is severe, it might be that by the time COMMIT is run,
  all changed pages for the transaction have already been spilled to WAL
  and there are no pages left to be written to WAL.
  And with no unwritten pages, there was nothing to put the commit flag
  on.  And without a commit flag, the transaction would end up being
  rolled back.

  The fix to this problem was that if all changed pages has already
  been written to the WAL when the commit was started, then page 1 of
  the database will be written to the WAL again, so that there will always
  be a page available on which to set the commit flag.
2011-04-17 - Version 3.7.6.2
  SQLite version 3.7.6.2 adds a one-line bug fix to 3.7.6.1 that enables
  pthreads to work correctly on NetBSD.  The problem was a faulty function
  signature for the open system call.  The problem does not appear to have
  any adverse impact on any system other than NetBSD.

  Upgrading from version 3.7.6.1 is only needed on NetBSD.
2011-04-13 - Version 3.7.6.1
  SQLite version 3.7.6.1 fixes a single bug in 3.7.6 that can cause a
  segfault if SQLITE_FCNTL_SIZE_HINT is used on a Unix build that has
  SQLITE_ENABLE_LOCKING_MODE set to 0 and is compiled with
  HAVE_POSIX_FALLOCATE.

  Upgrading from 3.7.6 is only needed for users effected by the
  configuration-specific bug described above.  There are no other changes
  to the code.
2011-04-12 - Version 3.7.6
  SQLite version 3.7.6 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.5 is optional.  Upgrading
  releases prior to 3.7.5 is recommended.
2011-02-01 - Version 3.7.5
  SQLite version 3.7.5 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Due to the discovery and fix of
  an obscure bug
  that could cause database corruption, upgrading from all prior
  releases of SQLite is recommended.  This bug was found during code
  review and has not been observed in the wild.

  This release adds new opcodes for the
  sqlite3_db_status() interface that allow more precise measurement of
  how the lookaside memory allocator is performing, which can be useful
  for tuning in applications with very tight memory constraints.

  The sqlite3_vsnprintf() interface was added.  This routine is simply
  a varargs version of the long-standing sqlite3_snprintf() interface.

  The output from sqlite3_trace() interface has been enhanced to work
  better (and faster) in systems that use recursive extensions such as
  FTS3 or RTREE.

  Testing with Valgrind shows that this release of SQLite is about 1%
  or 2% faster than the previous release for most operations.

  A fork of the popular ADO.NET adaptor for SQLite known as System.Data.SQLite
  is now available on http://System.Data.SQLite.org/.  The originator
  of System.Data.SQLite, Robert Simpson, is aware of this fork, has
  expressed his approval, and has commit privileges on the new Fossil
  repository.  The SQLite development team intends to maintain
  System.Data.SQLite moving forward.
2010-12-08 - Version 3.7.4
  SQLite version 3.7.4 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.2 and version 3.7.3
  is optional.  Upgrading from all other SQLite releases is recommended.

  This release features full-text search enhancements.  The older
  FTS3 virtual table is still fully supported, and should also run
  faster.  In addition, the new FTS4 virtual table is added.  FTS4
  follows the same syntax as FTS3 but holds additional metadata which
  facilitates some performance improvements and more advanced
  matchinfo() output.  Look for further full-text search enhancements
  in subsequent releases.

  Also in this release, the EXPLAIN QUERY PLAN output has been enhanced
  and new documentation is provided so that application developers can
  more easily understand how SQLite is performing their queries.

  Thanks to an account from the folks at http://www.devio.us/, OpenBSD
  has been added to the list of platforms upon which we
  test SQLite prior to every release.  That list of platforms
  now includes:

  
   Linux x86 &amp; x86_64
   MacOS 10.5 &amp; 10.6
   MacOS 10.2 PowerPC
   WinXP and Win7
   Android 2.2
   OpenBSD 4.7
  

  The previous release of SQLite (version 3.7.3) has proven to be very
  robust.  The only serious issue discovered was
  ticket 80ba201079 that
  describes an incorrect query result that can occur under very
  unusual circumstances.  The ticket description contains details of the
  problem.  Suffice it to say here that the problem is very obscure and
  is unlikely to effect most applications and so upgrading is optional.
  The problem is fixed, of course, in this release.
2010-October-08 - Version 3.7.3
  SQLite version 3.7.3 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.2 is optional.
  Upgrading from all other releases is recommended.

  This release adds two new interfaces (really just variations on existing
  interfaces).  The sqlite3_create_function_v2() interface adds a
  destructor for the application-data pointer.  The new
  sqlite3_soft_heap_limit64() interface allows the soft heap limit to
  be set to a value greater than 231.

  The RTREE extension has been enhanced with the ability to have an
  application-defined query region.  This might
  be used, for example, to locate all objects within
  the field of view of a camera.

  The 3.7.3 release also includes some performance enhancements, including
  query planner improvements, documentation updates,
  and fixes to some very obscure bugs.
2010-August-24 - Version 3.7.2
  SQLite version 3.7.2 fixes a long-standing bug that can cause the
  database free-page list to go corrupt if incremental_vacuum is used
  multiple times to
  partially reduce the size of a database file that contains many hundreds
  of unused database pages.  The original bug reports together with links
  to the patch that fixes it can be seen
  here.

  This bug has been in the code for at least a year and possibly longer.
  The bug has nothing to do with the versions 3.7.1 or 3.7.0 or any other
  recent release.  The fact that the bug was discovered (and fixed)
  within hours of the 3.7.1 release is purely a coincidence.

  The bug is impossible to hit without using incremental_vacuum and is
  very difficult to hit even with incremental_vacuum.  And the kind of
  corruption that the bug causes can usually be fixed
  simply by running VACUUM.  Nevertheless, because the bug can result
  in database corruption, it is recommended that all SQLite users upgrade
  to version 3.7.2 or later.
2010-August-23 - Version 3.7.1
  SQLite version 3.7.1 is a stabilization release for the 3.7.x series.
  Other than the filesize-in-header bug that was fixed in version 3.7.0.1,
  no major problems have been seen in 3.7.0.  Some minor corner-case
  performance regressions have been fixed.  A typo in the OS/2 interface
  has been repaired.

  A biggest part of the 3.7.1 release is a cleanup and refactoring of
  the pager module within SQLite.  This refactoring should have no
  application-visible effects.  The purpose was to reorganize the code
  in ways that make it easier to prove correctness.

  The 3.7.1 release adds new experimental methods for obtained more
  detailed memory usage information and for controlling database file
  fragmentation.  And the query planner now does a better job of
  optimizing the LIKE and GLOB operators.

  This release increases the maximum size of database pages from 32KiB to
  64KiB.  A database with 64KiB pages will not be readable or writable by
  older versions of SQLite.  Note that further increases in page size
  are not feasible since the file format uses 16-bit offsets to structures
  within each page.
2010-August-04 - Version 3.7.0.1
  SQLite version 3.7.0.1 is a patch release to fix a bug in the new
  filesize-in-header feature of the SQLite file format
  that could cause database corruption if the same database file is
  written alternately with version 3.7.0 and version 3.6.23.1 or earlier.
  A performance regression was also fixed in this release.
2010-07-22 - Version 3.7.0
  SQLite version 3.7.0 is a major release of SQLite that features
  a new transaction control mechanism using a write-ahead log or WAL.
  The traditional rollback-journal is still used as the default so there
  should be no visible change for legacy programs.  But newer programs
  can take advantage of improved performance and concurrency by enabling
  the WAL journaling mode.

  SQLite version 3.7.0 also contains some query planner enhancements and
  a few obscure bug fixes, but the only really big change is the addition
  of WAL mode.
2010-03-30 - Version 3.6.23.1
  SQLite version 3.6.23.1 is a patch release to fix a bug in the
  offsets() function of FTS3 at the request of the Mozilla.
2010-03-09 - Version 3.6.23
  SQLite version 3.6.23 is a regular bimonthly release of SQLite.
  Upgrading from the prior release is purely optional.

  This release contains new pragmas: the secure_delete pragma, and
  the compile_options pragma.
  There are a new SQL functions: sqlite_compileoption_used()
  and sqlite_compileoption_get().
  New C/C++ interfaces: sqlite3_compileoption_used(),
  sqlite3_compileoption_get(), SQLITE_CONFIG_LOG, and
  sqlite3_log().

  This release also includes several minor bug fixes and performance
  improvements.  Support for SQLITE_OMIT_FLOATING_POINT is enhanced.
  There are on-going improvements to FTS3.

  The ".genfkey" command in the Command Line Interface has been
  removed.  SQLite has supported standard SQL foreign key constraints
  since version 3.6.19 and so the ".genfkey" command was seen as
  an anachronism.
2010-01-06 - Version 3.6.22
  SQLite version 3.6.22 is a bug-fix release.  Two bugs have been fixed
  that might cause incorrect query results.
  
  Ticket 31338dca7e
  describes a
  problem with queries that have a WHERE clause of the form (x AND y) OR z
  where x and z come from one table of a join and y comes from a different
  table.
   Ticket eb5548a849
  describes
  a problem where the use of the CAST operator in the WHERE clause can lead
  to incorrect results if the column being cast to a new datatype is also
  used in the same WHERE clause without being cast.
  
  Both bugs are obscure,
  but because they could arise in an application after deployment, it is
  recommended that all applications upgrade SQLite to version 3.6.22.

  This release also includes other minor bug fixes and performance
  enhancements, especially in the FTS3 extension.
2009-12-07 - Version 3.6.21
  SQLite version 3.6.21 focuses on performance optimization.  For
  a certain set of traces, this version uses 12% fewer CPU instructions
  than the previous release (as measured by Valgrind).  In addition, the
  FTS3 extension has been through an extensive cleanup and rework and
  the sqlite3_trace() interface has been modified to insert
  bound parameter values into its output.
2009-11-04 - Version 3.6.20
  SQLite version 3.6.20 is a general maintenance release.  The
  query planner has been enhanced to work better with bound parameters
  in LIKE and GLOB operators and in range constraints and various minor
  bugs have been fixed.  Upgrading from 3.6.19 is optional.
2009-10-14 - Version 3.6.19
  SQLite version 3.6.19 adds native support for
  foreign key constraints, including deferred constraints and
  cascading deletes.  Enforcement of foreign keys is disabled by
  default for backwards compatibility and must be turned on using
  the foreign_keys pragma.

  Version 3.6.19 also adds support for the
  IS and IS NOT operators.  Formerly, SQLite (as most
  other SQL database engines) supported IS NULL and IS NOT NULL.  The
  IS and IS NOT operators are generalizations that allow the right-hand
  side to be an arbitrary expression.  IS and IS NOT work the same as
  == (equals) and != (not equals) except that with IS and IS NOT the
  NULL values compare equal to one another.
2009-09-11 - Version 3.6.18
  Beginning with this release, the SQLite source code is tracked and
  managed using the Fossil
  distributed configuration management system.  SQLite was previously
  versioned using CVS.  The entire CVS history has been imported into
  Fossil.  The older CVS repository remains on the website but is
  read-only.

  There are two major enhancements in SQLite version 3.6.18.  The first
  is a series or refinements to the query planner that help SQLite to
  choose better plans for joins where in the past it was selecting suboptimal
  query plans.  The SQLITE_ENABLE_STAT2 compile-time option has been
  added to cause SQLite to collect histogram data on indices when the
  ANALYZE command is run.  The use of histograms improve the query
  planning performance even more.

  The second major enhancement is that SQLite now support recursive
  triggers.  The older non-recursive behavior of triggers is still the
  default behavior.  Recursive triggers are activated using the
  recursive_triggers pragma.  In addition to allowing triggers to
  call themselves (either directly or indirectly) the new capability
  also fires DELETE triggers on rows that are removed from a table
  as a result of REPLACE conflict resolution processing.

  Non-recursive triggers are still the default behavior since this
  is least likely to cause problems for existing applications.  However,
  we anticipate that triggers will become
  recursive by default beginning with release 3.7.0.  At that point,
  applications that want to continue using the older non-recursive
  trigger behavior will need to use the recursive_triggers pragma
  to disable recursive triggers.

  This version of SQLite also contains bug fixes, though none of the
  bugs are serious and all are obscure, so upgrading is optional.

  The SQLite core continues to have 100% branch test coverage
  and so despite the many changes in this release, the developers
  believe that this version of SQLite is stable and ready for production
  use.
2009-08-10 - Version 3.6.17
  This is a monthly maintenance release with a focus of bug fixes,
  performance improvements, and increased test coverage.  This is the
  first release of SQLite since 100% branch test coverage
  was achieved on the SQLite core.

  In addition, a new interface sqlite3_strnicmp() is provided for the
  convenience of extension writers.

  None of the bugs fixed in this release are serious.  All bugs are
  obscure.  Upgrading is optional.
2009-07-25 - 100% Branch Test Coverage
  A subset of the TH3 test suite was measured by gcov to provide
  100% branch test coverage over the SQLite core
  (exclusive of the VFS backend and of extensions such as FTS3 and RTREE)
  when compiled for SuSE 10.1 Linux on x86.  The SQLite developers pledge
  to maintain branch test coverage at 100% in all future releases.
  Ongoing work will strive for 100% branch test coverage on the
  operating-system backends and extensions as well.
2009-06-27 - Version 3.6.16
  SQLite version 3.6.16 is another general maintenance release containing
  performance and robustness enhancements.  A single notable bug was fixed
  (ticket #3929).  This bug can cause INSERT or UPDATE statements to fail
  on indexed tables that have AFTER triggers that modify the same table and
  index.
2009-06-15 - Version 3.6.15
  SQLite version 3.6.15 is a general maintenance release containing
  performance and robustness enhancements and fixes for various obscure
  bugs.
2009-05-25 - Version 3.6.14.2
  SQLite version 3.6.14.2 fixes an obscure bug in the code generator
  (ticket #3879)
  section of SQLite which can potentially cause incorrect query results.
  The changes from the prior release consist of only this one bug fix,
  check-in [6676]
  and a change to the version number text.

  The bug was introduced in version 3.6.14.  It is recommended that
  users of version 3.6.14 and 3.6.14.1 upgrade to this release.  Applications
  are unlikely to hit this bug, but since it is difficult to predict which
  applications might hit it and which might not, we recommend that all
  users of 3.6.14 and 3.5.14.1 upgrade to this release.
2009-05-19 - Version 3.6.14.1
  SQLite version 3.6.14.1 is a patch release to version 3.6.14 with
  minimal changes that fixes three bugs.  Upgrading is only necessary
  for users who are impacted by one or more of those bugs.
2009-05-07 - Version 3.6.14
  SQLite version 3.6.14 provides new performance enhancements in
  the btree and pager layers and in the query optimizer.  Certain
  workloads can be as much as twice as fast as the previous release,
  though 10% faster is a more typical result.

  Queries against virtual tables that contain OR and IN operators
  in the WHERE clause are now able to use indexing.

  A new optional asynchronous I/O backend is available for
  Unix and Windows.  The asynchronous backend gives the illusion of faster
  response time by pushing slow write operations into a background thread.
  The tradeoff for faster response time is that more memory is required
  (to hold the content of the pending writes) and if a power failure or
  program crash occurs, some transactions that appeared to have committed
  might end up being rolled back upon restart.

  This release also contains many minor bug fixes, documentation enhancements,
  new test cases, and cleanups and simplifications to the source code.

  There is no compelling reason to upgrade from versions 3.6.12 or
  3.6.13 if those prior versions are working.  Though many users may
  benefit from the improved performance.
2008-12-16 - Version 3.6.7
  SQLite version 3.6.7 contains a major cleanup of the Unix driver,
  and support for the new Proxy Locking mechanism on Mac OS X.  Though
  the Unix driver is reorganized, its functionality is the same and so
  applications should not notice a difference.
2008-11-26 - Version 3.6.6.2
  This release fixes a bug that was introduced into SQLite version 3.6.6
  and which seems like it might be able to cause database corruption.  This
  bug was detected during stress testing.  It has not been seen in the wild.
  An analysis of the problem suggests that the bug might be able to cause
  database corruption, however focused efforts to find a real-world test
  cases that actually causes database corruption have so far been unsuccessful.
  Hence, the likelihood of this bug causing problems is low.  Nevertheless,
  we have decided to do an emergency branch release out of an abundance of
  caution.

  The version 3.6.6.2 release also fixes an obscure memory leak that
  can occur following a disk I/O error.
2008-11-22 - Version 3.6.6.1
  This release fixes a bug that was introduced into SQLite version 3.6.4
  and that can cause database corruption in obscure cases.  This bug has
  never been seen in the wild; it was first detected by internal stress
  tests and required substantial analysis before it could be shown to
  potentially lead to corruption.  So we feel that SQLite versions 3.6.4,
  3.6.5, and 3.6.6 are safe to use for development work.  But upgrading
  to this patch release or later is recommended prior to deploying
  products that incorporate SQLite.

  We have taken the unusual step of issuing a patch release in order to
  get the fix for this bug into circulation quickly.  SQLite version 3.6.7
  will continue on its normal path of development with an anticipated
  release in mid December.
2008-11-19 - Version 3.6.6
  SQLite version 3.6.5 is released.  This is a quick turn-around release
  that fixes a bug in virtual tables and FTS3 that snuck into
  version 3.6.5.  This release also adds the new
  application-defined page cache mechanism.
2008-11-12 - Version 3.6.5
  SQLite version 3.6.5 is released.  There are various minor feature
  enhancements and numerous obscure bug fixes.
  The change log contains the details.  Upgrading is
  optional.
2008-11-01 - Bloomberg Joins SQLite Consortium
  The SQLite developers are honored to announce that
  Bloomberg has joined the
  SQLite Consortium.
2008-10-15 - Version 3.6.4
  SQLite version 3.6.4 adds new features designed to help applications
  detect when indices are not being used on query.  There are also some
  important performance improvements.  Upgrading is optional.
2008-09-22 - Version 3.6.3
  SQLite version 3.6.3 fixes a bug in SELECT DISTINCT that was introduced
  by the previous version.  No new features are added.  Upgrading is
  recommended for all applications that make use of DISTINCT.
2008-08-30 - Version 3.6.2
  SQLite version 3.6.2 contains rewrites of the page-cache subsystem and
  the procedures for matching identifiers to table columns in SQL statements.
  These changes are designed to better modularize the code and make it more
  maintainable and reliable moving forward.  Nearly 5000 non-comment lines
  of core code (about 11.3%) have changed
  from the previous release.  Nevertheless, there should be no
  application-visible changes, other than bug fixes.
2008-08-06 - Version 3.6.1
  SQLite version 3.6.1 is a stabilization and performance enhancement
  release.
2008-07-16 - Version 3.6.0 beta
  Version 3.6.0 makes changes to the VFS object in order
  to make SQLite more easily portable to a wider variety of platforms.
  There are potential incompatibilities with some legacy applications.
  See the 35to36.html document for details.

  Many new interfaces are introduced in version 3.6.0.  The code is
  very well tested and is appropriate for use in stable systems.  We
  have attached the "beta" designation only so that we can make tweaks to
  the new interfaces in the next release without having to declare an
  incompatibility.
2008-05-12 - Version 3.5.9
  Version 3.5.9 adds a new experimental PRAGMA: journal_mode.
  Setting the journal mode to PERSIST can provide performance improvement
  on systems where deleting a file is expensive.  The PERSIST journal
  mode is still considered experimental and should be used with caution
  pending further testing.

  Version 3.5.9 is intended to be the last stable release prior to
  version 3.6.0.  Version 3.6.0 will make incompatible changes to the
  sqlite3_vfs VFS layer in order to address deficiencies in the original
  design.  These incompatibilities will only effect programmers who
  write their own custom VFS layers (typically embedded device builders).
  The planned VFS changes will be much smaller
  than the changes that occurred on the
  3.4.2 to 3.5.0 transaction that occurred last
  September.

  This release of SQLite is considered stable and ready for production use.
2008-04-16 - Version 3.5.8
  Version 3.5.8 includes some important new performance optimizations
  in the virtual machine code generator, including constant subexpression
  factoring and common subexpression elimination.  This release also
  creates new public interfaces:
  sqlite3_randomness() provides access to SQLite's internal
  pseudo-random number generator, sqlite3_limit() allows size
  limits to be set at run-time on a per-connection basis, and
  sqlite3_context_db_handle() is a convenience routine that allows
  an application-defined SQL function implementation to retrieve
  its database connection handle.

  This release of SQLite is considered stable and ready for production use.
2008-03-17 - Version 3.5.7
  Version 3.5.7 fixes several minor and obscure bugs, especially
  in the autoconf-generated makefile.  Upgrading is optional.
  This release of SQLite is considered stable and ready for production use.
2008-02-06 - Version 3.5.6
  Version 3.5.6 fixes a minor regression in 3.5.5 - a regression that
  had nothing to do with the massive change of the virtual machine
  to a register-based design.
  No problems have been reported with the new virtual machine.  This
  release of SQLite is considered stable and ready for production use.
2008-01-31 - Version 3.5.5
  Version 3.5.5 changes over 8% of the core source code of SQLite in order
  to convert the internal virtual machine from a stack-based design into
  a register-based design.  This change will allow future optimizations
  and will avoid an entire class of stack overflow bugs that have caused
  problems in the past.  Even though this change is large, extensive testing
  has found zero errors in the new virtual machine and so we believe this
  to be a very stable release.
2007-12-14 - Version 3.5.4
  Version 3.5.4 fixes a long-standing but obscure bug in UPDATE and
  DELETE which might cause database corruption.  (See ticket #2832.)
  Upgrading is recommended for all users.

  This release also brings the processing of ORDER BY statements into
  compliance with standard SQL.  This could, in theory, cause problems
  for existing applications that depend on the older, buggy behavior.
  See ticket #2822 for additional information.
2007-12-12 - SQLite Consortium Announced
  The SQLite Consortium was launched
  today with Mozilla and
  Symbian as charter members.
  As noted in the press release,
  the Consortium's goal is to promote the continuing vitality and
  independence of SQLite.
2007-11-27 - Version 3.5.3
  This is an incremental release that fixes several minor problems.
  Upgrading is optional.  If Version 3.5.2 or 3.5.1 is working fine
  for you, then there is no pressing need to change to 3.5.3.

  The prebuilt binaries and the amalgamation found on the
  download page include the FTS3 fulltext
  search extension module.  We are doing this on an experimental
  basis and are not promising to provide prebuilt binaries with
  FTS3 in the future.
2007-11-05 - Version 3.5.2
  This is an incremental release that fixes several minor problems,
  adds some obscure features, and provides some performance tweaks.
  Upgrading is optional.

  The experimental compile-time option
  SQLITE_OMIT_MEMORY_ALLOCATION is no longer supported.  On the other
  hand, it is now possible to compile SQLite so that it uses a static
  array for all its dynamic memory allocation needs and never calls
  malloc.  Expect to see additional radical changes to the memory
  allocation subsystem in future releases.
2007-10-04 - Version 3.5.1
  Fix a long-standing bug that might cause database corruption if a
  disk-full error occurs in the middle of a transaction and that
  transaction is not rolled back.
  Ticket #2686.

  The new VFS layer is stable.  However, we still reserve the right to
  make tweaks to the interface definition of the VFS if necessary.
2007-09-04 - Version 3.5.0 alpha
  The OS interface layer and the memory allocation subsystems in
  SQLite have been reimplemented.  The published API is largely unchanged
  but the (unpublished) OS interface has been modified extensively.
  Applications that implement their own OS interface will require
  modification.  See
  34to35.html for details.

  This is a large change.  Approximately 10% of the source code was
  modified.  We are calling this first release "alpha" in order to give
  the user community time to test and evaluate the changes before we
  freeze the new design.
2007-08-13 - Version 3.4.2
  While stress-testing the
  soft_heap_limit
  feature, a bug that could lead to
  database corruption was discovered and fixed.
  Though the consequences of this bug are severe, the chances of hitting
  it in a typical application are remote.  Upgrading is recommended
  only if you use the
  sqlite3_soft_heap_limit
  interface.
2007-07-20 - Version 3.4.1
  This release fixes a bug in VACUUM that
  can lead to database corruption.  The bug was introduced in version
  3.3.14.
  Upgrading is recommended for all users.  Also included are a slew of
  other more routine
  enhancements and bug fixes.
2007-06-18 - Version 3.4.0
  This release fixes two separate bugs either of which
  can lead to database corruption.  Upgrading
  is strongly recommended.  If you must continue using an older version
  of SQLite, please at least read about how to avoid these bugs
  at CorruptionFollowingBusyError and ticket #2418
  
  This release also adds explicit limits on the
  sizes and quantities of things SQLite will handle.  The new limits might
  causes compatibility problems for existing applications that
  use excessively large strings, BLOBs, tables, or SQL statements.
  The new limits can be increased at compile-time to work around any problems
  that arise.  Nevertheless, the version number of this release is
  3.4.0 instead of 3.3.18 in order to call attention to the possible
  incompatibility.
  
  There are also new features, including
  incremental BLOB I/O and
  incremental vacuum.
  See the change log
  for additional information.
2007-04-25 - Version 3.3.17
  This version fixes a bug in the forwards-compatibility logic of SQLite
  that was causing a database to become unreadable when it should have
  been read-only.  Upgrade from 3.3.16 only if you plan to deploy into
  a product that might need to be upgraded in the future.  For day to day
  use, it probably does not matter.
2007-04-18 - Version 3.3.16
  Performance improvements added in 3.3.14 but mistakenly turned off
  in 3.3.15 have been reinstated.  A bug has been fixed that prevented
  VACUUM from running if a NULL value was in a UNIQUE column.
2007-04-09 - Version 3.3.15
  An annoying bug introduced in 3.3.14 has been fixed.  There are
  also many enhancements to the test suite.
2007-04-02 - Version 3.3.14
  This version focuses on performance improvements.  If you recompile
  the amalgamation using GCC option -O3 (the precompiled binaries
  use -O2) you may see performance
  improvements of 35% or more over version 3.3.13 depending on your
  workload.  This version also
  adds support for 
  exclusive access mode.
2007-02-13 - Version 3.3.13
  This version fixes a subtle bug in the ORDER BY optimizer that can
  occur when using joins.  There are also a few minor enhancements.
  Upgrading is recommended.
2007-01-27 - Version 3.3.12
  The first published build of the previous version used the wrong
  set of source files.  Consequently, many people downloaded a build
  that was labeled as "3.3.11" but was really 3.3.10.  Version 3.3.12
  is released to clear up the ambiguity.  A couple more bugs have
  also been fixed and 
  PRAGMA integrity_check has been enhanced.
2007-01-22 - Version 3.3.11
  Version 3.3.11 fixes for a few more problems in version 3.3.9 that
  version 3.3.10 failed to catch.  Upgrading is recommended.
2007-01-09 - Version 3.3.10
  Version 3.3.10 fixes several bugs that were introduced by the previous
  release.  Upgrading is recommended.
2007-01-04 - Version 3.3.9
  Version 3.3.9 fixes bugs that can lead to database corruption under
  obscure and difficult to reproduce circumstances.  See
  DatabaseCorruption in the
  wiki for details.
  This release also adds the new
  sqlite3_prepare_v2()
  API and includes important bug fixes in the command-line
  shell and enhancements to the query optimizer.  Upgrading is
  recommended.
2006-10-09 - Version 3.3.8
  Version 3.3.8 adds support for full-text search using the
  FTS1 module.  There are also minor bug fixes.  Upgrade only if
  you want to try out the new full-text search capabilities or if
  you are having problems with 3.3.7.
2006-08-12 - Version 3.3.7
  Version 3.3.7 includes support for loadable extensions and virtual
  tables.  But both features are still considered "beta" and their
  APIs are subject to change in a future release.  This release is
  mostly to make available the minor bug fixes that have accumulated
  since 3.3.6.  Upgrading is not necessary.  Do so only if you encounter
  one of the obscure bugs that have been fixed or if you want to try
  out the new features.
2006-06-19 - New Book About SQLite
  The Definitive Guide to SQLite,
  a 2nd edition book by Mike Owens and Grant Allen,
  is now available from Apress.
  The books covers the latest SQLite internals as well as
  the native C interface and bindings for PHP, Python,
  Perl, Ruby, Tcl, and Java.  Recommended.
2006-06-6 - Version 3.3.6
  Changes include improved tolerance for Windows virus scanners
  and faster :memory: databases.  There are also fixes for several
  obscure bugs.  Upgrade if you are having problems.
2006-04-5 - Version 3.3.5
  This release fixes many minor bugs and documentation typos and
  provides some minor new features and performance enhancements.
  Upgrade only if you are having problems or need one of the new features.
2006-02-11 - Version 3.3.4
  This release fixes several bugs, including a
  blunder that might cause a deadlock on multithreaded systems.
  Anyone using SQLite in a multithreaded environment should probably upgrade.
2006-01-31 - Version 3.3.3 stable
  There have been no major problems discovered in version 3.3.2, so
  we hereby declare the new APIs and language features to be stable
  and supported.
2006-01-24 - Version 3.3.2 beta
  More bug fixes and performance improvements as we move closer to
  a production-ready version 3.3.x.
2006-01-16 - Version 3.3.1 alpha
  Many bugs found in last week's alpha release have now been fixed and
  the library is running much faster again.

  Database connections can now be moved between threads as long as the
  connection holds no locks at the time it is moved.  Thus the common
  paradigm of maintaining a pool of database connections and handing
  them off to transient worker threads is now supported.
  Please help test this new feature.
  See the MultiThreading wiki page for additional
  information.
2006-01-10 - Version 3.3.0 alpha
  Version 3.3.0 adds support for CHECK constraints, DESC indices,
  separate REAL and INTEGER column affinities, a new OS interface layer
  design, and many other changes.  The code passed a regression
  test but should still be considered alpha.  Please report any
  problems.

  The file format for version 3.3.0 has changed slightly to support
  descending indices and
  a more efficient encoding of boolean values.  SQLite 3.3.0 will read and
  write legacy databases created with any prior version of SQLite 3.  But
  databases created by version 3.3.0 will not be readable or writable
  by earlier versions of the SQLite.  The older file format can be
  specified at compile-time for those rare cases where it is needed.
2005-12-19 - Versions 3.2.8 and 2.8.17
  These versions contain one-line changes to 3.2.7 and 2.8.16 to fix a bug
  that has been present since March of 2002 and version 2.4.0.
  That bug might possibly cause database corruption if a large INSERT or
  UPDATE statement within a multi-statement transaction fails due to a
  uniqueness constraint but the containing transaction commits.
2005-09-24 - Version 3.2.7
  This version fixes several minor and obscure bugs.
  Upgrade only if you are having problems.
2005-09-16 - Version 3.2.6 - Critical Bug Fix
  This version fixes a bug that can result in database
  corruption if a VACUUM of a 1 gigabyte or larger database fails
  (perhaps do to running out of disk space or an unexpected power loss)
  and is later rolled back.
  
  Also in this release:
  The ORDER BY and GROUP BY processing was rewritten to use less memory.
  Support for COUNT(DISTINCT) was added.  The LIKE operator can now be
  used by the optimizer on columns with COLLATE NOCASE.
2005-08-27 - Version 3.2.5
  This release fixes a few more lingering bugs in the new code.
  We expect that this release will be stable and ready for production use.
2005-08-24 - Version 3.2.4
  This release fixes a bug in the new optimizer that can lead to segfaults
  when parsing very complex WHERE clauses.
2005-08-21 - Version 3.2.3
  This release adds the ANALYZE command,
  the CAST operator, and many
  very substantial improvements to the query optimizer.  See the
  change log for additional
  information.
2005-08-02 - 2005 Open Source Award for SQLite
  SQLite and its primary author D. Richard Hipp have been honored with a
  2005 Open Source Award
  from Google and O'Reilly.
2005-06-13 - Version 3.2.2
  This release includes numerous minor bug fixes, speed improvements,
  and code size reductions.  There is no reason to upgrade unless you
  are having problems or unless you just want to.
2005-03-29 - Version 3.2.1
  This release fixes a memory allocation problem in the new
  ALTER TABLE ADD COLUMN
  command.
2005-03-21 - Version 3.2.0
  The primary purpose for version 3.2.0 is to add support for
  ALTER TABLE ADD COLUMN.
  The new ADD COLUMN capability is made
  possible by AOL developers supporting and embracing great
  open-source software.  Thanks, AOL!

  Version 3.2.0 also fixes an obscure but serious bug that was discovered
  just prior to release.  If you have a multi-statement transaction and
  within that transaction an UPDATE or INSERT statement fails due to a
  constraint, then you try to rollback the whole transaction, the rollback
  might not work correctly.  See
  Ticket #1171 for details.  Upgrading is recommended for all users.
2005-03-16 - Version 3.1.6
  Version 3.1.6 fixes a critical bug that can cause database corruption
  when inserting rows into tables with around 125 columns. This bug was
  introduced in version 3.0.0.  See
  Ticket #1163 for additional information.
2005-03-11 - Versions 3.1.4 and 3.1.5 Released
  Version 3.1.4 fixes a critical bug that could cause database corruption
  if the autovacuum mode of version 3.1.0 is turned on (it is off by
  default) and a CREATE UNIQUE INDEX is executed within a transaction but
  fails because the indexed columns are not unique.  Anyone using the
  autovacuum feature and unique indices should upgrade.

  Version 3.1.5 adds the ability to disable
  the F_FULLFSYNC ioctl() in OS-X by setting "PRAGMA synchronous=on" instead
  of the default "PRAGMA synchronous=full".  There was an attempt to add
  this capability in 3.1.4 but it did not work due to a spelling error.
2005-02-19 - Version 3.1.3 Released
  Version 3.1.3 cleans up some minor issues discovered in version 3.1.2.
2005-02-15 - Versions 2.8.16 and 3.1.2 Released
  A critical bug in the VACUUM command that can lead to database
  corruption has been fixed in both the 2.x branch and the main
  3.x line.  This bug has existed in all prior versions of SQLite.
  Even though it is unlikely you will ever encounter this bug,
  it is suggested that all users upgrade.  See
  ticket #1116 for additional information.

  Version 3.1.2 is also the first stable release of the 3.1
  series.  SQLite 3.1 features added support for correlated
  subqueries, autovacuum, autoincrement, ALTER TABLE, and
  other enhancements.  See the
  release notes
  for version 3.1.0 for a detailed description of the
  changes available in the 3.1 series.
2005-02-01 - Version 3.1.1 (beta) Released
  Version 3.1.1 (beta) is now available on the
  website.  Version 3.1.1 is fully backwards compatible with the 3.0 series
  and features many new features including Autovacuum and correlated
  subqueries.  The
  release notes
  From version 3.1.0 apply equally to this release beta.  A stable release
  is expected within a couple of weeks.
2005-01-21 - Version 3.1.0 (alpha) Released
  Version 3.1.0 (alpha) is now available on the
  website.  Version 3.1.0 is fully backwards compatible with the 3.0 series
  and features many new features including Autovacuum and correlated
  subqueries.  See the
  release notes
  for details.

  This is an alpha release.  A beta release is expected in about a week
  with the first stable release to follow after two more weeks.
2004-11-09 - SQLite at the 2004 International PHP Conference
  There was a talk on the architecture of SQLite and how to optimize
  SQLite queries at the 2004 International PHP Conference in Frankfurt,
  Germany.
  Obsolete URL: https://www.sqlite.org/php2004/page-001.html 
  Slides from that talk are available.
2004-10-11 - Version 3.0.8
  Version 3.0.8 of SQLite contains several code optimizations and minor
  bug fixes and adds support for DEFERRED, IMMEDIATE, and EXCLUSIVE
  transactions.  This is an incremental release.  There is no reason
  to upgrade from version 3.0.7 if that version is working for you.
2004-10-10 - SQLite at the 11th
Annual Tcl/Tk Conference
  There will be a talk on the use of SQLite in Tcl/Tk at the
  11th Tcl/Tk Conference this week in
  New Orleans.  Visit http://www.tcl-lang.org/community/tcl2004/
  for details.
  Obsolete URL: https://www.sqlite.org/tclconf2004/page-001.html 
  Slides from the talk are available.
2004-09-18 - Version 3.0.7
  Version 3.0 has now been in use by multiple projects for several
  months with no major difficulties.   We consider it stable and
  ready for production use.
2004-09-02 - Version 3.0.6 (beta)
  Because of some important changes to sqlite3_step(),
  we have decided to
  do an additional beta release prior to the first "stable" release.
  If no serious problems are discovered in this version, we will
  release version 3.0 "stable" in about a week.
2004-08-29 - Version 3.0.5 (beta)
  The fourth beta release of SQLite version 3.0 is now available.
  The next release is expected to be called "stable".
2004-08-08 - Version 3.0.4 (beta)
  The third beta release of SQLite version 3.0 is now available.
  This new beta fixes several bugs including a database corruption
  problem that can occur when doing a DELETE while a SELECT is pending.
  Expect at least one more beta before version 3.0 goes final.
2004-07-22 - Version 3.0.3 (beta)
  The second beta release of SQLite version 3.0 is now available.
  This new beta fixes many bugs and adds support for databases with
  varying page sizes.  The next 3.0 release will probably be called
  a final or stable release.

  Version 3.0 adds support for internationalization and a new
  more compact file format.
  Details.
  The API and file format have been fixed since 3.0.2.  All
  regression tests pass (over 100000 tests) and the test suite
  exercises over 95% of the code.

  SQLite version 3.0 is made possible in part by AOL
  developers supporting and embracing great Open-Source Software.
2004-07-22 - Version 2.8.15
  SQLite version 2.8.15 is a maintenance release for the version 2.8
  series.  Version 2.8 continues to be maintained with bug fixes, but
  no new features will be added to version 2.8.  All the changes in
  this release are minor.  If you are not having problems, there is
  there is no reason to upgrade.
2004-06-30 - Version 3.0.2 (beta) Released
  The first beta release of SQLite version 3.0 is now available.
  Version 3.0 adds support for internationalization and a new
  more compact file format.
  Details.
  As of this release, the API and file format are frozen.  All
  regression tests pass (over 100000 tests) and the test suite
  exercises over 95% of the code.

  SQLite version 3.0 is made possible in part by AOL
  developers supporting and embracing great Open-Source Software.
2004-06-25 - Website hacked
  The www.sqlite.org website was hacked sometime around 2004-06-22
  because the lead SQLite developer failed to properly patch CVS.
  Evidence suggests that the attacker was unable to elevate privileges
  above user "cvs".  Nevertheless, as a precaution the entire website
  has been reconstructed from scratch on a fresh machine.  All services
  should be back to normal as of 2004-06-28.
2004-06-18 - Version 3.0.0 (alpha) Released
  The first alpha release of SQLite version 3.0 is available for
  public review and comment.  Version 3.0 enhances internationalization support
  through the use of UTF-16 and user-defined text collating sequences.
  BLOBs can now be stored directly, without encoding.
  A new file format results in databases that are 25% smaller (depending
  on content).  The code is also a little faster.  In spite of the many
  new features, the library footprint is still less than 240KB
  (x86, gcc -O1).
  Additional information.

  Our intent is to freeze the file format and API on 2004-07-01.
  Users are encouraged to review and evaluate this alpha release carefully
  and submit any feedback prior to that date.

  The 2.8 series of SQLite will continue to be supported with bug
  fixes for the foreseeable future.
2004-06-09 - Version 2.8.14 Released
  SQLite version 2.8.14 is a patch release to the stable 2.8 series.
  There is no reason to upgrade if 2.8.13 is working ok for you.
  This is only a bug-fix release.  Most development effort is
  going into version 3.0.0 which is due out soon.
2004-05-31 - CVS Access Temporarily Disabled
  Anonymous access to the CVS repository will be suspended
  for 2 weeks beginning on 2004-06-04.  Everyone will still
  be able to download
  prepackaged source bundles, create or modify trouble tickets, or view
  change logs during the CVS service interruption. Full open access to the
  CVS repository will be restored on 2004-06-18.
2004-04-23 - Work Begins On SQLite Version 3
  Work has begun on version 3 of SQLite.  Version 3 is a major
  changes to both the C-language API and the underlying file format
  that will enable SQLite to better support internationalization.
  The first beta is schedule for release on 2004-07-01.

  Plans are to continue to support SQLite version 2.8 with
  bug fixes.  But all new development will occur in version 3.0.

This page last modified on  2023-12-05 14:43:20 UTC


