# page_content

c2: Version Numbers in SQLite
c5: versionnumbers.html
c4:
1. SQLite Version Numbers


Beginning with version 3.9.0 (2015-10-14) SQLite uses 
semantic versioning.
Prior to that time, SQLite employed a version identifier that
contained between two and four numbers.

1.1. The New Version Numbering System (After 2015-10-14)


All SQLite releases starting with 3.9.0 use a three-number
"semantic version" of the form X.Y.Z.
The first number X is only increased when there is a change that
breaks backward compatibility.  The
current value for X is 3, and the SQLite developers plan to support
the current SQLite database file format, SQL syntax, and C interface
through at least the year 2050.  Hence, one
can expect that all future versions of SQLite for the next several
decades will begin with "3.".


The second number Y is incremented for any change that breaks forward
compatibility by adding new features.
Most future SQLite releases are expected
to increment the second number Y.  The Z is reset to zero whenever Y
is increased.


The third number Z is incremented for releases consisting of only
small changes that implement performance enhancements and/or bug fixes.


The rate of enhancement for SQLite over the previous five years
(2010-2015) is approximately 6 increments of Y per year.  The
numbering format used by for SQLITE_VERSION_NUMBER and
sqlite3_libversion_number() allows versions up to 3.999.999, which is
more than enough for the planned end-of-support date for SQLite
in 2050.  However, the current tarball naming conventions only
reserve two digits for the Y and so the naming format for downloads
will need to be revised in about 2030.

1.2. The Historical Numbering System (Before 2015-10-14)

This historical version numbering system used a two-, three-,
or four-number version:  W.X, W.X.Y, or W.X.Y.Z.
W was the file format: 1 or 2 or 3.
X was the major version.
Y was the minor version.
Z was used only for patch releases to fix bugs.


There have been three historical file formats for SQLite.
SQLite 1.0 through 1.0.32 used the
gdbm library as its storage
engine.
SQLite 2.0.0 through 2.8.17 used a custom b-tree storage engine that
supported only text keys and data.
All modern versions of SQLite (3.0.0 to present) use a b-tree storage
engine that has full support for binary data and Unicode.


This major version number X was historically incremented only for
large and important changes to the code.  What constituted "large
and important" was subjective.  The 3.6.23 to 3.7.0 change
was a result of adding support for WAL mode.
The 3.7.17 to 3.8.0 change was a result of rewrite known as the
next generation query planner.


The minor version number Y was historically incremented for new
features and/or new interfaces that did not significantly change
the structure of the code.  The addition of common table expressions,
partial indexes, and indexes on expressions are all examples of
"minor" changes.  Again, the distinction between "major" and "minor"
is subjective.


The patch level Z was historically only used for bug-fix releases
that changed only a small number of code lines.

1.3. Version History


Chronology
Change log

This page last modified on  2018-05-31 17:37:17 UTC


c2: Vulnerabilities
c5: cves.html
c4:
1. Executive Summary



CVEs about SQLite probably do not apply to your use of SQLite.


All historical vulnerabilities reported against SQLite require at least
one of these preconditions:


The attacker can submit and run arbitrary SQL statements.

The attacker can submit a maliciously crafted database file to the
application that the application will then open and query.



Few real-world applications meet either of these preconditions, and hence
few real-world applications are vulnerable, even if they use older
and unpatched versions of SQLite.


The SQLite development team fixes bugs promptly,
usually within hours of discovery.  New releases of SQLite
are issued if the bug seems likely to impact real-world
applications.


Grey-hat hackers are rewarded based on the number and severity of 
CVEs that they write.  This results in a proliferation of CVEs that
have minor impact, or no impact at all, but which make
exaggerated impact claims.


Very few CVEs written about SQLite are real vulnerabilities in the 
sense that they do not give any new capabilities to an attacker.
Consider:


    Almost all CVEs written against SQLite require the ability to
    inject and run arbitrary SQL.

    The advertised consequence of most CVEs is "denial of service",
    typically by causing a crash through a NULL pointer dereference or
    a division by zero, or similar.

    But if an attacker can already run
    arbitrary SQL, they do not need a bug to cause a denial of service.
    There are plenty of perfectly legal and valid SQL statements
    that will consume unlimited CPU, memory, and disk I/O in order
    to create a denial-of-service without requiring help from bugs.

    Hence, the mere fact that an attacker has a way to inject and run
    arbitrary SQL is in and of itself a denial-of-service attack.  That
    the arbitrary SQL might also tickle a bug in SQLite and cause a
    crash is not a new vulnerability.



The SQLite developers do not write CVEs.  Any CVEs you find on
SQLite are generated by third-parties, often without any input from the
core developers.  A common scenario is that someone will report a bug in
SQLite, which will promptly be fixed, then weeks later a CVE for that bug will
appear, unbeknownst to the developers.


You should not assume that a CVE about
SQLite contains authoritative information.
CVEs often contain inaccuracies.
The SQLite developers have attempted to add clarifications and
corrections to CVEs about SQLite.  



2. About CVEs

CVEs ("Common Vulnerabilities and Exposures") are reports of software
bugs that might allow a system to be hacked.  The idea
behind CVEs is sound.  They provide a common naming scheme whereby 
software bugs that might compromise information security can be easily
tracked.

While the original idea being CVEs is sound, the current processes for
creating and managing CVEs are inadequate.  There are countless grey-hat
hackers running fuzzers against a wide-variety of open-source software
products (SQLite as well as many others) and writing up CVEs against
any problems they find.  The grey-hats are rewarded, sometimes with
prestige and sometimes financially, by the number and severity of
the CVEs they write.  This incentive results in a proliferation
of CVEs which are often not well-vetted and which can have exaggerated
impact claims.  The quality-control procedures for CVEs are unable
to cope with this flood of inputs, making it difficult to correct
exaggerated, misleading, omitted, or inaccurate claims.

This is not to say that CVEs are useless.  CVEs do still (mostly)
report actual bugs.  But in most cases the bugs are not true vulnerabilities,
in the sense that they do not contribute to data loss or compromise
in and of themselves.
It is good that bugs are reported and fixed.  But not every bug is
accessible from every application.  In the case of SQLite, most of the
bugs reported by CVEs are inaccessible in most applications.  Upgrading
to the latest version of SQLite is always a good plan, but it need not
be an emergency just because an anonymous grey-hat on the internet
wrote up a CVE.

2.1. A separate SQL injection vulnerability is usually required


Other C-libraries that process complex structured inputs will
routinely be asked to deal with unvetted inputs from untrusted
sources.  Libraries like libjpeg, or libzip, or OpenSSL are
handed input streams that come directly from potentially hostile
agents.


But database engines like SQLite are usually not this way.
The SQL scripts that are passed into SQLite come from the
(trusted) application itself, not from an attacker.  Sometimes
applications contain bugs by which an external attacker can
trick the application into sending SQL of the attackers design
into the database engine.  This is a separate bug in the
application called an
SQL Injection
vulnerability.  Since SQL text is executable code, an
SQL Injection vulnerability is actually a special case of a
Remote
Code Execution (RCE) vulnerability.  An SQL Injection is perhaps not
quite as bad as other kinds of RCEs because,
while SQL is a powerful language, it is not as convenient
for crafting an exploit as Python or shell script or raw machine code.
Nevertheless, an SQL Injection is a serious problem.


Most CVEs written about SQLite assume that the attacker is
able to run arbitrary SQL scripts in SQLite.  In most applications,
this means that there must first be an SQL Injection vulnerability
that allows the attacker to inject the malicious SQL.


A few applications do allow untrusted SQL scripts received from
potentially hostile agents to be run direct in SQLite.  The main
example of this is the Chrome and Safari web browsers, which allow
an anonymous web page to run SQL using the WebSQL feature of Javascript.
This is done inside a sandbox with tightly controlled constraints on
resources, lest the SQL script try to soak up all available memory
or CPU cycles in a denial-of-service attack.  Chrome and Safari
have the infrastructure in place to allow a hostile agent to run
code which does not harm or compromise the rest of the machine.
They have to, as they also run Javascript which could, if not
tightly controlled, do even more damage than unrestrained SQL.
Apart from Chrome and Safari, no applications known to the
SQLite developers deliberately allows an anonymous remote agent
to run arbitrary SQL text.

However, most CVEs written against SQLite flippantly assume
that an attacker is free to run any arbitrary SQL in the database
engine.  So to a good approximation, this means most CVEs
written against SQLite really only apply to SQLite as it is
used in Chrome and Safari.  Or, in other words, most CVEs
for SQLite do not apply to you unless you are one of the
developers of Chrome or Safari.

2.2. Defense Against Dark Arts


Most applications can use SQLite without having to worry about
bugs in obscure SQL inputs.  If the application controls
the SQL, and the application is not deliberately trying to break
SQLite, then everything should just work.
It is not necessary to have the latest patched version of SQLite.
Any older version should work just fine.


However, there are some occasions where an application does need
to be able to safely run untrusted SQL. The SQLite developers work hard
to make SQLite safe for this purpose, though there are occasional
slip-ups.  It is good to keep up-to-date with the latest patches
in this case.  The separate defense against dark arts document
contains additional suggestions that can help prevent zero-day
attacks in cases where SQLite is given inputs that come directly
from untrusted sources. 

2.3. The SQLite Developer Policy Toward CVEs

SQLite developers fix all bugs in SQLite as soon as they are reported,
usually within a few hours.  The fixes are immediately available on the
public SQLite source tree.
If a bug seems like it might cause problems for existing applications,
a new patch release for SQLite will be issued.

However, the SQLite developers do not track CVEs.  There are 
various reasons for this:



The developers often do not find out about CVEs until long after the
bug is fixed.  You can see this by the fact that many CVEs reference the
bug fix in their initial report.


CVEs are a low-quality source of information about bugs in SQLite
that are likely to affect most applications.


Almost all bugs reported by CVEs are just bugs and not
true vulnerabilities.  Claiming that they are vulnerabilities is
stretching the meaning of the word "vulnerability" and the SQLite
developers do not wish to participate in that deception.


The developers have no editorial influence on the content of CVEs,
and they do not like to be controlled by groups in which they have
no voice.





3. Status Of Recent SQLite CVEs

Though the SQLite developers do not consider CVEs to be a reliable
source of information about bugs in SQLite, they recognize that many
groups, and especially small teams working at the bottom of tall
bureaucracies, sometimes need to track CVEs, whether they are useful
or not.  To aid in this chore, the following table of recent CVEs
affecting SQLite is provided.

If you notice new CVEs associated with SQLite that are not in
the table below, please bring them to the attention of the developers
on the SQLite Forum so they can
be added.




CVE Number
Fix
Comments





CVE-2022-46908

Bug not in the SQLite library
This is a bug in the --safe command-line option of the command-line shell
  program that is available for accessing SQLite database files.  The bug does
  not exist in the SQLite library.  Nor is it an issue for the CLI as long as
  the user does not depend on the --safe option.  It is not serious.  It is
  debatable whether or not this is a security issue.



CVE-2022-35737

3.39.2(2022-07-21)
This bug is an array-bounds overflow.  The bug is only accessible when using some
  of the C-language APIs provided by SQLite.  The bug cannot be reached using SQL
  nor can it be reached by providing SQLite with a corrupt database file.
  The bug only comes up when very long string inputs (greater than 2 billion bytes
  in length) are provided as arguments to a few specific C-language interfaces,
  and even then only under special circumstances.



CVE-2022-24854

Not a bug
This CVE describes a bug in an application that uses SQLite, not in SQLite itself.
  SQLite is doing everything correctly.  The application grants users the ability to
  run SQL statements, using SQLite, that can leak or change information that those users
  should not normally have access to.  This is purely an application bug.  It does not
  describe a malfunction or vulnerability in SQLite.



CVE-2022-21227

Not a bug
This CVE describes a bug in a third-party packages that provides a binding
  for SQLite to Node.js.  The bug reported is in the third-party Node.js binding,
  not in SQLite itself.  Do not be confused by the use of the word "SQLite" in the
  ambiguously-worded CVE description.



CVE-2021-45346

Not a bug
This CVE is misinformation.  See the discussion around
  SQLite forum post 53de8864ba114bf.



CVE-2021-42169

Not a bug
This CVE has nothing whatsoever to do with SQLite.  It is about a bug in
  application that happens to use SQLite.  Since SQLite is mentioned in the 
  CVE description, the CVE is included here to emphasize that
  this is not an SQLite bug.



CVE-2021-36690

Bug not in the SQLite library
This bug is not in the SQLite core library, but rather in an
  
  experimental extension that is used to implement the
  .expert command in the CLI.  The code that contains the bug
  does not appear in standard SQLite builds, though it
  is included in the sqlite3.exe command-line tool.
  Applications must link against the extra source code files that
  implement the extension and take other deliberate actions to
  activate the extension before the troublesome code can be run.
  For the rare application that uses the troublesome extension,
  the consequence of this bug is that malicious SQL can cause a
  NULL pointer deference and denial of service.
(details)



CVE-2021-28305

Not a bug
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite.  SQLite is mentioned by name in the CVE description,
  however, so we have included the CVE in the list.



CVE-2021-23404

Not a bug
This is not a bug in SQLite.  The bug is in a third-party application that
  uses SQLite and includes "sqlite" in its name.  This CVE is included on the
  list because it mentions SQLite even though the bug has nothing to do
  with SQLite.



CVE-2021-20227

3.34.1(2021-01-20)
Malicious SQL statement causes read-after-free.  No harm can come of this
  particular read-after-free instance, as far as anyone knows.  The bug is
  undetectable without a memory sanitizer. The CVE
  claims that this bug is an RCE - a Remote Code Execution
  vulnerability, but that claim is incorrect.
  The RCE claim is misinformation.
(details)



CVE-2021-20223

3.34.0(2020-12-01)
The problem identified by this CVE is not a vulnerability. 
  It is a malfunction. A coding error causes FTS5
  to sometimes return inconsistent and incorrect results under obscure circumstances,
  but no memory errors occur.
(details)



CVE-2020-15358

3.32.3(2020-06-18)
Malicious SQL statement causes an read past the end of a heap buffer.
(details)



CVE-2020-13871

3.32.3(2020-06-18)
Malicious SQL statement causes a read-only use-after-free memory error.
(details)



CVE-2020-13632

3.32.0(2020-05-22)
Malicious SQL statement causes a read of a NULL pointer in the
  matchinfo() SQL function of the FTS3 extension, resulting in
  denial of service.
(details)



CVE-2020-13631

3.32.0(2020-05-22)
Malicious SQL statement (an ALTER TABLE that tries to rename a
  virtual table into one of its own shadow tables)
  causes an infinite loop and denial of service.
(details)



CVE-2020-13630

3.32.0(2020-05-22)
Malicious SQL statement causes a read-only use-after-free,
  possibly resulting in a incorrect output from the snippet()
  SQL function of the FTS3 extension.  There is no known
  way to exfiltrate data or crash the application using this bug.
(details)



CVE-2020-13435

3.32.1(2020-05-25)
Malicious SQL statement causes a read access to a NULL pointer and
  denial of service.
(details)



CVE-2020-13434

3.32.1(2020-05-25)
Malicious SQL statement involving the printf() SQL function results
  in an integer overflow which can overwrite the stack with over 2
  billion bytes of 0x30 or 0x20 (ASCII '0' or ' ').
  Even though this is a stack overwrite, there is no known way to
  redirect control or otherwise escalate the level of harm.
  This is a denial-of-service attack only.
(details)



CVE-2020-11656

3.32.0(2020-05-22)
Malicious SQL statement causes read-only use-after-free of memory allocation
  if SQLite is compile with -DSQLITE_DEBUG.  Does not affect release
  builds.
(details)



CVE-2020-11655

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service.
(details)



CVE-2020-9327

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service
(details)



CVE-2020-6405

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service
(details)



CVE-2019-20218

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19959

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19926

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19925

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19924

3.31.0(2020-01-22)
Malicious SQL statement causes a uninitialized pointer reference and
  denial-of-service.
(details)



CVE-2019-19923

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service.
(details)



CVE-2019-19646

3.31.0(2020-01-22)
The PRAGMA integrity_check command might cause the byte-code for a prepared
  statement to loop indefinitely.  This might enable a denial-of-service, if the 
  application has not taken appropriate and prudent steps
  to limit the run-time of SQL statements.  This is not a vulnerability, as there
  are countless perfectly valid SQL queries, especially queries involving
  recursive common table expressions, that also run essentially forever.
(details)



CVE-2019-19317

3.31.0(2020-01-22)
This CVE identifies a bug in a development check-in of
  SQLite.  The bug never appeared in any official SQLite release.
(details)





This page last modified on  2022-12-14 16:29:06 UTC


c2: LoC Recommended Storage Format
c5: locrsf.html
c4:
1. Overview








SQLite is a
Recommended Storage Format
for datasets according to the
US Library of Congress.
Further information:




https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml#local

https://www.loc.gov/preservation/resources/rfs/data.html



As of this writing (2018-05-29) the only other recommended storage formats
for datasets are XML, JSON, and CSV.

1.1. What Is A Recommended Storage Format?


Recommended storage formats are formats which, in the opinion of the
preservationists at the Library of Congress, maximizes the chance of
survival and continued accessibility of digital content.
When selecting recommended storage formats, the following criteria are
considered (quoting from the LOC website):


Disclosure.
Degree to which complete specifications and tools for validating 
technical integrity exist and are accessible to those creating and
sustaining digital content.
A spectrum of disclosure levels can be observed for digital formats.
What is most significant is not approval by a recognized standards body,
but the existence of complete documentation.
Adoption.
Degree to which the format is already used by the primary creators,
disseminators, or users of information resources. 
This includes use as a master format, for delivery to end users, 
and as a means of interchange between systems.
Transparency.
Degree to which the digital representation is open to direct analysis 
with basic tools, such as human readability using a text-only editor.
Self-documentation.
Self-documenting digital objects contain basic descriptive, 
technical, and other administrative metadata.
External Dependencies.
Degree to which a particular format depends on particular hardware,
operating system, or software for rendering or use and the predicted
complexity of dealing with those dependencies in future technical 
environments.
Impact of Patents.
Degree to which the ability of archival institutions to sustain content 
in a format will be inhibited by patents.
Technical Protection Mechanisms.
Implementation of mechanisms such as encryption that prevent the 
preservation of content by a trusted repository. 

This page last modified on  2018-12-10 12:34:08 UTC


c2: The Error And Warning Log
c5: errlog.html
c4:
Overview 

SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.

1. Setting Up The Error Logging Callback

There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:


sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, pData);


The error logger callback function might look something like this:


void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}


The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to Syslog.  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.

Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.

2. Interface Details

The third argument to the sqlite3_config(SQLITE_CONFIG_LOG,...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.

The second argument to the error logger callback is an integer
extended error code.  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.

The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is not reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.

Applications can use the sqlite3_log(E,F,..) API to send new messages
to the log, if desired, but this is discouraged.  The sqlite3_log()
interface is intended for use by extensions only, not by applications.

3. Variety of Error Messages

The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.

The following is a partial list of the kinds of messages that might
appear in the error logger callback.



Any time there is an error either compiling an SQL statement 
(using sqlite3_prepare_v2() or its siblings) or running an SQL
statement (using sqlite3_step()) that error is logged.



When a schema change occurs that requires a prepared statement to be reparsed
and reprepared, that event is logged with the error code SQLITE_SCHEMA.
The reparse and reprepare is normally automatic (assuming that
sqlite3_prepare_v2() has been used to prepare the statements originally,
which is recommended) and so these logging events are normally the only
way to know that reprepares are taking place.


SQLITE_NOTICE messages are logged whenever a database has to be recovered
because the previous writer crashed without completing its transaction.
The error code is SQLITE_NOTICE_RECOVER_ROLLBACK when recovering a
rollback journal and SQLITE_NOTICE_RECOVER_WAL when recovering a 
write-ahead log.



SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See 1 and 2 for
additional information.)



Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.


I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. 

When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.


An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.


SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignoring certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.

4. Summary

The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occur with applications
after they get into the field.  The error logger callback has also 
proven useful in catching occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise
diagnostic capabilities later on.
This page last modified on  2022-01-20 21:38:08 UTC


c2: Using SQLite In Multi-Threaded Applications
c5: threadsafe.html
c4:
1. Overview

SQLite supports three different threading modes:


Single-thread.
In this mode, all mutexes are disabled and SQLite is unsafe to use in
more than a single thread at once.

Multi-thread.
In this mode, SQLite can be safely used by multiple threads provided that
no single database connection is used simultaneously in two or more threads.


Serialized.
In serialized mode, SQLite can be safely used by multiple threads with no
restriction.



The threading mode can be selected at compile-time (when the SQLite
library is being compiled from source code) or at start-time (when the
application that intends to use SQLite is initializing) or at
run-time (when a new SQLite database connection is being created).
Generally speaking, run-time overrides start-time and start-time
overrides compile-time.  Except, single-thread mode cannot be 
overridden once selected.



The default mode is serialized.


2. Compile-time selection of threading mode


Use the SQLITE_THREADSAFE compile-time parameter to select the
threading mode.  If no SQLITE_THREADSAFE compile-time parameter is
present, then serialized mode is used.
This can be made explicit with 
-DSQLITE_THREADSAFE=1.
With
-DSQLITE_THREADSAFE=0 the threading mode is
single-thread.  With
-DSQLITE_THREADSAFE=2 the threading mode is
multi-thread.



The return value of the sqlite3_threadsafe() interface is the value
of SQLITE_THREADSAFE set at compile-time. It does not reflect changes
to the threading mode made at runtime via the sqlite3_config()
interface or by flags given as the third argument to sqlite3_open_v2().



If single-thread mode is selected at compile-time, then critical
mutexing logic is omitted from the build and it is impossible to
enable either multi-thread or serialized modes at start-time or
run-time.


3. Start-time selection of threading mode


Assuming that the compile-time threading mode is not single-thread, then
the threading mode can be changed during initialization using the
sqlite3_config() interface.  The SQLITE_CONFIG_SINGLETHREAD verb
puts SQLite into single-thread mode, the SQLITE_CONFIG_MULTITHREAD
verb sets multi-thread mode, and the SQLITE_CONFIG_SERIALIZED verb
sets serialized mode.


4. Run-time selection of threading mode

If single-thread mode has not been selected at compile-time or start-time,
then individual database connections can be created as either multi-thread
or serialized.  It is not possible to downgrade an individual database
connection to single-thread mode.  Nor is it possible to escalate an
individual database connection if the compile-time or start-time mode
is single-thread.

The threading mode for an individual database connection is determined
by flags given as the third argument to sqlite3_open_v2().  The
SQLITE_OPEN_NOMUTEX flag causes the database connection to be in the
multi-thread mode and the SQLITE_OPEN_FULLMUTEX flag causes the connection
to be in serialized mode.  If neither flag is specified or if
sqlite3_open() or sqlite3_open16() are used instead of 
sqlite3_open_v2(), then the default
mode determined by the compile-time and start-time settings is used.

This page last modified on  2022-01-20 21:38:08 UTC


c2: Floating Point Numbers
c5: floatingpoint.html
c4:
1. How SQLite Stores Numbers


SQLite stores integer values in the 64-bit 
twos-complement
format&sup1.
This gives a storage range of -9223372036854775808 to +9223372036854775807,
inclusive.  Integers within this range are exact.


So-called "REAL" or floating point values are stored in the
IEEE 754
Binary-64
format&sup1.
This gives a range of positive values between approximately
1.7976931348623157e+308 and 4.9406564584124654e-324 with an equivalent
range of negative values.  A binary64 can also be 0.0 (and -0.0), positive
and negative infinity and "NaN" or "Not-a-Number".  Floating point
values are approximate.


Pay close attention to the last sentence in the previous paragraph:

Floating point values are approximate.



If you need an exact answer, you should not use binary64 floating-point
values, in SQLite or in any other product.  This is not an SQLite limitation.
It is a mathematical limitation inherent in the design of floating-point numbers.

&mdash;&sup1;
Exception:  The R-Tree extension stores information as 32-bit floating
point or integer values.

1.1. Floating-Point Accuracy


SQLite promises to preserve the 15 most significant digits of a floating
point value.  However, it makes no guarantees about the accuracy of
computations on floating point values, as no such guarantees are possible.
Performing math on floating-point values introduces error.
For example, consider what happens if you attempt to subtract two floating-point
numbers of similar magnitude:



1152693165.1106291898
-1152693165.1106280772

0.0000011126



The result shown above (0.0000011126) is the correct answer.  But if you
do this computation using binary64 floating-point, the answer you get is
0.00000095367431640625 - an error of about 14%.  If you do many similar
computations as part of your program, the errors add up so that your final
result might be completely meaningless.

The error arises because only about the first 15 significant digits of
each number are stored accurately, and the first difference between the two numbers
being subtracted is in the 16th digit.  

1.2. Floating Point Numbers


The binary64 floating-point format uses 64 bits per number.  Hence there
are 1.845e+19 different possible floating point values.  On the other hand
there are infinitely many real numbers in the range of 
1.7977e+308 and 4.9407e-324.  It follows then that binary64 cannot possibly
represent all possible real numbers within that range.  Approximations are
required.


An IEEE 754 floating-point value is an integer multiplied by a power
of two:


M &times 2E


The M value is the "mantissa" and E is the "exponent".  Both
M and E are integers.

For Binary64, M is a 53-bit integer and E is an 11-bit integer that is
offset so that represents a range of values between -1074 and +972, inclusive.

(NB:  The usual description of IEEE 754 is more complex, and it is important
to understand the added complexity if you really want to appreciate the details,
merits, and limitations of IEEE 754.  However, the integer description shown
here, while not exactly right, is easier to understand and is sufficient for
the purposes of this article.)

1.2.1. Unrepresentable numbers

Not every decimal number with fewer than 16 significant digits can be
represented exactly as a binary64 number.  In fact, most decimal numbers
with digits to the right of the decimal point lack an exact binary64
equivalent.  For example, if you have a database column that is intended
to hold an item price in dollars and cents, the only cents value that
can be exactly represented are 0.00, 0.25, 0.50, and 0.75.  Any other
numbers to the right of the decimal point result in an approximation.
If you provide a "price" value of 47.49, that number will be represented
in binary64 as:


6683623321994527 &times; 2-47


Which works out to be:


47.49000000000000198951966012828052043914794921875


That number is very close to 47.49, but it is not exact.  It is a little
too big.  If we reduce M by one to 6683623321994526 so that we have the
next smaller possible binary64 value, we get:


47.4899999999999948840923025272786617279052734375




This second number is too small.
The first number is closer to the desired value of 47.49, so that is the
one that gets used.  But it is not exact.  Most decimal values work this
way in IEEE 754.  Remember the key point we made above:


Floating point values are approximate.


If you remember nothing else about floating-point values, 
please don't forget this one key idea.

1.2.2. Is it close enough?

The precision provided by IEEE 754 Binary64 is sufficient for most computations.
For example, if "47.49" represents a price and inflation is running
at 2% per year, then the price is going up by about 0.0000000301 dollars per
second.  The error in the recorded value of 47.49 represents about 66 nanoseconds
worth of inflation.  So if the 47.49 price is exact
when you enter it, then the effects of inflation will cause the true value to
exactly equal the value actually stored
(47.4900000000000019895196601282805204391479492187) in less than 
one ten-millionth of a second.
Surely that level of precision is sufficient for most purposes?

2. Extensions For Dealing With Floating Point Numbers



2.1. The ieee754.c Extension

The ieee754 extension converts a floating point number between its
binary64 representation and the M&times;2E format.
In other words in the expression:


F = M &times 2E


The ieee754 extension converts between F and (M,E) and back again.

The ieee754 extension is not part of the amalgamation, but it is included
by default in the CLI.  If you want to include the ieee754 extension in your
application, you will need to compile and load it separately.



2.1.1. The ieee754() function

The ieee754(F) SQL function takes a single floating-point argument
as its input and returns a string that looks like this:


'ieee754(M,E)'


Except that the M and E are replaced by the mantissa and exponent of the
floating point number.  For example:

sqlite> .mode box
sqlite> SELECT ieee754(47.49) AS x;
┌───────────────────────────────┐
│               x               │
├───────────────────────────────┤
│ ieee754(6683623321994527,-47) │
└───────────────────────────────┘



Going in the other direction, the 2-argument version of ieee754() takes
the M and E values and converts them into the corresponding F value:

sqlite> select ieee754(6683623321994527,-47) as x;
┌───────┐
│   x   │
├───────┤
│ 47.49 │
└───────┘




2.1.2. The ieee754_mantissa() and ieee754_exponent() functions

The text output of the one-argument form of ieee754() is great for human
readability, but it is awkward to use as part of a larger expression.  Hence
the ieee754_mantissa() and ieee754_exponent() routines were added to return
the M and E values corresponding to their single argument F
value.
For example:

sqlite> .mode box
sqlite> SELECT ieee754_mantissa(47.49) AS M, ieee754_exponent(47.49) AS E;
┌──────────────────┬─────┐
│        M         │  E  │
├──────────────────┼─────┤
│ 6683623321994527 │ -47 │
└──────────────────┴─────┘




2.1.3. The ieee754_from_blob() and ieee754_to_blob() functions

The ieee754_to_blob(F) SQL function converts the floating point number F
into an 8-byte BLOB that is the big-endian binary64 encoding of that number.
The ieee754_from_blob(B) function goes the other way, converting an 8-byte
blob into the floating-point value that the binary64 encoding represents.

So, for example, if you read
on
Wikipedia that the encoding for the minimum positive binary64 value is
0x0000000000000001, then you can find the corresponding floating point value
like this:

sqlite> .mode box
sqlite> SELECT ieee754_from_blob(x'0000000000000001') AS F;
┌───────────────────────┐
│           F           │
├───────────────────────┤
│ 4.94065645841247e-324 │
└───────────────────────┘


Or go the other way:

sqlite> .mode box
sqlite> SELECT quote(ieee754_to_blob(4.94065645841247e-324)) AS binary64;
┌─────────────────────┐
│      binary64       │
├─────────────────────┤
│ X'0000000000000001' │
└─────────────────────┘




2.2. The decimal.c Extension

The decimal extension provides arbitrary-precision decimal arithmetic on
numbers stored as text strings.  Because the numbers are stored to arbitrary
precision and as text, no approximations are needed.  Computations can be
done exactly.

The decimal extension is not (currently) part of the SQLite amalgamation.
However, it is included in the CLI.

There are three math functions available:



 decimal_add(A,B)
 decimal_sub(A,B)
 decimal_mul(A,B)



These functions respectively add, subtract, and multiply their arguments
and return a new text string that is the decimal representation of the result.
There is no division operator at this time.

Use the decimal_cmp(A,B) to compare two decimal values.  The result will
be negative, zero, or positive if A is less than, equal to, or greater than B,
respectively.

The decimal_sum(X) function is an aggregate, like the built-in
sum() aggregate function, except that decimal_sum() computes its result
to arbitrary precision and is therefore precise.

Finally, the decimal extension provides the "decimal" collating sequences
that compares decimal text strings in numeric order.

3. Techniques


The following SQL illustrates how to use the ieee754 and decimal
extensions to compute the exact decimal equivalent
for a binary64 floating-point number.

-- The pow2 table will hold all the necessary powers of two.
CREATE TABLE pow2(x INTEGER PRIMARY KEY, v TEXT);
WITH RECURSIVE c(x,v) AS (
  VALUES(0,'1')
  UNION ALL
  SELECT x+1, decimal_mul(v,'2') FROM c WHERE x+1&lt;=971
) INSERT INTO pow2(x,v) SELECT x, v FROM c;
WITH RECURSIVE c(x,v) AS (
  VALUES(-1,'0.5')
  UNION ALL
  SELECT x-1, decimal_mul(v,'0.5') FROM c WHERE x-1&gt;=-1075
) INSERT INTO pow2(x,v) SELECT x, v FROM c;

-- This query finds the decimal representation of each value in the "c" table.
WITH c(n) AS (VALUES(47.49))
                 ----XXXXX----------- Replace with whatever you want
SELECT decimal_mul(ieee754_mantissa(c.n),pow2.v)
  FROM pow2, c WHERE pow2.x=ieee754_exponent(c.n);

This page last modified on  2022-11-21 14:37:06 UTC


c2: SQLite Library Footprint
c5: footprint.html
c4:
Size Of The SQLite Library


The code space used by the SQLite library depends on the target
platform, the compiler, and optimization settings.  These variables
also affect performance.



The chart below shows the relative size and performance for SQLite
as of 2017-10-08 for various compilers and optimization settings
as tested on Ubuntu 16.04.3 on x86_64.  General observations:



The Clang/LLVM compiler is not competitive with GCC.  Clang-generated
binaries are consistently larger and slower than GCC-generated binaries.


Profile guided optimization (PGO) is not helpful with SQLite.  PGO results
in binaries that are about 1% larger and about 0.33% slower.


GCC-7 generates binaries that are smaller and faster than GCC-5, though
the difference is not that great.


Compiling with GCC and -Os results in a binary that is slightly less 
than 500KB in size. (Update 2018-07-07: Due to the addition of new
features such as UPSERT and window functions, the library footprint
is now slightly larger than 500KB.)


The only significant design decision that developers need to make is
whether to use -Os (optimize for size) or -O6 (optimize for speed).
The -O6 setting makes binaries that run about 2% or 3% faster, but which
are also 66% larger.  The performance here is measured by counting CPU
cycles using cachegrind.  I-cache misses are not considered in the
analysis.  If I-cache misses are considered, builds with -O6 might not
be any faster than builds with -Os.


Taking into consideration all of the above, the SQLite developers recommend
compiling SQLite using GCC-7 with the -Os optimization setting.








Details



Measurements above were conducted using SQLite version 
5594a121bf132a98 from
2017-10-08.


The only SQLite compile-time option used was -DSQLITE_ENABLE_MEMSYS5.
The optional memsys5 memory allocator is used for performance testing
because it gives results that are more repeatable than the library-supplied
malloc()/free() on Ubuntu.


Performance can be improved and the size reduced by
enabling -DSQLITE_THREADSAFE=0,
-DSQLITE_DEFAULT_MEMSTATUS=0,
-DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1,
-DSQLITE_LIKE_DOESNT_MATCH_BLOBS,
-DSQLITE_MAX_EXPR_DEPTH=0,
-DSQLITE_OMIT_DECLTYPE,
-DSQLITE_OMIT_DEPRECATED,
-DSQLITE_OMIT_PROGRESS_CALLBACK,
-DSQLITE_OMIT_SHARED_CACHE, and
-DSQLITE_USE_ALLOCA.
All these options together result in about a 3.5% performance increase and
a 3.0% size reduction.


Adding optional features like -DSQLITE_ENABLE_JSON1,
-DSQLITE_ENABLE_FTS5, or -DSQLITE_ENABLE_RTREE will increase the
size of the library, obviously.


Performance was measured using the
speedtest1.c utility program,
which attempts to mimic a typical workload for SQLite.  Options to
the test runs are:

--shrink-memory --reprepare --stats --heap 10000000 64 --size 5

Performance was measured by running speedtest1 using cachegrind and
observing the "I refs" output.

This page last modified on  2018-07-07 17:44:20 UTC


c2: SQLite Support Options
c5: support.html
c4:
Public Forum

Connect with SQLite developers and other SQLite users on the
SQLite Forum.  This is the preferred
way to get free help with SQLite.




There was formerly a mailing list devoted to discussions
about SQLite, but as of 2020-03-12, the mailing list is deprecated.
Please use the 
SQLite Forum instead.
Historical mailing lists posts are preserved at

www.mail-archive.com



Paid Professional Support


If you would like professional support for SQLite
or if you want custom modifications performed by the
original author of SQLite, these services are available for a modest fee.
For additional information contact:



D. Richard Hipp 
Hwaci - Applied Software Research 
704.948.4565 
drh@hwaci.com

More Info



Proprietary SQLite Extensions

The core SQLite library found on this website is in the
public domain.  But there also exist
proprietary, licensed extensions to SQLite.



The SQLite Encryption Extension (SEE)
will read and write database files encrypted using
128-bit or 256-bit AES.


The SQLite Compressed and Encrypted Read-Only Database (CEROD)
extension will read a database file that is both compressed
and encrypted.


The ZIPVFS
extension will both read and write database files that are both compressed and optionally
encrypted using application-supplied compression and encryption routines.


The Test Harness #3 (TH3) is an alternative set of test
vectors for SQLite that verify the operation of SQLite in an
as-deployed configuration on embedded platforms with
100% branch test coverage.




This page last modified on  2021-01-16 12:18:53 UTC


c2: SQLite Frequently Asked Questions
c5: faq.html
c4:
Frequently Asked QuestionsHow do I create an AUTOINCREMENT field?What datatypes does SQLite support?SQLite lets me insert a string into a database column of type integer!Why doesn't SQLite allow me to use '0' and '0.0' as the primary
  key on two different rows of the same table?Can multiple applications or multiple instances of the same
  application access a single database file at the same time?Is SQLite threadsafe?How do I list all tables/indices contained in an SQLite databaseAre there any known size limits to SQLite databases?What is the maximum size of a VARCHAR in SQLite?Does SQLite support a BLOB type?How do I add, delete or rename columns from an existing table in SQLite?I deleted a lot of data but the database file did not get any
  smaller.  Is this a bug?Can I use SQLite in my commercial product without paying royalties?How do I use a string literal that contains an embedded single-quote (')
  character?What is an SQLITE_SCHEMA error, and why am I getting one?I get some compiler warnings when I compile SQLite.
  Isn't this a problem?  Doesn't it indicate poor code quality?Case-insensitive matching of Unicode characters does not work.INSERT is really slow - I can only do few dozen INSERTs per secondI accidentally deleted some important information from my SQLite database.
  How can I recover it?What is an SQLITE_CORRUPT error?  What does it mean for the database
  to be "malformed"? Why am I getting this error?Does SQLite support foreign keys?I get a compiler error if I use the SQLITE_OMIT_... 
  compile-time options when building SQLite.My WHERE clause expression column1="column1" does not work.
  It causes every row of the table to be returned, not just the rows
  where column1 has the value "column1".How are the syntax diagrams (a.k.a. "railroad" diagrams) for
  SQLite generated?The SQL standard requires that a UNIQUE constraint be enforced even if
  one or more of the columns in the constraint are NULL, but SQLite does
  not do this.  Isn't that a bug?What is the Export Control Classification Number (ECCN) for SQLite?My query does not return the column name that I expect.  Is this a bug?
(1) How do I create an AUTOINCREMENT field?
Short answer: A column declared INTEGER PRIMARY KEY will
  autoincrement.

  Longer answer:
  If you declare a column of a table to be INTEGER PRIMARY KEY, then
  whenever you insert a NULL
  into that column of the table, the NULL is automatically converted
  into an integer which is one greater than the largest value of that
  column over all other rows in the table, or 1 if the table is empty.
  Or, if the largest existing integer key 9223372036854775807 is in use then an
  unused key value is chosen at random.
  For example, suppose you have a table like this:

CREATE TABLE t1(
  a INTEGER PRIMARY KEY,
  b INTEGER
);

  With this table, the statement

INSERT INTO t1 VALUES(NULL,123);

  is logically equivalent to saying:

INSERT INTO t1 VALUES((SELECT max(a) FROM t1)+1,123);


  There is a function named
  sqlite3_last_insert_rowid() which will return the integer key
  for the most recent insert operation.

  Note that the integer key is one greater than the largest
  key that was in the table just prior to the insert.  The new key
  will be unique over all keys currently in the table, but it might
  overlap with keys that have been previously deleted from the
  table.  To create keys that are unique over the lifetime of the
  table, add the AUTOINCREMENT keyword to the INTEGER PRIMARY KEY
  declaration.  Then the key chosen will be one more than the
  largest key that has ever existed in that table.  If the largest
  possible key has previously existed in that table, then the INSERT
  will fail with an SQLITE_FULL error code.

(2) What datatypes does SQLite support?
SQLite uses dynamic typing.  Content can be stored as INTEGER,
  REAL, TEXT, BLOB, or as NULL.

(3) SQLite lets me insert a string into a database column of type integer!
This is a feature, not a bug.  SQLite uses dynamic typing. 
  It does not enforce data type  constraints.  Data of any type can
  (usually) be inserted into any column.  You can put arbitrary length
  strings into integer columns, floating point numbers in boolean columns,
  or dates in character columns.  The datatype you assign to a column in the
  CREATE TABLE command does not restrict what data can be put into
  that column.  Every column is able to hold
  an arbitrary length string.  (There is one exception: Columns of
  type INTEGER PRIMARY KEY may only hold a 64-bit signed integer.
  An error will result
  if you try to put anything other than an integer into an
  INTEGER PRIMARY KEY column.)

  But SQLite does use the declared type of a column as a hint
  that you prefer values in that format.  So, for example, if a
  column is of type INTEGER and you try to insert a string into
  that column, SQLite will attempt to convert the string into an
  integer.  If it can, it inserts the integer instead.  If not,
  it inserts the string.  This feature is called type affinity.
  

(4) Why doesn't SQLite allow me to use '0' and '0.0' as the primary
  key on two different rows of the same table?
This problem occurs when your primary key is a numeric type.  Change the
  datatype of your primary key to TEXT and it should work.

  Every row must have a unique primary key.  For a column with a
  numeric type, SQLite thinks that '0' and '0.0' are the
  same value because they compare equal to one another numerically.
  (See the previous question.)  Hence the values are not unique.

(5) Can multiple applications or multiple instances of the same
  application access a single database file at the same time?
Multiple processes can have the same database open at the same
  time.  Multiple processes can be doing a SELECT
  at the same time.  But only one process can be making changes to
  the database at any moment in time, however.

  SQLite uses reader/writer locks to control access to the database.
  (Under Win95/98/ME which lacks support for reader/writer locks, a
  probabilistic simulation is used instead.)
  But use caution: this locking mechanism might
  not work correctly if the database file is kept on an NFS filesystem.
  This is because fcntl() file locking is broken on many NFS implementations.
  You should avoid putting SQLite database files on NFS if multiple
  processes might try to access the file at the same time.  On Windows,
  Microsoft's documentation says that locking may not work under FAT
  filesystems if you are not running the Share.exe daemon.  People who
  have a lot of experience with Windows tell me that file locking of
  network files is very buggy and is not dependable.  If what they
  say is true, sharing an SQLite database between two or more Windows
  machines might cause unexpected problems.

  We are aware of no other embedded SQL database engine that
  supports as much concurrency as SQLite.  SQLite allows multiple processes
  to have the database file open at once, and for multiple processes to
  read the database at once.  When any process wants to write, it must
  lock the entire database file for the duration of its update.  But that
  normally only takes a few milliseconds.  Other processes just wait on
  the writer to finish then continue about their business.  Other embedded
  SQL database engines typically only allow a single process to connect to
  the database at once.

  However, client/server database engines (such as PostgreSQL, MySQL,
  or Oracle) usually support a higher level of concurrency and allow
  multiple processes to be writing to the same database at the same time.
  This is possible in a client/server database because there is always a
  single well-controlled server process available to coordinate access.
  If your application has a need for a lot of concurrency, then you should
  consider using a client/server database.  But experience suggests that
  most applications need much less concurrency than their designers imagine.
  

  When SQLite tries to access a file that is locked by another
  process, the default behavior is to return SQLITE_BUSY.  You can
  adjust this behavior from C code using the 
  sqlite3_busy_handler() or sqlite3_busy_timeout()
  API functions.

(6) Is SQLite threadsafe?
Threads are evil.
  Avoid them.

  SQLite is threadsafe.  We make this concession since many users choose
  to ignore the advice given in the previous paragraph.
  But in order to be thread-safe, SQLite must be compiled
  with the SQLITE_THREADSAFE preprocessor macro set to 1.  Both the Windows
  and Linux precompiled binaries in the distribution are compiled this way.
  If you are unsure if the SQLite library you are linking against is compiled
  to be threadsafe you can call the sqlite3_threadsafe()
  interface to find out.
  

  SQLite is threadsafe because it uses mutexes to serialize
  access to common data structures.  However, the work of acquiring and
  releasing these mutexes will slow SQLite down slightly.  Hence, if you
  do not need SQLite to be threadsafe, you should disable the mutexes
  for maximum performance.  See the threading mode documentation for
  additional information.

  Under Unix, you should not carry an open SQLite database across
  a fork() system call into the child process.

(7) How do I list all tables/indices contained in an SQLite database
If you are running the sqlite3 command-line access program
  you can type ".tables" to get a list of all tables.  Or you
  can type ".schema" to see the complete database schema including
  all tables and indices.  Either of these commands can be followed by
  a LIKE pattern that will restrict the tables that are displayed.

  From within a C/C++ program (or a script using Tcl/Ruby/Perl/Python
  bindings) you can get access to table and index names by doing a SELECT
  on a special table named "SQLITE_SCHEMA".  Every SQLite database
  has an SQLITE_SCHEMA table that defines the schema for the database.
  The SQLITE_SCHEMA table looks like this:

CREATE TABLE sqlite_schema (
  type TEXT,
  name TEXT,
  tbl_name TEXT,
  rootpage INTEGER,
  sql TEXT
);

  For tables, the type field will always be 'table' and the
  name field will be the name of the table.  So to get a list of
  all tables in the database, use the following SELECT command:

SELECT name FROM sqlite_schema
WHERE type='table'
ORDER BY name;

  For indices, type is equal to 'index', name is the
  name of the index and tbl_name is the name of the table to which
  the index belongs.  For both tables and indices, the sql field is
  the text of the original CREATE TABLE or CREATE INDEX statement that
  created the table or index.  For automatically created indices (used
  to implement the PRIMARY KEY or UNIQUE constraints) the sql field
  is NULL.

  The SQLITE_SCHEMA table cannot be modified using UPDATE, INSERT, 
  or DELETE (except under
  extraordinary conditions).  
  The SQLITE_SCHEMA table is automatically updated by commands like
  CREATE TABLE, CREATE INDEX, DROP TABLE, and DROP INDEX.

  Temporary tables do not appear in the SQLITE_SCHEMA table.  Temporary
  tables and their indices and triggers occur in another special table
  named SQLITE_TEMP_SCHEMA.  SQLITE_TEMP_SCHEMA works just like SQLITE_SCHEMA
  except that it is only visible to the application that created the 
  temporary tables.  To get a list of all tables, both permanent and
  temporary, one can use a command similar to the following:

SELECT name FROM 
   (SELECT * FROM sqlite_schema UNION ALL
    SELECT * FROM sqlite_temp_schema)
WHERE type='table'
ORDER BY name


(8) Are there any known size limits to SQLite databases?
See limits.html for a full discussion of
  the limits of SQLite.

(9) What is the maximum size of a VARCHAR in SQLite?
SQLite does not enforce the length of a VARCHAR.  You can declare
  a VARCHAR(10) and SQLite will be happy to store a 500-million character
  string there.  And it will keep all 500-million characters intact.
  Your content is never truncated.  SQLite understands the column type
  of "VARCHAR(N)" to be the same as "TEXT", regardless of the value
  of N.
  

(10) Does SQLite support a BLOB type?
SQLite allows you to store BLOB data in any 
  column, even columns that are declared to hold some other type.
  BLOBs can even be used as PRIMARY KEYs.

(11) How do I add, delete or rename columns from an existing table in SQLite?
SQLite has limited ALTER TABLE support that you can use to
  add, rename or drop columns or to change the name of a table
  as detailed at ALTER TABLE.

  If you want to make more complex changes in the structure or
  constraints of a table or its columns, you will have to recreate it.
  You can save existing data to a temporary table, drop the
  old table, create the new table, then copy the data back in from
  the temporary table. See 
  Making Other Kinds Of Table Schema Changes for procedure.

(12) I deleted a lot of data but the database file did not get any
  smaller.  Is this a bug?
No.  When you delete information from an SQLite database, the
  unused disk space is added to an internal "free-list" and is reused
  the next time you insert data.  The disk space is not lost.  But
  neither is it returned to the operating system.

  If you delete a lot of data and want to shrink the database file,
  run the VACUUM command.
  VACUUM will reconstruct
  the database from scratch.  This will leave the database with an empty
  free-list and a file that is minimal in size.  Note, however, that the
  VACUUM can take some time to run and it can use up to twice
  as much temporary disk space as the original file while it is running.
  

  An alternative to using the VACUUM command
  is auto-vacuum mode, enabled using the 
  auto_vacuum pragma.

(13) Can I use SQLite in my commercial product without paying royalties?
Yes.  SQLite is in the 
  public domain.  No claim of ownership is made
  to any part of the code.  You can do anything you want with it.

(14) How do I use a string literal that contains an embedded single-quote (')
  character?
The SQL standard specifies that single-quotes in strings are escaped
  by putting two single quotes in a row.  SQL works like the Pascal programming
  language in this regard. Example:
  

  
    INSERT INTO xyz VALUES('5 O''clock');
  

(15) What is an SQLITE_SCHEMA error, and why am I getting one?
An SQLITE_SCHEMA error is returned when a 
  prepared SQL statement is no longer valid and cannot be executed.
  When this occurs, the statement must be recompiled from SQL using 
  the sqlite3_prepare() API.
  An SQLITE_SCHEMA error can only occur when using the sqlite3_prepare(),
  and sqlite3_step() interfaces to run SQL.
  You will never receive an SQLITE_SCHEMA error from
  sqlite3_exec().  Nor will you receive an error if you
  prepare statements using sqlite3_prepare_v2() instead of
  sqlite3_prepare().

  The sqlite3_prepare_v2() interface creates a
  prepared statement that will automatically recompile itself if
  the schema changes.  The easiest way to deal with
  SQLITE_SCHEMA errors is to always use sqlite3_prepare_v2()
  instead of sqlite3_prepare().

(17) I get some compiler warnings when I compile SQLite.
  Isn't this a problem?  Doesn't it indicate poor code quality?
Quality assurance in SQLite is done using 
  full-coverage testing,
  not by compiler warnings or other static code analysis tools.
  In other words, we verify that SQLite actually gets the
  correct answer, not that it merely satisfies stylistic constraints.
  Most of the SQLite code base is devoted purely to testing.
  The SQLite test suite runs tens of thousands of separate test cases and
  many of those test cases are parameterized so that hundreds of millions
  of tests involving billions of SQL statements are run and evaluated
  for correctness prior to every release.  The developers use code
  coverage tools to verify that all paths through the code are tested.
  Whenever a bug is found in SQLite, new test cases are written to
  exhibit the bug so that the bug cannot recur undetected in the future.

  During testing, the SQLite library is compiled with special
  instrumentation that allows the test scripts to simulate a wide
  variety of failures in order to verify that SQLite recovers
  correctly.  Memory allocation is carefully tracked and no memory
  leaks occur, even following memory allocation failures.  A custom
  VFS layer is used to simulate operating system crashes and power
  failures in order to ensure that transactions are atomic across
  these events.  A mechanism for deliberately injecting I/O errors
  shows that SQLite is resilient to such malfunctions.  (As an
  experiment, try inducing these kinds of errors on other SQL database
  engines and see what happens!)

  We also run SQLite using Valgrind
  on Linux and verify that it detects no problems.

  Some people say that we should eliminate all warnings because
  benign warnings mask real warnings that might arise in future changes.
  This is true enough.  But in reply, the developers observe that all
  warnings have already been fixed in the builds
  used for SQLite development (various versions of GCC, MSVC,
  and clang).
  Compiler warnings usually only arise from compilers or compile-time 
  options that the SQLite developers do not use themselves.

(18) Case-insensitive matching of Unicode characters does not work.
The default configuration of SQLite only supports case-insensitive
  comparisons of ASCII characters.  The reason for this is that doing
  full Unicode case-insensitive comparisons and case conversions 
  requires tables and logic that would nearly double the size of
  the SQLite library.  The
  SQLite developers reason that any application that needs full
  Unicode case support probably already has the necessary tables and
  functions and so SQLite should not take up space to 
  duplicate this ability.

  Instead of providing full Unicode case support by default, 
  SQLite provides the ability to link against external
  Unicode comparison and conversion routines.
  The application can overload the built-in NOCASE collating
  sequence (using sqlite3_create_collation()) and the built-in
  like(), upper(), and lower() functions
  (using sqlite3_create_function()).  
  The SQLite source code includes an "ICU" extension that does 
  these overloads.  Or, developers can write their own overloads
  based on their own Unicode-aware comparison routines already
  contained within their project.

(19) INSERT is really slow - I can only do few dozen INSERTs per second
Actually, SQLite will easily do 50,000 or more INSERT statements per second
  on an average desktop computer.  But it will only do a few dozen transactions
  per second.  Transaction speed is limited by the rotational speed of
  your disk drive.  A transaction normally requires two complete rotations
  of the disk platter, which on a 7200RPM disk drive limits you to about
  60 transactions per second.

  Transaction speed is limited by disk drive speed because (by default)
  SQLite actually waits until the data really is safely stored on the disk
  surface before the transaction is complete.  That way, if you suddenly lose
  power or if your OS crashes, your data is still safe.  For details, 
  read about atomic commit in SQLite..

  By default, each INSERT statement is its own transaction.  But if you
  surround multiple INSERT statements with BEGIN...COMMIT then all the
  inserts are grouped into a single transaction.  The time needed to commit
  the transaction is amortized over all the enclosed insert statements and
  so the time per insert statement is greatly reduced.

  Another option is to run PRAGMA synchronous=OFF.  This command will
  cause SQLite to not wait on data to reach the disk surface, which will make
  write operations appear to be much faster.  But if you lose power in the
  middle of a transaction, your database file might go corrupt.

(20) I accidentally deleted some important information from my SQLite database.
  How can I recover it?
If you have a backup copy of your database file, recover the information
  from your backup.

  If you do not have a backup, recovery is very difficult.  You might
  be able to find partial string data in a binary dump of the raw database
  file.  Recovering numeric data might also be possible given special tools,
  though to our knowledge no such tools exist.  SQLite is sometimes compiled
  with the SQLITE_SECURE_DELETE option which overwrites all deleted content
  with zeros.  If that is the case then recovery is clearly impossible.
  Recovery is also impossible if you have run VACUUM since the data was
  deleted.  If SQLITE_SECURE_DELETE is not used and VACUUM has not been run,
  then some of the deleted content might still be in the database file, in
  areas marked for reuse.  But, again, there exist no procedures or tools
  that we know of to help you recover that data.

(21) What is an SQLITE_CORRUPT error?  What does it mean for the database
  to be "malformed"? Why am I getting this error?
An SQLITE_CORRUPT error is returned when SQLite detects an error
  in the structure, format, or other control elements of the
  database file.

  SQLite does not corrupt database files without external help.
  If your application crashes in the middle of an
  update, your data is safe.  The database is safe even if your OS
  crashes or takes a power loss.  The crash-resistance of SQLite has
  been extensively studied and tested and is attested by years of real-world 
  experience by billions of users.

  That said, there are a number of things that external programs or bugs
  in your hardware or OS can do to corrupt a database file.  See
  How To Corrupt An SQLite Database File for
  further information.

  You can use PRAGMA integrity_check 
  to do a thorough but time intensive test of the database integrity.

  You can use PRAGMA quick_check to do a faster 
  but less thorough test of the database integrity.

  Depending how badly your database is corrupted, you may be able to 
  recover some of the data by using the CLI to dump the schema and contents
  to a file and then recreate.  Unfortunately, once humpty-dumpty falls off 
  the wall, it is generally not possible to put him back together again.

(22) Does SQLite support foreign keys?

  As of version 3.6.19 (2009-10-14), 
  SQLite supports foreign key constraints.  But enforcement
  of foreign key constraints is turned off by default (for backwards compatibility).
  To enable foreign key constraint enforcement, run 
  PRAGMA foreign_keys=ON or compile with
  -DSQLITE_DEFAULT_FOREIGN_KEYS=1.

(23) I get a compiler error if I use the SQLITE_OMIT_... 
  compile-time options when building SQLite.
The SQLITE_OMIT_... compile-time options only work
  when building from canonical source files.  They do not work
  when you build from the SQLite amalgamation or from the pre-processed
  source files.

  It is possible to build a special amalgamation that will work with
  a predetermined set of SQLITE_OMIT_... options.  Instructions for doing
  so can be found with the SQLITE_OMIT_... documentation.

(24) My WHERE clause expression column1="column1" does not work.
  It causes every row of the table to be returned, not just the rows
  where column1 has the value "column1".
Use single-quotes, not double-quotes, around string literals in SQL.
  This is what the SQL standard requires.  Your WHERE clause expression
  should read: column1='column1'

  SQL uses double-quotes around identifiers (column or table names) that
  contains special characters or which are keywords.  So double-quotes are
  a way of escaping identifier names.  Hence, when you say
  column1="column1" that is equivalent to 
  column1=column1 which is obviously always true.

(25) How are the syntax diagrams (a.k.a. "railroad" diagrams) for
  SQLite generated?
Each diagram is hand-written using the Pikchr
  diagramming language.  These hand-written specifications are converted into
  SVG and inserted inline in the HTML files as part of the documentation build process.
  
  Many historical versions of the SQLite documentation used a different process for
  generating the syntax diagrams.  The historical process was based on Tcl/Tk and is
  described  at http://wiki.tcl-lang.org/21708.  The newer Pikchr-based syntax diagrams
  first landed on trunk on 2020-09-26.

(26) The SQL standard requires that a UNIQUE constraint be enforced even if
  one or more of the columns in the constraint are NULL, but SQLite does
  not do this.  Isn't that a bug?
Perhaps you are referring to the following statement from SQL92:

  
  A unique constraint is satisfied if and only if no two rows in a
  table have the same non-null values in the unique columns. 
  

  That statement is ambiguous, having at least two possible interpretations:

  
  A unique constraint is satisfied if and only if no two rows in a
  table have the same values and have non-null values in the unique columns.
  A unique constraint is satisfied if and only if no two rows in a 
  table have the same values in the subset of unique columns that are not null.
  

  SQLite follows interpretation (1), as does PostgreSQL, MySQL, Oracle,
  and Firebird.  It is true that Informix and Microsoft SQL Server use
  interpretation (2), however we the SQLite developers hold that 
  interpretation (1) is the most natural reading 
  of the requirement and we also want to maximize compatibility with other
  SQL database engines, and most other database engines also go with (1),
  so that is what SQLite does.

(27) What is the Export Control Classification Number (ECCN) for SQLite?
After careful review of the Commerce Control List (CCL), we are convinced
  that the core public-domain SQLite source code is not described by any ECCN,
  hence the ECCN should be reported as EAR99.

  The above is true for the core public-domain SQLite.  If you extend
  SQLite by adding new code, or if you statically link SQLite with your
  application, that might change the ECCN in your particular case.

(28) My query does not return the column name that I expect.  Is this a bug?
If the columns of your result set are named by AS clauses, then SQLite
  is guaranteed to use the identifier to the right of the AS keyword as the
  column name.  If the result set does not use an AS clause, then SQLite
  is free to name the column anything it wants.
  See the sqlite3_column_name() documentation for further information.

This page last modified on  2021-07-27 23:51:29 UTC


c2: Indexes On Expressions
c5: expridx.html
c4:
Normally, an SQL index references columns of a table.  But an index
can also be formed on expressions involving table columns.


As an example, consider the following table that tracks
dollar-amount changes on various "accounts":

CREATE TABLE account_change(
  chng_id INTEGER PRIMARY KEY,
  acct_no INTEGER REFERENCES account,
  location INTEGER REFERENCES locations,
  amt INTEGER,  -- in cents
  authority TEXT,
  comment TEXT
);
CREATE INDEX acctchng_magnitude ON account_change(acct_no, abs(amt));



Each entry in the account_change table records a deposit or a withdrawal
into an account.  Deposits have a positive "amt" and withdrawals have
a negative "amt".


The acctchng_magnitude index is over the account number ("acct_no") and
on the absolute value of the amount.  This index allows one to do 
efficient queries over the magnitude of a change to the account.
For example, to list all changes to account number $xyz that are
more than $100.00, one can say:


SELECT * FROM account_change WHERE acct_no=$xyz AND abs(amt)>=10000;



Or, to list all changes to one particular account ($xyz) in order of
decreasing magnitude, one can write:

SELECT * FROM account_change WHERE acct_no=$xyz
 ORDER BY abs(amt) DESC;



Both of the above example queries would work fine without the
acctchng_magnitude index.
The acctchng_magnitude index merely helps the queries to run
faster, especially on databases where there are many entries in
the table for each account.

1. How To Use Indexes On Expressions


Use a CREATE INDEX statement to create a new index on one or more
expressions just like you would to create an index on columns.  The only
difference is that expressions are listed as the elements to be indexed
rather than column names.


The SQLite query planner will consider using an index on an expression
when the expression that is indexed appears in the WHERE clause or in
the ORDER BY clause of a query, exactly as it is written in the
CREATE INDEX statement.  The query planner does not do algebra.  In order
to match WHERE clause constraints and ORDER BY terms to indexes, SQLite
requires that the expressions be the same, except for minor syntactic
differences such as white-space changes.  So if you have:

CREATE TABLE t2(x,y,z);
CREATE INDEX t2xy ON t2(x+y);



And then you run the query:

SELECT * FROM t2 WHERE y+x=22;



Then the index will not be used because 
the expression on the CREATE INDEX
statement (x+y) is not the same as the expression as it appears in the 
query (y+x).  The two expressions might be mathematically equivalent, but
the SQLite query planner insists that they be the same, not merely
equivalent.  Consider rewriting the query thusly:

SELECT * FROM t2 WHERE x+y=22;



This second query will likely use the index because now the expression
in the WHERE clause (x+y) matches the expression in the index exactly.


2. Restrictions


There are certain reasonable restrictions on expressions that appear in
CREATE INDEX statements:



Expressions in CREATE INDEX statements
may only refer to columns of the table being indexed, not to
columns in other tables.


Expressions in CREATE INDEX statements
may contain function calls, but only to functions whose output
is always determined completely by its input parameters (a.k.a.:
deterministic functions).  Obviously, functions like random() will not
work well in an index.  But also functions like sqlite_version(), though
they are constant across any one database connection, are not constant
across the life of the underlying database file, and hence may not be
used in a CREATE INDEX statement.


Note that application-defined SQL functions are by default considered
non-deterministic and may not be used in a CREATE INDEX statement unless
the SQLITE_DETERMINISTIC flag is used when the function is registered.


Expressions in CREATE INDEX statements may not use subqueries.


Expressions may only be used in CREATE INDEX statements, not within
UNIQUE or PRIMARY KEY constraints within the CREATE TABLE statement.



3. Compatibility


The ability to index expressions was added to SQLite with 
version 3.9.0 (2015-10-14).  A database that uses an index on
expressions will not be usable by earlier versions of SQLite.
This page last modified on  2022-01-20 21:38:08 UTC


c2: Query Language Understood by SQLite
c5: lang.html
c4:
SQL As Understood By SQLite

SQLite understands most of the standard SQL
language.  But it does omit some features
while at the same time
adding a few features of its own.  This document attempts to
describe precisely what parts of the SQL language SQLite does
and does not support.  A list of SQL keywords is 
also provided.  The SQL language syntax is described by
syntax diagrams.

The following syntax documentation topics are available:






aggregate functions
ALTER TABLE
ANALYZE
ATTACH DATABASE
BEGIN TRANSACTION
comment
COMMIT TRANSACTION
core functions
CREATE INDEX
CREATE TABLE
CREATE TRIGGER
CREATE VIEW
CREATE VIRTUAL TABLE
date and time functions
DELETE
DETACH DATABASE
DROP INDEX
DROP TABLE
DROP TRIGGER
DROP VIEW
END TRANSACTION
EXPLAIN
expression
INDEXED BY
INSERT
keywords
ON CONFLICT clause
PRAGMA
REINDEX
RELEASE SAVEPOINT
REPLACE
RETURNING clause
ROLLBACK TRANSACTION
SAVEPOINT
SELECT
UPDATE
UPSERT
VACUUM
WITH clause





The routines sqlite3_prepare_v2(), sqlite3_prepare(),
sqlite3_prepare16(), sqlite3_prepare16_v2(), 
sqlite3_exec(), and sqlite3_get_table() accept
an SQL statement list (sql-stmt-list) which is a semicolon-separated
list of statements.

sql-stmt-list:
 

 


Each SQL statement in the statement list is an instance of the
following:

sql-stmt:
 

 

This page last modified on  2021-02-02 12:11:09 UTC


c2: SQL Features That SQLite Does Not Implement
c5: omitted.html
c4:
SQL Features That SQLite Does Not Implement


SQLite implements most of the common features of SQL.
Rather than try to list all the features of SQL that SQLite does
support, it is much easier to list those that it does not.
Unsupported features of SQL are shown below.


See also the
Quirks, Caveats, and Gotchas of SQLite.



Complete ALTER TABLE support&nbsp;
  Only the RENAME TABLE, ADD COLUMN, RENAME COLUMN, and DROP COLUMN
  variants of the ALTER TABLE command are supported.  Other kinds of
  ALTER TABLE operations such as
  ALTER COLUMN, ADD CONSTRAINT, and so forth are omitted.
Complete trigger support&nbsp;
  FOR EACH ROW triggers are supported but not FOR EACH STATEMENT
  triggers.
Writing to VIEWs&nbsp;
  VIEWs in SQLite are read-only.  You may not execute a DELETE, INSERT, or
  UPDATE statement on a view. But you can create a trigger
  that fires on an attempt to DELETE, INSERT, or UPDATE a view and do
  what you need in the body of the trigger.
GRANT and REVOKE&nbsp;
  Since SQLite reads and writes an ordinary disk file, the
  only access permissions that can be applied are the normal
  file access permissions of the underlying operating system.
  The GRANT and REVOKE commands commonly found on client/server
  RDBMSes are not implemented because they would be meaningless
  for an embedded database engine.


This page last modified on  2022-04-13 13:11:44 UTC


c2: SQLite Session Module C/C++ Interface
c5: session.html
c4:
SQLite Session Module C/C++ Interface



This page defines the C-language interface to the SQLite 
   session extension.
   This is not a tutorial. These pages are designed to be precise, not easy 
   to read.  A tutorial is available separately.

This page contains all C-language interface information
in a single HTML file.  The same information is also
available broken out into lots of small pages
for easier viewing, if you prefer.

This document is created by a script which scans comments in the source 
code file sqlite3session.h.




Objects:


sqlite3_changegroup
sqlite3_changeset_iter
sqlite3_rebaser(exp)
sqlite3_session




Constants:


SQLITE_CHANGESETAPPLY_INVERT
SQLITE_CHANGESETAPPLY_NOSAVEPOINT
SQLITE_CHANGESETSTART_INVERT
SQLITE_CHANGESET_ABORT
SQLITE_CHANGESET_CONFLICT
SQLITE_CHANGESET_CONSTRAINT
SQLITE_CHANGESET_DATA
SQLITE_CHANGESET_FOREIGN_KEY
SQLITE_CHANGESET_NOTFOUND
SQLITE_CHANGESET_OMIT
SQLITE_CHANGESET_REPLACE
SQLITE_SESSION_CONFIG_STRMSIZE




Functions:


sqlite3changegroup_add
sqlite3changegroup_add_strm
sqlite3changegroup_delete
sqlite3changegroup_new
sqlite3changegroup_output
sqlite3changegroup_output_strm
sqlite3changeset_apply
sqlite3changeset_apply_strm
sqlite3changeset_apply_v2
sqlite3changeset_apply_v2_strm
sqlite3changeset_concat
sqlite3changeset_concat_strm
sqlite3changeset_conflict
sqlite3changeset_finalize
sqlite3changeset_fk_conflicts
sqlite3changeset_invert
sqlite3changeset_invert_strm
sqlite3changeset_new
sqlite3changeset_next
sqlite3changeset_old
sqlite3changeset_op
sqlite3changeset_pk
sqlite3changeset_start
sqlite3changeset_start_strm
sqlite3changeset_start_v2
sqlite3changeset_start_v2_strm
sqlite3rebaser_configure(exp)
sqlite3rebaser_create(exp)
sqlite3rebaser_delete(exp)
sqlite3rebaser_rebase(exp)
sqlite3rebaser_rebase_strm
sqlite3session_attach
sqlite3session_changeset
sqlite3session_changeset_size
sqlite3session_changeset_strm
sqlite3session_config
sqlite3session_create
sqlite3session_delete
sqlite3session_diff
sqlite3session_enable
sqlite3session_indirect
sqlite3session_isempty
sqlite3session_memory_used
sqlite3session_patchset
sqlite3session_patchset_strm
sqlite3session_table_filter




Flags for sqlite3changeset_start_v2#define SQLITE_CHANGESETSTART_INVERT        0x0002

The following flags may passed via the 4th parameter to
sqlite3changeset_start_v2 and sqlite3changeset_start_v2_strm:

SQLITE_CHANGESETAPPLY_INVERT 
  Invert the changeset while iterating through it. This is equivalent to
  inverting a changeset using sqlite3changeset_invert() before applying it.
  It is an error to specify this flag with a patchset.

Values for sqlite3session_config().#define SQLITE_SESSION_CONFIG_STRMSIZE 1

Changegroup Handletypedef struct sqlite3_changegroup sqlite3_changegroup;

A changegroup is an object used to combine two or more 
changesets or patchsets
Constructor: sqlite3changegroup_new()
Destructor: sqlite3changegroup_delete()
Methods:
 sqlite3changegroup_add(),
sqlite3changegroup_output()

Changeset Iterator Handletypedef struct sqlite3_changeset_iter sqlite3_changeset_iter;

An instance of this object acts as a cursor for iterating
over the elements of a changeset or patchset.
Constructors:
 sqlite3changeset_start(),
sqlite3changeset_start_v2()


sqlite3changeset_conflict
sqlite3changeset_finalize
sqlite3changeset_fk_conflicts
sqlite3changeset_new
sqlite3changeset_next
sqlite3changeset_old
sqlite3changeset_op
sqlite3changeset_pk




Rebasing changesetstypedef struct sqlite3_rebaser sqlite3_rebaser;
Important: This interface is experimental and is subject to change without notice.
Suppose there is a site hosting a database in state S0. And that
modifications are made that move that database to state S1 and a
changeset recorded (the "local" changeset). Then, a changeset based
on S0 is received from another site (the "remote" changeset) and 
applied to the database. The database is then in state 
(S1+"remote"), where the exact state depends on any conflict
resolution decisions (OMIT or REPLACE) made while applying "remote".
Rebasing a changeset is to update it to take those conflict 
resolution decisions into account, so that the same conflicts
do not have to be resolved elsewhere in the network. 

For example, if both the local and remote changesets contain an
INSERT of the same key on "CREATE TABLE t1(a PRIMARY KEY, b)":

  local:  INSERT INTO t1 VALUES(1, 'v1');
  remote: INSERT INTO t1 VALUES(1, 'v2');

and the conflict resolution is REPLACE, then the INSERT change is
removed from the local changeset (it was overridden). Or, if the
conflict resolution was "OMIT", then the local changeset is modified
to instead contain:

          UPDATE t1 SET b = 'v2' WHERE a=1;

Changes within the local changeset are rebased as follows:


Local INSERT
  This may only conflict with a remote INSERT. If the conflict 
  resolution was OMIT, then add an UPDATE change to the rebased
  changeset. Or, if the conflict resolution was REPLACE, add
  nothing to the rebased changeset.

Local DELETE
  This may conflict with a remote UPDATE or DELETE. In both cases the
  only possible resolution is OMIT. If the remote operation was a
  DELETE, then add no change to the rebased changeset. If the remote
  operation was an UPDATE, then the old.* fields of change are updated
  to reflect the new.* values in the UPDATE.

Local UPDATE
  This may conflict with a remote UPDATE or DELETE. If it conflicts
  with a DELETE, and the conflict resolution was OMIT, then the update
  is changed into an INSERT. Any undefined values in the new.* record
  from the update change are filled in using the old.* values from
  the conflicting DELETE. Or, if the conflict resolution was REPLACE,
  the UPDATE change is simply omitted from the rebased changeset.

  If conflict is with a remote UPDATE and the resolution is OMIT, then
  the old.* values are rebased using the new.* values in the remote
  change. Or, if the resolution is REPLACE, then the change is copied
  into the rebased changeset with updates to columns also updated by
  the conflicting remote UPDATE removed. If this means no columns would 
  be updated, the change is omitted.


A local change may be rebased against multiple remote changes 
simultaneously. If a single key is modified by multiple remote 
changesets, they are combined as follows before the local changeset
is rebased:


    If there has been one or more REPLACE resolutions on a
        key, it is rebased according to a REPLACE.

    If there have been no REPLACE resolutions on a key, then
        the local changeset is rebased according to the most recent
        of the OMIT resolutions.


Note that conflict resolutions from multiple remote changesets are 
combined on a per-field basis, not per-row. This means that in the 
case of multiple remote UPDATE operations, some fields of a single 
local change may be rebased for REPLACE while others are rebased for 
OMIT.

In order to rebase a local changeset, the remote changeset must first
be applied to the local database using sqlite3changeset_apply_v2() and
the buffer of rebase information captured. Then:


   An sqlite3_rebaser object is created by calling 
       sqlite3rebaser_create().
   The new object is configured with the rebase buffer obtained from
       sqlite3changeset_apply_v2() by calling sqlite3rebaser_configure().
       If the local changeset is to be rebased against multiple remote
       changesets, then sqlite3rebaser_configure() should be called
       multiple times, in the same order that the multiple
       sqlite3changeset_apply_v2() calls were made.
   Each local changeset is rebased by calling sqlite3rebaser_rebase().
   The sqlite3_rebaser object is deleted by calling
       sqlite3rebaser_delete().


Session Object Handletypedef struct sqlite3_session sqlite3_session;

An instance of this object is a session that can be used to
record changes to a database.
Constructor: sqlite3session_create()
Destructor: sqlite3session_delete()


sqlite3session_attach
sqlite3session_changeset
sqlite3session_changeset_size
sqlite3session_diff
sqlite3session_enable
sqlite3session_indirect
sqlite3session_patchset
sqlite3session_table_filter




Add A Changeset To A Changegroupint sqlite3changegroup_add(sqlite3_changegroup*, int nData, void *pData);

Add all changes within the changeset (or patchset) in buffer pData (size
nData bytes) to the changegroup. 

If the buffer contains a patchset, then all prior calls to this function
on the same changegroup object must also have specified patchsets. Or, if
the buffer contains a changeset, so must have the earlier calls to this
function. Otherwise, SQLITE_ERROR is returned and no changes are added
to the changegroup.

Rows within the changeset and changegroup are identified by the values in
their PRIMARY KEY columns. A change in the changeset is considered to
apply to the same row as a change already present in the changegroup if
the two rows have the same primary key.

Changes to rows that do not already appear in the changegroup are
simply copied into it. Or, if both the new changeset and the changegroup
contain changes that apply to a single row, the final contents of the
changegroup depends on the type of each change, as follows:


  Existing Change  
      New Change       
      Output Change
  INSERT INSERT 
      The new change is ignored. This case does not occur if the new
      changeset was recorded immediately after the changesets already
      added to the changegroup.
  INSERT UPDATE 
      The INSERT change remains in the changegroup. The values in the 
      INSERT change are modified as if the row was inserted by the
      existing change and then updated according to the new change.
  INSERT DELETE 
      The existing INSERT is removed from the changegroup. The DELETE is
      not added.
  UPDATE INSERT 
      The new change is ignored. This case does not occur if the new
      changeset was recorded immediately after the changesets already
      added to the changegroup.
  UPDATE UPDATE 
      The existing UPDATE remains within the changegroup. It is amended 
      so that the accompanying values are as if the row was updated once 
      by the existing change and then again by the new change.
  UPDATE DELETE 
      The existing UPDATE is replaced by the new DELETE within the
      changegroup.
  DELETE INSERT 
      If one or more of the column values in the row inserted by the
      new change differ from those in the row deleted by the existing 
      change, the existing DELETE is replaced by an UPDATE within the
      changegroup. Otherwise, if the inserted row is exactly the same 
      as the deleted row, the existing DELETE is simply discarded.
  DELETE UPDATE 
      The new change is ignored. This case does not occur if the new
      changeset was recorded immediately after the changesets already
      added to the changegroup.
  DELETE DELETE 
      The new change is ignored. This case does not occur if the new
      changeset was recorded immediately after the changesets already
      added to the changegroup.


If the new changeset contains changes to a table that is already present
in the changegroup, then the number of columns and the position of the
primary key columns for the table must be consistent. If this is not the
case, this function fails with SQLITE_SCHEMA. If the input changeset
appears to be corrupt and the corruption is detected, SQLITE_CORRUPT is
returned. Or, if an out-of-memory condition occurs during processing, this
function returns SQLITE_NOMEM. In all cases, if an error occurs the state
of the final contents of the changegroup is undefined.

If no error occurs, SQLITE_OK is returned.

Delete A Changegroup Objectvoid sqlite3changegroup_delete(sqlite3_changegroup*);

Create A New Changegroup Objectint sqlite3changegroup_new(sqlite3_changegroup **pp);

An sqlite3_changegroup object is used to combine two or more changesets
(or patchsets) into a single changeset (or patchset). A single changegroup
object may combine changesets or patchsets, but not both. The output is
always in the same format as the input.

If successful, this function returns SQLITE_OK and populates (*pp) with
a pointer to a new sqlite3_changegroup object before returning. The caller
should eventually free the returned object using a call to 
sqlite3changegroup_delete(). If an error occurs, an SQLite error code
(i.e. SQLITE_NOMEM) is returned and *pp is set to NULL.

The usual usage pattern for an sqlite3_changegroup object is as follows:


   It is created using a call to sqlite3changegroup_new().

   Zero or more changesets (or patchsets) are added to the object
       by calling sqlite3changegroup_add().

   The result of combining all input changesets together is obtained 
       by the application via a call to sqlite3changegroup_output().

   The object is deleted using a call to sqlite3changegroup_delete().


Any number of calls to add() and output() may be made between the calls to
new() and delete(), and in any order.

As well as the regular sqlite3changegroup_add() and 
sqlite3changegroup_output() functions, also available are the streaming
versions sqlite3changegroup_add_strm() and sqlite3changegroup_output_strm().

Obtain A Composite Changeset From A Changegroupint sqlite3changegroup_output(
  sqlite3_changegroup*,
  int *pnData,                    /* OUT: Size of output buffer in bytes */
  void **ppData                   /* OUT: Pointer to output buffer */
);

Obtain a buffer containing a changeset (or patchset) representing the
current contents of the changegroup. If the inputs to the changegroup
were themselves changesets, the output is a changeset. Or, if the
inputs were patchsets, the output is also a patchset.

As with the output of the sqlite3session_changeset() and
sqlite3session_patchset() functions, all changes related to a single
table are grouped together in the output of this function. Tables appear
in the same order as for the very first changeset added to the changegroup.
If the second or subsequent changesets added to the changegroup contain
changes for tables that do not appear in the first changeset, they are
appended onto the end of the output changeset, again in the order in
which they are first encountered.

If an error occurs, an SQLite error code is returned and the output
variables (*pnData) and (*ppData) are set to 0. Otherwise, SQLITE_OK
is returned and the output variables are set to the size of and a 
pointer to the output buffer, respectively. In this case it is the
responsibility of the caller to eventually free the buffer using a
call to sqlite3_free().

Concatenate Two Changeset Objectsint sqlite3changeset_concat(
  int nA,                         /* Number of bytes in buffer pA */
  void *pA,                       /* Pointer to buffer containing changeset A */
  int nB,                         /* Number of bytes in buffer pB */
  void *pB,                       /* Pointer to buffer containing changeset B */
  int *pnOut,                     /* OUT: Number of bytes in output changeset */
  void **ppOut                    /* OUT: Buffer containing output changeset */
);

This function is used to concatenate two changesets, A and B, into a 
single changeset. The result is a changeset equivalent to applying
changeset A followed by changeset B. 

This function combines the two input changesets using an 
sqlite3_changegroup object. Calling it produces similar results as the
following code fragment:


  sqlite3_changegroup *pGrp;
  rc = sqlite3_changegroup_new(&pGrp);
  if( rc==SQLITE_OK ) rc = sqlite3changegroup_add(pGrp, nA, pA);
  if( rc==SQLITE_OK ) rc = sqlite3changegroup_add(pGrp, nB, pB);
  if( rc==SQLITE_OK ){
    rc = sqlite3changegroup_output(pGrp, pnOut, ppOut);
  }else{
    *ppOut = 0;
    *pnOut = 0;
  }


Refer to the sqlite3_changegroup documentation below for details.

Obtain Conflicting Row Values From A Changeset Iteratorint sqlite3changeset_conflict(
  sqlite3_changeset_iter *pIter,  /* Changeset iterator */
  int iVal,                       /* Column number */
  sqlite3_value **ppValue         /* OUT: Value from conflicting row */
);

This function should only be used with iterator objects passed to a
conflict-handler callback by sqlite3changeset_apply() with either
SQLITE_CHANGESET_DATA or SQLITE_CHANGESET_CONFLICT. If this function
is called on any other iterator, SQLITE_MISUSE is returned and *ppValue
is set to NULL.

Argument iVal must be greater than or equal to 0, and less than the number
of columns in the table affected by the current change. Otherwise,
SQLITE_RANGE is returned and *ppValue is set to NULL.

If successful, this function sets *ppValue to point to a protected
sqlite3_value object containing the iVal'th value from the 
"conflicting row" associated with the current conflict-handler callback
and returns SQLITE_OK.

If some other error occurs (e.g. an OOM condition), an SQLite error code
is returned and *ppValue is set to NULL.

Finalize A Changeset Iteratorint sqlite3changeset_finalize(sqlite3_changeset_iter *pIter);

This function is used to finalize an iterator allocated with
sqlite3changeset_start().

This function should only be called on iterators created using the
sqlite3changeset_start() function. If an application calls this
function with an iterator passed to a conflict-handler by
sqlite3changeset_apply(), SQLITE_MISUSE is immediately returned and the
call has no effect.

If an error was encountered within a call to an sqlite3changeset_xxx()
function (for example an SQLITE_CORRUPT in sqlite3changeset_next() or an 
SQLITE_NOMEM in sqlite3changeset_new()) then an error code corresponding
to that error is returned by this function. Otherwise, SQLITE_OK is
returned. This is to allow the following pattern (pseudo-code):


  sqlite3changeset_start();
  while( SQLITE_ROW==sqlite3changeset_next() ){
    // Do something with change.
  }
  rc = sqlite3changeset_finalize();
  if( rc!=SQLITE_OK ){
    // An error has occurred 
  }


Determine The Number Of Foreign Key Constraint Violationsint sqlite3changeset_fk_conflicts(
  sqlite3_changeset_iter *pIter,  /* Changeset iterator */
  int *pnOut                      /* OUT: Number of FK violations */
);

This function may only be called with an iterator passed to an
SQLITE_CHANGESET_FOREIGN_KEY conflict handler callback. In this case
it sets the output variable to the total number of known foreign key
violations in the destination database and returns SQLITE_OK.

In all other cases this function returns SQLITE_MISUSE.

Invert A Changesetint sqlite3changeset_invert(
  int nIn, const void *pIn,       /* Input changeset */
  int *pnOut, void **ppOut        /* OUT: Inverse of input */
);

This function is used to "invert" a changeset object. Applying an inverted
changeset to a database reverses the effects of applying the uninverted
changeset. Specifically:


   Each DELETE change is changed to an INSERT, and
   Each INSERT change is changed to a DELETE, and
   For each UPDATE change, the old.* and new.* values are exchanged.


This function does not change the order in which changes appear within
the changeset. It merely reverses the sense of each individual change.

If successful, a pointer to a buffer containing the inverted changeset
is stored in *ppOut, the size of the same buffer is stored in *pnOut, and
SQLITE_OK is returned. If an error occurs, both *pnOut and *ppOut are
zeroed and an SQLite error code returned.

It is the responsibility of the caller to eventually call sqlite3_free()
on the *ppOut pointer to free the buffer allocation following a successful 
call to this function.

WARNING/TODO: This function currently assumes that the input is a valid
changeset. If it is not, the results are undefined.

Obtain new.* Values From A Changeset Iteratorint sqlite3changeset_new(
  sqlite3_changeset_iter *pIter,  /* Changeset iterator */
  int iVal,                       /* Column number */
  sqlite3_value **ppValue         /* OUT: New value (or NULL pointer) */
);

The pIter argument passed to this function may either be an iterator
passed to a conflict-handler by sqlite3changeset_apply(), or an iterator
created by sqlite3changeset_start(). In the latter case, the most recent
call to sqlite3changeset_next() must have returned SQLITE_ROW. 
Furthermore, it may only be called if the type of change that the iterator
currently points to is either SQLITE_UPDATE or SQLITE_INSERT. Otherwise,
this function returns SQLITE_MISUSE and sets *ppValue to NULL.

Argument iVal must be greater than or equal to 0, and less than the number
of columns in the table affected by the current change. Otherwise,
SQLITE_RANGE is returned and *ppValue is set to NULL.

If successful, this function sets *ppValue to point to a protected
sqlite3_value object containing the iVal'th value from the vector of 
new row values stored as part of the UPDATE or INSERT change and
returns SQLITE_OK. If the change is an UPDATE and does not include
a new value for the requested column, *ppValue is set to NULL and 
SQLITE_OK returned. The name of the function comes from the fact that 
this is similar to the "new.*" columns available to update or delete 
triggers.

If some other error occurs (e.g. an OOM condition), an SQLite error code
is returned and *ppValue is set to NULL.

Advance A Changeset Iteratorint sqlite3changeset_next(sqlite3_changeset_iter *pIter);

This function may only be used with iterators created by the function
sqlite3changeset_start(). If it is called on an iterator passed to
a conflict-handler callback by sqlite3changeset_apply(), SQLITE_MISUSE
is returned and the call has no effect.

Immediately after an iterator is created by sqlite3changeset_start(), it
does not point to any change in the changeset. Assuming the changeset
is not empty, the first call to this function advances the iterator to
point to the first change in the changeset. Each subsequent call advances
the iterator to point to the next change in the changeset (if any). If
no error occurs and the iterator points to a valid change after a call
to sqlite3changeset_next() has advanced it, SQLITE_ROW is returned. 
Otherwise, if all changes in the changeset have already been visited,
SQLITE_DONE is returned.

If an error occurs, an SQLite error code is returned. Possible error 
codes include SQLITE_CORRUPT (if the changeset buffer is corrupt) or 
SQLITE_NOMEM.

Obtain old.* Values From A Changeset Iteratorint sqlite3changeset_old(
  sqlite3_changeset_iter *pIter,  /* Changeset iterator */
  int iVal,                       /* Column number */
  sqlite3_value **ppValue         /* OUT: Old value (or NULL pointer) */
);

The pIter argument passed to this function may either be an iterator
passed to a conflict-handler by sqlite3changeset_apply(), or an iterator
created by sqlite3changeset_start(). In the latter case, the most recent
call to sqlite3changeset_next() must have returned SQLITE_ROW. 
Furthermore, it may only be called if the type of change that the iterator
currently points to is either SQLITE_DELETE or SQLITE_UPDATE. Otherwise,
this function returns SQLITE_MISUSE and sets *ppValue to NULL.

Argument iVal must be greater than or equal to 0, and less than the number
of columns in the table affected by the current change. Otherwise,
SQLITE_RANGE is returned and *ppValue is set to NULL.

If successful, this function sets *ppValue to point to a protected
sqlite3_value object containing the iVal'th value from the vector of 
original row values stored as part of the UPDATE or DELETE change and
returns SQLITE_OK. The name of the function comes from the fact that this 
is similar to the "old.*" columns available to update or delete triggers.

If some other error occurs (e.g. an OOM condition), an SQLite error code
is returned and *ppValue is set to NULL.

Obtain The Current Operation From A Changeset Iteratorint sqlite3changeset_op(
  sqlite3_changeset_iter *pIter,  /* Iterator object */
  const char **pzTab,             /* OUT: Pointer to table name */
  int *pnCol,                     /* OUT: Number of columns in table */
  int *pOp,                       /* OUT: SQLITE_INSERT, DELETE or UPDATE */
  int *pbIndirect                 /* OUT: True for an 'indirect' change */
);

The pIter argument passed to this function may either be an iterator
passed to a conflict-handler by sqlite3changeset_apply(), or an iterator
created by sqlite3changeset_start(). In the latter case, the most recent
call to sqlite3changeset_next() must have returned SQLITE_ROW. If this
is not the case, this function returns SQLITE_MISUSE.

Arguments pOp, pnCol and pzTab may not be NULL. Upon return, three
outputs are set through these pointers: 

*pOp is set to one of SQLITE_INSERT, SQLITE_DELETE or SQLITE_UPDATE,
depending on the type of change that the iterator currently points to;

*pnCol is set to the number of columns in the table affected by the change; and

*pzTab is set to point to a nul-terminated utf-8 encoded string containing
the name of the table affected by the current change. The buffer remains
valid until either sqlite3changeset_next() is called on the iterator
or until the conflict-handler function returns.

If pbIndirect is not NULL, then *pbIndirect is set to true (1) if the change
is an indirect change, or false (0) otherwise. See the documentation for
sqlite3session_indirect() for a description of direct and indirect
changes.

If no error occurs, SQLITE_OK is returned. If an error does occur, an
SQLite error code is returned. The values of the output variables may not
be trusted in this case.

Obtain The Primary Key Definition Of A Tableint sqlite3changeset_pk(
  sqlite3_changeset_iter *pIter,  /* Iterator object */
  unsigned char **pabPK,          /* OUT: Array of boolean - true for PK cols */
  int *pnCol                      /* OUT: Number of entries in output array */
);

For each modified table, a changeset includes the following:


   The number of columns in the table, and
   Which of those columns make up the tables PRIMARY KEY.


This function is used to find which columns comprise the PRIMARY KEY of
the table modified by the change that iterator pIter currently points to.
If successful, *pabPK is set to point to an array of nCol entries, where
nCol is the number of columns in the table. Elements of *pabPK are set to
0x01 if the corresponding column is part of the tables primary key, or
0x00 if it is not.

If argument pnCol is not NULL, then *pnCol is set to the number of columns
in the table.

If this function is called when the iterator does not point to a valid
entry, SQLITE_MISUSE is returned and the output variables zeroed. Otherwise,
SQLITE_OK is returned and the output variables populated as described
above.

Configure a changeset rebaser object.int sqlite3rebaser_configure(
  sqlite3_rebaser*, 
  int nRebase, const void *pRebase
); 
Important: This interface is experimental and is subject to change without notice.
Configure the changeset rebaser object to rebase changesets according
to the conflict resolutions described by buffer pRebase (size nRebase
bytes), which must have been obtained from a previous call to
sqlite3changeset_apply_v2().

Create a changeset rebaser object.int sqlite3rebaser_create(sqlite3_rebaser **ppNew);
Important: This interface is experimental and is subject to change without notice.
Allocate a new changeset rebaser object. If successful, set (*ppNew) to
point to the new object and return SQLITE_OK. Otherwise, if an error
occurs, return an SQLite error code (e.g. SQLITE_NOMEM) and set (*ppNew) 
to NULL. 

Delete a changeset rebaser object.void sqlite3rebaser_delete(sqlite3_rebaser *p); 
Important: This interface is experimental and is subject to change without notice.
Delete the changeset rebaser object and all associated resources. There
should be one call to this function for each successful invocation
of sqlite3rebaser_create().

Rebase a changesetint sqlite3rebaser_rebase(
  sqlite3_rebaser*,
  int nIn, const void *pIn, 
  int *pnOut, void **ppOut 
);
Important: This interface is experimental and is subject to change without notice.
Argument pIn must point to a buffer containing a changeset nIn bytes
in size. This function allocates and populates a buffer with a copy
of the changeset rebased according to the configuration of the
rebaser object passed as the first argument. If successful, (*ppOut)
is set to point to the new buffer containing the rebased changeset and 
(*pnOut) to its size in bytes and SQLITE_OK returned. It is the
responsibility of the caller to eventually free the new buffer using
sqlite3_free(). Otherwise, if an error occurs, (*ppOut) and (*pnOut)
are set to zero and an SQLite error code returned.

Attach A Table To A Session Objectint sqlite3session_attach(
  sqlite3_session *pSession,      /* Session object */
  const char *zTab                /* Table name */
);

If argument zTab is not NULL, then it is the name of a table to attach
to the session object passed as the first argument. All subsequent changes 
made to the table while the session object is enabled will be recorded. See 
documentation for sqlite3session_changeset() for further details.

Or, if argument zTab is NULL, then changes are recorded for all tables
in the database. If additional tables are added to the database (by 
executing "CREATE TABLE" statements) after this call is made, changes for 
the new tables are also recorded.

Changes can only be recorded for tables that have a PRIMARY KEY explicitly
defined as part of their CREATE TABLE statement. It does not matter if the 
PRIMARY KEY is an "INTEGER PRIMARY KEY" (rowid alias) or not. The PRIMARY
KEY may consist of a single column, or may be a composite key.

It is not an error if the named table does not exist in the database. Nor
is it an error if the named table does not have a PRIMARY KEY. However,
no changes will be recorded in either of these scenarios.

Changes are not recorded for individual rows that have NULL values stored
in one or more of their PRIMARY KEY columns.

SQLITE_OK is returned if the call completes without error. Or, if an error 
occurs, an SQLite error code (e.g. SQLITE_NOMEM) is returned.

Special sqlite_stat1 Handling

As of SQLite version 3.22.0, the "sqlite_stat1" table is an exception to 
some of the rules above. In SQLite, the schema of sqlite_stat1 is:
 
 &nbsp;     CREATE TABLE sqlite_stat1(tbl,idx,stat)  
 

Even though sqlite_stat1 does not have a PRIMARY KEY, changes are 
recorded for it as if the PRIMARY KEY is (tbl,idx). Additionally, changes 
are recorded for rows for which (idx IS NULL) is true. However, for such
rows a zero-length blob (SQL value X'') is stored in the changeset or
patchset instead of a NULL value. This allows such changesets to be
manipulated by legacy implementations of sqlite3changeset_invert(),
concat() and similar.

The sqlite3changeset_apply() function automatically converts the 
zero-length blob back to a NULL value when updating the sqlite_stat1
table. However, if the application calls sqlite3changeset_new(),
sqlite3changeset_old() or sqlite3changeset_conflict on a changeset 
iterator directly (including on a changeset iterator passed to a
conflict-handler callback) then the X'' value is returned. The application
must translate X'' to NULL itself if required.

Legacy (older than 3.22.0) versions of the sessions module cannot capture
changes made to the sqlite_stat1 table. Legacy versions of the
sqlite3changeset_apply() function silently ignore any modifications to the
sqlite_stat1 table that are part of a changeset or patchset.

Generate A Changeset From A Session Objectint sqlite3session_changeset(
  sqlite3_session *pSession,      /* Session object */
  int *pnChangeset,               /* OUT: Size of buffer at *ppChangeset */
  void **ppChangeset              /* OUT: Buffer containing changeset */
);

Obtain a changeset containing changes to the tables attached to the 
session object passed as the first argument. If successful, 
set *ppChangeset to point to a buffer containing the changeset 
and *pnChangeset to the size of the changeset in bytes before returning
SQLITE_OK. If an error occurs, set both *ppChangeset and *pnChangeset to
zero and return an SQLite error code.

A changeset consists of zero or more INSERT, UPDATE and/or DELETE changes,
each representing a change to a single row of an attached table. An INSERT
change contains the values of each field of a new database row. A DELETE
contains the original values of each field of a deleted database row. An
UPDATE change contains the original values of each field of an updated
database row along with the updated values for each updated non-primary-key
column. It is not possible for an UPDATE change to represent a change that
modifies the values of primary key columns. If such a change is made, it
is represented in a changeset as a DELETE followed by an INSERT.

Changes are not recorded for rows that have NULL values stored in one or 
more of their PRIMARY KEY columns. If such a row is inserted or deleted,
no corresponding change is present in the changesets returned by this
function. If an existing row with one or more NULL values stored in
PRIMARY KEY columns is updated so that all PRIMARY KEY columns are non-NULL,
only an INSERT is appears in the changeset. Similarly, if an existing row
with non-NULL PRIMARY KEY values is updated so that one or more of its
PRIMARY KEY columns are set to NULL, the resulting changeset contains a
DELETE change only.

The contents of a changeset may be traversed using an iterator created
using the sqlite3changeset_start() API. A changeset may be applied to
a database with a compatible schema using the sqlite3changeset_apply()
API.

Within a changeset generated by this function, all changes related to a
single table are grouped together. In other words, when iterating through
a changeset or when applying a changeset to a database, all changes related
to a single table are processed before moving on to the next table. Tables
are sorted in the same order in which they were attached (or auto-attached)
to the sqlite3_session object. The order in which the changes related to
a single table are stored is undefined.

Following a successful call to this function, it is the responsibility of
the caller to eventually free the buffer that *ppChangeset points to using
sqlite3_free().

Changeset Generation

Once a table has been attached to a session object, the session object
records the primary key values of all new rows inserted into the table.
It also records the original primary key and other column values of any
deleted or updated rows. For each unique primary key value, data is only
recorded once - the first time a row with said primary key is inserted,
updated or deleted in the lifetime of the session.

There is one exception to the previous paragraph: when a row is inserted,
updated or deleted, if one or more of its primary key columns contain a
NULL value, no record of the change is made.

The session object therefore accumulates two types of records - those
that consist of primary key values only (created when the user inserts
a new record) and those that consist of the primary key values and the
original values of other table columns (created when the users deletes
or updates a record).

When this function is called, the requested changeset is created using
both the accumulated records and the current contents of the database
file. Specifically:


   For each record generated by an insert, the database is queried
       for a row with a matching primary key. If one is found, an INSERT
       change is added to the changeset. If no such row is found, no change 
       is added to the changeset.

   For each record generated by an update or delete, the database is 
       queried for a row with a matching primary key. If such a row is
       found and one or more of the non-primary key fields have been
       modified from their original values, an UPDATE change is added to 
       the changeset. Or, if no such row is found in the table, a DELETE 
       change is added to the changeset. If there is a row with a matching
       primary key in the database, but all fields contain their original
       values, no change is added to the changeset.


This means, amongst other things, that if a row is inserted and then later
deleted while a session object is active, neither the insert nor the delete
will be present in the changeset. Or if a row is deleted and then later a 
row with the same primary key values inserted while a session object is
active, the resulting changeset will contain an UPDATE change instead of
a DELETE and an INSERT.

When a session object is disabled (see the sqlite3session_enable() API),
it does not accumulate records when rows are inserted, updated or deleted.
This may appear to have some counter-intuitive effects if a single row
is written to more than once during a session. For example, if a row
is inserted while a session object is enabled, then later deleted while 
the same session object is disabled, no INSERT record will appear in the
changeset, even though the delete took place while the session was disabled.
Or, if one field of a row is updated while a session is disabled, and 
another field of the same row is updated while the session is enabled, the
resulting changeset will contain an UPDATE change that updates both fields.

Return An Upper-limit For The Size Of The Changesetsqlite3_int64 sqlite3session_changeset_size(sqlite3_session *pSession);

By default, this function always returns 0. For it to return
a useful result, the sqlite3_session object must have been configured
to enable this API using sqlite3session_object_config() with the
SQLITE_SESSION_OBJCONFIG_SIZE verb.

When enabled, this function returns an upper limit, in bytes, for the size 
of the changeset that might be produced if sqlite3session_changeset() were
called. The final changeset size might be equal to or smaller than the
size in bytes returned by this function.

Configure global parametersint sqlite3session_config(int op, void *pArg);

The sqlite3session_config() interface is used to make global configuration
changes to the sessions module in order to tune it to the specific needs 
of the application.

The sqlite3session_config() interface is not threadsafe. If it is invoked
while any other thread is inside any other sessions method then the
results are undefined. Furthermore, if it is invoked after any sessions
related objects have been created, the results are also undefined. 

The first argument to the sqlite3session_config() function must be one
of the SQLITE_SESSION_CONFIG_XXX constants defined below. The 
interpretation of the (void*) value passed as the second parameter and
the effect of calling this function depends on the value of the first
parameter.


SQLITE_SESSION_CONFIG_STRMSIZE
   By default, the sessions module streaming interfaces attempt to input
   and output data in approximately 1 KiB chunks. This operand may be used
   to set and query the value of this configuration setting. The pointer
   passed as the second argument must point to a value of type (int).
   If this value is greater than 0, it is used as the new streaming data
   chunk size for both input and output. Before returning, the (int) value
   pointed to by pArg is set to the final value of the streaming interface
   chunk size.


This function returns SQLITE_OK if successful, or an SQLite error code
otherwise.

Create A New Session Objectint sqlite3session_create(
  sqlite3 *db,                    /* Database handle */
  const char *zDb,                /* Name of db (e.g. "main") */
  sqlite3_session **ppSession     /* OUT: New session object */
);

Create a new session object attached to database handle db. If successful,
a pointer to the new object is written to *ppSession and SQLITE_OK is
returned. If an error occurs, *ppSession is set to NULL and an SQLite
error code (e.g. SQLITE_NOMEM) is returned.

It is possible to create multiple session objects attached to a single
database handle.

Session objects created using this function should be deleted using the
sqlite3session_delete() function before the database handle that they
are attached to is itself closed. If the database handle is closed before
the session object is deleted, then the results of calling any session
module function, including sqlite3session_delete() on the session object
are undefined.

Because the session module uses the sqlite3_preupdate_hook() API, it
is not possible for an application to register a pre-update hook on a
database handle that has one or more session objects attached. Nor is
it possible to create a session object attached to a database handle for
which a pre-update hook is already defined. The results of attempting 
either of these things are undefined.

The session object will be used to create changesets for tables in
database zDb, where zDb is either "main", or "temp", or the name of an
attached database. It is not an error if database zDb is not attached
to the database when the session object is created.

Delete A Session Objectvoid sqlite3session_delete(sqlite3_session *pSession);

Delete a session object previously allocated using 
sqlite3session_create(). Once a session object has been deleted, the
results of attempting to use pSession with any other session module
function are undefined.

Session objects must be deleted before the database handle to which they
are attached is closed. Refer to the documentation for 
sqlite3session_create() for details.

Load The Difference Between Tables Into A Sessionint sqlite3session_diff(
  sqlite3_session *pSession,
  const char *zFromDb,
  const char *zTbl,
  char **pzErrMsg
);

If it is not already attached to the session object passed as the first
argument, this function attaches table zTbl in the same manner as the
sqlite3session_attach() function. If zTbl does not exist, or if it
does not have a primary key, this function is a no-op (but does not return
an error).

Argument zFromDb must be the name of a database ("main", "temp" etc.)
attached to the same database handle as the session object that contains 
a table compatible with the table attached to the session by this function.
A table is considered compatible if it:


   Has the same name,
   Has the same set of columns declared in the same order, and
   Has the same PRIMARY KEY definition.


If the tables are not compatible, SQLITE_SCHEMA is returned. If the tables
are compatible but do not have any PRIMARY KEY columns, it is not an error
but no changes are added to the session object. As with other session
APIs, tables without PRIMARY KEYs are simply ignored.

This function adds a set of changes to the session object that could be
used to update the table in database zFrom (call this the "from-table") 
so that its content is the same as the table attached to the session 
object (call this the "to-table"). Specifically:


   For each row (primary key) that exists in the to-table but not in 
    the from-table, an INSERT record is added to the session object.

   For each row (primary key) that exists in the to-table but not in 
    the from-table, a DELETE record is added to the session object.

   For each row (primary key) that exists in both tables, but features 
    different non-PK values in each, an UPDATE record is added to the
    session.  


To clarify, if this function is called and then a changeset constructed
using sqlite3session_changeset(), then after applying that changeset to 
database zFrom the contents of the two compatible tables would be 
identical.

It an error if database zFrom does not exist or does not contain the
required compatible table.

If the operation is successful, SQLITE_OK is returned. Otherwise, an SQLite
error code. In this case, if argument pzErrMsg is not NULL, *pzErrMsg
may be set to point to a buffer containing an English language error 
message. It is the responsibility of the caller to free this buffer using
sqlite3_free().

Enable Or Disable A Session Objectint sqlite3session_enable(sqlite3_session *pSession, int bEnable);

Enable or disable the recording of changes by a session object. When
enabled, a session object records changes made to the database. When
disabled - it does not. A newly created session object is enabled.
Refer to the documentation for sqlite3session_changeset() for further
details regarding how enabling and disabling a session object affects
the eventual changesets.

Passing zero to this function disables the session. Passing a value
greater than zero enables it. Passing a value less than zero is a 
no-op, and may be used to query the current state of the session.

The return value indicates the final state of the session object: 0 if 
the session is disabled, or 1 if it is enabled.

Set Or Clear the Indirect Change Flagint sqlite3session_indirect(sqlite3_session *pSession, int bIndirect);

Each change recorded by a session object is marked as either direct or
indirect. A change is marked as indirect if either:


   The session object "indirect" flag is set when the change is
       made, or
   The change is made by an SQL trigger or foreign key action 
       instead of directly as a result of a users SQL statement.


If a single row is affected by more than one operation within a session,
then the change is considered indirect if all operations meet the criteria
for an indirect change above, or direct otherwise.

This function is used to set, clear or query the session object indirect
flag.  If the second argument passed to this function is zero, then the
indirect flag is cleared. If it is greater than zero, the indirect flag
is set. Passing a value less than zero does not modify the current value
of the indirect flag, and may be used to query the current state of the 
indirect flag for the specified session object.

The return value indicates the final state of the indirect flag: 0 if 
it is clear, or 1 if it is set.

Test if a changeset has recorded any changes.int sqlite3session_isempty(sqlite3_session *pSession);

Return non-zero if no changes to attached tables have been recorded by 
the session object passed as the first argument. Otherwise, if one or 
more changes have been recorded, return zero.

Even if this function returns zero, it is possible that calling
sqlite3session_changeset() on the session handle may still return a
changeset that contains no changes. This can happen when a row in 
an attached table is modified and then later on the original values 
are restored. However, if this function returns non-zero, then it is
guaranteed that a call to sqlite3session_changeset() will return a 
changeset containing zero changes.

Query for the amount of heap memory used by a session object.sqlite3_int64 sqlite3session_memory_used(sqlite3_session *pSession);

This API returns the total amount of heap memory in bytes currently 
used by the session object passed as the only argument.

Generate A Patchset From A Session Objectint sqlite3session_patchset(
  sqlite3_session *pSession,      /* Session object */
  int *pnPatchset,                /* OUT: Size of buffer at *ppPatchset */
  void **ppPatchset               /* OUT: Buffer containing patchset */
);

The differences between a patchset and a changeset are that:


   DELETE records consist of the primary key fields only. The 
       original values of other fields are omitted.
   The original values of any modified fields are omitted from 
       UPDATE records.


A patchset blob may be used with up to date versions of all 
sqlite3changeset_xxx API functions except for sqlite3changeset_invert(), 
which returns SQLITE_CORRUPT if it is passed a patchset. Similarly,
attempting to use a patchset blob with old versions of the
sqlite3changeset_xxx APIs also provokes an SQLITE_CORRUPT error. 

Because the non-primary key "old.*" fields are omitted, no 
SQLITE_CHANGESET_DATA conflicts can be detected or reported if a patchset
is passed to the sqlite3changeset_apply() API. Other conflict types work
in the same way as for changesets.

Changes within a patchset are ordered in the same way as for changesets
generated by the sqlite3session_changeset() function (i.e. all changes for
a single table are grouped together, tables appear in the order in which
they were attached to the session object).

Set a table filter on a Session Object.void sqlite3session_table_filter(
  sqlite3_session *pSession,      /* Session object */
  int(*xFilter)(
    void *pCtx,                   /* Copy of third arg to _filter_table() */
    const char *zTab              /* Table name */
  ),
  void *pCtx                      /* First argument passed to xFilter */
);

The second argument (xFilter) is the "filter callback". For changes to rows 
in tables that are not attached to the Session object, the filter is called
to determine whether changes to the table's rows should be tracked or not. 
If xFilter returns 0, changes are not tracked. Note that once a table is 
attached, xFilter will not be called again.

Flags for sqlite3changeset_apply_v2#define SQLITE_CHANGESETAPPLY_NOSAVEPOINT   0x0001
#define SQLITE_CHANGESETAPPLY_INVERT        0x0002

The following flags may passed via the 9th parameter to
sqlite3changeset_apply_v2 and sqlite3changeset_apply_v2_strm:


SQLITE_CHANGESETAPPLY_NOSAVEPOINT 
  Usually, the sessions module encloses all operations performed by
  a single call to apply_v2() or apply_v2_strm() in a SAVEPOINT. The
  SAVEPOINT is committed if the changeset or patchset is successfully
  applied, or rolled back if an error occurs. Specifying this flag
  causes the sessions module to omit this savepoint. In this case, if the
  caller has an open transaction or savepoint when apply_v2() is called, 
  it may revert the partially applied changeset by rolling it back.

SQLITE_CHANGESETAPPLY_INVERT 
  Invert the changeset before applying it. This is equivalent to inverting
  a changeset using sqlite3changeset_invert() before applying it. It is
  an error to specify this flag with a patchset.

Constants Returned By The Conflict Handler#define SQLITE_CHANGESET_OMIT       0
#define SQLITE_CHANGESET_REPLACE    1
#define SQLITE_CHANGESET_ABORT      2

A conflict handler callback must return one of the following three values.


SQLITE_CHANGESET_OMIT
  If a conflict handler returns this value no special action is taken. The
  change that caused the conflict is not applied. The session module 
  continues to the next change in the changeset.

SQLITE_CHANGESET_REPLACE
  This value may only be returned if the second argument to the conflict
  handler was SQLITE_CHANGESET_DATA or SQLITE_CHANGESET_CONFLICT. If this
  is not the case, any changes applied so far are rolled back and the 
  call to sqlite3changeset_apply() returns SQLITE_MISUSE.

  If CHANGESET_REPLACE is returned by an SQLITE_CHANGESET_DATA conflict
  handler, then the conflicting row is either updated or deleted, depending
  on the type of change.

  If CHANGESET_REPLACE is returned by an SQLITE_CHANGESET_CONFLICT conflict
  handler, then the conflicting row is removed from the database and a
  second attempt to apply the change is made. If this second attempt fails,
  the original row is restored to the database before continuing.

SQLITE_CHANGESET_ABORT
  If this value is returned, any changes applied so far are rolled back 
  and the call to sqlite3changeset_apply() returns SQLITE_ABORT.


Constants Passed To The Conflict Handler#define SQLITE_CHANGESET_DATA        1
#define SQLITE_CHANGESET_NOTFOUND    2
#define SQLITE_CHANGESET_CONFLICT    3
#define SQLITE_CHANGESET_CONSTRAINT  4
#define SQLITE_CHANGESET_FOREIGN_KEY 5

Values that may be passed as the second argument to a conflict-handler.


SQLITE_CHANGESET_DATA
  The conflict handler is invoked with CHANGESET_DATA as the second argument
  when processing a DELETE or UPDATE change if a row with the required
  PRIMARY KEY fields is present in the database, but one or more other 
  (non primary-key) fields modified by the update do not contain the 
  expected "before" values.

  The conflicting row, in this case, is the database row with the matching
  primary key.

SQLITE_CHANGESET_NOTFOUND
  The conflict handler is invoked with CHANGESET_NOTFOUND as the second
  argument when processing a DELETE or UPDATE change if a row with the
  required PRIMARY KEY fields is not present in the database.

  There is no conflicting row in this case. The results of invoking the
  sqlite3changeset_conflict() API are undefined.

SQLITE_CHANGESET_CONFLICT
  CHANGESET_CONFLICT is passed as the second argument to the conflict
  handler while processing an INSERT change if the operation would result 
  in duplicate primary key values.

  The conflicting row in this case is the database row with the matching
  primary key.

SQLITE_CHANGESET_FOREIGN_KEY
  If foreign key handling is enabled, and applying a changeset leaves the
  database in a state containing foreign key violations, the conflict 
  handler is invoked with CHANGESET_FOREIGN_KEY as the second argument
  exactly once before the changeset is committed. If the conflict handler
  returns CHANGESET_OMIT, the changes, including those that caused the
  foreign key constraint violation, are committed. Or, if it returns
  CHANGESET_ABORT, the changeset is rolled back.

  No current or conflicting row information is provided. The only function
  it is possible to call on the supplied sqlite3_changeset_iter handle
  is sqlite3changeset_fk_conflicts().

SQLITE_CHANGESET_CONSTRAINT
  If any other constraint violation occurs while applying a change (i.e. 
  a UNIQUE, CHECK or NOT NULL constraint), the conflict handler is 
  invoked with CHANGESET_CONSTRAINT as the second argument.

  There is no conflicting row in this case. The results of invoking the
  sqlite3changeset_conflict() API are undefined.



Streaming Versions of API functions.int sqlite3changeset_apply_strm(
  sqlite3 *db,                    /* Apply change to "main" db of this handle */
  int (*xInput)(void *pIn, void *pData, int *pnData), /* Input function */
  void *pIn,                                          /* First arg for xInput */
  int(*xFilter)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    const char *zTab              /* Table name */
  ),
  int(*xConflict)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    int eConflict,                /* DATA, MISSING, CONFLICT, CONSTRAINT */
    sqlite3_changeset_iter *p     /* Handle describing change and conflict */
  ),
  void *pCtx                      /* First argument passed to xConflict */
);
int sqlite3changeset_apply_v2_strm(
  sqlite3 *db,                    /* Apply change to "main" db of this handle */
  int (*xInput)(void *pIn, void *pData, int *pnData), /* Input function */
  void *pIn,                                          /* First arg for xInput */
  int(*xFilter)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    const char *zTab              /* Table name */
  ),
  int(*xConflict)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    int eConflict,                /* DATA, MISSING, CONFLICT, CONSTRAINT */
    sqlite3_changeset_iter *p     /* Handle describing change and conflict */
  ),
  void *pCtx,                     /* First argument passed to xConflict */
  void **ppRebase, int *pnRebase,
  int flags
);
int sqlite3changeset_concat_strm(
  int (*xInputA)(void *pIn, void *pData, int *pnData),
  void *pInA,
  int (*xInputB)(void *pIn, void *pData, int *pnData),
  void *pInB,
  int (*xOutput)(void *pOut, const void *pData, int nData),
  void *pOut
);
int sqlite3changeset_invert_strm(
  int (*xInput)(void *pIn, void *pData, int *pnData),
  void *pIn,
  int (*xOutput)(void *pOut, const void *pData, int nData),
  void *pOut
);
int sqlite3changeset_start_strm(
  sqlite3_changeset_iter **pp,
  int (*xInput)(void *pIn, void *pData, int *pnData),
  void *pIn
);
int sqlite3changeset_start_v2_strm(
  sqlite3_changeset_iter **pp,
  int (*xInput)(void *pIn, void *pData, int *pnData),
  void *pIn,
  int flags
);
int sqlite3session_changeset_strm(
  sqlite3_session *pSession,
  int (*xOutput)(void *pOut, const void *pData, int nData),
  void *pOut
);
int sqlite3session_patchset_strm(
  sqlite3_session *pSession,
  int (*xOutput)(void *pOut, const void *pData, int nData),
  void *pOut
);
int sqlite3changegroup_add_strm(sqlite3_changegroup*, 
    int (*xInput)(void *pIn, void *pData, int *pnData),
    void *pIn
);
int sqlite3changegroup_output_strm(sqlite3_changegroup*,
    int (*xOutput)(void *pOut, const void *pData, int nData), 
    void *pOut
);
int sqlite3rebaser_rebase_strm(
  sqlite3_rebaser *pRebaser,
  int (*xInput)(void *pIn, void *pData, int *pnData),
  void *pIn,
  int (*xOutput)(void *pOut, const void *pData, int nData),
  void *pOut
);

The six streaming API xxx_strm() functions serve similar purposes to the 
corresponding non-streaming API functions:


  Streaming functionNon-streaming equivalent
  sqlite3changeset_apply_strmsqlite3changeset_apply 
  sqlite3changeset_apply_strm_v2sqlite3changeset_apply_v2 
  sqlite3changeset_concat_strmsqlite3changeset_concat 
  sqlite3changeset_invert_strmsqlite3changeset_invert 
  sqlite3changeset_start_strmsqlite3changeset_start 
  sqlite3session_changeset_strmsqlite3session_changeset 
  sqlite3session_patchset_strmsqlite3session_patchset 


Non-streaming functions that accept changesets (or patchsets) as input
require that the entire changeset be stored in a single buffer in memory. 
Similarly, those that return a changeset or patchset do so by returning 
a pointer to a single large buffer allocated using sqlite3_malloc(). 
Normally this is convenient. However, if an application running in a 
low-memory environment is required to handle very large changesets, the
large contiguous memory allocations required can become onerous.

In order to avoid this problem, instead of a single large buffer, input
is passed to a streaming API functions by way of a callback function that
the sessions module invokes to incrementally request input data as it is
required. In all cases, a pair of API function parameters such as

 
 &nbsp;     int nChangeset,
 &nbsp;     void *pChangeset,
 

Is replaced by:

 
 &nbsp;     int (*xInput)(void *pIn, void *pData, int *pnData),
 &nbsp;     void *pIn,
 

Each time the xInput callback is invoked by the sessions module, the first
argument passed is a copy of the supplied pIn context pointer. The second 
argument, pData, points to a buffer (*pnData) bytes in size. Assuming no 
error occurs the xInput method should copy up to (*pnData) bytes of data 
into the buffer and set (*pnData) to the actual number of bytes copied 
before returning SQLITE_OK. If the input is completely exhausted, (*pnData) 
should be set to zero to indicate this. Or, if an error occurs, an SQLite 
error code should be returned. In all cases, if an xInput callback returns
an error, all processing is abandoned and the streaming API function
returns a copy of the error code to the caller.

In the case of sqlite3changeset_start_strm(), the xInput callback may be
invoked by the sessions module at any point during the lifetime of the
iterator. If such an xInput callback returns an error, the iterator enters
an error state, whereby all subsequent calls to iterator functions 
immediately fail with the same error code as returned by xInput.

Similarly, streaming API functions that return changesets (or patchsets)
return them in chunks by way of a callback function instead of via a
pointer to a single large buffer. In this case, a pair of parameters such
as:

 
 &nbsp;     int *pnChangeset,
 &nbsp;     void **ppChangeset,
 

Is replaced by:

 
 &nbsp;     int (*xOutput)(void *pOut, const void *pData, int nData),
 &nbsp;     void *pOut
 

The xOutput callback is invoked zero or more times to return data to
the application. The first parameter passed to each call is a copy of the
pOut pointer supplied by the application. The second parameter, pData,
points to a buffer nData bytes in size containing the chunk of output
data being returned. If the xOutput callback successfully processes the
supplied data, it should return SQLITE_OK to indicate success. Otherwise,
it should return some other SQLite error code. In this case processing
is immediately abandoned and the streaming API function returns a copy
of the xOutput error code to the application.

The sessions module never invokes an xOutput callback with the third 
parameter set to a value less than or equal to zero. Other than this,
no guarantees are made as to the size of the chunks of data returned.

Apply A Changeset To A Databaseint sqlite3changeset_apply(
  sqlite3 *db,                    /* Apply change to "main" db of this handle */
  int nChangeset,                 /* Size of changeset in bytes */
  void *pChangeset,               /* Changeset blob */
  int(*xFilter)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    const char *zTab              /* Table name */
  ),
  int(*xConflict)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    int eConflict,                /* DATA, MISSING, CONFLICT, CONSTRAINT */
    sqlite3_changeset_iter *p     /* Handle describing change and conflict */
  ),
  void *pCtx                      /* First argument passed to xConflict */
);
int sqlite3changeset_apply_v2(
  sqlite3 *db,                    /* Apply change to "main" db of this handle */
  int nChangeset,                 /* Size of changeset in bytes */
  void *pChangeset,               /* Changeset blob */
  int(*xFilter)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    const char *zTab              /* Table name */
  ),
  int(*xConflict)(
    void *pCtx,                   /* Copy of sixth arg to _apply() */
    int eConflict,                /* DATA, MISSING, CONFLICT, CONSTRAINT */
    sqlite3_changeset_iter *p     /* Handle describing change and conflict */
  ),
  void *pCtx,                     /* First argument passed to xConflict */
  void **ppRebase, int *pnRebase, /* OUT: Rebase data */
  int flags                       /* SESSION_CHANGESETAPPLY_* flags */
);

Apply a changeset or patchset to a database. These functions attempt to
update the "main" database attached to handle db with the changes found in
the changeset passed via the second and third arguments. 

The fourth argument (xFilter) passed to these functions is the "filter
callback". If it is not NULL, then for each table affected by at least one
change in the changeset, the filter callback is invoked with
the table name as the second argument, and a copy of the context pointer
passed as the sixth argument as the first. If the "filter callback"
returns zero, then no attempt is made to apply any changes to the table.
Otherwise, if the return value is non-zero or the xFilter argument to
is NULL, all changes related to the table are attempted.

For each table that is not excluded by the filter callback, this function 
tests that the target database contains a compatible table. A table is 
considered compatible if all of the following are true:


   The table has the same name as the name recorded in the 
       changeset, and
   The table has at least as many columns as recorded in the 
       changeset, and
   The table has primary key columns in the same position as 
       recorded in the changeset.


If there is no compatible table, it is not an error, but none of the
changes associated with the table are applied. A warning message is issued
via the sqlite3_log() mechanism with the error code SQLITE_SCHEMA. At most
one such warning is issued for each table in the changeset.

For each change for which there is a compatible table, an attempt is made 
to modify the table contents according to the UPDATE, INSERT or DELETE 
change. If a change cannot be applied cleanly, the conflict handler 
function passed as the fifth argument to sqlite3changeset_apply() may be 
invoked. A description of exactly when the conflict handler is invoked for 
each type of change is below.

Unlike the xFilter argument, xConflict may not be passed NULL. The results
of passing anything other than a valid function pointer as the xConflict
argument are undefined.

Each time the conflict handler function is invoked, it must return one
of SQLITE_CHANGESET_OMIT, SQLITE_CHANGESET_ABORT or 
SQLITE_CHANGESET_REPLACE. SQLITE_CHANGESET_REPLACE may only be returned
if the second argument passed to the conflict handler is either
SQLITE_CHANGESET_DATA or SQLITE_CHANGESET_CONFLICT. If the conflict-handler
returns an illegal value, any changes already made are rolled back and
the call to sqlite3changeset_apply() returns SQLITE_MISUSE. Different 
actions are taken by sqlite3changeset_apply() depending on the value
returned by each invocation of the conflict-handler function. Refer to
the documentation for the three 
available return values for details.


DELETE Changes
  For each DELETE change, the function checks if the target database 
  contains a row with the same primary key value (or values) as the 
  original row values stored in the changeset. If it does, and the values 
  stored in all non-primary key columns also match the values stored in 
  the changeset the row is deleted from the target database.

  If a row with matching primary key values is found, but one or more of
  the non-primary key fields contains a value different from the original
  row value stored in the changeset, the conflict-handler function is
  invoked with SQLITE_CHANGESET_DATA as the second argument. If the
  database table has more columns than are recorded in the changeset,
  only the values of those non-primary key fields are compared against
  the current database contents - any trailing database table columns
  are ignored.

  If no row with matching primary key values is found in the database,
  the conflict-handler function is invoked with SQLITE_CHANGESET_NOTFOUND
  passed as the second argument.

  If the DELETE operation is attempted, but SQLite returns SQLITE_CONSTRAINT
  (which can only happen if a foreign key constraint is violated), the
  conflict-handler function is invoked with SQLITE_CHANGESET_CONSTRAINT
  passed as the second argument. This includes the case where the DELETE
  operation is attempted because an earlier call to the conflict handler
  function returned SQLITE_CHANGESET_REPLACE.

INSERT Changes
  For each INSERT change, an attempt is made to insert the new row into
  the database. If the changeset row contains fewer fields than the
  database table, the trailing fields are populated with their default
  values.

  If the attempt to insert the row fails because the database already 
  contains a row with the same primary key values, the conflict handler
  function is invoked with the second argument set to 
  SQLITE_CHANGESET_CONFLICT.

  If the attempt to insert the row fails because of some other constraint
  violation (e.g. NOT NULL or UNIQUE), the conflict handler function is 
  invoked with the second argument set to SQLITE_CHANGESET_CONSTRAINT.
  This includes the case where the INSERT operation is re-attempted because 
  an earlier call to the conflict handler function returned 
  SQLITE_CHANGESET_REPLACE.

UPDATE Changes
  For each UPDATE change, the function checks if the target database 
  contains a row with the same primary key value (or values) as the 
  original row values stored in the changeset. If it does, and the values 
  stored in all modified non-primary key columns also match the values
  stored in the changeset the row is updated within the target database.

  If a row with matching primary key values is found, but one or more of
  the modified non-primary key fields contains a value different from an
  original row value stored in the changeset, the conflict-handler function
  is invoked with SQLITE_CHANGESET_DATA as the second argument. Since
  UPDATE changes only contain values for non-primary key fields that are
  to be modified, only those fields need to match the original values to
  avoid the SQLITE_CHANGESET_DATA conflict-handler callback.

  If no row with matching primary key values is found in the database,
  the conflict-handler function is invoked with SQLITE_CHANGESET_NOTFOUND
  passed as the second argument.

  If the UPDATE operation is attempted, but SQLite returns 
  SQLITE_CONSTRAINT, the conflict-handler function is invoked with 
  SQLITE_CHANGESET_CONSTRAINT passed as the second argument.
  This includes the case where the UPDATE operation is attempted after 
  an earlier call to the conflict handler function returned
  SQLITE_CHANGESET_REPLACE.  


It is safe to execute SQL statements, including those that write to the
table that the callback related to, from within the xConflict callback.
This can be used to further customize the application's conflict
resolution strategy.

All changes made by these functions are enclosed in a savepoint transaction.
If any other error (aside from a constraint failure when attempting to
write to the target database) occurs, then the savepoint transaction is
rolled back, restoring the target database to its original state, and an 
SQLite error code returned.

If the output parameters (ppRebase) and (pnRebase) are non-NULL and
the input is a changeset (not a patchset), then sqlite3changeset_apply_v2()
may set (*ppRebase) to point to a "rebase" that may be used with the 
sqlite3_rebaser APIs buffer before returning. In this case (*pnRebase)
is set to the size of the buffer in bytes. It is the responsibility of the
caller to eventually free any such buffer using sqlite3_free(). The buffer
is only allocated and populated if one or more conflicts were encountered
while applying the patchset. See comments surrounding the sqlite3_rebaser
APIs for further details.

The behavior of sqlite3changeset_apply_v2() and its streaming equivalent
may be modified by passing a combination of
supported flags as the 9th parameter.

Note that the sqlite3changeset_apply_v2() API is still experimental
and therefore subject to change.

Create An Iterator To Traverse A Changeset int sqlite3changeset_start(
  sqlite3_changeset_iter **pp,    /* OUT: New changeset iterator handle */
  int nChangeset,                 /* Size of changeset blob in bytes */
  void *pChangeset                /* Pointer to blob containing changeset */
);
int sqlite3changeset_start_v2(
  sqlite3_changeset_iter **pp,    /* OUT: New changeset iterator handle */
  int nChangeset,                 /* Size of changeset blob in bytes */
  void *pChangeset,               /* Pointer to blob containing changeset */
  int flags                       /* SESSION_CHANGESETSTART_* flags */
);

Create an iterator used to iterate through the contents of a changeset.
If successful, *pp is set to point to the iterator handle and SQLITE_OK
is returned. Otherwise, if an error occurs, *pp is set to zero and an
SQLite error code is returned.

The following functions can be used to advance and query a changeset 
iterator created by this function:


   sqlite3changeset_next()
   sqlite3changeset_op()
   sqlite3changeset_new()
   sqlite3changeset_old()


It is the responsibility of the caller to eventually destroy the iterator
by passing it to sqlite3changeset_finalize(). The buffer containing the
changeset (pChangeset) must remain valid until after the iterator is
destroyed.

Assuming the changeset blob was created by one of the
sqlite3session_changeset(), sqlite3changeset_concat() or
sqlite3changeset_invert() functions, all changes within the changeset 
that apply to a single table are grouped together. This means that when 
an application iterates through a changeset using an iterator created by 
this function, all changes that relate to a single table are visited 
consecutively. There is no chance that the iterator will visit a change 
the applies to table X, then one for table Y, and then later on visit 
another change for table X.

The behavior of sqlite3changeset_start_v2() and its streaming equivalent
may be modified by passing a combination of
supported flags as the 4th parameter.

Note that the sqlite3changeset_start_v2() API is still experimental
and therefore subject to change.


c2: SQLite Developers
c5: crew.html
c4:
D. Richard Hipp
 began the SQLite project on 2000-05-29
and continues to serve as the project architect.  Richard was born,
lives, and works in 
Charlotte, North Carolina.  He holds degrees from
Georgia Tech (MSEE, 1984) and
Duke University (PhD, 1992) and is
the founder of the consulting firm
Hwaci.








Dan Kennedy is an Australian currently based in South-East Asia.
He holds a degree in Computer System Engineering from the University of
Queensland and has worked in a variety of fields, including industrial
automation, computer graphics and embedded software development.
Dan has been a key contributor to SQLite since 2002.







Joe Mistachkin
(pronounced "miss-tash-kin") is a software engineer and one
of the maintainers of Tcl/Tk.  He is also the author of the TclBridge
component and the Eagle scripting language.  He has been working in the
software industry since 1994.


This page last modified on  2018-05-31 17:37:17 UTC


c2: SQLite Archive Files
c5: sqlar.html
c4:
1. Introduction


An "SQLite Archive" is a file container similar to a 
ZIP archive or
Tarball but
based on an SQLite database.


An SQLite Archive is an ordinary SQLite database file that contains the
following table as part of its schema:

CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);



Each row of the SQLAR table holds the content of a single file.
The filename (the full pathname relative to the root of the archive)
is in the "name" field.
The "mode" field is an integer which is the unix-style access permissions
for the file.  "mtime" is the modification time of the file in seconds
since 1970.  "sz" is the original uncompressed size of the file.
The "data" field contains the file content.  The content is usually
compressed using Deflate, though not always.  If the
"sz" field is equal to the size of the "data" field, then the content
is stored uncompressed.



1.1. Database As Container Object


An SQLite Archive is one example of a more general
idea that an SQLite database can behave as a container object holding
lots of smaller data components.


With client/server databases like PostgreSQL or Oracle, users and
developers tend to think of the database as a service or a "node", not
as an object.  This is because the database content is spread out across
multiple files on the server, or possibly across multiple servers in a
service cluster.  One cannot point to a single file or even a single
directory and say "this is the database".


SQLite, in contrast, stores all content in a single file on disk.
That single file is something you can point to and say
"this is the database".  It behaves as an object.
An SQLite database file can be copied, renamed, sent as an
email attachment, passed as the argument a POST HTTP request,
or otherwise treated as other data object such as an image,
document, or media file.


Studies show that many applications already use
SQLite as a container object.  For example,
Kennedy
(no relation to the SQLite developer) reports that 14% of
Android applications never write to their SQLite databases.  It is
believed that these applications are downloading entire databases
from the cloud and then using the information locally as needed.  In other
words, the applications are using SQLite not so much as a database but as
a queryable wire-transfer format.

1.2. Applications Using SQLite Archives


The Fossil Distributed Version Control system
provides users with the option to download check-ins as either Tarballs,
ZIP Archives, or SQLite Archives.
2. Advantages Of SQLite Archives



An SQLite Archive is flexible.
ZIP Archives and Tarballs are limited to storing only files.  An
SQLite Archive stores files plus whatever other tabular
and/or relational data seems useful to the application.


An SQLite Archive is transactional.
Updates are atomic and durable, even if there are crashes
or power losses in the middle of the update.
Readers see a consistent and unchanging version of the content even
is some other process is simultaneously updating the archive.


An SQLite Archive can be updated incrementally.
Individual files can be added or removed or replaced without having
to rewrite the entire archive.


An SQLite Archive can be queried using a high-level query language (SQL).
Some examples:

 What is the total size of all files in the archive whose names
     end in ".h" or ".cpp"?
 What percentage of the files are compressed by less than 25%?
 How many executable files are in the archive?

Questions like these (and countless others) can be answered without
having to uncompress or extract any content.


Applications that already use SQLite for other purposes can easily
add support for SQLite Archives using a small extension
(https://sqlite.org/src/file/ext/misc/sqlar.c) to handle the compression
and decompression of content.  Even this tiny extension can be omitted
if the files in the archive are uncompressed.  In contrast, supporting
ZIP Archives and/or Tarballs requires either separate libraries or
lots of extra custom code, or sometimes both.


An SQLite Archive can work around firewall-imposed censorship.
For example, certain file types that are considered "dangerous" 
(examples: DLLs) will be
blocked by Gmail
and probably many other email services and firewalls, even if the
files are wrapped inside a ZIP Archive or Tarball.
But these firewalls usually do not (yet) know about SQLite Archives and
so content can be put inside an SQLite Archive to evade censorship.


3. Disadvantages Of SQLite Archives



The SQLite Archive is a relatively new format.  It was first described in
in 2014.  ZIP Archives and Tarballs, on the other hand, have been around
for decades and are well-entrenched as standard formats.  Most programmers
know what a ZIP Archive or Tarball is, but if you say "SQLite Archive" you
are more likely to get a reply of "What?"  Tooling to process ZIP Archives
and Tarballs is more likely to be installed on stock computers.


Since an SQLite database is a more general format (it is designed to do
much more than simply store a bunch of files) it is not as compact as either
the ZIP Archive or Tarball formats.  An SQLite Archive is usually about 1%
larger than the equivalent ZIP Archive.  Tarballs are compressed as a single
unit rather than compressing each file separately as is done by both
SQLite and ZIP Archives.  For these reason, Tarballs tend to be smaller
than either ZIP or SQLite Archives.

As an example, the following table show the relative sizes for an
SQLite Archive, a ZIP Archive, and a Tarball of the 1,743 files
in the SQLite 3.22.0 source tree:

  SQLite Archive10,754,048
  ZIP Archive (using Info-ZIP 3.0)10,662,365
  ZIP Archive (using zipfile)10,390,215
  Tarball&#x2007;9,781,109



An SQLite Archive supports only the Deflate compression
method.  Tarballs and ZIP Archive support a wider assortment of
compression methods.




4. Managing An SQLite Archive From The Command-Line


The recommended way of creating, updating, listing, and extracting
an SQLite Archive is to use the sqlite3.exe command-line shell 
for SQLite version 3.23.0 (2018-04-02) or later.  This CLI
supports the -A command-line option that allows easy management
of SQLite Archives.
The CLI for SQLite version 3.22.0 (2018-01-22) has the
.archive command for managing SQLite Archives, but that requires
interacting with the shell.


To list all of the files in an SQLite Archive named "example.sqlar"
using one of these commands:

sqlite3 example.sqlar -At
sqlite3 example.sqlar -Atv



To extract all files from an SQLite Archive named "example.sqlar":

sqlite3 example.sqlar -Ax



To create a new SQLite Archive named "alltxt.sqlar" containing all *.txt
files in the current directory:

sqlite3 alltxt.sqlar -Ac *.txt



To add or update files in an existing SQLite Archive:

sqlite3 example.sqlar -Au *.md



For usage hints and a summary of all options, simply give the CLI
the -A option with no additional arguments:

sqlite3 -A



All of these commands work the same way if the filename argument is
is a ZIP Archive instead of an SQLite database.

4.1. Other command-line tools


Just as there is the "zip" program to manage ZIP Archives, and the
"tar" program to manage Tarballs, the 
"sqlar" program exists to manage SQL Archives.
The "sqlar" program is able to create a new SQLite Archive, list the
content of an existing archive, add or remove files from the archive,
and/or extract files from the archive.
A separate "sqlarfs" program is able to mount the SQLite Archive as
a Fuse Filesystem.

5. Managing SQLite Archives From Application Code


Applications can easily read or write SQLite Archives by linking against
SQLite and including the 
ext/misc/sqlar.c extension
to handle the compression and decompression.  The sqlar.c extension
creates two new SQL functions.


sqlar_compress(X)

The sqlar_compress(X) function attempts to compress a copy of the
string or blob X using the Default algorithm and
returns the result as a blob.  If the input X is incompressible, then
a copy of X is returned.  This routine is used when inserting content
into an SQLite Archive.
sqlar_uncompress(Y,SZ)

The sqlar_uncompress(Y,SZ) function will undo the compression accomplished
by sqlar_compress(X).  The Y parameter is the compressed content (the output
from a prior call to sqlar_compress()) and SZ is the original uncompressed
size of the input X that generated Y.  If SZ is less than or equal to the
size of Y, that indicates that no compression occurred, and so
sqlar_uncompress(Y,SZ) returns a copy of Y.  Otherwise, sqlar_uncompress(Y,SZ)
runs the Inflate algorithm on Y to uncompress it and restore it to its
original form and returns the uncompressed content.
This routine is used when extracting content from an SQLite Archive.



Using the two routines above, it is simple for applications to insert
new records into or extract existing records from an SQLite Archive.
Insert a new into an SQLite Archive using code like this:

INSERT INTO sqlar(name,mode,mtime,sz,data)
 VALUES ($name,$mode,strftime('%s',$mtime),
         length($content),sqlar_compress($content));



Extract an entry from the SQLite Archive using code like this:

SELECT name, mode, datetime(mtime,'unixepoch'), sqlar_uncompress(data,sz)
  FROM sqlar
 WHERE ...;



The code above is for the general case.  For the special case of an
SQLite Archive that only stores uncompressed or uncompressible content
(this might come up, for example, in an SQLite Archive that stores only
JPEG, GIF, and/or PNG images) then the content can be inserted into
and extracted from the database without using the sqlar_compress()
and sqlar_uncompress() functions, and the sqlar.c extension is not
required.
This page last modified on  2020-12-02 18:42:06 UTC


c2: The Lemon LALR(1) Parser Generator
c5: lemon.html
c4:
1. Overview

The SQL language parser for SQLite is generated using a code-generator
program called "Lemon".  The Lemon program reads a grammar of the input
language and emits C-code to implement a parser for that language.


1.1. Lemon Source Files And Documentation

Lemon does not have its own source repository.  Rather, Lemon consists
of a few files in the SQLite source tree:



     lemon.html &rarr;
     The original detailed usage documentation and programmers reference
     for Lemon.

     lemon.c &rarr; The source code
     for the utility program that reads a grammar file and generates 
     corresponding parser C-code.

     lempar.c &rarr; A template
     for the generated parser C-code.  The "lemon" utility program reads this
     template and inserts additional code in order to generate a parser.


2. Advantages of Lemon

Lemon generates an LALR(1) parser.  Its operation is similar to the
more familiar tools Yacc and
Bison, but Lemon adds important
improvements, including:



     The grammar syntax is less error prone - using symbolic names for
     semantic values rather that the "$1"-style positional notation
     of Yacc.

     In Lemon, the tokenizer calls the parser.  Yacc operates the other
     way around, with the parser calling the tokenizer.  The Lemon
     approach is reentrant and threadsafe, whereas Yacc uses global 
     variables and is therefore neither.  Reentrancy is especially
     important for SQLite since some SQL statements make recursive calls
     to the parser.  For example, when parsing a CREATE TABLE statement,
     SQLite invokes the parser recursively to generate an INSERT statement
     to make a new entry in the sqlite_schema table.

     Lemon has the concept of a non-terminal destructor that can be
     used to reclaim memory or other resources following a syntax error
     or other aborted parse.


2.1. Use of Lemon Within SQLite

Lemon is used in two places in SQLite.

The primary use of Lemon is to create the SQL language parser.
A grammar file (parse.y) is
compiled by Lemon into parse.c and parse.h.  The parse.c file is
incorporated into the amalgamation without further modification.

Lemon is also used to generate the parser for the query pattern
expressions in the FTS5 extension.  In this case, the input grammar
file is fts5parse.y.

2.2. Lemon Customizations Especially For SQLite

One of the advantages of hosting code generator tools as part of
the project is that the tools can be optimized to serve specific needs of
the overall project.  Lemon has benefited from this effect. Over the years,
the Lemon parser generator has been extended and enhanced to provide
new capabilities and improved performance to SQLite.  A few of the
specific enhancements to Lemon that are specifically designed for use
by SQLite include:



Lemon has the concept of a "fallback" token.
The SQL language contains a large number of keywords and these keywords
have the potential to collide with identifier names.
Lemon has the ability to designate some keywords as being able to
"fallback" to an identifier.  If the keyword appears in the input token
stream in a context that would otherwise be a syntax error, the token
is automatically transformed into its fallback before the syntax error
is raised.  This feature allows the parser to be very forgiving of
reserved words used as identifiers, which is a problem that comes up
frequently in the SQL language.


In support of the 100% MC/DC testing goal for SQLite, 
the parser code generated by Lemon has no unreachable branches,
and contains extra (compile-time selected) instrumentation useful
for measuring test coverage.


Lemon supports conditional compilation of grammar file rules, so that
a different parser can be generated depending on compile-time options.


As a performance optimization, reduce actions in the Lemon input grammar
are allowed to contain comments of the form "/*A-overwrites-Z*/" to indicate
that the semantic value "A" on the right-hand side of the rule is allowed
to directly overwrite the semantic value "Z" on the left-hand side.
This simple optimization reduces the number of stack operations in the
push-down automaton used to parse the input grammar, and thus improve
performance of the parser.  It also makes the generated code a little smaller.


The parsing of SQL statements is a significant consumer of CPU cycles 
in any SQL database engine.  On-going efforts to optimize SQLite have caused
the developers to spend a lot of time tweaking Lemon to generate faster
parsers.  These efforts have benefited all users of the Lemon parser generator,
not just SQLite.  But if Lemon had been a separately maintained tool, it
would have been more difficult to make coordinated changes to both SQLite
and Lemon, and as a result not as much optimization would have been
accomplished.  Hence, the fact that the parser generator tool is included
in the source tree for SQLite has turned out to be a net benefit for both
the tool itself and for SQLite.

3. History Of Lemon

Lemon was originally written by D. Richard Hipp (also the creator of SQLite)
while he was in graduate school at Duke University between 1987 and 1992.
The original creation date of Lemon has been lost, but was probably sometime
around 1990.  Lemon generates an LALR(1) parser.  There was a companion 
LL(1) parser generator tool named "Lime", but the source code for Lime
has been lost.

The Lemon source code was originally written as separate source files,
and only later merged into a single "lemon.c" source file.

The author of Lemon and SQLite (Hipp) reports that his C programming
skills were greatly enhanced by studying John Ousterhout's original
source code to Tcl.  Hipp discovered and studied Tcl in 1993.  Lemon
was written before then, and SQLite afterwards.  There is a clear
difference in the coding styles of these two products, with SQLite seeming
to be cleaner, more readable, and easier to maintain.
This page last modified on  2021-08-21 20:50:48 UTC


c2: In-Memory Databases
c5: inmemorydb.html
c4:
In-Memory Databases

An SQLite database is normally stored in a single ordinary disk
file. However, in certain circumstances, the database might be stored in
memory.

The most common way to force an SQLite database to exist purely 
in memory is to open the database using the special filename
":memory:".  In other words, instead of passing the name of
a real disk file into one of the sqlite3_open(), sqlite3_open16(), or
sqlite3_open_v2() functions, pass in the string ":memory:".  For
example:


rc = sqlite3_open(":memory:", &amp;db);


When this is done, no disk file is opened.  
Instead, a new database is created
purely in memory.  The database ceases to exist as soon as the database
connection is closed.  Every :memory: database is distinct from every
other.  So, opening two database connections each with the filename
":memory:" will create two independent in-memory databases.

The special filename ":memory:" can be used anywhere that a database
filename is permitted.  For example, it can be used as the
filename in an ATTACH command:


ATTACH DATABASE ':memory:' AS aux1;


Note that in order for the special ":memory:" name to apply and to
create a pure in-memory database, there must be no additional text in the
filename.  Thus, a disk-based database can be created in a file by prepending
a pathname, like this:  "./:memory:".

The special ":memory:" filename also works when using URI filenames.
For example:


rc = sqlite3_open("file::memory:", &amp;db);


Or,


ATTACH DATABASE 'file::memory:' AS aux1;




In-memory Databases And Shared Cache

In-memory databases are allowed to use shared cache if they are
opened using a URI filename.  If the unadorned ":memory:" name is used
to specify the in-memory database, then that database always has a private
cache and is only visible to the database connection that originally
opened it.  However, the same in-memory database can be opened by two or
more database connections as follows:


rc = sqlite3_open("file::memory:?cache=shared", &amp;db);


Or,


ATTACH DATABASE 'file::memory:?cache=shared' AS aux1;


This allows separate database connections to share the same
in-memory database.  Of course, all database connections sharing the
in-memory database need to be in the same process.  The database is
automatically deleted and memory is reclaimed when the last connection
to the database closes.

If two or more distinct but shareable in-memory databases are needed
in a single process, then the mode=memory query parameter can
be used with a URI filename to create a named in-memory database:


rc = sqlite3_open("file:memdb1?mode=memory&amp;cache=shared", &amp;db);


Or,


ATTACH DATABASE 'file:memdb1?mode=memory&amp;cache=shared' AS aux1;


When an in-memory database is named in this way, it will only share its
cache with another connection that uses exactly the same name.





Temporary Databases

When the name of the database file handed to sqlite3_open() or to
ATTACH is an empty string, then a new temporary file is created to hold
the database.


rc = sqlite3_open("", &amp;db);



ATTACH DATABASE '' AS aux2;


A different temporary file is created each time so that, just as
with the special ":memory:" string, two database connections to temporary
databases each have their own private database.  Temporary databases are
automatically deleted when the connection that created them closes.

Even though a disk file is allocated for each temporary database, in
practice the temporary database usually resides in the in-memory pager cache
and hence there is very little difference between a pure in-memory database
created by ":memory:" and a temporary database created by an empty filename.
The sole difference is that a ":memory:" database must remain in memory
at all times whereas parts of a temporary database might be flushed to disk
if the database becomes large or if SQLite comes under memory pressure.

The previous paragraphs describe the behavior of temporary databases
under the default SQLite configuration.  An application can use the
temp_store pragma and the SQLITE_TEMP_STORE compile-time parameter to
force temporary databases to behave as pure in-memory databases, if desired.

This page last modified on  2022-01-20 21:38:08 UTC


c2: SQLite Pro Support
c5: prosupport.html
c4:
Professional Support &amp; Extension Products




Support
(details below)



1.Public Forum. 
  Help and advice from a world-wide community of experts.

Free
More Info
Recent Posts

2.Annual Maintenance Subscription. 
  Private, expert email advice from the developers of SQLite.

$1500/year
More Info
Purchase

3.Technical Support. 
  High-priority email and phone support directly from the SQLite developers.
  Guaranteed response time available as an option.

$8K-85K/year
More Info
Request A Quote

4.SQLite Consortium Membership. 
  Premium enterprise support including on-site visits and
  access to all proprietary extensions and test suites.

$120K/year
More Info
Request A Quote





Public Forum. 
  Help and advice from a world-wide community of experts.

Cost: Free
More Info
Recent Posts


Annual Maintenance Subscription. 
  Private, expert email advice from the developers of SQLite.

Cost: $1500/year
More Info
Purchase


Technical Support. 
  High-priority email and phone support directly from the SQLite developers.
  Guaranteed response time available as an option.

Cost: $8K-85K/year
More Info
Request A Quote


SQLite Consortium Membership. 
  Premium enterprise support including on-site visits and
  access to all proprietary extensions and test suites.

Cost: $120K/year
More Info
Request A Quote




Licenses
(details below)



5.SQLite License. 
  Warranty of title and perpetual right-to-use for the SQLite source code. 

$6000 one time fee
More Info
Purchase

6.SQLite Encryption Extension (SEE). 
   A drop-in replacement for public-domain SQLite source code that has 
   the added ability to read/write AES-encrypted databases. 

$2000 one time fee
More Info
Purchase

7.Compressed And Encrypted Read-Only Database (CEROD). 
   A drop-in replacement for public-domain SQLite source code
   that has the added ability to read database that has been both 
   compressed and encrypted. 

$2000 one time fee
More Info
Buy Now!

8.The ZIPVFS Read/Write Compressed Database Extension. 
   The ZIPVFS extension automatically compresses and encrypts
   your data as it is written to disk, and decompressed and decrypts
   it while reading.

$4000 one time fee
More Info
Buy Now!





SQLite License. 
  Warranty of title and perpetual right-to-use for the SQLite source code. 

Cost: $6000 one time fee
More Info
Purchase


SQLite Encryption Extension (SEE). 
   A drop-in replacement for public-domain SQLite source code that has 
   the added ability to read/write AES-encrypted databases. 

Cost: $2000 one time fee
More Info
Purchase


Compressed And Encrypted Read-Only Database (CEROD). 
   A drop-in replacement for public-domain SQLite source code
   that has the added ability to read database that has been both 
   compressed and encrypted. 

Cost: $2000 one time fee
More Info
Buy Now!


The ZIPVFS Read/Write Compressed Database Extension. 
   The ZIPVFS extension automatically compresses and encrypts
   your data as it is written to disk, and decompressed and decrypts
   it while reading.

Cost: $4000 one time fee
More Info
Buy Now!




Special Services
(details below)



9.TH3 Testing Support. 
  The TH3 test harness
  is an aviation-grade test suite for SQLite. SQLite developers 
  can run TH3 on specialized hardware and/or using specialized
  compile-time options, according to customer specification, 
  either remotely or on customer premises. Pricing for this 
  services is on a case-by-case basis depending on requirements.

call
More Info
Request A Quote





TH3 Testing Support. 
  The TH3 test harness
  is an aviation-grade test suite for SQLite. SQLite developers 
  can run TH3 on specialized hardware and/or using specialized
  compile-time options, according to customer specification, 
  either remotely or on customer premises. Pricing for this 
  services is on a case-by-case basis depending on requirements.

Cost: call
More Info
Request A Quote





Further Information

SQLite is free and works great.
Most people use SQLite without
any kind of license or support.

Free support for SQLite is available on the public
SQLite Forum.
The forum is monitored by a large
community of experts, including the core SQLite development team,
who are able to resolve just about
any problems with SQLite that you are likely to have.

If you do not want to discuss your issues on a public and open forum, an
Annual Maintenance 
Subscription or "AMS" might
serve your needs better.
An AMS entitles you to private email support directly from the developers of
SQLite.  The level of support with an AMS is similar to the support one
gets from the public forum except that an AMS includes a unilateral 
promise of non-disclosure, so with an AMS you can discuss your issues
with the SQLite developers in private, without the whole world watching.
An AMS costs $1500 per year.

Users with more advanced support needs can opt for a
Technical Support
Agreement.
Technical support agreements are customized to the needs of each
individual client, but generally include direct telephone support
and priority handling of issues and bugs.  Guaranteed response time
is available as an option.  The cost of
technical support varies but is generally
in the range of $8000 to $35000 per year.

If SQLite is "mission critical" to your company, then you may
want to become an 
SQLite Consortium 
Member.  The SQLite
Consortium is a collaboration of companies who sponsor ongoing development
of SQLite in exchange for enterprise-level technical support, on-site
visits from the SQLite developers, unlimited access to all licensed
products, and strong guarantees that SQLite will remain in the public
domain, free and independent, and will not come under the control of
a competitor.


Software Licenses

The SQLite source code is in the 
public domain,
and is free for use
by anyone and for any purpose.  No license is required.  However, some
users desire a license so that they can have warranty of title, or just
because their company lawyers say they need one.  A
perpetual license
and warranty of title
for the core SQLite source code is available for this purpose.

The 
SQLite Encryption
Extension (SEE),
the ZIPVFS Extension,
and the Compressed and
Encrypted ReadOnly Database (CEROD) extension are enhanced versions 
of SQLite that handle encrypted
and/or compressed databases.  SEE can read and write encrypted databases.
SEE encrypts all database content, including metadata, so that the database
file appears as white noise.  ZIPVFS 
compresses the database on-the-fly using application-supplied 
compression and decompression functions.
CEROD reads a compressed database that is
also optionally encrypted.  All of SEE, ZIPVFS, and CEROD are 
supplied in source code form only; the licensee is responsible for 
compiling the products for their chosen platform.  It is not difficult 
to compile any of these extension.  All products come in the form of an
amalgamated source file
named "sqlite3.c".  So compiling SEE, ZIPVFS, or CEROD into an application 
is simply a matter of substituting the SEE-, ZIPVFS-, or CEROD-enabled sqlite3.c 
source file in place of the public-domain sqlite3.c source file and recompiling.
Licenses for SEE, ZIPVFS, and CEROD are perpetual.
All three extension can read and write ordinary,
uncompressed and unencrypted database files.


Testing Services

The Test Harness #3 (TH3) is 
a suite of test cases for SQLite that provide 100% branch test coverage
(and 100% modified condition/decision coverage) for the core SQLite in
an as-deployed configuration using only published and documented interfaces.
TH3 is designed for use with embedded devices, and is compatible with
DO-178B.  Every release of the public-domain SQLite is tested using TH3,
and so all users benefit from the TH3 tests.  But the TH3 tests are not
themselves public.  Hardware or system manufactures who want to have
TH3 test run on their systems can negotiate a service agreement to have
the SQLite Developers run those tests.

About The SQLite Team

Paid support options and products are provided by
Hipp, Wyrick &amp; Company, Inc., (Hwaci), a 
Georgia
corporation 
with headquarters in

Charlotte, North Carolina and has been in business since
1992.
Hwaci has an international team of
employees and associates representing the best available talent.  
We are a 100% engineering company.  There is
no sales staff.
Our goal is to provide outstanding service and honest advice
without spin or sales-talk.

Hwaci is a small company but it is
also closely held and debt-free and has low
fixed costs, which means that it is largely immune to buy-outs,
take-overs, and market down-turns.  Hwaci intends to 
continue operating in its current form, and at roughly its current 
size until at least the year 2050.
We expect to be here when you need us,
even if that need is many years in the future.


c2: Appropriate Uses For SQLite
c5: whentouse.html
c4:
Appropriate Uses For SQLite


SQLite is not directly comparable to client/server SQL database engines such
as MySQL, Oracle, PostgreSQL, or SQL Server since SQLite is trying to
solve a different problem.



Client/server SQL database engines strive to implement a shared repository
of enterprise data.  They emphasize scalability, concurrency, centralization,
and control.
SQLite strives to provide local data storage for
individual applications and devices.  SQLite emphasizes economy,
efficiency, reliability, independence, and simplicity.




SQLite does not compete with client/server databases.
SQLite competes with fopen().


Situations Where SQLite Works Well





Embedded devices and the internet of things

Because an SQLite database requires no administration,
it works well in devices that must operate without expert human support.
SQLite is a good fit for use in 
cellphones, set-top boxes, televisions, game consoles,
cameras, watches, kitchen appliances, thermostats, automobiles, 
machine tools, airplanes, remote sensors, drones, medical devices,
and robots:  the "internet of things".


Client/server database engines are designed to live inside a
lovingly-attended datacenter at the core of the network.
SQLite works there too, but SQLite also thrives at the edge of the network,
fending for itself while providing fast and
reliable data services to applications that would otherwise
have dodgy connectivity.


Application file format


SQLite is often used as the on-disk file format
for desktop applications such as version control systems,
financial analysis tools, media cataloging and editing suites, CAD
packages, record keeping programs, and so forth.  The traditional
File/Open operation calls sqlite3_open() to attach to the database
file.  Updates happen automatically as application content is revised
so the File/Save menu option becomes superfluous.  The File/Save_As
menu option can be implemented using the backup API.


There are many benefits to this approach, including improved
performance, reduced cost and complexity, and
improved reliability.  See technical notes
"aff_short.html" and 
"appfileformat.html" and
"fasterthanfs.html" for more information.
This use case is closely related to the
data transfer format and
data container use cases below.




Websites

SQLite works great as the database engine for most low to
medium traffic websites (which is to say, most websites).
The amount of web traffic that SQLite can handle depends
on how heavily the website uses its database.  Generally
speaking, any site that gets fewer than 100K hits/day should work
fine with SQLite.
The 100K hits/day figure is a conservative estimate, not a
hard upper bound.
SQLite has been demonstrated to work with 10 times that amount
of traffic.

The SQLite website (https://www.sqlite.org/) uses SQLite itself,
of course, and as of this writing (2015) it handles about 400K to 500K
HTTP requests per day, about 15-20% of which are dynamic pages touching
the database.  Dynamic content uses about 200 SQL statements per webpage.
This setup runs on a single VM that shares a physical server with 23 others
and yet still keeps the load average below 0.1 most of the time.

See also: Hacker New discussion from 2022-12-13.


Data analysis


People who understand SQL can employ the 
sqlite3 command-line shell (or various third-party
SQLite access programs) to analyze large
datasets. Raw data can be imported from CSV files, then that
data can be sliced and diced to generate a myriad of summary
reports.  More complex analysis can be done using simple scripts written
in Tcl or Python (both of which come with SQLite built-in) or in R or
other languages using readily available adaptors.
Possible uses include website log analysis, sports
statistics analysis, compilation of programming metrics, and
analysis of experimental results.  Many bioinformatics researchers
use SQLite in this way.



The same thing can be done with an enterprise client/server
database, of course.  The advantage of SQLite is
that it is easier to install and use and the resulting database 
is a single file that can be written to a USB memory stick
or emailed to a colleague.



Cache for enterprise data


Many applications use SQLite as a cache of relevant content from
an enterprise RDBMS.
This reduces latency, since most queries now occur against the local
cache and avoid a network round-trip.  It also reduces the load 
on the network and on the central database server.  And in many cases, 
it means that the client-side application can continue operating during
network outages.





Server-side database


Systems designers
report success using SQLite as a data store on server applications
running in the datacenter, or in other words, using SQLite as the underlying
storage engine for an application-specific database server.

With this pattern, the overall system is still client/server:
clients send requests to the server and get back replies over the network.
But instead of sending generic SQL and getting back raw table content, 
the client requests and server responses are high-level and 
application-specific.
The server translates requests into multiple SQL queries, gathers the
results, does post-processing, filtering, and analysis, then constructs
a high-level reply containing only the essential information.

Developers report that SQLite is often faster than a client/server
SQL database engine in this scenario.
Database requests are serialized by the server, so concurrency is not
an issue.  Concurrency is also improved by "database sharding":
using separate database files for different subdomains.  For
example, the server might have a separate SQLite database for each
user, so that the server can handle hundreds or thousands of simultaneous
connections, but each SQLite database is only used by one connection.




Data transfer format

Because an SQLite database is a single compact file in a
well-defined cross-platform format, it is often used
as a container for transferring content from one system to another.
The sender gathers content into an SQLite database file, transfers
that one file to the receiver, then the receiver uses SQL to extract
the content as needed.

An SQLite database facilitates data transfer between systems even
when the endpoints have different word sizes and/or byte orders.
The data can be a complex mix of large binary blobs, text, and small
numeric or boolean values.  The data format can be easily extended
by adding new tables and/or columns, without breaking legacy receivers.
The SQL query language means that receivers are not required to parse
the entire transfer all at once, but can instead query the
received content as needed.  The data format is "transparent" in the
sense that it is easily decoded for human viewing using 
a variety of universally available, open-source tools, from multiple
vendors.




File archive and/or data container


The SQLite Archive idea shows how
SQLite can be used as a substitute for ZIP archives or Tarballs.
An archive of files stored in SQLite is only very slightly larger, and
in some cases actually smaller, than the equivalent ZIP archive.
And an SQLite archive features incremental and atomic updating
and the ability to store much richer metadata.


Fossil version 2.5 and later offers
SQLite Archive files as a download format, in addition
to traditional tarball and ZIP archive.
The sqlite3.exe command-line shell version 3.22.0 and later will create,
list, or unpack an SQL archiving using the 
.archive command.


SQLite is a good solution for any situation that requires bundling
diverse content into a self-contained and self-describing package 
for shipment across a network.
Content is encoded in a 
well-defined, cross-platform, and stable file format.
The encoding is efficient, and receivers can extract small subsets
of the content without having to read and parse the entire file.


SQL archives are useful as the distribution format for software
or content updates that are broadcast to many clients.  Variations
on this idea are used, for example, to transmit TV programming guides
to set-top boxes and to send over-the-air updates to vehicle navigation
systems.


Replacement for ad hoc disk files

Many programs use 
fopen(),
fread(), and 
fwrite() to create and
manage files of data in home-grown formats.  SQLite works 
particularly well as a
replacement for these ad hoc data files.
Contrary to intuition, SQLite can be faster than the filesystem
for reading and writing content to disk.


Internal or temporary databases


For programs that have a lot of data that must be sifted and sorted
in diverse ways, it is often easier and quicker to load the data into
an in-memory SQLite database and use queries with joins and ORDER BY
clauses to extract the data in the form and order needed rather than
to try to code the same operations manually.
Using an SQL database internally in this way also gives the program
greater flexibility since new columns and indices can be added without
having to recode every query.



Stand-in for an enterprise database during demos or testing


Client applications typically use a generic database interface that allows
connections to various SQL database engines.  It makes good sense to 
include SQLite in the mix of supported databases and to statically
link the SQLite engine in with the client.  That way the client program
can be used standalone with an SQLite data file for testing or for
demonstrations.



Education and Training


Because it is simple to setup and use (installation is trivial: just
copy the sqlite3 or sqlite3.exe executable to the target machine
and run it) SQLite makes a good database engine for use in teaching SQL.
Students can easily create as many databases as they like and can
email databases to the instructor for comments or grading.  For more
advanced students who are interested in studying how an RDBMS is
implemented, the modular and well-commented and documented SQLite code
can serve as a good basis.



Experimental SQL language extensions

The simple, modular design of SQLite makes it a good platform for
prototyping new, experimental database language features or ideas.






Situations Where A Client/Server RDBMS May Work Better


Client/Server Applications

If there are many client programs sending SQL to the same 
database over a network, then use a client/server database
engine instead of SQLite.  SQLite will work over a network filesystem,
but because of the latency associated with most network filesystems,
performance will not be great.  Also, file locking logic is buggy in
many network filesystem implementations (on both Unix and Windows).
If file locking does not work correctly,
two or more clients might try to modify the
same part of the same database at the same time, resulting in 
corruption.  Because this problem results from bugs in
the underlying filesystem implementation, there is nothing SQLite
can do to prevent it.

A good rule of thumb is to avoid using SQLite
in situations where the same database will be accessed directly
(without an intervening application server) and simultaneously
from many computers over a network.


High-volume Websites

SQLite will normally work fine as the database backend to a website.
But if the website is write-intensive or is so busy that it requires
multiple servers, then consider using an enterprise-class client/server 
database engine instead of SQLite.


Very large datasets

An SQLite database is limited in size to 281 terabytes 
(248 bytes, 256 tibibytes).
And even if it could handle larger databases, SQLite stores the entire
database in a single disk file and many filesystems limit the maximum
size of files to something less than this.  So if you are contemplating
databases of this magnitude, you would do well to consider using a
client/server database engine that spreads its content across multiple
disk files, and perhaps across multiple volumes.



High Concurrency


SQLite supports an unlimited number of simultaneous readers, but it 
will only allow one writer at any instant in time.
For many situations, this is not a problem.  Writers queue up. Each application
does its database work quickly and moves on, and no lock lasts for more
than a few dozen milliseconds. But there are some applications that require
more concurrency, and those applications may need to seek a different
solution.







Checklist For Choosing The Right Database Engine


Is the data separated from the application by a network?
       &rarr; choose client/server

Relational database engines act as bandwidth-reducing data filters.
So it is best to keep the database engine and the data on
the same physical device so that the high-bandwidth engine-to-disk
link does not have to traverse the network, only the lower-bandwidth
application-to-engine link.

But SQLite is built into the application.  So if the data is on a
separate device from the application, it is required that the higher
bandwidth engine-to-disk link be across the network.  This works, but
it is suboptimal.  Hence, it is usually better to select a client/server
database engine when the data is on a separate device from the
application.

Nota Bene:
In this rule, "application" means the code that issues SQL statements.
If the "application" is an application server and
if the content resides on the same physical machine as the application server,
then SQLite might still be appropriate even though the end user is
another network hop away.


Many concurrent writers? &rarr; choose client/server

If many threads and/or processes need to write the
database at the same instant (and they cannot queue up and take turns)
then it is best to select a database engine that supports that
capability, which always means a client/server database engine.

SQLite only supports one writer at a time per database file.
But in most cases, a write transaction only takes milliseconds and
so multiple writers can simply take turns.  SQLite will handle
more write concurrency than many people suspect.  Nevertheless,
client/server database systems, because they have a long-running
server process at hand to coordinate access, can usually handle 
far more write concurrency than SQLite ever will.


Big data? &rarr; choose client/server

If your data will grow to a size that you are uncomfortable
or unable to fit into a single disk file, then you should select
a solution other than SQLite.  SQLite supports databases up to
281 terabytes in size, assuming you can find a disk drive and filesystem
that will support 281-terabyte files.  Even so, when the size of the
content looks like it might creep into the terabyte range, it would
be good to consider a centralized client/server database.


Otherwise &rarr; choose SQLite!

For device-local storage with low writer concurrency and less than a
terabyte of content, SQLite is almost always a better solution.  SQLite
is fast and reliable and it requires no configuration or maintenance.
It keeps things simple.  SQLite "just works".


This page last modified on  2022-12-14 16:04:04 UTC


c2: SQLite Changes From Version 3.4.2 To 3.5.0
c5: 34to35.html
c4:
Moving From SQLite 3.4.2 to 3.5.0
  SQLite version 3.5.0 (2007-09-04) introduces a new OS interface layer that
  is incompatible with all prior versions of SQLite.  In addition,
  a few existing interfaces have been generalized to work across all
  database connections within a process rather than just all
  connections within a thread.  The purpose of this article
  is to describe the changes to 3.5.0 in detail so that users
  of prior versions of SQLite can judge what, if any, effort will
  be required to upgrade to newer versions.

1.0 Overview Of Changes
  A quick enumeration of the changes in SQLite version 3.5.0
  is provided here.  Subsequent sections will describe these
  changes in more detail.


  
  The OS interface layer has been completely reworked:
  
  The undocumented sqlite3_os_switch() interface has
      been removed.
  The SQLITE_ENABLE_REDEF_IO compile-time flag no longer functions.
      I/O procedures are now always redefinable.
  Three new objects are defined for specifying I/O procedures:
      sqlite3_vfs, sqlite3_file, and sqlite3_io_methods.
  Three new interfaces are used to create alternative OS interfaces:
      sqlite3_vfs_register(), sqlite3_vfs_unregister(), and
      sqlite3_vfs_find().
  A new interface has been added to provided additional control over
      the creation of new database connections: sqlite3_open_v2().
      The legacy interfaces of sqlite3_open() and
      sqlite3_open16() continue to be fully supported.
  
  The optional shared cache and memory management features that
      were introduced in version 3.3.0 can now be used across multiple
      threads within the same process.  Formerly, these extensions only
      applied to database connections operating within a single thread.
   
   The sqlite3_enable_shared_cache() interface now applies to all
       threads within a process, not to just the one thread in which it
       was run.
   The sqlite3_soft_heap_limit() interface now applies to all threads
       within a process, not to just the one thread in which it was run.
   The sqlite3_release_memory() interface will now attempt to reduce
       the memory usages across all database connections in all threads, not
       just connections in the thread where the interface is called.
   The sqlite3_thread_cleanup() interface has become a no-op.
   
  Restrictions on the use of the same database connection by multiple
      threads have been dropped.  It is now safe for
      multiple threads to use the same database connection at the same
      time.
  There is now a compile-time option that allows an application to
      define alternative malloc()/free() implementations without having
      to modify any core SQLite code.
  There is now a compile-time option that allows an application to
      define alternative mutex implementations without having
      to modify any core SQLite code.
  


  Of these changes, only 1a and 2a through 2c are incompatibilities
  in any formal sense.
  But users who have previously made custom modifications to the
  SQLite source (for example to add a custom OS layer for embedded
  hardware) might find that these changes have a larger impact.
  On the other hand, an important goal of these changes is to make
  it much easier to customize SQLite for use on different operating
  systems. 

2.0 The OS Interface Layer
  If your system defines a custom OS interface for SQLite or if you
  were using the undocumented sqlite3_os_switch()
  interface, then you will need to make modifications in order to
  upgrade to SQLite version 3.5.0.  This may seem painful at first
  glance.  But as you look more closely, you will probably discover
  that your changes are made smaller and easier to understand and manage
  by the new SQLite interface.  It is likely that your changes will
  now also work seamlessly with the SQLite amalgamation.  You will
  no longer need to make any changes to the code SQLite source code.
  All of your changes can be effected by application code and you can
  link against a standard, unmodified version of the SQLite amalgamation.
  Furthermore, the OS interface layer, which was formerly undocumented,
  is now an officially support interface for SQLite.  So you have
  some assurance that this will be a one-time change and that your
  new backend will continue to work in future versions of SQLite.

2.1 The Virtual File System Object
  The new OS interface for SQLite is built around an object named
  sqlite3_vfs.  The "vfs" stands for "Virtual File System".
  The sqlite3_vfs object is basically a structure containing pointers
  to functions that implement the primitive disk I/O operations that
  SQLite needs to perform in order to read and write databases.  
  In this article, we will often refer to an sqlite3_vfs objects as a "VFS".


  SQLite is able to use multiple VFSes at the same time.  Each
  individual database connection is associated with just one VFS.
  But if you have multiple database connections, each connection
  can be associated with a different VFS.


  There is always a default VFS.
  The legacy interfaces sqlite3_open() and sqlite3_open16() always
  use the default VFS.
  The new interface for creating database connections,
  sqlite3_open_v2(), allows you to specify which VFS you want to
  use by name.

2.1.1 Registering New VFS Objects
  Standard builds of SQLite for Unix or Windows come with a single
  VFS named "unix" or "win32", as appropriate.  This one VFS is also 
  the default.  So if you are using the legacy open functions, everything
  will continue to operate as it has before.  The change is that an application
  now has the flexibility of adding new VFS modules to implement a
  customized OS layer.  The sqlite3_vfs_register() API can be used
  to tell SQLite about one or more application-defined VFS modules:


int sqlite3_vfs_register(sqlite3_vfs*, int makeDflt);

  Applications can call sqlite3_vfs_register() at any time, though of course
  a VFS needs to be registered before it can be used.  The first argument
  is a pointer to a customized VFS object that the application has prepared.
  The second argument is true to make the new VFS the default VFS so that
  it will be used by the legacy sqlite3_open() and sqlite3_open16() APIs.
  If the new VFS is not the default, then you will probably have to use
  the new sqlite3_open_v2() API to use it.  Note, however, that if
  a new VFS is the only VFS known to SQLite (if SQLite was compiled without
  its usual default VFS or if the precompiled default VFS was removed
  using sqlite3_vfs_unregister()) then the new VFS automatically becomes the
  default VFS regardless of the makeDflt argument to sqlite3_vfs_register().


  Standard builds include the default "unix" or "win32" VFSes.
  But if you use the -DOS_OTHER=1 compile-time option, then SQLite is
  built without a default VFS.  In that case, the application must
  register at least one VFS prior to calling sqlite3_open().
  This is the approach that embedded applications should use.
  Rather than modifying the SQLite source to insert an alternative
  OS layer as was done in prior releases of SQLite, instead compile
  an unmodified SQLite source file (preferably the amalgamation)
  with the -DOS_OTHER=1 option, then invoke sqlite3_vfs_register()
  to define the interface to the underlying filesystem prior to
  creating any database connections.

2.1.2 Additional Control Over VFS Objects
  The sqlite3_vfs_unregister() API is used to remove an existing
  VFS from the system.


int sqlite3_vfs_unregister(sqlite3_vfs*);

  The sqlite3_vfs_find() API is used to locate a particular VFS
  by name.  Its prototype is as follows:


sqlite3_vfs *sqlite3_vfs_find(const char *zVfsName);

  The argument is the symbolic name for the desired VFS.  If the
  argument is a NULL pointer, then the default VFS is returned.
  The function returns a pointer to the sqlite3_vfs object that
  implements the VFS.  Or it returns a NULL pointer if no object
  could be found that matched the search criteria.

2.1.3 Modifications Of Existing VFSes
  Once a VFS has been registered, it should never be modified.  If
  a change in behavior is required, a new VFS should be registered.
  The application could, perhaps, use sqlite3_vfs_find() to locate
  the old VFS, make a copy of the old VFS into a new sqlite3_vfs
  object, make the desired modifications to the new VFS, unregister
  the old VFS, then register the new VFS in its place.  Existing
  database connections would continue to use the old VFS even after
  it is unregistered, but new database connections would use the
  new VFS.

2.1.4 The VFS Object
  A VFS object is an instance of the following structure:


typedef struct sqlite3_vfs sqlite3_vfs;
struct sqlite3_vfs {
  int iVersion;            /* Structure version number */
  int szOsFile;            /* Size of subclassed sqlite3_file */
  int mxPathname;          /* Maximum file pathname length */
  sqlite3_vfs *pNext;      /* Next registered VFS */
  const char *zName;       /* Name of this virtual file system */
  void *pAppData;          /* Pointer to application-specific data */
  int (*xOpen)(sqlite3_vfs*, const char *zName, sqlite3_file*,
               int flags, int *pOutFlags);
  int (*xDelete)(sqlite3_vfs*, const char *zName, int syncDir);
  int (*xAccess)(sqlite3_vfs*, const char *zName, int flags);
  int (*xGetTempName)(sqlite3_vfs*, char *zOut);
  int (*xFullPathname)(sqlite3_vfs*, const char *zName, char *zOut);
  void *(*xDlOpen)(sqlite3_vfs*, const char *zFilename);
  void (*xDlError)(sqlite3_vfs*, int nByte, char *zErrMsg);
  void *(*xDlSym)(sqlite3_vfs*,void*, const char *zSymbol);
  void (*xDlClose)(sqlite3_vfs*, void*);
  int (*xRandomness)(sqlite3_vfs*, int nByte, char *zOut);
  int (*xSleep)(sqlite3_vfs*, int microseconds);
  int (*xCurrentTime)(sqlite3_vfs*, double*);
  /* New fields may be appended in figure versions.  The iVersion
  ** value will increment whenever this happens. */
};

  To create a new VFS, an application fills in an instance of this
  structure with appropriate values and then calls sqlite3_vfs_register().


  The iVersion field of sqlite3_vfs should be 1 for SQLite version 3.5.0.
  This number may increase in future versions of SQLite if we have to
  modify the VFS object in some way.  We hope that this never happens,
  but the provision is made in case it does.


  The szOsFile field is the size in bytes of the structure that defines
  an open file: the sqlite3_file object.  This object will be described
  more fully below.  The point here is that each VFS implementation can
  define its own sqlite3_file object containing whatever information
  the VFS implementation needs to store about an open file.  SQLite needs
  to know how big this object is, however, in order to preallocate enough
  space to hold it.


  The mxPathname field is the maximum length of a file pathname that
  this VFS can use.  SQLite sometimes has to preallocate buffers of
  this size, so it should be as small as reasonably possible.  Some
  filesystems permit huge pathnames, but in practice pathnames rarely
  extend beyond 100 bytes or so.  You do not have to put the longest
  pathname that the underlying filesystem can handle here.  You only
  have to put the longest pathname that you want SQLite to be able to
  handle.  A few hundred is a good value in most cases.


  The pNext field is used internally by SQLite.  Specifically, SQLite
  uses this field to form a linked list of registered VFSes.


  The zName field is the symbolic name of the VFS.  This is the name 
  that the sqlite3_vfs_find() compares against when it is looking for
  a VFS.


  The pAppData pointer is unused by the SQLite core.  The pointer is
  available to store auxiliary information that a VFS information might
  want to carry around.


  The remaining fields of the sqlite3_vfs object all store pointers
  to functions that implement primitive operations.  We call these
  "methods".  The first method, xOpen, is used to open files on
  the underlying storage media.  The result is an sqlite3_file
  object.  There are additional methods, defined by the sqlite3_file
  object itself that are used to read and write and close the file.
  The additional methods are detailed below.  The filename is in UTF-8.
  SQLite will guarantee that the zFilename string passed to
  xOpen() is a full pathname as generated by xFullPathname() and
  that the string will be valid and unchanged until xClose() is
  called.  So the sqlite3_file can store a pointer to the
   filename if it needs to remember the filename for some reason.
   The flags argument to xOpen() is a copy of the flags argument
   to sqlite3_open_v2().  If sqlite3_open() or sqlite3_open16()
   is used, then flags is SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE.
   If xOpen() opens a file read-only then it sets *pOutFlags to
   include SQLITE_OPEN_READONLY.  Other bits in *pOutFlags may be
   set.
   SQLite will also add one of the following flags to the xOpen()
   call, depending on the object being opened:
   
     SQLITE_OPEN_MAIN_DB
     SQLITE_OPEN_MAIN_JOURNAL
     SQLITE_OPEN_TEMP_DB
     SQLITE_OPEN_TEMP_JOURNAL
     SQLITE_OPEN_TRANSIENT_DB
     SQLITE_OPEN_SUBJOURNAL
     SQLITE_OPEN_SUPER_JOURNAL
   
   The file I/O implementation can use the object type flags to
   changes the way it deals with files.  For example, an application
   that does not care about crash recovery or rollback, might make
   the open of a journal file a no-op.  Writes to this journal are
   also a no-op.  Any attempt to read the journal returns SQLITE_IOERR.
   Or the implementation might recognize the a database file will
   be doing page-aligned sector reads and writes in a random order
   and set up its I/O subsystem accordingly.
   SQLite might also add one of the following flags to the xOpen
   method:
   
    SQLITE_OPEN_DELETEONCLOSE
    SQLITE_OPEN_EXCLUSIVE
   
   The SQLITE_OPEN_DELETEONCLOSE flag means the file should be
   deleted when it is closed.  This will always be set for TEMP 
   databases and journals and for subjournals.  The 
   SQLITE_OPEN_EXCLUSIVE flag means the file should be opened
   for exclusive access.  This flag is set for all files except
   for the main database file.
   The sqlite3_file structure passed as the third argument to
   xOpen is allocated by the caller.  xOpen just fills it in.  The
   caller allocates a minimum of szOsFile bytes for the sqlite3_file
   structure.


  The differences between an SQLITE_OPEN_TEMP_DB database and an
  SQLITE_OPEN_TRANSIENT_DB database is this:  The SQLITE_OPEN_TEMP_DB
  is used for explicitly declared and named TEMP tables (using the
  CREATE TEMP TABLE syntax) or for named tables in a temporary database
  that is created by opening a database with a filename that is an empty
  string.  An SQLITE_OPEN_TRANSIENT_DB holds a database table that
  SQLite creates automatically in order to evaluate a subquery or
  ORDER BY or GROUP BY clause.  Both TEMP_DB and TRANSIENT_DB databases
  are private and are deleted automatically.  TEMP_DB databases last
  for the duration of the database connection.  TRANSIENT_DB databases
  last only for the duration of a single SQL statement.


  The xDelete method is used to delete a file.  The name of the file is
  given in the second parameter.  The filename will be in UTF-8.
  The VFS must convert the filename into whatever character representation
  the underlying operating system expects.  If the syncDir parameter is
  true, then the xDelete method should not return until the change
  to the directory contents for the directory containing the
  deleted file have been synced to disk in order to ensure that the
  file does not "reappear" if a power failure occurs soon after.


  The xAccess method is used to check for access permissions on a file.
  The filename will be UTF-8 encoded.  The flags argument will be
  SQLITE_ACCESS_EXISTS to check for the existence of the file,
  SQLITE_ACCESS_READWRITE to check to see if the file is both readable
  and writable, or SQLITE_ACCESS_READ to check to see if the file is
  at least readable.  The "file" named by the second parameter might
  be a directory or folder name.


  The xGetTempName method computes the name of a temporary file that
  SQLite can use.  The name should be written into the buffer given
  by the second parameter.  SQLite will size that buffer to hold
  at least mxPathname bytes.  The generated filename should be in UTF-8.
  To avoid security problems, the generated temporary filename should
  contain enough randomness to prevent an attacker from guessing the
  temporary filename in advance.


  The xFullPathname method is used to convert a relative pathname
  into a full pathname.  The resulting full pathname is written into
  the buffer provided by the third parameter.  SQLite will size the
  output buffer to at least mxPathname bytes.  Both the input and
  output names should be in UTF-8.


  The xDlOpen, xDlError, xDlSym, and xDlClose methods are all used for
  accessing shared libraries at run-time.  These methods may be omitted
  (and their pointers set to zero) if the library is compiled with
  SQLITE_OMIT_LOAD_EXTENSION or if the sqlite3_enable_load_extension()
  interface is never used to enable dynamic extension loading.  The
  xDlOpen method opens a shared library or DLL and returns a pointer to
  a handle.  NULL is returned if the open fails.  If the open fails,
  the xDlError method can be used to obtain a text error message.
  The message is written into the zErrMsg buffer of the third parameter
  which is at least nByte bytes in length.  The xDlSym returns a pointer
  to a symbol in the shared library.  The name of the symbol is given
  by the second parameter.  UTF-8 encoding is assumed.  If the symbol
  is not found a NULL pointer is returned.  The xDlClose routine closes
  the shared library.


  The xRandomness method is used exactly once to initialize the 
  pseudo-random number generator (PRNG) inside of SQLite.  Only
  the xRandomness method on the default VFS is used.  The xRandomness
  methods on other VFSes are never accessed by SQLite.
  The xRandomness routine requests that nByte bytes of randomness
  be written into zOut.  The routine returns the actual number of
  bytes of randomness obtained.  The quality of the randomness so obtained
  will determine the quality of the randomness generated by built-in 
  SQLite functions such as random() and randomblob().  SQLite also
  uses its PRNG to generate temporary file names.  On some platforms
  (ex: Windows) SQLite assumes that temporary file names are unique
  without actually testing for collisions, so it is important to have
  good-quality randomness even if the random() and randomblob() 
  functions are never used.


  The xSleep method is used to suspend the calling thread for at
  least the number of microseconds given.  This method is used to
  implement the sqlite3_sleep() and sqlite3_busy_timeout() APIs.
  In the case of sqlite3_sleep() the xSleep method of the default
  VFS is always used.  If the underlying system does not have a
  microsecond resolution sleep capability, then the sleep time should
  be rounded up.  xSleep returns this rounded-up value.


  The xCurrentTime method finds the current time and date and writes
  the result as a double-precision floating point value into pointer
  provided by the second parameter.  The time and date is in
  coordinated universal time (UTC) and is a fractional Julian day number.

2.1.5 The Open File Object
  The result of opening a file is an instance of an sqlite3_file object.
  The sqlite3_file object is an abstract base class defined as follows:


typedef struct sqlite3_file sqlite3_file;
struct sqlite3_file {
  const struct sqlite3_io_methods *pMethods;
};

  Each VFS implementation will subclass the sqlite3_file by adding
  additional fields at the end to hold whatever information the VFS
  needs to know about an open file.  It does not matter what information
  is stored as long as the total size of the structure does not exceed
  the szOsFile value recorded in the sqlite3_vfs object.


  The sqlite3_io_methods object is a structure that contains pointers
  to methods for reading, writing, and otherwise dealing with files.
  This object is defined as follows:


typedef struct sqlite3_io_methods sqlite3_io_methods;
struct sqlite3_io_methods {
  int iVersion;
  int (*xClose)(sqlite3_file*);
  int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst);
  int (*xWrite)(sqlite3_file*, const void*, int iAmt, sqlite3_int64 iOfst);
  int (*xTruncate)(sqlite3_file*, sqlite3_int64 size);
  int (*xSync)(sqlite3_file*, int flags);
  int (*xFileSize)(sqlite3_file*, sqlite3_int64 *pSize);
  int (*xLock)(sqlite3_file*, int);
  int (*xUnlock)(sqlite3_file*, int);
  int (*xCheckReservedLock)(sqlite3_file*);
  int (*xFileControl)(sqlite3_file*, int op, void *pArg);
  int (*xSectorSize)(sqlite3_file*);
  int (*xDeviceCharacteristics)(sqlite3_file*);
  /* Additional methods may be added in future releases */
};

  The iVersion field of sqlite3_io_methods is provided as insurance
  against future enhancements.  The iVersion value should always be
  1 for SQLite version 3.5.


  The xClose method closes the file.  The space for the sqlite3_file
  structure is deallocated by the caller.  But if the sqlite3_file
  contains pointers to other allocated memory or resources, those
  allocations should be released by the xClose method.


  The xRead method reads iAmt bytes from the file beginning at a byte
  offset to iOfst.  The data read is stored in the pointer of the
  second parameter.  xRead returns the SQLITE_OK on success,
  SQLITE_IOERR_SHORT_READ if it was not able to read the full number
  of bytes because it reached end-of-file, or SQLITE_IOERR_READ for
  any other error.


  The xWrite method writes iAmt bytes of data from the second parameter
  into the file beginning at an offset of iOfst bytes.  If the size of
  the file is less than iOfst bytes prior to the write, then xWrite should
  ensure that the file is extended with zeros up to iOfst bytes prior
  to beginning its write.  xWrite continues to extends the file as
  necessary so that the size of the file is at least iAmt+iOfst bytes 
  at the conclusion of the xWrite call.  The xWrite method returns
  SQLITE_OK on success.  If the write cannot complete because the
  underlying storage medium is full, then SQLITE_FULL is returned.
  SQLITE_IOERR_WRITE should be returned for any other error.


  The xTruncate method truncates a file to be nByte bytes in length.
  If the file is already nByte bytes or less in length then this
  method is a no-op.  The xTruncate method returns SQLITE_OK on
  success and SQLITE_IOERR_TRUNCATE if anything goes wrong.


  The xSync method is used to force previously written data out of
  operating system cache and into non-volatile memory.  The second
  parameter is usually SQLITE_SYNC_NORMAL.  If the second parameter
  is SQLITE_SYNC_FULL then the xSync method should make sure that
  data has also been flushed through the disk controllers cache.
  The SQLITE_SYNC_FULL parameter is the equivalent of the F_FULLSYNC
  ioctl() on Mac OS X. The xSync method returns
  SQLITE_OK on success and SQLITE_IOERR_FSYNC if anything goes wrong.


  The xFileSize() method determines the current size of the file
  in bytes and writes that value into *pSize.  It returns SQLITE_OK
  on success and SQLITE_IOERR_FSTAT if something goes wrong.


  The xLock and xUnlock methods are used to set and clear file locks.
  SQLite supports five levels of file locks, in order:
  
   SQLITE_LOCK_NONE
   SQLITE_LOCK_SHARED
   SQLITE_LOCK_RESERVED
   SQLITE_LOCK_PENDING
   SQLITE_LOCK_EXCLUSIVE
  
  The underlying implementation can support some subset of these locking
  levels as long as it meets the other requirements of this paragraph.
  The locking level is specified as the second argument to both xLock
  and xUnlock.  The xLock method increases the locking level to the
  specified locking level or higher.  The xUnlock method decreases the
  locking level to no lower than the level specified.  
  SQLITE_LOCK_NONE means that the file is unlocked.  SQLITE_LOCK_SHARED
  gives permission to read the file.  Multiple database connections can
  hold SQLITE_LOCK_SHARED at the same time.
  SQLITE_LOCK_RESERVED is like SQLITE_LOCK_SHARED in that it is permission
  to read the file.  But only a single connection can hold a reserved lock
  at any point in time.  The SQLITE_LOCK_PENDING is also permission to
  read the file.  Other connections can continue to read the file as well,
  but no other connection is allowed to escalate a lock from none to shared.
  SQLITE_LOCK_EXCLUSIVE is permission to write on the file.  Only a single
  connection can hold an exclusive lock and no other connection can hold
  any lock (other than "none") while one connection holds an exclusive
  lock.  The xLock returns SQLITE_OK on success, SQLITE_BUSY if it
  is unable to obtain the lock, or SQLITE_IOERR_RDLOCK if something else
  goes wrong.  The xUnlock method returns SQLITE_OK on success and
  SQLITE_IOERR_UNLOCK for problems.


  The xCheckReservedLock() method checks to see if another connection or
  another process is currently holding a reserved, pending, or exclusive
  lock on the file.  It returns true or false.


  The xFileControl() method is a generic interface that allows custom
  VFS implementations to directly control an open file using the
  (new and experimental)
  sqlite3_file_control() interface.  The second "op" argument
  is an integer opcode.  The third
  argument is a generic pointer which is intended to be a pointer
  to a structure that may contain arguments or space in which to
  write return values.  Potential uses for xFileControl() might be
  functions to enable blocking locks with timeouts, to change the
  locking strategy (for example to use dot-file locks), to inquire
  about the status of a lock, or to break stale locks.  The SQLite
  core reserves opcodes less than 100 for its own use. 
  A list of opcodes less than 100 is available.
  Applications that define a custom xFileControl method should use opcodes 
  greater than 100 to avoid conflicts.


  The xSectorSize returns the "sector size" of the underlying
  non-volatile media.  A "sector" is defined as the smallest unit of
  storage that can be written without disturbing adjacent storage.
  On a disk drive the "sector size" has until recently been 512 bytes,
  though there is a push to increase this value to 4KiB.  SQLite needs
  to know the sector size so that it can write a full sector at a
  time, and thus avoid corrupting adjacent storage space if a power
  loss occurs in the middle of a write.


  The xDeviceCharacteristics method returns an integer bit vector that
  defines any special properties that the underlying storage medium might
  have that SQLite can use to increase performance.  The allowed return
  is the bit-wise OR of the following values:
  
   SQLITE_IOCAP_ATOMIC
   SQLITE_IOCAP_ATOMIC512
   SQLITE_IOCAP_ATOMIC1K
   SQLITE_IOCAP_ATOMIC2K
   SQLITE_IOCAP_ATOMIC4K
   SQLITE_IOCAP_ATOMIC8K
   SQLITE_IOCAP_ATOMIC16K
   SQLITE_IOCAP_ATOMIC32K
   SQLITE_IOCAP_ATOMIC64K
   SQLITE_IOCAP_SAFE_APPEND
   SQLITE_IOCAP_SEQUENTIAL
  
  The SQLITE_IOCAP_ATOMIC bit means that all writes to this device are
  atomic in the sense that either the entire write occurs or none of it
  occurs.  The other 
  SQLITE_IOCAP_ATOMICnnn values indicate that
  writes of aligned blocks of the indicated size are atomic.
  SQLITE_IOCAP_SAFE_APPEND means that when extending a file with new
  data, the new data is written first and then the file size is updated.
  So if a power failure occurs, there is no chance that the file might have
  been extended with randomness.  The SQLITE_IOCAP_SEQUENTIAL bit means
  that all writes occur in the order that they are issued and are not
  reordered by the underlying file system.

2.1.6 Checklist For Constructing A New VFS
  The preceding paragraphs contain a lot of information.
  To ease the task of constructing
  a new VFS for SQLite we offer the following implementation checklist:


  
   Define an appropriate subclass of the sqlite3_file object.
   Implement the methods required by the sqlite3_io_methods object.
   Create a static and 
       constant sqlite3_io_methods object containing pointers
       to the methods from the previous step.
   Implement the xOpen method that opens a file and populates an
       sqlite3_file object, including setting pMethods to
       point to the sqlite3_io_methods object from the previous step.
   Implement the other methods required by sqlite3_vfs.
   Define a static (but not constant) sqlite3_vfs structure that
       contains pointers to the xOpen method and the other methods and
       which contains the appropriate values for iVersion, szOsFile,
       mxPathname, zName, and pAppData.
   Implement a procedure that calls sqlite3_vfs_register() and
       passes it a pointer to the sqlite3_vfs structure from the previous
       step.  This procedure is probably the only exported symbol in the
       source file that implements your VFS.
  


  Within your application, call the procedure implemented in the last
  step above as part of your initialization process before any
  database connections are opened.  

3.0 The Memory Allocation Subsystem
  Beginning with version 3.5, SQLite obtains all of the heap memory it
  needs using the routines sqlite3_malloc(), sqlite3_free(), and
  sqlite3_realloc().  These routines have existed in prior versions
  of SQLite, but SQLite has previously bypassed these routines and used
  its own memory allocator.  This all changes in version 3.5.0.


  The SQLite source tree actually contains multiple versions of the
  memory allocator.  The default high-speed version found in the
  "mem1.c" source file is used for most builds.  But if the SQLITE_MEMDEBUG
  flag is enabled, a separate memory allocator the "mem2.c" source file
  is used instead.  The mem2.c allocator implements lots of hooks to
  do error checking and to simulate memory allocation failures for testing
  purposes.  Both of these allocators use the malloc()/free() implementation
  in the standard C library.


  Applications are not required to use either of these standard memory
  allocators.  If SQLite is compiled with SQLITE_OMIT_MEMORY_ALLOCATION
  then no implementation for the sqlite3_malloc(), sqlite3_realloc(),
  and sqlite3_free() functions is provided.  Instead, the application
  that links against SQLite must provide its own implementation of these
  functions.  The application provided memory allocator is not required
  to use the malloc()/free() implementation in the standard C library.
  An embedded application might provide an alternative memory allocator
  that uses memory for a fixed memory pool set aside for the exclusive
  use of SQLite, for example.


  Applications that implement their own memory allocator must provide
  implementation for the usual three allocation functions 
  sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free().
  And they must also implement a fourth function:


int sqlite3_memory_alarm(
  void(*xCallback)(void *pArg, sqlite3_int64 used, int N),
  void *pArg,
  sqlite3_int64 iThreshold
);

 The sqlite3_memory_alarm routine is used to register
 a callback on memory allocation events.
 This routine registers or clears a callback that fires when
 the amount of memory allocated exceeds iThreshold.  Only
 a single callback can be registered at a time.  Each call
 to sqlite3_memory_alarm() overwrites the previous callback.
 The callback is disabled by setting xCallback to a NULL
 pointer.


 The parameters to the callback are the pArg value, the 
 amount of memory currently in use, and the size of the
 allocation that provoked the callback.  The callback will
 presumably invoke sqlite3_free() to free up memory space.
 The callback may invoke sqlite3_malloc() or sqlite3_realloc()
 but if it does, no additional callbacks will be invoked by
 the recursive calls.


 The sqlite3_soft_heap_limit() interface works by registering
 a memory alarm at the soft heap limit and invoking 
 sqlite3_release_memory() in the alarm callback.  Application
 programs should not attempt to use the sqlite3_memory_alarm()
 interface because doing so will interfere with the
 sqlite3_soft_heap_limit() module.  This interface is exposed
 only so that applications can provide their own
 alternative implementation when the SQLite core is
 compiled with SQLITE_OMIT_MEMORY_ALLOCATION.


  The built-in memory allocators in SQLite also provide the following
  additional interfaces:


sqlite3_int64 sqlite3_memory_used(void);
sqlite3_int64 sqlite3_memory_highwater(int resetFlag);

  These interfaces can be used by an application to monitor how
  much memory SQLite is using.  The sqlite3_memory_used() routine
  returns the number of bytes of memory currently in use and the
  sqlite3_memory_highwater() returns the maximum instantaneous
  memory usage.  Neither routine includes the overhead associated
  with the memory allocator.  These routines are provided for use
  by the application.  SQLite never invokes them itself.  So if
  the application is providing its own memory allocation subsystem,
  it can omit these interfaces if desired.

4.0 The Mutex Subsystem
  SQLite has always been threadsafe in the sense that it is safe to
  use different SQLite database connections in different threads at the
  same time.  The constraint was that the same database connection
  could not be used in two separate threads at once.  SQLite version 3.5.0
  relaxes this constraint. 


  In order to allow multiple threads to use the same database connection
  at the same time, SQLite must make extensive use of mutexes.  And for
  this reason a new mutex subsystem as been added.  The mutex subsystem
  as the following interface:


sqlite3_mutex *sqlite3_mutex_alloc(int);
void sqlite3_mutex_free(sqlite3_mutex*);
void sqlite3_mutex_enter(sqlite3_mutex*);
int sqlite3_mutex_try(sqlite3_mutex*);
void sqlite3_mutex_leave(sqlite3_mutex*);

  Though these routines exist for the use of the SQLite core, 
  application code is free to use these routines as well, if desired.
  A mutex is an sqlite3_mutex object.  The sqlite3_mutex_alloc()
  routine allocates a new mutex object and returns a pointer to it.
  The argument to sqlite3_mutex_alloc() should be 
  SQLITE_MUTEX_FAST or SQLITE_MUTEX_RECURSIVE for non-recursive
  and recursive mutexes, respectively.  If the underlying system does
  not provide non-recursive mutexes, then a recursive mutex can be
  substituted in that case.  The argument to sqlite3_mutex_alloc()
  can also be a constant designating one of several static mutexes:
  
    SQLITE_MUTEX_STATIC_MAIN
    SQLITE_MUTEX_STATIC_MEM
    SQLITE_MUTEX_STATIC_MEM2
    SQLITE_MUTEX_STATIC_PRNG
    SQLITE_MUTEX_STATIC_LRU
  
  These static mutexes are reserved for use internally by SQLite
  and should not be used by the application.  The static mutexes
  are all non-recursive.


  The sqlite3_mutex_free() routine should be used to deallocate
  a non-static mutex.  If a static mutex is passed to this routine
  then the behavior is undefined.


  The sqlite3_mutex_enter() attempts to enter the mutex and blocks
  if another threads is already there.  sqlite3_mutex_try() attempts
  to enter and returns SQLITE_OK on success or SQLITE_BUSY if another
  thread is already there.  sqlite3_mutex_leave() exits a mutex.
  The mutex is held until the number of exits matches the number of
  entrances.  If sqlite3_mutex_leave() is called on a mutex that 
  the thread is not currently holding, then the behavior is undefined.
  If any routine is called for a deallocated mutex, then the behavior
  is undefined.


  The SQLite source code provides multiple implementations of these
  APIs, suitable for varying environments.  If SQLite is compiled with
  the SQLITE_THREADSAFE=0 flag then a no-op mutex implementation that 
  is fast but does no real mutual exclusion is provided.  That 
  implementation is suitable for use in single-threaded applications
  or applications that only use SQLite in a single thread.  Other
  real mutex implementations are provided based on the underlying
  operating system.


  Embedded applications may wish to provide their own mutex implementation.
  If SQLite is compiled with the -DSQLITE_MUTEX_APPDEF=1 compile-time flag
  then the SQLite core provides no mutex subsystem and a mutex subsystem
  that matches the interface described above must be provided by the
  application that links against SQLite.

5.0 Other Interface Changes
  Version 3.5.0 of SQLite changes the behavior of a few APIs in ways
  that are technically incompatible.  However, these APIs are seldom
  used and even when they are used it is difficult to imagine a
  scenario where the change might break something.  The changes
  actually makes these interface much more useful and powerful.


  Prior to version 3.5.0, the sqlite3_enable_shared_cache() API
  would enable and disable the shared cache feature for all connections
  within a single thread - the same thread from which the 
  sqlite3_enable_shared_cache() routine was called.  Database connections
  that used the shared cache were restricted to running in the same
  thread in which they were opened.  Beginning with version 3.5.0,
  the sqlite3_enable_shared_cache() applies to all database connections
  in all threads within the process.  Now database connections running
  in separate threads can share a cache.  And database connections that
  use shared cache can migrate from one thread to another.


  Prior to version 3.5.0 the sqlite3_soft_heap_limit() set an upper
  bound on heap memory usage for all database connections within a
  single thread.  Each thread could have its own heap limit.  Beginning
  in version 3.5.0, there is a single heap limit for the entire process.
  This seems more restrictive (one limit as opposed to many) but in
  practice it is what most users want.


  Prior to version 3.5.0 the sqlite3_release_memory() function would
  try to reclaim memory from all database connections in the same thread
  as the sqlite3_release_memory() call.  Beginning with version 3.5.0,
  the sqlite3_release_memory() function will attempt to reclaim memory
  from all database connections in all threads.

6.0 Summary
  The transition from SQLite version 3.4.2 to 3.5.0 is a major change.
  Every source code file in the SQLite core had to be modified, some
  extensively.  And the change introduced some minor incompatibilities
  in the C interface.  But we feel that the benefits of the transition
  from 3.4.2 to 3.5.0 far outweigh the pain of porting.  The new
  VFS layer is now well-defined and stable and should simplify future
  customizations.  The VFS layer, and the separable memory allocator
  and mutex subsystems allow a standard SQLite source code amalgamation
  to be used in an embedded project without change, greatly simplifying
  configuration management.  And the resulting system is much more
  tolerant of highly threaded designs.


This page last modified on  2020-06-18 21:18:56 UTC


c2: Features Of SQLite
c5: features.html
c4:
Features Of SQLite


Transactions
    are atomic, consistent, isolated, and durable (ACID)
    even after system crashes and power failures.
Zero-configuration
     - no setup or administration needed.
Full-featured SQL implementation
    with advanced capabilities like partial indexes,
    indexes on expressions, JSON,
    common table expressions, and window functions.
    (Omitted features)
A complete database is stored in a 
    single cross-platform disk file.
    Great for use as an application file format.
Supports terabyte-sized databases and gigabyte-sized strings
    and blobs.  (See limits.html.)
Small code footprint: 
    less than 750KiB fully configured or much less
    with optional features omitted.
Simple, easy to use API.
Fast:  In some cases, SQLite is 
    faster than direct filesystem I/O
Written in ANSI-C.  TCL bindings included.
    Bindings for dozens of other languages available separately.
Well-commented source code with
    100% branch test coverage.
Available as a 
    single ANSI-C source-code file 
    that is easy to compile and hence is easy
    to add into a larger project.
Self-contained:
    no external dependencies.
Cross-platform: Android, *BSD, iOS, Linux, Mac, Solaris, VxWorks, 
    and Windows (Win32, WinCE, WinRT)
    are supported out of the box.  Easy to port to other systems.
Sources are in the public domain.
    Use for any purpose.
Comes with a standalone command-line interface
    (CLI) client that can be used to administer SQLite databases.



Suggested Uses For SQLite:


Database For The Internet Of Things.
SQLite is popular choice for the database engine in cellphones,
PDAs, MP3 players, set-top boxes, and other electronic gadgets.
SQLite has a small code footprint, makes efficient use of memory,
disk space, and disk bandwidth, is highly reliable, and requires
no maintenance from a Database Administrator.

Application File Format.
Rather than using fopen() to write XML, JSON, CSV,
or some proprietary format into
disk files used by your application, use an SQLite database.
You'll avoid having to write and troubleshoot a parser, your data
will be more easily accessible and cross-platform, and your updates
will be transactional.  (more...)

Website Database.
Because it requires no configuration and stores information in ordinary
disk files, SQLite is a popular choice as the database to back small
to medium-sized websites.

Stand-in For An Enterprise RDBMS.
SQLite is often used as a surrogate for an enterprise RDBMS for
demonstration purposes or for testing.  SQLite is fast and requires
no setup, which takes a lot of the hassle out of testing and which
makes demos perky and easy to launch.

More suggestions...

This page last modified on  2022-02-21 19:15:54 UTC


c2: Well-Known Users Of SQLite
c5: famous.html
c4:
Well-Known Users of SQLite


SQLite is used by literally millions of applications
with literally billions and billions of deployments.
SQLite is the most widely deployed database engine in the
world today.


A few of the better-known users of SQLite are shown below
in alphabetical order.
This is not a complete list.
SQLite is in the 
public domain and so most
developers use it in their projects without ever telling us.







  Adobe uses SQLite as the 
  application 
  file format for their 
  Photoshop Lightroom
  product.
  SQLite is also a standard part of the
  Adobe Integrated Runtime (AIR).
  It is reported that 
  Acrobat Reader
  also uses SQLite.






  Airbus confirms that SQLite is being
  used in the flight software for the 
  A350 XWB family
  of aircraft.






  Apple uses SQLite in many (most?) of the
  native applications running on Mac OS-X desktops and servers and
  on iOS devices such as iPhones and iPods.  SQLite is also used in
  iTunes, even on non-Apple hardware.






  Bentley Systems uses SQLite as the
  application file format for their
  Microstation
  CAD/CAM product.






  Bosch uses SQLite in the multimedia
  systems install on GM, Nissan, and Suzuki automobiles.
  (link)






  The increasingly popular Dropbox file
  archiving and synchronization service is reported to use SQLite as
  the primary data store on the client side.






  Expensify uses SQLite as a server-side
  database engine for their enterprise-scale expense reporting software.






  Facebook uses SQLite as the SQL database
  engine in their 
  osquery product.






  We believe that General Electric
  uses SQLite in some product or 
  another because they have written to the SQLite developers at 
  least four separate times requesting 
  the US Export Control Number for SQLite. So presumably GE is using
  SQLite in something that they are exporting. But nobody
  (outside of GE) seems to know what that might be.






  uses SQLite in their 
  Android cell-phone 
  operating system, and in the 
  Chrome Web Browser.






  Intuit apparently uses SQLite in 
  QuickBooks and in
  TurboTax to judge from some error
  reports from users seen 
  here and
  here.






  The United States Library of Congress recognizes SQLite as a
  recommended storage format for preservation of digital content.






  McAfee uses SQLite in its antivirus 
  programs.  
  Mentioned here and implied 
  here.






  Microsoft uses SQLite as a core component
  of Windows 10, and in other products.






  SQLite is the primary meta-data storage format for
  the Firefox Web Browser and the
  Thunderbird Email Reader from
  Mozilla.






  The Navigation Data Standard uses 
  SQLite as its application file format.






  The popular PHP
  programming language comes with both SQLite2 and SQLite3 built in.






  All Python
  distributions since Python 2.5 include SQLite.






  The RedHat Package Manager (RPM)
  uses SQLite to track its state.






  There are 
  multiple
  sightings
  of SQLite in the Skype client for Mac OS X and Windows.






  The Tcl/Tk programming language now comes with SQLite built-in.  SQLite works
  particularly well with Tcl since SQLite was originally a Tcl extension that
  subsequently "escaped" into the wild.






  SQLite comes bundled with the Xojo
  programming environment
  (formerly REALbasic)










  Adobe uses SQLite as the 
  application 
  file format for their 
  Photoshop Lightroom
  product.
  SQLite is also a standard part of the
  Adobe Integrated Runtime (AIR).
  It is reported that 
  Acrobat Reader
  also uses SQLite.






  Airbus confirms that SQLite is being
  used in the flight software for the 
  A350 XWB family
  of aircraft.






  Apple uses SQLite in many (most?) of the
  native applications running on Mac OS-X desktops and servers and
  on iOS devices such as iPhones and iPods.  SQLite is also used in
  iTunes, even on non-Apple hardware.






  Bentley Systems uses SQLite as the
  application file format for their
  Microstation
  CAD/CAM product.






  Bosch uses SQLite in the multimedia
  systems install on GM, Nissan, and Suzuki automobiles.
  (link)






  The increasingly popular Dropbox file
  archiving and synchronization service is reported to use SQLite as
  the primary data store on the client side.






  Expensify uses SQLite as a server-side
  database engine for their enterprise-scale expense reporting software.






  Facebook uses SQLite as the SQL database
  engine in their 
  osquery product.






  We believe that General Electric
  uses SQLite in some product or 
  another because they have written to the SQLite developers at 
  least four separate times requesting 
  the US Export Control Number for SQLite. So presumably GE is using
  SQLite in something that they are exporting. But nobody
  (outside of GE) seems to know what that might be.






  uses SQLite in their 
  Android cell-phone 
  operating system, and in the 
  Chrome Web Browser.






  Intuit apparently uses SQLite in 
  QuickBooks and in
  TurboTax to judge from some error
  reports from users seen 
  here and
  here.






  The United States Library of Congress recognizes SQLite as a
  recommended storage format for preservation of digital content.






  McAfee uses SQLite in its antivirus 
  programs.  
  Mentioned here and implied 
  here.






  Microsoft uses SQLite as a core component
  of Windows 10, and in other products.






  SQLite is the primary meta-data storage format for
  the Firefox Web Browser and the
  Thunderbird Email Reader from
  Mozilla.






  The Navigation Data Standard uses 
  SQLite as its application file format.






  The popular PHP
  programming language comes with both SQLite2 and SQLite3 built in.






  All Python
  distributions since Python 2.5 include SQLite.






  The RedHat Package Manager (RPM)
  uses SQLite to track its state.






  There are 
  multiple
  sightings
  of SQLite in the Skype client for Mac OS X and Windows.






  The Tcl/Tk programming language now comes with SQLite built-in.  SQLite works
  particularly well with Tcl since SQLite was originally a Tcl extension that
  subsequently "escaped" into the wild.






  SQLite comes bundled with the Xojo
  programming environment
  (formerly REALbasic)




This page last modified on  2022-06-18 09:43:52 UTC


c2: NUL Characters In Strings
c5: nulinstr.html
c4:
1. Introduction


SQLite allows NUL characters (ASCII 0x00, Unicode \u0000) in the middle
of string values stored in the database.  However, the use of NUL within
strings can lead to surprising behaviors:



The length() SQL function only counts characters up to and excluding
the first NUL.



The quote() SQL function only shows characters up to and excluding
the first NUL.


The .dump command in the CLI omits the first NUL character and all
subsequent text in the SQL output that it generates.  In fact, the
CLI omits everything past the first NUL character in all contexts.



The use of NUL characters in SQL text strings is not recommended.

2. Unexpected Behavior


Consider the following SQL:

CREATE TABLE t1(
  a INTEGER PRIMARY KEY,
  b TEXT
);
INSERT INTO t1(a,b) VALUES(1, 'abc'||char(0)||'xyz');

SELECT a, b, length(b) FROM t1;



The SELECT statement above shows output of:

1,'abc',3



(Through this document, we assume that the CLI has ".mode quote" set.)
But if you run:

SELECT * FROM t1 WHERE b='abc';



Then no rows are returned.  SQLite knows that the t1.b column actually
holds a 7-character string, and the 7-character string 'abc'||char(0)||'xyz'
is not equal to the 3-character string 'abc', and so no rows are returned.
But a user might be easily confused by this because the CLI output
seems to show that the string has only 3 characters.  This seems like
a bug.  But it is how SQLite works.

3. How To Tell If You Have NUL Characters In Your Strings


If you CAST a string into a BLOB, then the entire length of the
string is shown.  For example:

SELECT a, CAST(b AS BLOB) FROM t1;



Gives this result:

1,X'6162630078797a'



In the BLOB output, you can clearly see the NUL character as the 4th
character in the 7-character string.


Another, more automated, way
to tell if a string value X contains embedded NUL characters is to
use an expression like this:

instr(X,char(0))



If this expression returns a non-zero value N, then there exists an 
embedded NUL at the N-th character position.  Thus to count the number
of rows that contain embedded NUL characters:

SELECT count(*) FROM t1 WHERE instr(b,char(0))>0;


4. Removing NUL Characters From A Text Field


The following example shows how to remove NUL character, and all text
that follows, from a column of a table.  So if you have a database file
that contains embedded NULs and you would like to remove them, running
UPDATE statements similar to the following might help:

UPDATE t1 SET b=substr(b,1,instr(b,char(0)))
 WHERE instr(b,char(0));

This page last modified on  2022-05-23 22:21:54 UTC


c2: History Of SQLite Releases
c5: chronology.html
c4:
History Of SQLite Releases




DateVersion


2022-12-28
3.40.1
2022-11-16
3.40.0
2022-09-29
3.39.4
2022-09-05
3.39.3
2022-07-21
3.39.2
2022-07-13
3.39.1
2022-06-25
3.39.0
2022-05-06
3.38.5
2022-05-04
3.38.4
2022-04-27
3.38.3
2022-03-26
3.38.2
2022-03-12
3.38.1
2022-02-22
3.38.0
2022-01-06
3.37.2
2021-12-30
3.37.1
2021-11-27
3.37.0
2021-06-18
3.36.0
2021-04-19
3.35.5
2021-04-02
3.35.4
2021-03-26
3.35.3
2021-03-17
3.35.2
2021-03-15
3.35.1
2021-03-12
3.35.0
2021-01-20
3.34.1
2020-12-01
3.34.0
2020-08-14
3.33.0
2020-06-18
3.32.3
2020-06-04
3.32.2
2020-05-25
3.32.1
2020-05-22
3.32.0
2020-01-27
3.31.1
2020-01-22
3.31.0
2019-10-10
3.30.1
2019-10-04
3.30.0
2019-07-10
3.29.0
2019-04-16
3.28.0
2019-02-25
3.27.2
2019-02-08
3.27.1
2019-02-07
3.27.0
2018-12-01
3.26.0
2018-11-05
3.25.3
2018-09-25
3.25.2
2018-09-18
3.25.1
2018-09-15
3.25.0
2018-06-04
3.24.0
2018-04-10
3.23.1
2018-04-02
3.23.0
2018-01-22
3.22.0
2017-10-24
3.21.0
2017-08-24
3.20.1
2017-08-01
3.20.0
2017-06-17
3.18.2
2017-06-16
3.18.1
2017-06-08
3.19.3
2017-05-25
3.19.2
2017-05-24
3.19.1
2017-05-22
3.19.0
2017-03-28
3.18.0
2017-02-13
3.17.0
2017-01-06
3.16.2
2017-01-03
3.16.1
2017-01-02
3.16.0
2016-11-28
3.15.2
2016-11-04
3.15.1
2016-10-14
3.15.0
2016-09-12
3.14.2
2016-08-11
3.14.1
2016-08-08
3.14
2016-05-18
3.13.0
2016-04-18
3.12.2
2016-04-08
3.12.1
2016-03-31
3.9.3
2016-03-29
3.12.0
2016-03-03
3.11.1
2016-02-15
3.11.0
2016-01-20
3.10.2
2016-01-14
3.10.1
2016-01-06
3.10.0
2015-11-02
3.9.2
2015-10-16
3.9.1
2015-10-14
3.9.0
2015-07-29
3.8.11.1
2015-07-27
3.8.11
2015-05-20
3.8.10.2
2015-05-09
3.8.10.1
2015-05-07
3.8.10
2015-04-08
3.8.9
2015-02-25
3.8.8.3
2015-01-30
3.8.8.2
2015-01-20
3.8.8.1
2015-01-16
3.8.8
2014-12-09
3.8.7.4
2014-12-05
3.8.7.3
2014-11-18
3.8.7.2
2014-10-29
3.8.7.1
2014-10-22
3.8.6.1
2014-10-17
3.8.7
2014-08-15
3.8.6
2014-06-04
3.8.5
2014-04-03
3.8.4.3
2014-03-26
3.8.4.2
2014-03-11
3.8.4.1
2014-03-10
3.8.4
2014-02-11
3.8.3.1
2014-02-03
3.8.3
2013-12-06
3.8.2
2013-10-17
3.8.1
2013-09-03
3.8.0.2
2013-08-29
3.8.0.1
2013-08-26
3.8.0
2013-05-20
3.7.17
2013-04-12
3.7.16.2
2013-03-29
3.7.16.1
2013-03-18
3.7.16
2013-01-09
3.7.15.2
2012-12-19
3.7.15.1
2012-12-12
3.7.15
2012-10-04
3.7.14.1
2012-09-03
3.7.14
2012-06-11
3.7.13
2012-05-22
3.7.12.1
2012-05-14
3.7.12
2012-05-14
3.7.12
2012-05-14
3.7.12
2012-03-20
3.7.11
2012-01-16
3.7.10
2011-11-01
3.7.9
2011-09-19
3.7.8
2011-06-28
3.7.7.1
2011-06-23
3.7.7
2011-05-19
3.7.6.3
2011-04-17
3.7.6.2
2011-04-13
3.7.6.1
2011-04-12
3.7.6
2011-02-01
3.7.5
2010-12-07
3.7.4
2010-10-08
3.7.3
2010-08-24
3.7.2
2010-08-23
3.7.1
2010-08-04
3.7.0.1
2010-07-21
3.7.0
2010-03-26
3.6.23.1
2010-03-09
3.6.23
2010-01-06
3.6.22
2009-12-07
3.6.21
2009-11-04
3.6.20
2009-10-30
3.6.16.1
2009-10-14
3.6.19
2009-09-11
3.6.18
2009-08-10
3.6.17
2009-06-27
3.6.16
2009-06-15
3.6.15
2009-05-25
3.6.14.2
2009-05-19
3.6.14.1
2009-05-07
3.6.14
2009-04-13
3.6.13
2009-03-31
3.6.12
2009-02-18
3.6.11
2009-01-15
3.6.10
2009-01-14
3.6.9
2009-01-12
3.6.8
2008-12-16
3.6.7
2008-11-26
3.6.6.2
2008-11-22
3.6.6.1
2008-11-19
3.6.6
2008-11-12
3.6.5
2008-10-15
3.6.4
2008-09-22
3.6.3
2008-08-30
3.6.2
2008-08-06
3.6.1
2008-07-16
3.6.0
2008-05-14
3.5.9
2008-04-16
3.5.8
2008-03-17
3.5.7
2008-02-06
3.5.6
2008-01-31
3.5.5
2007-12-14
3.5.4
2007-11-27
3.5.3
2007-11-05
3.5.2
2007-10-04
3.5.1
2007-09-04
3.5.0
2007-08-13
3.4.2
2007-07-20
3.4.1
2007-06-18
3.4.0
2007-04-25
3.3.17
2007-04-18
3.3.16
2007-04-09
3.3.15
2007-04-02
3.3.14
2007-02-13
3.3.13
2007-01-27
3.3.12
2007-01-22
3.3.11
2007-01-10
3.3.10
2007-01-04
3.3.9
2006-10-09
3.3.8
2006-08-12
3.3.7
2006-06-06
3.3.6
2006-04-05
3.3.5
2006-02-11
3.3.4
2006-01-31
3.3.3
2006-01-24
3.3.2
2006-01-16
3.3.1
2006-01-11
3.3.0
2005-12-19
2.8.17
2005-12-19
3.2.8
2005-09-24
3.2.7
2005-09-17
3.2.6
2005-08-27
3.2.5
2005-08-24
3.2.4
2005-08-21
3.2.3
2005-06-12
3.2.2
2005-03-29
3.2.1
2005-03-21
3.2.0
2005-03-17
3.1.6
2005-03-11
3.1.5
2005-03-11
3.1.4
2005-02-28
3.1.3.1
2005-02-20
3.1.3
2005-02-15
3.1.2
2005-02-15
2.8.16
2005-02-01
3.1.1
2005-01-21
3.1.0
2004-10-12
3.0.8
2004-09-18
3.0.7
2004-09-02
3.0.6
2004-08-29
3.0.5
2004-08-09
3.0.4
2004-07-22
3.0.3
2004-07-22
2.8.15
2004-06-30
3.0.2
2004-06-22
3.0.1
2004-06-18
3.0.0
2004-06-09
2.8.14
2004-03-08
2.8.13
2004-02-08
2.8.12
2004-01-14
2.8.11
2004-01-14
2.8.10
2004-01-06
2.8.9
2003-12-18
2.8.8
2003-12-04
2.8.7
2003-08-22
2.8.6
2003-07-22
2.8.5
2003-06-29
2.8.4
2003-06-04
2.8.3
2003-05-17
2.8.2
2003-05-17
2.8.1
2003-02-16
2.8.0
2003-01-25
2.7.6
2002-12-28
2.7.5
2002-12-17
2.7.4
2002-10-31
2.7.3
2002-09-25
2.7.2
2002-08-31
2.7.1
2002-08-25
2.7.0
2002-08-13
2.6.3
2002-07-31
2.6.2
2002-07-19
2.6.1
2002-07-18
2.6.0
2002-07-07
2.5.6
2002-07-06
2.5.5
2002-07-01
2.5.4
2002-06-25
2.5.3
2002-06-25
2.5.2
2002-06-19
2.5.1
2002-06-17
2.5.0
2002-05-10
2.4.12
2002-05-08
2.4.11
2002-05-03
2.4.10
2002-04-22
2.4.9
2002-04-20
2.4.8
2002-04-12
2.4.7
2002-04-02
2.4.6
2002-04-02
2.4.5
2002-03-30
2.4.4
2002-03-23
2.4.3
2002-03-20
2.4.2
2002-03-13
2.4.1
2002-03-13
2.4.1
2002-03-11
2.4.0
2002-02-19
2.3.3
2002-02-14
2.3.2
2002-02-13
2.3.1
2002-02-03
2.3.0
2002-01-28
2.2.5
2002-01-22
2.2.4
2002-01-16
2.2.3
2002-01-14
2.2.2
2002-01-09
2.2.1
2001-12-22
2.2.0
2001-12-15
2.1.7
2001-12-14
2.1.6
2001-12-06
2.1.5
2001-12-05
2.1.4
2001-11-24
2.1.3
2001-11-23
2.1.2
2001-11-13
2.1.1
2001-11-12
2.1.0
2001-11-04
2.0.8
2001-10-22
2.0.7
2001-10-19
2.0.6
2001-10-15
2.0.5
2001-10-13
2.0.4
2001-10-13
2.0.3
2001-10-09
2.0.2
2001-10-02
2.0.1
2001-09-28
2.0.0
2001-07-23
1.0.32
2001-04-15
1.0.31
2001-04-06
1.0.30
2001-04-05
1.0.29
2001-04-04
1.0.28
2001-03-20
1.0.27
2001-03-20
1.0.26
2001-03-15
1.0.25
2001-03-14
1.0.24
2001-02-20
1.0.23
2001-02-19
1.0.22
2001-02-19
1.0.21
2001-02-11
1.0.20
2001-02-06
1.0.19
2001-01-04
1.0.18
2000-12-10
1.0.17
2000-11-28
1.0.16
2000-10-23
1.0.15
2000-10-19
1.0.14
2000-10-19
1.0.13
2000-10-17
1.0.12
2000-10-11
1.0.10
2000-10-09
1.0.9
2000-09-30
1.0.8
2000-09-14
1.0.5
2000-08-28
1.0.4
2000-08-22
1.0.3
2000-08-18
1.0.1
2000-08-17
1.0






334 releases
over 22.4 years.

This page last modified on  2022-12-28 14:21:19 UTC


c2: The SQLITE_STMT Virtual Table
c5: stmt.html
c4:
1. Overview


The SQLITE_STMT extension implements an eponymous-only virtual table that
provides information about all prepared statements associated with
the database connection.



The SQLITE_STMT extension is included in the amalgamation though 
it is disabled
by default.  Use the SQLITE_ENABLE_STMTVTAB compile-time option to enable
the SQLITE_STMT extension.  The SQLITE_STMT extension can also be 
loaded at run-time
by compiling the extension into a shared library or DLL using the source
code at https://sqlite.org/src/file/ext/misc/stmt.c and following the
instructions for how to compile loadable extensions.



The SQLITE_STMT extension is enabled in default builds
of the command-line shell.

2. Usage


The SQLITE_STMT virtual table is a read-only table that can be directly
queried to access information about all prepared statements on the 
current database connection.  For example:

SELECT * FROM sqlite_stmt;



A statement such as the above can be run immediately prior to invoking
sqlite3_close() to confirm that all prepared statements have been 
finalized and to help identify and track down prepared
statements that have "leaked" and missed finalization.


The SQLITE_STMT virtual table can also be used to access performance
information about prepared statements, to aid in optimization an application.
For example,
to find out how much memory is being used by prepared statements that have
never been used, one could run:

SELECT sum(mem) FROM sqlite_stmt WHERE run=0;


2.1. Columns


The columns are provided by the SQLITE_STMT virtual table are summarized by
the hypothetical CREATE TABLE statement show here:

CREATE TABLE sqlite_stmt(
  sql    TEXT,    -- Original SQL text
  ncol   INT,     -- Number of output columns
  ro     BOOLEAN, -- True for "read only" statements
  busy   BOOLEAN, -- True if the statement is current running
  nscan  INT,     -- Number of full-scan steps
  nsort  INT,     -- Number of sort operations
  naidx  INT,     -- Number of automatic index inserts
  nstep  INT,     -- Number of byte-code engine steps
  reprep INT,     -- Number of reprepare operations
  run    INT,     -- Number of times this statement has been run
  mem    INT      -- Heap memory used by this statement
);


Future releases may add new output columns and may change the order
of legacy columns.
Further detail about the meaning of each column in the SQLITE_STMT virtual
table is provided below:


sql:
The original SQL text of the prepared statement.  If the prepared
statement is compiled using the sqlite3_prepare() interface, then
the SQL text might not have been saved, in which case this column
will be NULL.

ncol:
The number of columns in the result set of a query.
For DML statements, this column has a value of 0.

ro:
The "read only" column.  This column is true (non-zero) if the
SQL statement is a query and false (zero) if it is a DML statement.

busy:
This field is true if the prepared statement is currently running.
In other words, this field is true if sqlite3_step() has been called
on the prepared statement at least once but sqlite3_reset() has
not yet been called to reset it.

nscan:
This field is the number of times that the bytecode engine has stepped
through a table as part of a full-table scan.  A large number if this
field may indicate an opportunity to improve performance by adding an
index.  This field is equivalent to the SQLITE_STMTSTATUS_FULLSCAN_STEP
value.

nsort:
This field is the number of times that the bytecode engine had to sort.
A positive value in this field may indicate an opportunity to improve
performance by adding an index that will cause the query results to
appear naturally in the desired order.  
This field is equivalent to the SQLITE_STMTSTATUS_SORT value.

naidx:
This field is the number of rows that have been inserted into
automatic indexes.  A positive value in this field may indicate 
an opportunity to improve performance by adding a named index that
take the place of the automatic index.
This field is equivalent to the SQLITE_STMTSTATUS_AUTOINDEX value.

nstep:
This field is the number of bytecode engine operations that have
been performed for the prepared statement.  This field can be used
as a proxy for how much CPU time a statement has used.
This field is equivalent to the SQLITE_STMTSTATUS_VM_STEP value.

reprep:
This field is the number of times that the statement has had to be
reprepared due to schema changes or changes to parameter bindings.
This field is equivalent to the SQLITE_STMTSTATUS_REPREPARE value.

run:
This field is the number of times that the statement has been run.
This field is equivalent to the SQLITE_STMTSTATUS_RUN value.

mem:
This field is the number of bytes of heap storage used by the
prepared statement.
This field is equivalent to the SQLITE_STMTSTATUS_MEMUSED value.

This page last modified on  2017-07-14 15:38:02 UTC


c2: The COMPLETION() Table-Valued Function
c5: completion.html
c4:
1. Overview

The COMPLETION extension implements a table-valued function named
"completion" that can be used to suggest completions of partially entered
words during interactive SQL input.  The completion table can be
used to help implement tab-completion, for example.

2. Details

The designed query interface is:

SELECT DISTINCT candidate COLLATE nocase
  FROM completion($prefix, $wholeline)
 ORDER BY 1;


The query above will return suggestions for the whole input word that
begins with $prefix.  The $wholeline parameter is all text from the beginning
of the line up to the insertion point.  The $wholeline parameter is used
for context.

The $prefix parameter may be NULL, in which case the prefix is deduced
from $wholeline.  Or, the $wholeline parameter may be NULL or omitted if 
context information is unavailable or if context-aware completion is not
desired.

The completion table might return the same candidate more than once, and
it will return candidates in an arbitrary order.  The DISTINCT keyword and
the ORDER BY in the sample query above are added to make the answers unique
and in lexicographical order.

2.1. Example Usage

The completion table is used to implement tab-completion in the
command-line shell in conjunction with either the readline or linenoise
input line editing packages for unix.  See the
https://sqlite.org/src/file/src/shell.c.in source file for example
code.  Search for "FROM completion" to find the relevant code sections.

Because the completion table is built into the command-line shell in order
to provide for tab-completions, you can run test queries against the
completion table directly in the command-line shell.  Simply type a
query such as the example shown above, filling in appropriate values
for $prefix and $wholeline, and observe the output.

3. Limitations

The completion table is designed for interactive use.  It will return
answers at a speed appropriate for human typing.  No effort is made to
be unusually efficient, so long as the response time is nearly instantaneous
in a user interface.

As of this writing (2017-07-13), the completion virtual table only
looks for SQL keywords, and schema, table, and column names.  The
context contained in $wholeline is completely ignored.  Future enhancements
will try to return new completions taken from function and pragma names
and other sources, as well as consider more context.  The completion
table should be considered a work-in-progress.
This page last modified on  2017-07-14 03:53:46 UTC


c2: SQLite is a Self Contained System
c5: selfcontained.html
c4:
SQLite is "stand-alone" or "self-contained" in the sense that it
has very few dependencies.  It runs on any operating system, even
stripped-down bare-bones embedded operating systems.  SQLite uses
no external libraries or interfaces (other than a few standard C-library 
calls described below).  The entire SQLite library is
encapsulated in a single source code file that requires
no special facilities or tools to build.


A minimal build of SQLite requires just these routines from the
standard C library:

memcmp()
memcpy()
memmove()
memset()
strcmp()
strlen()
strncmp()



Most builds also use the system memory allocation routines:

malloc()
realloc()
free()

But those routines are optional and can be omitted
using a compile-time option.


Default builds of SQLite contain appropriate VFS objects for talking
to underlying operating system, and those VFS objects will contain
operating system calls such as open(), read(), write(), fsync(), and
so forth.  All of these interfaces are readily available on most platforms,
and custom VFSes can be designed to run SQLite on even the most
austere embedded devices.
This page last modified on  2022-08-10 18:45:48 UTC


c2: Books About SQLite
c5: books.html
c4:
Books About SQLite




SQLite Forensics (2018)


Author: Paul Sanderson
Publisher: Amazon


This text by noted digital forensics expert, Paul Sanderson, provides
investigators with low-level technical details useful in analysing
SQLite database files.


Every computer and phone uses hundreds of SQLite databases and there are
over one trillion SQLite databases in active use. Hence, the importance
of examining the data held in these databases in an
investigation, including deleted data when possible, is paramount. This
book fully explains the format of the SQLite database file. It shows how
records are encoded, how to decode them manually and how to decode
records that are partially overwritten. It also describe how the
workings of SQLite, and in particular the journal and WAL, can be used
to ascertain what has happened in a manner that cannot be determined
from the data alone. The book covers basic SQL queries and how they can
be used to create a custom report that includes data from different
tables, and shows how one can use SQL queries to test hypotheses about
the relationships of data in different tables. 

This book is aimed mainly at
forensic practitioners, and it is assumed that the reader has some basic
knowledge of computer forensics; it will also be of interest to computer
professionals in general particularly those who have an interest in the
SQLite file format.






Learning SQLite for iOS (2016)


Author: Gene Da Rocha
Publisher: Packt Publishing


This book starts with the architecture of SQLite database and introduces
you to concepts in SQL. You will find yourself equipped to design your
own database system, administer it, and maintain it. Further, you will
learn how to operate your SQLite databases smoothly using SQL commands. 


You will be able to extend the functionality of SQLite by using its vast
arsenal of C API calls to build some interesting, exciting, new, and
intelligent data-driven applications. Understand how Xcode, HTML5, and
Phonegap can be used to build a cross-platform modern app which can
benefit from all these technologies - all through creating a complete,
customizable application skeleton that you can build on for your own
apps. This book provides a comprehensive description of SQLite database
system. It describes design principles, engineering trade-offs,
implementation issues, and operations of SQLite.





SQLite Database System Design and Implementation (2015)


Author: Sibsankar Haldar
Publisher: https://books.google.com/


This book provides a comprehensive description of SQLite database system. 
It describes design principles, engineering trade-offs, implementation issues,
and operations of SQLite. 





Android SQLite Essentials (2014)


Authors: Sunny Kumar Aditya and Vikash Kumar Karn
Publisher: 
Packt Publishing


Android SQLite Essentials focuses on the core concepts behind building
database-driven applications. This book covers the basic and advanced
topics with equivalent simplicity and detail, in order to enable
readers to quickly grasp and implement the concepts to build an
application database.

This book takes a hands-on, example-based approach to help readers
understand the core topics of SQLite and Android database-driven
applications. This book focuses on providing you with latent as well
as widespread knowledge about practices and approaches towards
development in an easily understandable manner.





SQLite for Mobile Apps Simplified (2014)


Author: Sribatsa Das
Publisher: Amazon
Amazon


SQLite for Mobile Apps Simplified is devoted to presenting approach
and implementation methodology for using SQLite database in mobile apps.
It presents step-by-step examples to create schema, execute transactions
and access data from Android, BlackBerry and iOS applications.
In addition, it presents ADB Shell and SQLite command-line shell from
ADB Shell to access the SQLite Database created by the Android apps.
For BlackBerry and iOS application, the book presents ways to access
the data using the command line shell. 





The Definitive Guide to SQLite (2nd edition, 2010)


Authors: Mike Owens and Grant Allen
Publisher: Apress
Amazon


Outside of the world of enterprise computing, there is one database 
that enables a huge range of software and hardware to flex relational 
database capabilities, without the baggage and cost of traditional 
database management systems. That database is SQLite - an embeddable
database with an amazingly small footprint, yet able to handle databases 
of enormous size. SQLite comes equipped with an array of powerful 
features available through a host of programming and development 
environments. It is supported by languages such as C, Java, Perl,
PHP, Python, Ruby, TCL, and more.

The Definitive Guide to SQLite, Second Edition 
is devoted to complete coverage of the latest version of this powerful 
database. It offers a thorough overview of SQLite's capabilities 
and APIs. The book also uses SQLite as the basis for helping newcomers
make their first foray into database development. In only a short time 
you can be writing programs as diverse as a server-side browser plug-in 
or the next great iPhone or Android application! 






Using SQLite (2010)


Author: Jay A. Kreibich
Publisher: O'Reilly Media
O'Reilly

Developers, take note: databases aren't just for the IS group any more.
You can build database-backed applications for the desktop, Web,
embedded systems, or operating systems without linking to heavy-duty
client-server databases such as Oracle and MySQL. 
This book shows how you to use SQLite, a small and lightweight 
database that you can build right into your application during development.
Applications that handle data have an enormous advantage today, and
with SQLite, you'll discover how to develop a database-backed application
that remains manageable in size and complexity. This book guides
you every step of the way. You'll get a crash course in data modeling, 
become familiar with SQLite's dialect of the SQL database language, 
and learn how you to work with SQLite using either a scripting 
language or a C-based language, such as C# or Objective C.Now, 
even relatively small and nimble applications can be a part of 
the data revolution. Using SQLite shows you how.






SQLite 3 - Einstieg in die Datenbankwelt (2010)


Author: Key Droessler
Publisher: Lulu.com
Amazon

Die Datenbanksprache SQL ( Structured Query Language ) wird in Datenbanken
zur Definition, Manipulation, Sicherung, aber hauptsaechlich zur Abfrage 
von Daten aus der Datenbank eingesetzt. Unabhaengig vom Betriebssystem oder 
aufwendigen, benutzerfreundlichen, graphischen Oberflaechen bleibt die 
Logik aber immer gleich.SQLite ist eine freie Desktop-Datenbank, 
sie kostet nichts, ist fuer viele Betriebssysteme verfuegbar, 
schnell heruntergeladen und installiert und auf das Notwendigste 
reduziert. Fuer den Einsteiger sind das die besten Voraussetzungen, 
ohne viel Aufwand schnell in die Welt der Datenbanken und Datenbanksprache 
reinzuschnuppern.Wer nach den Uebungen aber auf den Geschmack gekommen ist,
hat schon den groessten Teil an Datenbanken und SQL gelernt, denn alles 
Besprochene ist Wissen, welches auch auf jedes andere der vielen 
Datenbanken grundlegend anwendbar ist. Nun koennen Sie auf die richtig 
Grossen zugehen, vom grossen Fachbuch bis zum riesigen Datenbanksystem.






Inside Symbian SQL (2010)

Authors: Ivan Litovski &amp; Richard Maynard
Publisher: Wiley
wiley.com


This is the definitive reference book on the Symbian SQL database
which is based on SQLite. The authors (both members of the Symbian
engineering team responsible for the implementation of the code)
show you how to design code
and ease migration from an internal and external point of view, 
plus they reveal the dos and don'ts of writing high-performance 
database applications. Packed with resources and sample code, 
this timely book reveals how to design and tune applications 
that use the Symbian SQL framework to ultimately improve performance.

With its sample code and insider expertise, this text has everything 
you need to keep you ahead of the curve. 






The SQL Guide to SQLite (2009)

Author: Rick F. van der Lans
Publisher: Lulu.com
Amazon

SQLite is a small, fast, embeddable, SQL-based database server.
It is easy to install, needs no management, and is open source.
This book describes SQLite in detail. With hundreds of examples, plus a
proven approach and structure, the book teaches you how to use SQLite
efficiently and effectively. It contains a complete description of the 
SQL dialect as implemented in SQLite version 3.6. The book can be seen 
as a tutorial and a reference book. Source code for the numerous SQL 
examples and exercises included in this book can be downloaded from www.r20.nl.






An Introduction to SQLite - 2nd Edition (2009)

Author: Naoki Nishizawa
Publisher: Shoeisha
Amazon.jp

This text is written in fluent Japanese specifically for a Japanese
audience.  This is the second edition of the book - the first edition
was published in 2005.  






Inside SQLite (2007)


Author: Sibsankar Haldar
Publisher: O'Reilly Media
O'Reilly

SQLite is a small, zero-configuration, custom-tailored, embeddable, 
thread-safe, easily maintainable, transaction-oriented, SQL-based,
relational database management system. There is no separate install or 
setup procedure to initialize SQLite before using it. 
There is no configuration file. 
SQLite is open source, and is available in the public domain 
(for more information on open source, visit http://opensource.org). 
You can download SQLite source code from its homepage http://www.sqlite.org, 
compile it using your favorite C compiler, and start using the compiled
library. SQLite runs on Linux, Windows, Mac OS X, and a few other operating systems. It has been widely used in low-to-medium tier database applications. 
This Short Cut discusses design principles, engineering trade-offs, 
implementation issues, and operations of SQLite. It presents a 
comprehensive description of all important components of the SQLite engine.






SQLite (2004)

Author: Chris Newman
Publisher: Sams
Amazon


SQLite is a small, fast, embeddable database. What makes it popular is 
the combination of the database engine and interface into a single library 
as well as the ability to store all the data in a single file.
Its functionality lies between MySQL and PostgreSQL, however it is faster 
than both databases.

In SQLite, 
author Chris Newman provides a thorough, practical guide to 
using, administering and programming this up-and-coming database. 
If you want to learn about SQLite or about its use in conjunction with 
PHP this is the book for you.

This page last modified on  2019-09-28 19:02:24 UTC


c2: Imposter Tables
c5: imposter.html
c4:
1. Introduction


An imposter table is a table that is attached to the same b-tree
as an index.
An imposter table allows the content of an index to be queried or
modified as if the index were an ordinary table.



Imposter tables are intended for analysis and debugging only.
This is not a feature that most application developers should understand
or even know about.
Imposter tables are for experts only.



Improper use of imposter tables can cause index corruption, though
any corruption created this way can be fixed by running REINDEX.


2. Details


Each table and each index in SQLite is stored in a separate b-tree
in the database file.  Each b-tree is identified by its root page
number.  The root page number for any index or table can be found
by querying the "rootpage" column of the sqlite_schema table.
See the indexing tutorial and the file format documentation
for further background on this design.



Usually the b-trees for tables and indexes are slightly different.
A table b-tree contains a 64-bit integer key and arbitrary data.
The 64-bit integer key is the ROWID.  Index b-trees contain
an arbitrary binary key and no data.  So table b-trees and index
b-trees are not directly compatible.



However, the b-tree for a WITHOUT ROWID table is in the same format
as an index b-tree.  Thus, an index b-tree can be accessed as if it
were a WITHOUT ROWID table.


2.1. Manually Created Imposter Tables


One way to create an imposter table is to directly edit the sqlite_schema
table to insert a new row that describes the table.
For example, suppose the schema is like this:


CREATE TABLE t1(a INTEGER PRIMARY KEY,b TEXT,c INT, d INT);
CREATE INDEX t1bc ON t1(b,c);



The WITHOUT ROWID table that has the same structure as the t1bc index
would look like this:


CREATE TABLE t2(b TEXT,c INT,a INT, PRIMARY KEY(b,c,a)) WITHOUT ROWID;



To create a permanent imposter table "t2" against index "t1bc" one
should first enable editing of the sqlite_schema table by running
"PRAGMA writable_schema=ON".
(Be careful to observe the warnings that accompany this PRAGMA.
A mistake can cause severe database corruption.)
Then insert a new entry into the sqlite_schema table like this:


INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)
 SELECT 'table','t2','t2',rootpage,
   'CREATE TABLE t2(b,c,a,PRIMARY KEY(b,c,a))WITHOUT ROWID'
   FROM sqlite_schema
  WHERE name='t1bc';



The INSERT statement above adds a new row to the sqlite_schema table that
defines a table "t2" having the same on-disk format as index "t1bc" and 
pointing to the same b-tree.
After adding this sqlite_schema table entry, it is necessary to close and
reopen the database in order to get SQLite to reread the schema.
Then the "t2" table can be queried to see the content of the "t1bc" index.


2.1.1. Corrupted Database


A serious problem with the manual imposter table approach described above
is that after adding the new "t2" entry to the "sqlite_schema" table, the
database file will technically be corrupt.  Both the "t1bc" index and the
"t2" table will point to the same b-tree.  This will not cause
any immediate problems, though one should avoid running VACUUM.



It is possible to write into the "t2" table, thus changing the content
of the index.
But doing so will get the "t1bc" index out of synchronization with its
parent table "t1".  An out-of-sync index can result in incorrect query
results.



Since the "t2" imposter table is a form of database corruption, the
manual approach to creating imposter tables is not recommended.
Actually, any use of imposter tables is discouraged for all but
expert developers, but manually created imposter tables are
especially discouraged because they are permanent.


2.2. Transient Imposter Tables


Another (safer) approach to creating an imposter table is to add an
entry for the imposter table to SQLite's internal symbol table without
updating the "sqlite_schema" table on disk.
That way, the imposter table exists in only a single database connection
and is automatically removed whenever the schema is reloaded.



Creation of a transient imposter table involves a special
sqlite3_test_control() call.  Unlike all other SQLite APIs,
sqlite3_test_control() interface is subject to incompatible changes
from one release to the next, and so the mechanism described below
is not guaranteed to work in future releases of SQLite.  The
SQLite developers do not consider this a problem because imposter
tables should not be used in applications.  Imposter tables are for
analysis and testing use only.



To create a transient imposter table, first call sqlite3_test_control()
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 1, tnum);



The "db" parameter is a pointer to the database connection.
The "main" argument is the name of the schema in which the imposter
table is to be created.  The "1" argument enables the imposter table
mechanism.  "tnum" is the root page of the index that the imposter
table should mirror.



After the sqlite3_test_control() call above, then run a CREATE TABLE
statement the defines the imposter table.
With the imposter mechanism enabled, this CREATE TABLE statement does
not create a real table but instead merely adds an entry in SQLite's
internal symbol table.  Note that the CREATE TABLE statement must
be in the correct format for the index.  If the imposter table has the
wrong number of columns or is not a WITHOUT ROWID table or is otherwise
incompatible with the index b-tree, then SQLITE_CORRUPT errors will result
when the imposter table is used.



After running the CREATE TABLE statement, disable the imposter mechanism
as follows:


sqlite3_test_control(SQLITE_TESTCTRL_IMPOSTER, db, "main", 0, 0);



In other words, make the same sqlite3_test_control() call except change
the last two parameters to zero.



After the imposter table is loaded into SQLite's internal schema as
described above, the imposter table can be used as any other table.
But the imposter table will only be visible to the one database
connection that created it.  No changes are made to the database file
on disk.  And the imposter table will disappear the next time the schema 
is loaded.




2.3. The .imposter Shell Command


As of SQLite 3.16.0 (2017-01-02), the command-line shell contains
a dot-command ".imposter" that does all of the work of setting up a
transient imposter table.
Instead of making multiple calls to sqlite3_test_control() and figuring
out and invoking a compatible CREATE TABLE statement, a transient
imposter table can be constructed as follows:


.imposter t1bc t2



Of course, substitute the desired index and imposter table names in
place of the "t1bc" and "t2" shown in the example.
The ".imposter" command reads the schema of the "t1bc" index, uses
that information to construct a compatible CREATE TABLE statement for
the imposter table, then makes all the necessary calls to create the
transient imposter table automatically.


3. Summary And Final Warning


The imposter table mechanism is a power analysis and debugging tool
for SQLite.  But as with all sharp tools, it can also be dangerous and
can result in corrupt database files if misused.  Do not attempt to
use imposter tables in an application.  Imposter tables are intended
for use in the laboratory by experts.

This page last modified on  2020-06-18 21:18:56 UTC


c2: The CSV Virtual Table
c5: csv.html
c4:
1. Overview


The CSV virtual table reads
RFC 4180 formatted comma-separated
values, and returns that content as if it were rows and columns of an SQL
table.



The CSV virtual table is useful to applications that need to bulk-load
large amounts of comma-separated value content.
The CSV virtual table is also useful as a template source file for
implementing other virtual tables.




The CSV virtual table is not built into the SQLite amalgamation.
It is available as a
separate source file
that can be compiled into a loadable extension.
Typical usage of the CSV virtual table from the
command-line shell would be something like this:

.load ./csv
CREATE VIRTUAL TABLE temp.t1 USING csv(filename='thefile.csv');
SELECT * FROM t1;



The first line of the script above causes the command-line shell to
read and activate the run-time loadable extension for CSV.  For an
application, the equivalent C-language API is
sqlite3_load_extension().
Observe that the filename extension (ex: ".dll" or ".so" or ".dylib") is
omitted from the extension filename.  Omitting the filename extension is
not required, but it helps in making the script cross-platform.  SQLite
will automatically append the appropriate extension.


The second line above creates a virtual table named "t1" that reads
the content of the file named in the argument.  The number and names of
the columns is determined automatically by reading the first line of
content.  Other options to the CSV virtual table provide the ability to
take the CSV content from a string rather than a separate file, and give 
the programmer more control over the number and names of the columns.
The options are detailed below.  The CSV virtual table is usually
created as a TEMP table so that it exists only for the current database
connection and does not become a permanent part of the database schema.
Note that there is no "CREATE TEMP VIRTUAL TABLE" command in SQLite.
Instead, prepend the "temp." schema prefix to the name of the virtual
table.


The third line of the example shows the virtual table being used, to read
all content of the CSV file.  This is perhaps the simplest possible use
of the virtual table.  The CSV virtual table can be used anywhere an ordinary
virtual table can be used.  One can use the CSV virtual table inside subqueries,
or common table expressions or add WHERE, GROUP BY, HAVING, ORDER BY,
and LIMIT clauses as required.

2. Arguments


The example above showed a single filename='thefile.csv' argument
for the CSV virtual table.  But other arguments are also possible.


filename=FILENAME
The filename= argument specifies an external file from which
CSV content is read.  Every CSV virtual table must have either a 
filename= argument or a data= argument and not both.

data=TEXT
The data= argument specifies that TEXT is the literal
content of the CSV file.

schema=SCHEMA
 The schema= argument specifies a CREATE TABLE statement that
the CSV virtual table passes to the sqlite3_declare_vtab() interface in
order to define the names of the columns in the virtual table.

columns=N
The columns=N argument specifies the number of columns
in the CSV file.
If the input data contains more columns than this,
then the excess columns are ignored.  If the input data contains fewer columns,
then extra columns are filled with NULL.
If the columns=N argument is omitted, the first line of the
CSV file is read to determine the number of columns.

header=BOOLEAN
or just
header
If the header argument is true then the first row of the CSV file
to be treated as a header rather than as data.  The second line of the CSV
file becomes the first row of content.
If the schema= options is omitted, then the first line of the CSV
file determines the names of the columns.


3. Column Names


The column names of the virtual table are determined primarily by the
schema= argument.
If the schema= argument is omitted, but header is true, then
the values found in the first line of the CSV file become the column names.
If the schema= argument is omitted and header is false, then
the columns are named "c0", "c1", "c2", and so forth.
This page last modified on  2018-11-16 01:45:15 UTC


c2: Recovering Data From A Corrupt SQLite Database
c5: recovery.html
c4:
1. Recovering (Some) Data From A Corrupt SQLite Database


SQLite databases are remarkably rebust.  Application faults and
power failures typically leave the content of the database intact.
However, it is possible to corrupt an SQLite database.
For example, hardware malfunctions can damage the database file, or a
rogue process can open the database and overwrite parts of it.


Given a corrupt database file, it is sometimes desirable to try to
salvage as much data from the file as possible.  The recovery API
is designed to facilitate this.

1.1. Limitations


It is sometimes possible to perfectly restore a database that has
gone corrupt, but that is the exception.  Usually
the recovered database will be defective in a number of ways:



Some content might be permanently deleted and unrecoverable.
This can happen, for example, if a rogue process overwrites part
of the database file.


Previously deleted content might reappear.  Normally when SQLite
does a DELETE operation, it does not actually overwrite the old content
but instead remembers that space is available for reuse during the next
INSERT.  If such deleted content is still in the file when a recovery
is attempted, it might be extracted and "resurrected".


Recovered content might be altered.
For example, the value stored in a particular row
might change from 48 to 49.  Or it might change from an integer into
a string or blob.  A value that was NULL might become an integer.
A string value might become a BLOB.  And so forth.


Constraints may not be valid after recovery.  CHECK constraints, 
FOREIGN KEY constraints, UNIQUE constraints, type constraints on
STRICT tables - any of these might be violated in the recovered
database.


Content might be moved from one table into another.



The recovery API does as good of a job as it can at restoring a database,
but the results will always be suspect.  Sometimes (for example if the
corruption is restricted to indexes) the recovery will perfectly restore
the database content.  However in other cases, the recovery will be imperfect.
The impact of this imperfection depends on the application.  A database that
holds a list of bookmarks is still a list of bookmarks after recovery.
A few bookmarks might be missing or added or altered after recovery, but
the list is "fuzzy" and imperfect to begin with so adding a bit more
uncertainty will not be fatal to the application.  But if an accounting
database goes corrupt and is subsequently recovered, the books might be
out of balance.


It is best to think of the recovery API as a salvage undertaking.
Recovery will extract as much usable data as it can from the wreck
of the old database, but some parts may be damaged beyond repair and
some rework and testing should be performed prior to returning the
recovered database to service.

2. Recovery Using The ".recover" Command In The CLI


The easiest way to manually recover a corrupt database is using
the Command Line Interface or "CLI" for SQLite.  The CLI is a program
named "sqlite3".  Use it to recover a corrupt database file using
a command similar to the following:

sqlite3 corrupt.db .recover &gt;data.sql



This will generate SQL text in the file named "data.sql" that can be used
to reconstruct the original database:

sqlite3 recovered.db &lt;data.sql



The ".recover" option is actually a command that is issued to the
CLI.  That command can accept arguments.  For example, by running:

sqlite3 corruptdb ".recover --ignore-freelist" &gt;data.sql



Notice that the ".recover" command and its arguments must be contained
in quotes.  The following options are supported:



--ignore-freelist

Ignore pages of the database that appear to be part of the
freelist.  Normally the freelist is scanned, and if it contains
pages that look like they have content, that content is output.
But if the page really is on the freelist, that can mean that
previously deleted information is reintroduced into the database.


--lost-and-found TABLE

If content is found during recovery that cannot be associated
with a particular table, it is put into the "lost_and_found"
table.  Use this option to change the name of the
"lost_and_found" table to "TABLE".


--no-rowids

If this option is provided, then rowid values that are not also
INTEGER PRIMARY KEY values are not extracted from the
corrupt database.




3. Building The Recovery API Into An Application

3.1. Source Code Files

If you want to build the recovery API into your application, you will
need to add some source files to your build, above and beyond the usual
"sqlite3.c" and "sqlite3.h" source files.  You will need:






sqlite3recover.c

This is the main source file that implements the recovery API.





sqlite3recover.h

This is the header file that goes with sqlite3recover.h.





dbdata.c

This file implements two virtual tables name "sqlite_dbdata" and
"sqlite_dbptr" that required by sqlite3recover.c.






The two C source file above need to be linked into your application in the
same way as "sqlite3.c" is linked in.  And the header file needs to be
accessible to the compiler when the C files are being compiled.

3.2. How To Implement Recovery

These are the basic steps needed to recover content from a corrupt
Database:



Creates an sqlite3_recover handle by calling either
sqlite3_recover_init() or sqlite3_recover_init_sql().
Use sqlite3_recover_init() to store the recovered content
in a separate database and use sqlite3_recover_init_sql()
to generate SQL text that will reconstruct the database.


Make zero or more calls to sqlite3_recover_config() to set
options on the new sqlite3_recovery handle.


Invoke sqlite3_recover_step() repeatedly
until it returns something other than SQLITE_OK. If it
returns SQLITE_DONE, then the recovery operation completed without 
error. If it returns some other non-SQLITE_OK value, then an error 
has occurred.  The sqlite3_recover_run() interface is also
available as a convenience wrapper that simply invokes
sqlite3_recover_step() repeatedly until it returns something other
than SQLITE_DONE.


Retrieves any error code and English language error message using the
sqlite3_recover_errcode() and sqlite3_recover_errmsg() interfaces,
respectively.


Invoke sqlite3_recover_finish() to destroy the sqlite3_recover object.



Details of the interface are described in comments in the
sqlite3_recover.h header file.

3.3. Example Implementations



Examples of how the recovery extension is used by SQLite itself
can be seen at the following links:


https://sqlite.org/src/info/30475c820dc5ab8a8?ln=999,1026

An example of the recovery extension found in the 
"fuzzcheck" testing utility in the SQLite tree.

https://sqlite.org/src/info/84bb08d8762920285f08f1c0?ln=7299,7361

The code that implements the ".recover" command in the CLI.

This page last modified on  2022-11-04 15:23:18 UTC


c2: Clustered Indexes and the WITHOUT ROWID Optimization
c5: withoutrowid.html
c4:
1. Introduction

By default, every row in SQLite has a special column, usually called the
"rowid", that uniquely identifies that row within the table.  However
if the phrase "WITHOUT ROWID" is added to the end of a CREATE TABLE statement,
then the special "rowid" column is omitted.  There are sometimes
space and performance advantages to omitting the rowid.

A WITHOUT ROWID table is a table that uses a 
Clustered Index
as the primary key.

1.1. Syntax

To create a WITHOUT ROWID table, simply add the keywords "WITHOUT ROWID"
to the end of the CREATE TABLE statement.  For example:


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
) WITHOUT ROWID;


As with all SQL syntax, the case of the keywords does not matter.  
One can write "WITHOUT rowid" or "without rowid" or "WiThOuT rOwId" and
it will mean the same thing.

Every WITHOUT ROWID table must have a PRIMARY KEY.  An error is raised
if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.

In most contexts, the special "rowid" column of normal tables can 
also be called "oid" or "_rowid_".  However, only "rowid" works as 
the keyword in the CREATE TABLE statement.

1.2. Compatibility

SQLite version 3.8.2 (2013-12-06) or later 
is necessary in order to use a WITHOUT
ROWID table.  An attempt to open a database that contains one or more WITHOUT
ROWID tables using an earlier version of SQLite will result in a
"malformed database schema" error.

1.3. Quirks

WITHOUT ROWID is found only in SQLite and is not compatible
with any other SQL database engine, as far as we know.
In an elegant system, all tables would behave as WITHOUT ROWID
tables even without the WITHOUT ROWID keyword.  However, when SQLite was
first designed, it used only integer rowids for row keys 
to simplify the implementation.
This approach worked well for many years.  But as the demands on
SQLite grew, the need for tables in which the PRIMARY KEY really did
correspond to the underlying row key grew more acute.  The WITHOUT ROWID
concept was added
in order to meet that need without breaking backwards
compatibility with the billions of SQLite databases already in use at
the time (circa 2013).

2. Differences From Ordinary Rowid Tables

The WITHOUT ROWID syntax is an optimization.  It provides no new
capabilities.  Anything that can be done using a WITHOUT ROWID table
can also be done in exactly the same way, and exactly the same syntax,
using an ordinary rowid table.  The only advantage of a WITHOUT ROWID
table is that it can sometimes use less disk space and/or perform a little
faster than an ordinary rowid table.

For the most part, ordinary rowid tables and WITHOUT ROWID tables
are interchangeable.  But there are some additional restrictions on
WITHOUT ROWID tables that do not apply to ordinary rowid tables:



Every WITHOUT ROWID table must have a PRIMARY KEY.
An attempt to create a WITHOUT ROWID table without a PRIMARY KEY results
in an error.


The special behaviors associated "INTEGER PRIMARY KEY" do not apply
on WITHOUT ROWID tables.
In an ordinary table, "INTEGER PRIMARY KEY" means that the column is an
alias for the rowid.  But since there is no rowid in a WITHOUT ROWID
table, that special meaning no longer applies.  An "INTEGER PRIMARY KEY" 
column in a WITHOUT ROWID table works
like an "INT PRIMARY KEY" column in an ordinary table: It is a PRIMARY KEY
that has integer affinity.


AUTOINCREMENT does not work on WITHOUT ROWID tables.
The AUTOINCREMENT mechanism assumes the presence of a rowid and so it
does not work on a WITHOUT ROWID table.  An error is raised if the
 "AUTOINCREMENT" keyword is used in the CREATE TABLE statement for
a WITHOUT ROWID table.


NOT NULL is enforced on every column of the PRIMARY KEY in a WITHOUT
ROWID table.
This is in accordance with the SQL standard.  Each column of a PRIMARY KEY
is supposed to be individually NOT NULL.  However, NOT NULL was not enforced
on PRIMARY KEY columns by early versions of SQLite due to a bug.  By the
time that this bug was discovered, so many SQLite databases were already
in circulation that the decision was made not to fix this bug for fear of
breaking compatibility.  So, ordinary rowid tables in SQLite violate the
SQL standard and allow NULL values in PRIMARY KEY fields.  But WITHOUT ROWID
tables do follow the standard and will throw an error on any attempt to
insert a NULL into a PRIMARY KEY column.


The sqlite3_last_insert_rowid() function
does not work for WITHOUT ROWID tables.
Inserts into a WITHOUT ROWID do not change the value returned by the
sqlite3_last_insert_rowid() function.  The last_insert_rowid() SQL
function is also unaffected since it is just a wrapper around
sqlite3_last_insert_rowid().


The incremental blob I/O mechanism does not work
for WITHOUT ROWID tables.
Incremental BLOB I/O uses the rowid to create an sqlite3_blob object for
doing the direct I/O.  However, WITHOUT ROWID tables do not have a rowid,
and so there is no way to create an sqlite3_blob object for a WITHOUT
ROWID table.


The sqlite3_update_hook() interface does not fire callbacks for changes
to a WITHOUT ROWID table.
Part of the callback from sqlite3_update_hook() is the rowid of the table
row that has changed.  However, WITHOUT ROWID tables do not have a rowid.
Hence, the update hook is not invoked when a WITHOUT ROWID table changes.




3. Benefits Of WITHOUT ROWID Tables

A WITHOUT ROWID table is an optimization that can reduce storage and
processing requirements.

In an ordinary SQLite table, the PRIMARY KEY is really just a 
UNIQUE index.  The key used to look up records on disk
is the rowid.
The special "INTEGER PRIMARY KEY" column type in ordinary SQLite tables 
causes the column to be an alias for the rowid, and so an INTEGER PRIMARY
KEY is a true PRIMARY KEY.  But any other kind of PRIMARY KEYs, including
"INT PRIMARY KEY" are just unique indexes in an ordinary rowid table.

Consider a table (shown below) intended to store a
vocabulary of words together with a count of the number of occurrences of
each word in some text corpus:


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
);


As an ordinary SQLite table, "wordcount" is implemented as two
separate B-Trees.  The main table uses the hidden rowid value as the key
and stores the "word" and "cnt" columns as data.  The "TEXT PRIMARY KEY"
phrase of the CREATE TABLE statement
causes the creation of an unique index on the "word" column.  This index is a
separate B-Tree that uses "word" and the "rowid" as the key and stores no
data at all.  Note that the complete text of every "word" is stored twice:
once in the main table and again in the index.

Consider querying this table to find the number of occurrences of the
word "xyzzy".:


SELECT cnt FROM wordcount WHERE word='xyzzy';


This query first has to search the index B-Tree looking for any entry
that contains the matching value for "word".  When an entry is found in
the index, the rowid is extracted and used to search the main table.
Then the "cnt" value is read out of the main table and returned.  Hence, two
separate binary searches are required to fulfill the request.

A WITHOUT ROWID table uses a different data design for the equivalent
table.


CREATE TABLE IF NOT EXISTS wordcount(
  word TEXT PRIMARY KEY,
  cnt INTEGER
) WITHOUT ROWID;


In this latter table, there is only a single B-Tree which uses the "word"
column as its key and the "cnt" column as its data.  (Technicality:  the
low-level implementation actually stores both "word" and "cnt" in the "key"
area of the B-Tree.  But unless you are looking at the low-level byte encoding
of the database file, that fact is unimportant.)  Because there is only
a single B-Tree, the text of the "word" column is only stored once in the
database.  Furthermore, querying the "cnt" value for a specific "word"
only involves a single binary search into the main B-Tree, since the "cnt"
value can be retrieved directly from the record found by that first search
and without the need to do a second binary search on the rowid.

Thus, in some cases, a WITHOUT ROWID table can use about half the amount
of disk space and can operate nearly twice as fast.  Of course, in a 
real-world schema, there will typically be secondary indices and/or
UNIQUE constraints, and the situation is more complicated.  But even then,
there can often be space and performance advantages to using WITHOUT ROWID
on tables that have non-integer or composite PRIMARY KEYs.



4. When To Use WITHOUT ROWID

The WITHOUT ROWID optimization is likely to be helpful for tables
that have non-integer or composite (multi-column) PRIMARY KEYs and that do
not store large strings or BLOBs.

WITHOUT ROWID tables will work correctly (that is to say, they
provide the correct answer) for tables with a single INTEGER PRIMARY KEY. 
However, ordinary rowid tables will run faster in that case.  
Hence, it is good design
to avoid creating WITHOUT ROWID tables with single-column PRIMARY KEYs 
of type INTEGER.

WITHOUT ROWID tables work best when individual rows are not too large.
A good rule-of-thumb is that the average size of a single row in a
WITHOUT ROWID table should be less than about 1/20th the size of 
a database page.  That means that rows should not contain more than about
50 bytes each for a 1KiB page size or about 200 bytes each for 4KiB
page size.  WITHOUT ROWID tables will work (in the sense that
they get the correct answer) for arbitrarily large rows - up to 2GB in size -
but traditional rowid tables tend to work faster for large row sizes.
This is because rowid tables are implemented as B*-Trees where
all content is stored in the leaves of the tree, whereas WITHOUT ROWID 
tables are implemented using ordinary B-Trees with content stored on both
leaves and intermediate nodes.  Storing content in 
intermediate nodes causes each intermediate node entry to take up more
space on the page and thus reduces the fan-out, increasing the search cost.

The "sqlite3_analyzer.exe" utility program, available as source code
in the SQLite source tree or as a precompiled binary on the
SQLite Download page, can be
used to measure the average sizes of table rows in an existing SQLite
database.

Note that except for a few corner-case differences detailed above,
WITHOUT ROWID tables and rowid tables work the same.  They both generate
the same answers given the same SQL statements.  So it is a simple matter
to run experiments on an application, late in the development cycle,
to test whether or not the use of WITHOUT ROWID tables will be helpful.
A good strategy is to simply not worry about WITHOUT ROWID until near
the end of product development, then go back and run tests to see
if adding WITHOUT ROWID to tables with non-integer PRIMARY KEYs helps
or hurts performance, and retaining the WITHOUT ROWID only in those cases
where it helps.

5. Determining If An Existing Table Is WITHOUT ROWID

A WITHOUT ROWID table returns the same content for
PRAGMA table_info and PRAGMA table_xinfo as does an ordinary
table.  But unlike an ordinary table, a WITHOUT ROWID also
responds to the PRAGMA index_info command.  The PRAGMA index_info
on a WITHOUT ROWID table returns information abou the PRIMARY KEY
for the table.  In this way, the PRAGMA index_info command can be
used to unabiguously determine whether a particular table is a 
WITHOUT ROWID table or an ordinary table - an ordinary table will
always return no rows but a WITHOUT ROWID table will always return
one or more rows.
This page last modified on  2022-01-20 21:38:08 UTC


c2: SQLite Copyright
c5: copyright.html
c4:
SQLite Is Public Domain




SQLite is in the
Public Domain




All of the code and documentation in SQLite has been dedicated to the
public domain
by the authors.
All code authors, and representatives of the companies they work for,
have signed affidavits dedicating their contributions to
the public domain and originals of
those signed affidavits are stored in a firesafe at the main offices
of Hwaci.  All contributors are citizens
of countries that allow creative works to be dedicated into the public
domain.
Anyone is free to copy, modify, publish, use, compile, sell, or distribute
the original SQLite code, either in source code form or as a compiled binary,
for any purpose, commercial or non-commercial, and by any means.



The previous paragraph applies to the deliverable code and documentation
in SQLite - those parts of the SQLite library that you actually bundle and
ship with a larger application.  Some scripts used as part of the
build process (for example the "configure" scripts generated by autoconf)
might fall under other open-source licenses.  Nothing from these build 
scripts ever reaches the final deliverable SQLite library, however, and 
so the licenses associated with those scripts should not be 
a factor in assessing your rights to copy and use the SQLite library.



All of the deliverable code in SQLite has been written from scratch.
No code has been taken from other projects or from the open
internet.  Every line of code can be traced back to its original
author, and all of those authors have public domain dedications
on file.  So the SQLite code base is clean and is
uncontaminated with licensed code from other projects.




Open-Source, not Open-Contribution


SQLite is open-source, meaning that you can make as many copies of it as
you want and do whatever you want with those copies, without limitation.
But SQLite is not open-contribution.  In order to keep SQLite in the public
domain and ensure that the code does not become contaminated with proprietary
or licensed content, the project does not accept patches from people who
have not submitted an affidavit dedicating their contribution into the
public domain.


All of the code in SQLite is original, having been written
specifically for use by SQLite.  No code has been copied from unknown
sources on the internet.










Warranty of Title


SQLite is in the public domain and does not require a license.
Even so, some organizations want legal proof of their right to use
SQLite.  Circumstances where this might occurs include the following:



 Your company desires indemnity against claims of copyright infringement.
 You are using SQLite in a jurisdiction that does not recognize
     the public domain.  
 You are using SQLite in a jurisdiction that does not recognize
     the right of an author to dedicate their work to the public
     domain. 
 You want to hold a tangible legal document
     as evidence that you have the legal right to use and distribute
     SQLite. 
 Your legal department tells you that you must purchase a license.
     



If any of the above circumstances apply to you,
Hwaci, the company that employs 
all the developers of SQLite, will 
sell you
a Warranty of Title for SQLite.
A Warranty of Title is a legal document that asserts that the claimed
authors of SQLite are the true authors, and that the authors 
have the legal right to dedicate the SQLite to the public domain, and 
that Hwaci will vigorously defend against challenges to those claims.
All proceeds from the sale of SQLite Warranties of Title are used to fund
continuing improvement and support of SQLite.


Contributed Code


In order to keep SQLite completely free and unencumbered by copyright,
the project does not accept patches.  If you would like to suggest a
change and you include a patch as a proof-of-concept, that would
be great.  However, please do not be offended if we rewrite your patch
from scratch.
This page last modified on  2021-11-10 19:19:27 UTC


c2: Benefits of SQLite As A File Format
c5: aff_short.html
c4:
SQLite As An Application File Format


(Note:  The current page is a brief summary of why SQLite makes
a good application file format.  The topic is considered at greater
detail in a separate technical note.)


SQLite has been used with great success as the on-disk file format
for desktop applications such as version control systems,
financial analysis tools, media cataloging and editing suites, CAD
packages, record keeping programs, and so forth.  The traditional
File/Open operation calls sqlite3_open() to attach to the database
file.  Updates happen automatically as application content is revised
so the File/Save menu option becomes superfluous.  The File/Save_As
menu option can be implemented using the backup API.



There are many advantages to using SQLite as an application file format,
including:



Better performance

 Reading and writing from an SQLite database
     is often faster than reading and writing individual files from disk.
     See 35% Faster Than The Filesystem
     and Internal Versus External BLOBs.
 The application only has to load the data it needs, rather
     than reading the entire file and holding a complete parse
     in memory.
 Small edits only overwrite the parts of the file that change,
     reducing write time and wear on SSD drives.

Reduced application cost and complexity

 No application file I/O code to write and debug.
 Content can be accessed and updated using concise SQL queries instead
     of lengthy and error-prone procedural routines.
 The file format can be extended in future releases simply
     by adding new tables and/or column, preserving backwards compatibility.
 Applications can leverage the
     full-text search and RTREE indexes and use triggers to implement
     an automated undo/redo stack.
 Performance problems can often be resolved, even late in the
     development cycle, using CREATE INDEX, avoiding costly
     redesign, rewrite, and retest efforts.

Portability

 The application file is portable across all operating systems,
     32-bit and 64-bit and big- and little-endian architectures.
 A federation of programs, perhaps all written in different programming
     languages, can access the same application file with no
     compatibility concerns.
 Multiple processes can attach to the same application
     file and can read and write without interfering with each another.
 Diverse content which might otherwise be stored as a "pile-of-files"
     is encapsulated into a single disk file for simpler transport
     via scp/ftp, USB stick, and/or email attachment.

Reliability

 Content can be updated continuously and atomically so 
     that little or no work is lost in a power failure or crash.
 Bugs are far less likely in SQLite than in custom-written file I/O code.
 SQL queries are many times smaller than the equivalent procedural
     code, and since the number of bugs per line of code is roughly
     constant, this means fewer bugs overall.

Accessibility

 SQLite database content can be viewed using a wide variety
     third-party tools.
 Content stored in an SQLite database is more likely to be 
     recoverable decades in the future, long after all traces of
     the original application have been lost. Data lives longer than code.
 SQLite database files are recommended by the US Library of Congress
     as a storage format for long-term preservation of digital content.




SQLite allows database files to have any desired filename extension, so
an application can choose a custom filename extension for its own use, if
desired.  The application_id pragma can be used to set an "Application ID"
integer in the database file so that tools like
file(1) can determine that the file
is associated with your application and is not just a generic
SQL database.
This page last modified on  2018-05-29 18:35:58 UTC


c2: Why Is SQLite Coded In C
c5: whyc.html
c4:
1. C Is Best


Note: Sections 2.0 and 3.0 of this article were added in response
to comments on 
Hacker News and
Reddit.



Since its inception on 2000-05-29, SQLite has been implemented in generic C.
C was and continues to be the best language for implementing a software
library like SQLite.  There are no plans to recode SQLite in any other
programming language at this time.


The reasons why C is the best language to implement SQLite include:



 Performance
 Compatibility
 Low-dependency
 Stability


1.1. Performance

An intensively used low-level library like SQLite needs to be fast.
(And SQLite is fast, see Internal Versus External BLOBs and
35% Faster Than The Filesystem for example.)

C is a great language for writing fast code.  C is sometimes
described as "portable assembly language".  It enables to developers
to code as close to the underlying hardware as possible while still
remaining portable across platforms.

Other programming languages sometimes claim to be "as fast as C".
But no other language claims to be faster than C for general-purpose
programming, because none are.

1.2. Compatibility

Nearly all systems have the ability to call libraries
written in C.  This is not true of other implementation languages.

So, for example, Android applications written in Java are able to
invoke SQLite (through an adaptor).  Maybe it would have been more
convenient for Android if SQLite had been coded in Java as that would
make the interface simpler.  However, on iPhone applications are coded
in Objective-C or Swift, neither of which have the ability to call
libraries written in Java.  Thus, SQLite would be unusable on iPhones
had it been written in Java.

1.3. Low-Dependency

Libraries written in C do not have a huge run-time dependency.
In its minimum configuration, SQLite requires only the following
routines from the standard C library:






 memcmp()
 memcpy()
 memmove()
 memset()


&nbsp;&nbsp;&nbsp;


 strcmp()
 strlen()
 strncmp()







In a more complete build, SQLite also uses library routines like
malloc() and free() and operating system interfaces for opening, reading,
writing, and closing files.  But even then, the number of dependencies
is very small.  Other "modern" language, in contrast, often require
multi-megabyte runtimes loaded with thousands and thousands of interfaces.

1.4. Stability


The C language is old and boring.
It is a well-known and well-understood language.
This is exactly what one wants when developing a module like SQLite.
Writing a small, fast, and reliable database engine is hard enough as it
is without the implementation language changing out from under you with
each update to the implementation language specification.

2. Why Isn't SQLite Coded In An Object-Oriented Language?


Some programmers cannot imagine developing a complex system like
SQLite in a language that is not "object oriented".  So why is
SQLite not coded in C++ or Java?



Libraries written in C++ or Java can generally only be used by
applications written in the same language. It is difficult to
get an application written in Haskell or Java to invoke a library
written in C++.  On the other hand, libraries written in C are
callable from any programming language.


Object-Oriented is a design pattern, not a programming language.
You can do object-oriented programming in any language you want,
including assembly language.  Some languages (ex: C++ or Java) make
object-oriented easier.  But you can still do object-oriented programming
in languages like C.


Object-oriented is not the only valid design pattern.
Many programmers have been taught to think purely in terms of
objects.  And, to be fair, objects are often a good way to
decompose a problem.  But objects are not the only way, and are
not always the best way to decompose a problem.  Sometimes good old
procedural code is easier to write, easier to maintain and understand,
and faster than object-oriented code.


When SQLite was first being developed, Java was a young and immature
language.  C++ was older, but was undergoing such growing pains that
it was difficult to find any two C++ compilers that worked the same
way.  So C was definitely a better choice back when SQLite was first
being developed.  The situation is less stark now, but there is little
to no benefit in recoding SQLite at this point.


3. Why Isn't SQLite Coded In A "Safe" Language?


There has lately been a lot of interest in "safe" programming languages
like Rust or Go in which it is impossible, or is at least difficult, to make
common programming errors like memory leaks or array overruns.  So the
question often arises as to why SQLite is not coded in a "safe" language.



None of the safe programming languages existed for the first 10 years
of SQLite's existence.  SQLite could be recoded in Go or Rust, but doing
so would probably introduce far more bugs than would be fixed, and it
seems also likely to result in slower code.


Safe languages insert additional machine branches to do things like
verify that array accesses are in-bounds.  In correct code, those
branches are never taken.  That means that the machine code cannot
be 100% branch tested, which is an important component of SQLite's
quality strategy.


Safe languages usually want to abort if they encounter an out-of-memory
(OOM) situation.  SQLite is designed to recover gracefully from an OOM.
It is unclear how this could be accomplished in the current crop of
safe languages.


All of the existing safe languages are new.  The developers of SQLite
applaud the efforts of computer language researchers in trying to
develop languages that are easier to program safely.  We encourage these
efforts to continue.  But we ourselves are more interested in old and
boring languages when it comes to implementing SQLite.



All that said, it is possible that SQLite might
one day be recoded in Rust.  Recoding SQLite in Go is unlikely
since Go hates assert().  But Rust is a possibility.  Some
preconditions that must occur before SQLite is recoded in Rust
include:



 Rust needs to mature a little more, stop changing so fast, and
     move further toward being old and boring.
 Rust needs to demonstrate that it can be used to create general-purpose
     libraries that are callable from all other programming languages.
 Rust needs to demonstrate that it can produce object code that
     works on obscure embedded devices, including devices that lack
     an operating system.
 Rust needs to pick up the necessary tooling that enables one to
     do 100% branch coverage testing of the compiled binaries.
 Rust needs a mechanism to recover gracefully from OOM errors.
 Rust needs to demonstrate that it can do the kinds of work that
     C does in SQLite without a significant speed penalty.



If you are a "rustacean" and feel that Rust already meets the
preconditions listed above, and that SQLite should be recoded in
Rust, then you are welcomed and encouraged
to contact the SQLite developers privately
and argue your case.
This page last modified on  2022-07-29 00:41:26 UTC


c2: Memory-Mapped I/O
c5: mmap.html
c4:
Memory-Mapped I/O

The default mechanism by which SQLite accesses and updates database disk
files is the xRead() and xWrite() methods of the
sqlite3_io_methods VFS object.  These methods are typically implemented as
"read()" and "write()" system calls which cause the operating system
to copy disk content between the kernel buffer cache and user space.

Beginning with version 3.7.17 (2013-05-20), SQLite has the option of 
accessing disk content directly using memory-mapped I/O and the new
xFetch() and xUnfetch() methods on sqlite3_io_methods.

There are advantages and disadvantages to using memory-mapped I/O.
Advantages include:


Many operations, especially I/O intensive operations, can be
    faster since content need not be copied between kernel space
    and user space.

The SQLite library may need less RAM since it shares pages with
    the operating-system page cache and does not always need its own copy of
    working pages.


But there are also disadvantages:


An I/O error on a memory-mapped file cannot be caught and dealt with by
    SQLite.  Instead, the I/O error causes a signal which, if not caught
    by the application, results in a program crash.

The operating system must have a unified buffer cache in order for
    the memory-mapped I/O extension to work correctly, especially in
    situations where two processes are accessing the same database
    file and one process is using memory-mapped I/O while the other
    is not.  Not all operating systems have a unified buffer cache.
    In some operating systems that claim to have a unified buffer cache,
    the implementation is buggy and can lead to corrupt databases.

Performance does not always increase with memory-mapped I/O.  In fact,
    it is possible to construct test cases where performance is reduced
    by the use of memory-mapped I/O.

Windows is unable to truncate a memory-mapped file.  Hence, on Windows,
    if an operation such as VACUUM or auto_vacuum tries to reduce the
    size of a memory-mapped database file, the size reduction attempt will
    silently fail, leaving unused space at the end of the database file.
    No data is lost due to this problem, and the unused space will be
    reused again the next time the database grows.  However if a version 
    of SQLite prior to 3.7.0 runs PRAGMA integrity_check on such a 
    database, it will (incorrectly) report database corruption due to 
    the unused space at the end.  Or if a version of SQLite prior to 3.7.0
    writes to the database while it still has unused space at the end, it
    may make that unused space inaccessible and unavailable for reuse until
    after the next VACUUM.


Because of the potential disadvantages, memory-mapped I/O is disabled
by default.  To activate memory-mapped I/O, use the mmap_size pragma
and set the mmap_size to some large number, usually 256MB or larger, depending
on how much address space your application can spare.  The rest is
automatic.  The PRAGMA mmap_size statement will be a silent no-op on
systems that do not support memory-mapped I/O.

How Memory-Mapped I/O Works

To read a page of database content using the legacy xRead() method,
SQLite first allocates a page-size chunk of heap memory then invokes
the xRead() method which causes the database page content to be copied
into the newly allocated heap memory.  This involves (at a minimum)
a copy of the entire page.

But if SQLite wants to access a page of the database file and
memory mapped I/O is enabled, it first calls the xFetch() method.
The xFetch() method asks the operating system to return a pointer to
the requested page, if possible.  If the requested page has been or
can be mapped into the application address space, then xFetch returns
a pointer to that page for SQLite to use without having to copy anything.
Skipping the copy step is what makes memory mapped I/O faster.

SQLite does not assume that the xFetch() method will work.  If
a call to xFetch() returns a NULL pointer (indicating that the requested
page is not currently mapped into the applications address space) then
SQLite silently falls back to using xRead().  An error is only reported
if xRead() also fails.

When updating the database file, SQLite always makes a copy of the
page content into heap memory before modifying the page.  This is necessary
for two reasons.  First, changes to the database
are not supposed to be visible to other processes until
after the transaction commits and so the changes must occur in private memory.
Second, SQLite uses a read-only memory map to prevent stray pointers in the 
application from overwriting and corrupting the database file.


After all needed changes are completed, xWrite() is used to move the content
back into the database file.
Hence the use of memory mapped I/O does not significantly change the
performance of database changes.
Memory mapped I/O is mostly a benefit for queries.

Configuring Memory-Mapped I/O

The "mmap_size" is the maximum number of bytes of the database file that
SQLite will try to map into the process address space at one time.  The
mmap_size applies separately to each database file, so the total amount
of process address space that could potentially be used is the mmap_size
times the number of open database files.

To activate memory-mapped I/O, an application can set the mmap_size to some
large value.  For example:


PRAGMA mmap_size=268435456;


To disable memory-mapped I/O, simply set the mmap_size to zero:


PRAGMA mmap_size=0;


If mmap_size is set to N then all current implementations map the first
N bytes of the database file and use legacy xRead() calls for any content
beyond N bytes.  If the database file is smaller than N bytes, then the entire
file is mapped.  In the future, new OS interfaces could, in theory, map
regions of the file other than the first N bytes, but no such 
implementation currently exists.

The mmap_size is set separately for each database file using the
"PRAGMA mmap_size" statement.  The usual default mmap_size is zero,
meaning that memory mapped I/O is disabled by default.  However, the
default mmap_size can be increased either at compile-time using
the SQLITE_DEFAULT_MMAP_SIZE macro or at start-time using the
sqlite3_config(SQLITE_CONFIG_MMAP_SIZE,...) interface.

SQLite also maintains a hard upper bound on the mmap_size.  Attempts
to increase the mmap_size above this hard upper bound (using
PRAGMA mmap_size) will automatically cap the mmap_size at the hard
upper bound.  If the hard upper bound is zero, then memory mapped I/O
is impossible.  The hard upper bound can be set at compile-time using
the SQLITE_MAX_MMAP_SIZE macro.  If SQLITE_MAX_MMAP_SIZE is set to
zero, then the code used to implement memory mapped I/O is omitted from
the build.  The hard upper bound is automatically set to zero on certain
platforms (ex: OpenBSD) where memory mapped I/O does not work due to the
lack of a unified buffer cache.

If the hard upper bound on mmap_size is non-zero at compilation time,
it may still be reduced or zeroed at start-time using the
sqlite3_config(SQLITE_CONFIG_MMAP_SIZE,X,Y) interface.  The X and
Y parameters must both be 64-bit signed integers. The X parameter
is the default mmap_size of the process and the Y is the new hard upper bound.
The hard upper bound cannot be increased above its compile-time setting
using SQLITE_CONFIG_MMAP_SIZE but it can be reduced or zeroed.
This page last modified on  2022-01-20 21:38:08 UTC


c2: Application-Defined SQL Functions
c5: appfunc.html
c4:
1. Executive Summary

Applications that use SQLite can define custom SQL functions that call
back into application code to compute their results.  The custom SQL
function implementations can be embedded in the application code itself,
or can be loadable extensions.

Application-defined or custom SQL functions are created using the
sqlite3_create_function() family of interfaces.
Custom SQL functions can be scalar functions, aggregate functions,
or window functions.
Custom SQL functions can have any number of arguments from 0 up to
SQLITE_MAX_FUNCTION_ARG.
The sqlite3_create_function() interface specifies callbacks that are
invoked to carry out the processing for the new SQL function.

SQLite also supports custom table-valued functions, but they are
implemented by a different mechanism that is not covered in this document.

2. Defining New SQL Functions


The sqlite3_create_function() family of interfaces is used to create
new custom SQL functions.  Each member of this family is a wrapper around
a common core.  All family members accomplish the same thing; they merely
have different calling signatures.


sqlite3_create_function() &rarr;
The original version of sqlite3_create_function() allows the application
to create a single new SQL function that can be either a scalar or an
aggregate.  The name of the function is specified using UTF8.

sqlite3_create_function16() &rarr;
This variant works exactly like the sqlite3_create_function() original
except that the name of the function itself is specified as a UTF16
string rather than as a UTF8 string.

sqlite3_create_function_v2() &rarr;
This variant works like the original sqlite3_create_function() except
that it includes an additional parameter that is a pointer to a
destructor for the sqlite3_user_data() pointer that is passed in
as the 5th argument to all of the sqlite3_create_function() variants.
That destructor function (if it is non-NULL) is called when the
custom function is deleted - usually when the database connection is
closing.

sqlite3_create_window_function() &rarr;
This variant works like the original sqlite3_create_function() except
that it accepts a different set of callback pointers - the callback
pointers used by window function definitions.


2.1. Common Parameters

Many of the parameters passed to the sqlite3_create_function()
family of interfaces are common across the entire family.


db &rarr;
The 1st parameter is always a pointer to the database connection
on which the custom SQL function will work.  Custom SQL functions are
created separately for each database connection.  There is no short-hand
mechanism for creating SQL functions that work across all database
connections.

zFunctionName &rarr;
The 2nd parameter is the name of the SQL function that is being
created.  The name is usually in UTF8, except that the name should
be in UTF16 in the native byte order for sqlite3_create_function16().

The maximum length of a SQL function name is 255 bytes of UTF8.
Any attempt to create a function with a longer name will result in
an SQLITE_MISUSE error.

The SQL function creation interfaces may be called multiple
times with the same function name.
If two calls have the same function number but a different number of
arguments, for example, then two variants of the SQL function will
be registered, each taking a different number of arguments.

nArg &rarr;
The 3rd parameter is always the number of arguments that the function
accepts.  The value must be an integer between -1 and 
SQLITE_MAX_FUNCTION_ARG (default value: 127).  A value of -1 means
that the SQL function is a variadic function that can take any number
of arguments between 0 and SQLITE_MAX_FUNCTION_ARG.

eTextRep &rarr;
The 4th parameter is a 32-bit integer flag whose bits convey various
properties about the new function.  The original purpose of this
parameter was to specify the preferred text encoding for the function,
using one of the following constants:

 SQLITE_UTF8
 SQLITE_UTF16BE
 SQLITE_UTF16LE

All custom SQL functions will accept text in any encoding.  Encoding
conversions will happen automatically.  The preferred encoding merely
specifies the encoding for which the function implementation is optimized.
It is possible to specify multiple functions with the same name and the
same number of arguments, but different preferred encodings and different
callbacks used to implement the function, and SQLite will chose the
set of callbacks for which the input encodings most closely match the
preferred encoding.

The 4th parameter as more recently be extended with additional flag bits
to convey additional information about the function.  The additional
bits include:

 SQLITE_DETERMINISTIC
 SQLITE_DIRECTONLY
 SQLITE_INNOCUOUS
 SQLITE_SUBTYPE


Additional bits may be added in future versions of SQLite.

pApp &rarr;
The 5th parameter is an arbitrary pointer that is passed through
into the callback routines.  SQLite itself does nothing with this
pointer, except to make it available to the callbacks, and to pass
it into the destructor when the function is unregistered.


2.2. Multiple Calls To sqlite3_create_function() For The Same Function


It is common for an application to invoke sqlite3_create_function() multiple
times for the same SQL function.  For example, if an SQL function can take
either 2 or 3 arguments, then sqlite3_create_function() would be invoked
once for the 2-argument version and a second time for the 3-argument version.
The underlying implementation (the callbacks) can be different for both
variants.


An application can also register multiple SQL functions with the same name
and same number of arguments, but a different preferred text encoding.
In that case, SQLite will invoke the function using the callbacks for
the version whose preferred text encoding most closely matches the database
text encoding.  In this way, multiple implementations of the same function
can be provided that are optimized for UTF8 or UTF16.


If multiple calls to sqlite3_create_function() specify the same function name,
and the same number of arguments, and the same preferred text encoding, then
the callbacks and other parameters of the second call overwrite the first,
and the destructor callback from the first call (if it exists) is invoked.


2.3. Callbacks


SQLite evaluates an SQL function by invoking callback routines.

2.3.1. The Scalar Function Callback

Scalar SQL functions are implemented by a single callback in the
xFunc parameter to sqlite3_create_function().
The following code demonstrations the implementation of a "noop(X)"
scalar SQL function that merely returns its argument:

static void noopfunc(
  sqlite3_context *context,
  int argc,
  sqlite3_value **argv
){
  assert( argc==1 );
  sqlite3_result_value(context, argv&#91;0&#93;);
}



The 1st parameter, context, is a pointer to an opaque object
that describes the content from which the SQL function was invoked.  This
context point becomes the first parameter to many other routines that
the function implement might to invoke, including:



sqlite3_aggregate_context
sqlite3_context_db_handle
sqlite3_get_auxdata
sqlite3_result_blob
sqlite3_result_blob64
sqlite3_result_double
sqlite3_result_error
sqlite3_result_error16
sqlite3_result_error_code
sqlite3_result_error_nomem
sqlite3_result_error_toobig
sqlite3_result_int
sqlite3_result_int64
sqlite3_result_null
sqlite3_result_pointer
sqlite3_result_subtype
sqlite3_result_text
sqlite3_result_text16
sqlite3_result_text16be
sqlite3_result_text16le
sqlite3_result_text64
sqlite3_result_value
sqlite3_result_zeroblob
sqlite3_result_zeroblob64
sqlite3_set_auxdata
sqlite3_user_data




The sqlite3_result() family of functions are
used to specify the result of the scalar SQL function.  One or more of
these should be invoked by the callback to set the function return value.
If none of these routines are invoked for a specific callback, then the
return value will be NULL.

The sqlite3_user_data() routine returns a copy of the pArg
pointer that was given to sqlite3_create_function() when the SQL
function was created.

The sqlite3_context_db_handle() routine returns a pointer to the
database connection object.

The sqlite3_aggregate_context() routine is used only in the
implementations of aggregate and window functions.  Scalar functions
may not use sqlite3_aggregate_context().  The sqlite3_aggregate_context()
function is included in the interface list only for completeness.


The 2nd and 3rd arguments to the scalar SQL function implemenetation,
argc and argv, are
the number of arguments to the SQL function itself and the values for
each argument of the SQL function.
Argument values can be of any datatype and are thus stored in
instances of the sqlite3_value object.
Specific C-language values can be extracted from this object using
the sqlite3_value() family of interfaces.

2.3.2. The Aggregate Function Callbacks

Aggregate SQL functions are implemented by using two callback
functions, xStep and xFinal.  The xStep() function 
is called for each row of the aggregate and the xFinal() function
is invoked to compute the final answer at the end.
The following (slightly simplified) version of the built-in
count() function illustrates:

typedef struct CountCtx CountCtx;
struct CountCtx {
  i64 n;
};
static void countStep(sqlite3_context *context, int argc, sqlite3_value **argv){
  CountCtx *p;
  p = sqlite3_aggregate_context(context, sizeof(*p));
  if( (argc==0 || SQLITE_NULL!=sqlite3_value_type(argv&#91;0&#93;)) && p ){
    p->n++;
  }
}   
static void countFinalize(sqlite3_context *context){
  CountCtx *p;
  p = sqlite3_aggregate_context(context, 0);
  sqlite3_result_int64(context, p ? p->n : 0);
}


Recall that there are two versions of the count() aggregate.
With zero arguments, count() returns a count of the number of rows.
With one argument, count() returns the number of times that the
argument was non-NULL.

The countStep() callback is invoked once for each row in the aggregate.
As you can see, the count is incremented if either there are no arguments,
or if the one argument is not NULL.

The step function for an aggregate should always begin with a call
to the sqlite3_aggregate_context() routine to fetch the persistent
state of the aggregate function.  On the first invocation of the step()
function, the aggregate context is initialized to a block of memory
that is N bytes in size, where N is the second parameter to
sqlite3_aggregate_context() and that memory is zeroed.  On all subsequent
calls to the step() function, the same block of memory is returned.
Except, sqlite3_aggregate_context() might return NULL in the case of
an out-of-memory error, so aggregate functions should be prepared to
deal with that case.

After all rows are processed the countFinalize() routine is called
exactly once.  This routine computes the final result and invokes
one of the sqlite3_result() family of functions
to set the final result.  The aggregate context will be freed automatically
by SQLite, though the xFinalize() routine must clean up any substructure
associated with the aggregate context before it returns.  If the xStep()
method is called one or more times, then SQLite guarantees thta the
xFinal() method will be called at once, even if the query aborts.

2.3.3. The Window Function Callbacks

Window functions use the same xStep() and xFinal() callbacks that
aggregate functions use, plus two others: xValue and xInverse.
See the documentation on
application-defined window functions for further details.

2.3.4. Examples

There are dozens and dozens of SQL function implementations scattered
throughout the SQLite source code that can be used as example applications.
The built-in SQL functions use the same interface as application-defined
SQL functions, so built-in functions can be used as examples too.
Search for "sqlite3_context" in the SQLite source code to find examples.



3. Security Implications


Application-defined SQL functions can become security vulnerabilities if
not carefully managed.  Suppose, for example, an application defines
a new "system(X)" SQL function that runs its argument X as a command and
returns the integer result code.  Perhaps the implementation is like this:

static void systemFunc(
  sqlite3_context *context,
  int argc,
  sqlite3_value **argv
){
  const char *zCmd = (const char*)sqlite3_value_text(argv&#91;0&#93;);
  if( zCmd!=0 ){
    int rc = system(zCmd);
    sqlite3_result_int(context, rc);
  }
}



This is a function with powerful side-effects.  Most programmers would
be naturally cautious about using it, but probably would not see the
harm in merely having it available.  But there is great risk in merely
defining such a function, even if the application itself never invokes
it!


Suppose the application normally does a query against table TAB1
when it starts up.  If an attacker can gain access to the database
file and modify the schema like this:

ALTER TABLE tab1 RENAME TO tab1_real;
CREATE VIEW tab1 AS SELECT * FROM tab1 WHERE system('rm -rf *') IS NOT NULL;



Then, when the application attempts to open the database, register the
system() function, then run an innocent query against the "tab1" table,
it instead deletes all the files in its working directory.  Yikes!


To prevent this kind of mischief, applications that create their own
custom SQL functions should take one or more of the following safety
precautions.  The more precautions taken the better:



Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
on each database connection as soon as it is opened.
This prevents application-defined functions from being used in places
where an attacker might be able to surreptiously invoke them by modifying
a database schema:

 In in VIEWs.
 In in TRIGGERSs.
 In CHECK constraints of a table definition.
 In DEFAULT constraints of a table definition.
 In the definitions of generated columns.
 In the expression part of an index on an expression.
 In the WHERE clause of a partial index.


To put it another way, this setting requires that application-defined
functions only be run directly by top-level SQL invoked from the application
itself, not as a consequence of doing some other innocent-looking query.


Use the PRAGMA trusted_schema=OFF SQL statement to disable trusted
schema.  This has the same effect as the previous bullet, but does not
require the use of C-code and hence can be performed in programs written
in another programming language and that do not have access SQLite
C-language APIs.


Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
This make SQLite distrust application-defined functions inside of
the schema by default.


If any application-defined SQL functions have potentially dangerous
side-effects, or if they could potentially leak sensitive information
to an attacker if misused, then tag those functions using the
SQLITE_DIRECTONLY option on the "enc" parameter.  This means
that the function can never be run from schema-code even if the
trusted-schema option is on.


Never tag an application-defined SQL function with SQLITE_INNOCUOUS
unless you really need to and you have checked the implementation closely
and are certain that it can do no harm even if it falls under the
control of an attacker.

This page last modified on  2020-09-20 12:35:50 UTC


c2: pressrelease-20071212.html
c5: pressrelease-20071212.html
c4:
FOR IMMEDIATE RELEASE:

SQLite Consortium Launches With Mozilla And Symbian As Charter Members


Ensures independent technical direction for world's
most deployed SQL database engine

Charlotte, North Carolina - December 12, 2007 - The SQLite Consortium,
a new membership association dedicated to maintaining SQLite as a fully open 
and independent product, was formally announced today. Mozilla and Symbian Ltd. 
have joined the SQLite Consortium as charter members.

SQLite is a compact, high efficiency, high reliability, embeddable SQL database 
engine. The source code to SQLite is in the public domain and is available with 
no associated fees. SQLite is the most deployed SQL database engine in the world 
and is currently used in a wide range of commercial software products and electronic 
devices from leading manufacturers. SQLite is found today in many mobile phones, 
MP3 players, set-top boxes, and PCs.

The mission of the SQLite Consortium is to continue developing and enhancing
SQLite as a product that anyone may use without paying royalties or licensing fees. 
Members of the SQLite Consortium provide funding to enable this mission and in
return receive enterprise-level technical support. Technical control and direction
of SQLite remains entirely with the SQLite developers.

Mozilla, developer of the popular open-source Firefox web browser, and Symbian,
the market-leading open operating system for advanced data-enabled smartphones, 
both deploy the SQLite database engine in their products. As charter members of 
the Consortium, Mozilla and Symbian are ensuring the development and support of 
SQLite as a freely accessible and public domain software asset.

"SQLite has become a popular embedded database because it is lightweight, fast,
and open source," said Michael Schroepfer, Vice President of Engineering, Mozilla.
"As a charter member of the SQLite Consortium, Mozilla is excited to help ensure
SQLite remains a vibrant and open technology, in line with our mission to promote
choice and innovation on the Internet."

"The SQLite Consortium will help set the standards for database management 
which are essential in smartphone functionality and will also help create a pool
of developers, highly-skilled in SQLite for future mobile phone development, "
said Bruce Carney, Director, Developer Programmes & Services, Symbian. 
"Our involvement with the SQL Consortium not only demonstrates Symbian's commitment 
to open standards in the industry, but as mobile phones become increasingly powerful
and smartphones become increasingly popular, we are focused on ensuring that
desktop developers, who move to the mobile space, have the easiest and most
productive experience possible."

SQLite is a winner of the 2005 Google/O'Reilly Open Source Award. Additional
information regarding the SQLite Consortium is available at the SQLite website,
http://www.sqlite.org/.


# # #


About SQLite

SQLite is a software library that implements a self-contained, embeddable, 
serverless, zero-configuration, transactional SQL database engine. The code for 
SQLite is in the public domain and is free for any use, commercial or private. 
SQLite is currently found in countless software titles and electronic devices.

SQLite was originally developed and released 2000 by Dr. D. Richard Hipp.
The code continues to be maintained and enhanced by an international team 
of developers under Hipp's direction.



CONTACT INFORMATION:
SQLite Consortium
Dr. D. Richard Hipp
6200 Maple Cove Lane
Charlotte, NC 28269
Email: drh@sqlite.org
http://www.sqlite.org/
Tel: +1.704.948.4565



About Mozilla

Mozilla is a global community dedicated to building free, open source products 
and technologies that improve the online experience for people everywhere. 
Mozilla works in the open with a highly disciplined, transparent and cooperative 
development process, under the umbrella of the non-profit Mozilla Foundation. 
As a wholly owned subsidiary, the Mozilla Corporation organizes the development 
and marketing of Mozilla products. This unique structure has enabled Mozilla to 
financially support and cultivate competitive, viable community innovation. 
For more information, visit www.mozilla.com.

CONTACT INFORMATION:
Mozilla Corporation
Jessica Waight
Tel: +1.415.345.4754
Email: jwaight@outcastpr.com


About Symbian Limited

Symbian is a software licensing company that develops and licenses Symbian OS, 
the market-leading open operating system for advanced, data-enabled mobile phones 
known as smartphones.

Symbian licenses Symbian OS to the world's leading handset manufacturers 
and has built close co-operative business relationships with leading companies 
across the mobile industry. During Q3 2007, 20.4 million Symbian smartphones 
were sold worldwide to over 250 major network operators, bringing the total
number of Symbian smartphones shipped up to 30 September 2007 to 165 million.

Symbian has its headquarters in London, United Kingdom, with offices in the 
United States, United Kingdom, Asia (India, P.R. China, and Korea) and Japan.
For more information, please visit www.symbian.com.

CONTACT INFORMATION:
Symbian Global Press Office
Karen Hamblen
+44 207 154 1707
press@symbian.com


c2: Recent SQLite News
c5: news.html
c4:
Recent News

2022-12-28 - Version 3.40.1
Version 3.40.1 is a patch release that fixes some obscure problems
in version 3.40.0.  The problems fixed have no impact on most applications.
Upgrading is only necessary if you encounter problems.

The two most important fixes are these:


Fix the --safe command-line option on the CLI so that it
correctly disallows functions with side-effects.  This is a bug
in the CLI &mdash; not a bug in the
SQLite library &mdash; and it only affects the --safe command-line
option, making that option less than fully "safe".  As the number
of systems that use the --safe command-line option in the CLI is
approximately zero, this is not considered an important bug.  However,
a third-party wrote a CVE against it which caused considerable angst
among maintainers, so it seems good to get the fix into circulation
sooner rather than wait on the next major release.

The optional memsys5 memory allocator picked up a bug that might
put it into an infinite loop for very large (500MiB) allocations.
Almost all systems use their native memory allocator, not memsys5.
Memsys5 is only used if SQLite is compiled using SQLITE_ENABLE_MEMSYS5
and then initialized using sqlite3_config(SQLITE_CONFIG_HEAP).
Very, very few systems do this, and so the problem is not considered
to be serious.


See the branch-3.40 timeline
for a complete listing of changes that have gone into this patch release.
2022-11-16 - Version 3.40.0
Version 3.40.0 is a new feature release of SQLite.  See the
change log for details.
Key enhancements in this release include:


Official support for compiling 
SQLite to WASM
and running it in a web browser.

New and improved recovery extension for extracting
data from corrupted database files.


This release also includes performance enhancements and
improvements to the query planner.

A psychological milestone:  The performance benchmark that
the SQLite developers have used for many years has for
the first time dropped below 1 billion CPU cycles
(measured using cachegrind) when run in WAL mode.
This is less than half the number of CPU cycles used as
recently as 8 years ago.  (The graph below shows SQLite using
a rollback journal which uses fewer CPU cycles at the expense
of extra I/O.  The benchmark passed through the 1 billion cycle
barrier for rollback journals for the previous release.)



2022-09-29 - Version 3.39.4
Version 3.39.4 is a security release that addresses a single
long-standing problem in the FTS3 extension.  An attacker
who is able to execute arbitrary SQL statements and who can create
a corrupt database that is 2GB or larger in size might be able to trick
FTS3 into overflowing an integer used to size a memory allocation,
causing the allocation to be too small and ultimately resulting in
a buffer overrun.  The release also includes fixes for other
obscure weaknesses, as described in the release notes.
2022-09-05 - Version 3.39.3
Version 3.39.3 is a patch release that fixes a few obscure problems
in the 3.39.0 release.  See the release notes for details.
2022-07-21 - Version 3.39.2
Version 3.39.2 is a security release that addresses multiple long-standing
issues in SQLite.  The most severe problem is identified by CVE-2022-35737.
That issue is associated with the auxiliary C-language APIs
of SQLite and cannot be reached using SQL or database inputs, and hence is
unlikely to affect your application.  Nevertheless, upgrading is recommended.
The other issues are comparatively minor.
This release also fixes a performance regression that appeared in version
3.39.0 that might affect some multi-way joins that use LEFT JOIN.
2022-07-13 - Version 3.39.1
Version 3.39.1 is a patch release that fixes a few minor problem in version
3.39.0.  Upgrading is optional.
2022-06-25 - Version 3.39.0
Version 3.39.0 is regular maintenance release of SQLite.  The key enhancement
in this release is added support for RIGHT and FULL JOIN.  There are other
language and performance enhancements as well &mdash; see the
release notes for details.
2022-05-06 - Version 3.38.5
The 3.38.4 patch release included a minor change to the CLI source code
that did not work.  The release manager only ran a subset of the normal
release tests, and hence did not catch the problem. As a result, the CLI
will segfault when using columnar output modes in version 3.38.4.  This
blunder did not affect the core SQLite library.  It only affected the CLI.

Take-away lesson:  Always run all of your tests prior to
a release - even a trival patch release.  Always.

The 3.38.5 patch release fixes the 3.38.4 blunder.
2022-05-04 - Version 3.38.4
Another user-discovered problem in the new Bloom filter optimization
is fixed in this patch release.  Without the fix, it is possible for
a multi-way join that uses a Bloom filters for two or more tables in
the join to enter an infinite loop if the key constraint on one of those
tables contains a NULL value.
2022-04-27 - Version 3.38.3
Version 3.38.3 fixes a bug in the automatic-index and Bloom filter
construction logic that might cause SQLite to be overly aggressive
in the use of ON clause constraints, resulting in a incorret
automatic-index or Bloom filter that excludes some valid rows from
output.  The bug was introduced in version 3.38.0.  Other minor
changes were tossed in to complete the patch.
2022-03-26 - Version 3.38.2
Version 3.38.2 fixes another bug in the new Bloom filter
optimization that might cause incorrect answers for a
LEFT JOIN that has an IS NULL constraint on the right-hand
table.
2022-03-12 - Version 3.38.1
Version 3.38.1 fixes a pair of bugs in the Bloom filter
optimization that was introduced in version 3.38.0.  These
bugs might cause incorrect answers for some obscure queries.
Various other minor problems and documentation typos were
fixed at the same time.
2022-02-22 - Version 3.38.0
Version 3.38.0 is a routine maintenance release of
SQLite.  There are various minor enhancements and
about a 0.5% reduction in the number of CPU cycles
used. See the
release notes for
more detail.
2022-01-06 - Version 3.37.2
Version 3.37.2 fixes a
database corruption bug.  You are
encouraged to upgrade, especially if you are using SAVEPOINT.

The problem first appeared in version 3.35.0 (2021-03-12) and
affects all subsequent releases through 3.37.1.
If temporary files are store in memory (which is not the default
behavior, but is sometimes selected by applications using either
-DSQLITE_TEMP_STORE or PRAGMA temp_store) and
if a SAVEPOINT is rolled back and then subsequent changes
within the same transaction are committed, the database file might
(with low but non-zero probability) go corrupt.
2021-12-30 - Version 3.37.1
Version 3.37.1 fixes a bug in the UPSERT logic, introduced by
the UPSERT enhancements of version 3.35.0, that can cause
incorrect byte-code to be generated in some cases, resulting
in an infinite loop in the byte code, or a NULL-pointer dereference.
This patch release also fixes some other minor problems with
assert() statements and in the CLI.
2021-11-27 - Version 3.37.0
Version 3.37.0 is a routine maintenance release of SQLite.
The biggest new feature in this release is support for
STRICT tables.  Other enhancements are described in
the release notes.
2021-06-18 - Version 3.36.0
Version 3.36.0 is a routine maintenance release of SQLite.
There are no new major features, only incremental improvements
to existing features and small performance improvements.
2021-04-19 - Patch release 3.35.5
The new ALTER TABLE DROP COLUMN capability that was added
in the 3.35.0 release contained a bug that might cause the
table content to go corrupt when the table was rewritten
to remove the dropped column.  Fixed by this patch.
2021-04-02 - Patch release 3.35.4
Version 3.35.4 is yet another patch release to fix
obscure problems in features associated with the 3.35.0.
2021-03-26 - Patch release 3.35.3
Version 3.35.3 contains patches for a handful of minor
problems discovered in prior releases.
2021-03-17 - Patch release 3.35.2
Version 3.35.2 is a small patch release to fix some minor problems
that were discovered shortly after the 3.35.1 release.
2021-03-15 - Patch release 3.35.1
A user discovered an issue with the new DROP COLUMN capability
in version 3.35.0, and so version 3.35.1 was created to fix it.
No need to upgrade if you are not using DROP COLUMN.
2021-03-12 - Release 3.35.0
SQLite version 3.35.0 is a routine maintenance release.  This
release adds a number of new language features, including
support for ALTER TABLE DROP COLUMN, built-in math functions,
generalized UPSERT, and the MATERIALIZED hint on common table
expressions.  There are also query planner optimizations and
incremental CLI improvements.
2020-01-20 - Release 3.34.1
SQLite version 3.34.1 is a patch releases that fixes a possible
use-after-free bug that can be provoked by malicious SQL.  Other
minor issues in extensions and documentation are also fixed.
2020-12-01 - Release 3.34.0
SQLite version 3.34.0 is a routine maintenance release.  This
release adds incremental improvements to performance and features,
including enhancements to the query planner, multiple recursive
SELECTS in recursive common table expressions, and better error messages
from CHECK constraint failures.  See the change log for details.
2020-08-14 - Release 3.33.0
SQLite version 3.33.0 is a routine maintenance release.  This
release features added support for "UPDATE FROM" following the
PostgreSQL syntax, and a doubling of the maximum database size
to 281 TB, as well as many other improvements.  See the change
log for details.
2020-06-18 - Release 3.32.3
The 3.32.3 release is a patch release that contains fixes for
various issues discovered by fuzzers.  None of the issues fixed
are likely to be encountered by applications that use SQLite in
ordinary ways, though upgrading never hurts.

Map of all changes since the 3.32.0 release:
https://www.sqlite.org/src/timeline?p=version-3.32.3&bt=version-3.32.0
2020-06-04 - Release 3.32.2
The 3.32.2 release is a one-line change relative to 3.32.1
that fixes a long-standing bug in the COMMIT command.  Since
version 3.17.0, if you were to retry a COMMIT command over
and over after it returns SQLITE_BUSY, it might eventually
report success, even though it was still blocked.  This patch
fixes the problem.
2020-05-25 - Release 3.32.1
Grey-hats published
information about two SQLite bugs approximately 24 hours after
the release of version 3.32.0.  These bugs enable maliciously
crafted SQL to crash the process that is running SQLite.  Both
bugs are long-standing problems that affect releases prior to
3.32.0. The 3.32.1 release fixes both problems.
2020-05-22 - Release 3.32.0
Version 3.32.0 is an ordinary maintenance release of SQLite.
This release features the ability to run an
approximate ANALYZE to gather database statistics for
use by the query planner, without having to scan every row
of every index.
See the change log for additional enhancements
and improvements.
2020-01-27 - Release 3.31.1
Applications that use SQLite should only interface with SQLite
through the officially published APIs.  Applications should not
depend upon or use the internal data structures of SQLite as those
structures might change from one release to another.  However, there
is a popular application that does depend on the details of the
internal layout of data in an internal SQLite data structure, and
those details changed in version 3.31.0, breaking the application.
This is, technically, a bug in the application, not in SQLite.
But it is within the power of SQLite to fix it, by reverting the
internal data structure change, and so that is what we have done
for the 3.31.1 release.
2020-01-22 - Release 3.31.0
Version 3.31.0 is an ordinary maintenance release of SQLite.
This release features the ability to define
generated columns for tables as well as many other enhancements.
See the change log for additional information.
2019-10-11 - Release 3.30.1
Version 3.30.1 is a bug-fix release that addresses a problem
that can occur when an aggregate function in a nested query
makes use of the new FILTER clause capability.  Some addition
patches for various obscure issues are also included, for
completeness.
2019-10-04 - Release 3.30.0
Version 3.30.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  This release adds support fo the NULLS FIRST
and NULLS LAST clauses on ORDER BY statements and the 
addition of FILTER clauses on all aggregate functions.
See the change log for details.
2019-07-10 - Release 3.29.0
Version 3.29.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

Beginning with this release, the
double-quoted string literal misfeature is deprecated.
The misfeature is still enabled by default, for legacy
compatibility, however developers are encouraged to
disable it at compile-time using the
-DSQLITE_DQS=0 option, or at run-time using
the SQLITE_DBCONFIG_DQS_DML and SQLITE_DBCONFIG_DQS_DDL
actions to the sqlite3_db_config() interface.  This is
especially true for double-quoted string literals in
CREATE TABLE and CREATE INDEX statements, as those elements
can cause unexpected problems following an ALTER TABLE.
See ticket 9b78184be266fd70
for an example.
2019-04-16 - Release 3.28.0
Version 3.28.0 is a regularly scheduled maintenance release
of SQLite containing miscellaneous performance and feature
enhancements.  See the change log for
details.

This release fixes an obscure security issue. Applications
using older versions of SQLite may be vulnerable if

 SQLite is compiled with certain optional extensions
     enabled, and
 the attacker is able to inject arbitrary SQL.

We are not aware of any applications that are vulnerable to
this problem. On the other hand, we do not know about
every application that uses SQLite.  
If your application allows unauthenticated users on the internet
(and hence potential attackers) to run arbitrary SQL and if
you build SQLite with any of the optional extensions enabled,
then you should take this upgrade at your earliest opportunity.

For further information about improving SQLite's robustness
in internet-facing applications, see the our
security recommendations.
2019-02-25 - Release 3.27.2
Version 3.27.2 is a patch release that
fixes a two bugs and various documentation
errors found in the version 3.27.1.  The
changes from version 3.27.1 and 3.27.0 are
minimal.
2019-02-08 - Release 3.27.1
After release 3.27.0 was tagged, but before the build could
be completed and uploaded, a 
long-standing bug
in the query optimizer was reported via System.Data.SQLite.
Since people will be upgrading anyhow, we decided 
publish the fix for this newly discovered problems right
away.  Hence, 3.27.1 was released less than 24 hours after
3.27.0.

It would have been better if the query optimizer bug had come to
our attention one day earlier, so that we could have incorporated
a fix into 3.27.0, but sometimes that's the way things go.
2019-02-07 - Release 3.27.0
SQLite version 3.27.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-12-01 - Release 3.26.0
SQLite version 3.26.0 is a routine maintenance release with various
performance and feature enhancements.  See the
release notes for details.
2018-11-05 - Release 3.25.3
SQLite version 3.25.3 is a third patch against 3.25.0 that fixes various
problems that have come to light and which seem serious enough to 
justify a patch.
2018-09-25 - Release 3.25.2
SQLite version 3.25.2 is another patch against 3.25.0 that fixes still
more problems associated with the new window function feature and the
ALTER TABLE enhancements.  Of particular note is the new
PRAGMA legacy_alter_table=ON command, which causes the ALTER TABLE RENAME
command to behave in the same goofy way that it did before the enhancements
found in version 3.25.0 &rarr; references to renamed tables that are inside
the bodies of triggers and views are not updated.  The legacy behavior is
arguably a bug, but some programs depend on the older buggy behavior.  The
3.25.2 release also contains a fix to window function processing for
VIEWs.  There also a slew of other minor fixes that affect obscure
compile-time options.  See the
Fossil Timeline for
details.
2018-09-18 - Release 3.25.1
SQLite version 3.25.1 is a patch against version 3.25.0 that contains
two one-line fixes for bug that were introduced in version 3.25.0.
See the change log for details.  Upgrading from 3.25.0 is recommended.
2018-09-15 - Release 3.25.0
SQLite version 3.25.0 is a regularly scheduled maintenance release.
Two big enhancements in this release:


Support for window functions was added, using the PostgreSQL documentation
as the baseline.

The ALTER TABLE command was enhanced to support renaming of columns, and
so that column and table renames are propagated into trigger bodies and views.

In addition, there are various performance enhancements and minor fixes.
One bug of note is 
ticket 9936b2fa443fec
which describes a hard-to-reach condition where the ORDER BY LIMIT
optimization might cause an infinite loop during query evaluation.  
This ticket raised a lot of
concern on 
HackerNews and
Reddit,
probably due to my choice of the ticket
title.  "Infinite Loop" sounds scary.  But I argue that the bug isn't really
all that bad in that it is very difficult to reach, will show up during
testing (rather than magically appearing after a product is
deployed), does not cause any data loss, and does not return an
incorrect result.  It was an important error, but not nearly as dire
as many people interpreted it to be.  And, in any event, it is fixed now.
2018-06-04 - Release 3.24.0
SQLite version 3.24.0 is a regularly scheduled maintenance release.
Highlights of this release include support for PostgreSQL-style
UPSERT and improved performance, especially for ORDER BY LIMIT queries.
2018-04-10 - Release 3.23.1
The version 3.23.1 release fixes a bug in the new
LEFT JOIN strength reduction optimization added to version 3.23.0.
A few other minor and obscure fixes were also inserted, as well as
a small performance optimization.  Code changes relative to
version 3.23.0 are minimal.
2018-04-02 - Release 3.23.0
The version 3.23.0 release is a regularly scheduled maintenance release.
See the change log for a list of enhancements and bug
fixes.
2018-01-22 - Release 3.22.0
The version 3.22.0 release is a regularly scheduled maintenance release.
There are many minor, though interesting, enhancements in this release.
See the change log for details.
2017-10-24 - Release 3.21.0
The version 3.21.0 release is a regularly scheduled maintenance release.
There are lots of enhancements in this release.
See the change log for details.
2017-08-24 - Release 3.20.1
The version 3.20.1 patch release changes two lines of code in
the sqlite3_result_pointer() interface in order to fix a rare
memory leak.  There are no other changes relative to version 3.20.0.
2017-08-01 - Release 3.20.0
SQLite version 3.20.0 is a regularly scheduled maintenance release
of SQLite.

This release contains many minor enhancements, including:

 Several new extensions
 Enhancements to the "sqlite3.exe" command-line shell
 Query planner enhancements
 Miscellaneous code optimizations for improved performance
 Fixes for some obscure bugs


See the release notes for more information.
2017-06-17 - Release 3.18.2
SQLite version 3.18.2 is another backport of a bug fix found
in SQLite version 3.19.0, specifically the fix for
ticket 61fe9745.  Changes
against version 3.18.0 are minimal.
2017-06-16 - Release 3.18.1
SQLite version 3.18.1 is a bug-fix release against version 3.18.0
that fixes the auto_vacuum corruption bug described in ticket
fda22108.  This release was
created for users who need that bug fix but do not yet want to upgrade 
to version 3.19.3.  
2017-06-08 - Release 3.19.3
Version 3.19.3 is an emergency patch release to fix a 
bug in 
auto_vacuum logic that can lead to database corruption.
The bug was introduced in version 3.16.0 
(2017-01-02).  Though the bug is obscure and rarely
encountered, upgrading is recommended for all users, and
especially for users who turn on auto_vacuum.
2017-05-25 - Release 3.19.2
Still more problems have been found in the LEFT JOIN
flattening optimization
that was added in the 3.19.0 release.  This patch release fixes all known
issues with that optimization and adds new test cases.  Hopefully this
will be the last patch.
2017-05-24 - Release 3.19.1
One of the new query planner optimizations in the 3.19.0 release contained
bugs.  The 3.19.1 patch release fixes them.

Beginning with 3.19.0, subqueries and views on the right-hand side of
a LEFT JOIN operator could sometimes be
flattened into the
main query.  The new optimization worked well for all of the test cases
that the developers devised, and for millions of legacy test cases, but
once 3.19.0 was released, users found some other cases where the optimization
failed.  Ticket
cad1ab4cb7b0fc344 contains
examples.

These problems exist only in 3.19.0.  Users of SQLite 3.19.0 should
upgrade, but users of all prior versions of SQLite are safe.
2017-05-22 - Release 3.19.0
SQLite version 3.19.0 is a regularly scheduled maintenance release.

The emphasis on this release is improvements to the query planner.
There are also some obscure bug fixes.  There is no reason to upgrade
unless you are having problems with a prior release.
2017-03-30 - Release 3.18.0
SQLite version 3.18.0 is a regularly scheduled maintenance release.

This release features an initial implementation the 
"PRAGMA optimize" command.  This command can now be used to cause
ANALYZE to be run on an as-needed basis.  Applications should invoke
"PRAGMA optimize" just before closing the database connection.
The "PRAGMA optimize" statement will likely be enhanced to do other
kinds of automated database maintenance in future releases.

The Fossil version control system that is
used to manage the SQLite project has been upgraded to use SHA3-256 hashes
instead of SHA1.  Therefore, the version identifications for SQLite now
show a 64-hex-digit SHA3-256 hash rather than the 40-hex-digit SHA1 hash.

See the change log for other enhancements and optimizations
in this release.
2017-02-13 - Release 3.17.0
SQLite version 3.17.0 is a regularly scheduled maintenance release.

Most of the changes in this release are performance optimizations.
Optimizations to the R-Tree extension are especially noticeable.

In this release, the default size of the 
lookaside buffer allocated for each database connection
is increased from 64,000 to 120,000 bytes.  This provides improved
performance on many common workloads in exchange for a small increase
in memory usage.
Applications that value a small memory footprint over raw speed
can change the lookaside buffer size back to its old value (or to zero)
using the SQLITE_DEFAULT_LOOKASIDE compile-time option, or the
sqlite3_config(SQLITE_CONFIG_LOOKASIDE)
start-time setting, or the
sqlite3_db_config(SQLITE_DBCONFIG_LOOKASIDE)
run-time setting.
2017-01-06 - Release 3.16.2
One of the performance optimizations added in 3.16.0 caused triggers
and foreign keys to malfunction for the REPLACE statement on
WITHOUT ROWID tables that lack secondary indexes.  This patch
release fixes the problem.  See ticket 
30027b613b4 for details.
2017-01-03 - Release 3.16.1
SQLite version 3.16.1 fixes a bug in the row-value logic for UPDATE
statements inside of triggers.  The bug has been there since row-values
were added by release 3.15.0, but was not discovered until just a few
minutes after the 3.16.0 release was published, and so it was not fixed
by 3.16.0.  This patch release is version 3.16.0 with the row-value bug fix.
2017-01-02 - Release 3.16.0
SQLite version 3.16.0 is a regularly schedule maintenance release.


This release includes many microoptimizations that collectively reduce
the CPU cycle count by about 9%,
add there have been important enhancements to the command-line shell.

Support for PRAGMA functions is added,
so that many pragma statements can be used as part of a larger SQL query.
This is considered an experimental feature.
We do not anticipate any changes to the PRAGMA function interface, but
will keep continue to call this interface "experimental" for a few release
cycles in case unforeseen issues arise.

See the change log for other enhancements.
2016-11-28 - Release 3.15.2
SQLite version 3.15.2 is a bug-fix patch release that fixes several minor
issues in the 3.15.0 and 3.15.1 releases.
2016-11-04 - Release 3.15.1
SQLite version 3.15.1 is a bug-fix patch release that fixes some minor
issues in the 3.15.0 release.
2016-10-14 - Release 3.15.0
SQLite version 3.15.0 is a regularly scheduled maintenance release.
The key feature in this release is the added support for
row values.  There are also other enhancements and
fixes for a number of obscure bugs.

The 3.15.0 release uses about 7% fewer CPU cycles than 3.14.2.
Most of the improvement in this release
is in the SQL parser, query planner, and
byte-code generator (the front-end) corresponding
to the sqlite3_prepare_v2() interface.  Overall,
version 3.15.0 uses about half as much CPU time as
version 3.8.1 (2013-10-17).  These
performance measurements are made using the "speedtest1.c"
workload on x64 compiled with gcc and -Os. Performance
improvements may vary with different platforms and
workloads.
2016-09-12 - Release 3.14.2
SQLite version 3.14.2 fixes several obscure bugs and adds 
improved support for building SQLite using the STDCALL calling
convention on 32-bit windows systems.  Upgrading from versions
3.14 and 3.14.1 is optional.
2016-08-11 - Release 3.14.1
SQLite version 3.14.1 adds a small patch to improve the performance
of the pcache1TruncateUnsafe() routine for cases when the only a few
pages on the end of the cache are being removed.  This causes COMMITs
to run faster when there is a very large page cache.  Upgrading from
version 3.14 is optional.
2016-08-08 - Release 3.14
SQLite version 3.14 (the "&#960;" release)
is a regularly scheduled maintenance
release containing performance enhancements, new features, and fixes for
obscure bugs.
2016-05-18 - Release 3.13.0
SQLite version 3.13.0 is a regularly schedule maintenance release containing
performance enhancements and fixes for obscure bugs.
2016-04-18 - Release 3.12.2
Yikes!  The 3.12.0 and 3.12.1 releases contain a backwards compatibility bug!
   Tables that declare a column with type "INTEGER" PRIMARY KEY
   (where the datatype name INTEGER is quoted) generate an incompatible
   database file.  The mistake came about because the developers have never
   thought to put a typename in quotes before, and so there was no documentation 
   of that capability nor any tests.  (There are tests now, though, of course.)
   Instances of quoting the datatype name are probably infrequent in the wild,
   so we do not expect the impact of this bug to be too severe.
   Upgrading is still strongly recommended.
Fixes for three other minor issues were included in this patch release.
   The other issues would have normally been deferred until the next scheduled
   release, but since a patch release is being issued anyhow, they might as
   well be included.
2016-04-08 - Release 3.12.1
SQLite version 3.12.1 is an emergency patch release to address a 
   crash bug that snuck
   into version 3.12.0.  Upgrading from version 3.12.0 is highly
   recommended.
Another minor problem involving datatypes on view columns, and
   a query planner deficiency are fixed at the same time.  These two
   issues did not justify a new release on their own, but since a release
   is being issued to deal with the crash bug, we included these other
   fixes for good measure.
2016-03-29 - Release 3.12.0
SQLite version 3.12.0 is a regularly scheduled maintenance release.
   A notable change in this release is an
   increase in the default page size for newly created database files.
   There are also various performance improvements.
   See the change log for details.
2016-03-03 - Release 3.11.1
SQLite version 3.11.1 is a patch release that fixes problems in the
   new FTS5 extension and increases a default setting in the spellfix1
   extension, and implements enhancements to some of the Windows makefiles.
   The SQLite core is unchanged from 3.11.0. Upgrading is optional.
2016-02-15 - Release 3.11.0
SQLite version 3.11.0 is a regularly scheduled maintenance release.
2016-01-20 - Release 3.10.2
Yikes!  An optimization attempt gone bad resulted in a 
bug in the LIKE operator
which is fixed by this patch release.
Three other minor but low-risk fixes are also included in the patch.
2016-01-14 - Release 3.10.1
SQLite version 3.10.1 is a bug-fix release primarily targeting the
fix for the query planner bug
cb3aa0641d9a4 discovered by
Mapscape.  Also included is a minor API enhancement requested by
the Firefox developers at Mozilla.  The differences from version 
3.10.0 are minimal.
2016-01-06 - Release 3.10.0
SQLite version 3.10.0 is a regularly scheduled maintenance release.


Old news...
This page last modified on  2022-12-27 17:04:09 UTC


c2: SQLite Unlock-Notify API
c5: unlock_notify.html
c4:
Using the sqlite3_unlock_notify() API


/* This example uses the pthreads API */
#include &lt;pthread.h&gt;

/*
** A pointer to an instance of this structure is passed as the user-context
** pointer when registering for an unlock-notify callback.
*/
typedef struct UnlockNotification UnlockNotification;
struct UnlockNotification {
  int fired;                         /* True after unlock event has occurred */
  pthread_cond_t cond;               /* Condition variable to wait on */
  pthread_mutex_t mutex;             /* Mutex to protect structure */
};

/*
** This function is an unlock-notify callback registered with SQLite.
*/
static void unlock_notify_cb(void **apArg, int nArg){
  int i;
  for(i=0; i&lt;nArg; i++){
    UnlockNotification *p = (UnlockNotification *)apArg[i];
    pthread_mutex_lock(&p-&gt;mutex);
    p-&gt;fired = 1;
    pthread_cond_signal(&p-&gt;cond);
    pthread_mutex_unlock(&p-&gt;mutex);
  }
}

/*
** This function assumes that an SQLite API call (either sqlite3_prepare_v2() 
** or sqlite3_step()) has just returned SQLITE_LOCKED. The argument is the
** associated database connection.
**
** This function calls sqlite3_unlock_notify() to register for an 
** unlock-notify callback, then blocks until that callback is delivered 
** and returns SQLITE_OK. The caller should then retry the failed operation.
**
** Or, if sqlite3_unlock_notify() indicates that to block would deadlock 
** the system, then this function returns SQLITE_LOCKED immediately. In 
** this case the caller should not retry the operation and should roll 
** back the current transaction (if any).
*/
static int wait_for_unlock_notify(sqlite3 *db){
  int rc;
  UnlockNotification un;

  /* Initialize the UnlockNotification structure. */
  un.fired = 0;
  pthread_mutex_init(&un.mutex, 0);
  pthread_cond_init(&un.cond, 0);

  /* Register for an unlock-notify callback. */
  rc = sqlite3_unlock_notify(db, unlock_notify_cb, (void *)&un);
  assert( rc==SQLITE_LOCKED || rc==SQLITE_OK );

  /* The call to sqlite3_unlock_notify() always returns either SQLITE_LOCKED 
  ** or SQLITE_OK. 
  **
  ** If SQLITE_LOCKED was returned, then the system is deadlocked. In this
  ** case this function needs to return SQLITE_LOCKED to the caller so 
  ** that the current transaction can be rolled back. Otherwise, block
  ** until the unlock-notify callback is invoked, then return SQLITE_OK.
  */
  if( rc==SQLITE_OK ){
    pthread_mutex_lock(&un.mutex);
    if( !un.fired ){
      pthread_cond_wait(&un.cond, &un.mutex);
    }
    pthread_mutex_unlock(&un.mutex);
  }

  /* Destroy the mutex and condition variables. */
  pthread_cond_destroy(&un.cond);
  pthread_mutex_destroy(&un.mutex);

  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_step().
** It functions in the same way as step(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API step()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_step(sqlite3_stmt *pStmt){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_step(pStmt)) ){
    rc = wait_for_unlock_notify(sqlite3_db_handle(pStmt));
    if( rc!=SQLITE_OK ) break;
    sqlite3_reset(pStmt);
  }
  return rc;
}

/*
** This function is a wrapper around the SQLite function sqlite3_prepare_v2().
** It functions in the same way as prepare_v2(), except that if a required
** shared-cache lock cannot be obtained, this function may block waiting for
** the lock to become available. In this scenario the normal API prepare_v2()
** function always returns SQLITE_LOCKED.
**
** If this function returns SQLITE_LOCKED, the caller should rollback
** the current transaction (if any) and try again later. Otherwise, the
** system may become deadlocked.
*/
int sqlite3_blocking_prepare_v2(
  sqlite3 *db,              /* Database handle. */
  const char *zSql,         /* UTF-8 encoded SQL statement. */
  int nSql,                 /* Length of zSql in bytes. */
  sqlite3_stmt **ppStmt,    /* OUT: A pointer to the prepared statement */
  const char **pz           /* OUT: End of parsed string */
){
  int rc;
  while( SQLITE_LOCKED==(rc = sqlite3_prepare_v2(db, zSql, nSql, ppStmt, pz)) ){
    rc = wait_for_unlock_notify(db);
    if( rc!=SQLITE_OK ) break;
  }
  return rc;
}




  When two or more connections access the same database in shared-cache 
  mode, read and write (shared and exclusive) locks on individual tables
  are used to ensure that concurrently executing transactions are kept 
  isolated. Before writing to a table, a write (exclusive) lock must be 
  obtained on that table. Before reading, a read (shared) lock must be
  obtained. A connection releases all held table locks when it concludes
  its transaction. If a connection cannot obtain a required lock, then
  the call to sqlite3_step() returns SQLITE_LOCKED.


  Although it is less common, a call to sqlite3_prepare() or
  sqlite3_prepare_v2() may also return SQLITE_LOCKED if it cannot obtain
  a read-lock on the sqlite_schema table of each attached database. These
  APIs need to read the schema data contained in the sqlite_schema table
  in order to compile SQL statements to sqlite3_stmt* objects.


  This article presents a technique using the SQLite sqlite3_unlock_notify()
  interface such that calls to sqlite3_step() and sqlite3_prepare_v2()
  block until the required locks are available instead of returning 
  SQLITE_LOCKED immediately. If the
  sqlite3_blocking_step() or sqlite3_blocking_prepare_v2() functions presented 
  to the left return SQLITE_LOCKED, this indicates that to block would 
  deadlock the system.


  The sqlite3_unlock_notify() API, which is only available if the library is
  compiled with the pre-processor symbol SQLITE_ENABLE_UNLOCK_NOTIFY defined,
  is documented here. This article is not a
  substitute for reading the full API documentation!


  The sqlite3_unlock_notify() interface is designed for use in systems
  that have a separate thread assigned to each database connection.  There
  is nothing in the implementation that prevents a single thread from running
  multiple database connections.  However, the sqlite3_unlock_notify()
  interface only works on a single connection at a time, so the lock
  resolution logic presented here will only work for a single 
  database connection per thread.

The sqlite3_unlock_notify() API


  After a call to sqlite3_step() or sqlite3_prepare_v2() returns 
  SQLITE_LOCKED, the sqlite3_unlock_notify() API may be invoked to register 
  for an unlock-notify callback. The unlock-notify callback is invoked by 
  SQLite after the database connection holding the table-lock that prevented
  the call to sqlite3_step() or sqlite3_prepare_v2() from succeeding has
  finished its transaction and released all locks. For example, if a call to
  sqlite3_step() is an attempt to read from table X, and some other connection
  Y is holding a write-lock on table X, then sqlite3_step() will return
  SQLITE_LOCKED. If sqlite3_unlock_notify() is then called, the unlock-notify
  callback will be invoked after connection Y's transaction is concluded. The
  connection that the unlock-notify callback is waiting on, in this case
  connection Y, is known as the "blocking connection".


  If a call to sqlite3_step() that attempts to write to a database table
  returns SQLITE_LOCKED, then more than one other connection may be holding 
  a read-lock on the database table in question. In this case SQLite simply
  selects one of those other connections arbitrarily and issues the 
  unlock-notify callback when that connection's transaction is finished.
  Whether the call to sqlite3_step() was blocked by one or many connections, 
  when the corresponding unlock-notify callback is issued it is not 
  guaranteed that the required lock is available, only that it may be.


  When the unlock-notify callback is issued, it is issued from within a
  call to sqlite3_step() (or sqlite3_close()) associated with the blocking 
  connection. It is illegal to invoke any sqlite3_XXX() API functions from
  within an unlock-notify callback. The expected use is that the unlock-notify
  callback will signal some other waiting thread or schedule some action
  to take place later.


  The algorithm used by the sqlite3_blocking_step() function is as follows:


   Call sqlite3_step() on the supplied statement handle. If the call
       returns anything other than SQLITE_LOCKED, then return this value
       to the caller. Otherwise, continue.

   Invoke sqlite3_unlock_notify() on the database connection handle
       associated with the supplied statement handle to register for an
       unlock-notify callback. If the call to unlock_notify() returns
       SQLITE_LOCKED, then return this value to the caller.

   Block until the unlock-notify callback is invoked by another thread.

   Call sqlite3_reset() on the statement handle. Since an 
       SQLITE_LOCKED error may only occur on the first call to sqlite3_step()
       (it is not possible for one call to sqlite3_step() to return 
       SQLITE_ROW and then the next SQLITE_LOCKED), the statement handle may 
       be reset at this point without affecting the results of the query
       from the point of view of the caller. If sqlite3_reset() were not
       called at this point, the next call to sqlite3_step() would return
       SQLITE_MISUSE.

   Return to step 1.



  The algorithm used by the sqlite3_blocking_prepare_v2() function is similar,
  except that step 4 (resetting the statement handle) is omitted.


Writer Starvation


  Multiple connections may hold a read-lock simultaneously.  If many threads
  are acquiring overlapping read-locks, it might be the case that at least
  one thread is always holding a read lock.
  Then a table waiting for a write-lock will wait forever.
  This scenario is called "writer starvation."


  SQLite helps applications avoid writer starvation. After any attempt to
  obtain a write-lock on a table fails (because one or more other 
  connections are holding read-locks), all attempts to open new transactions
  on the shared-cache fail until one of the following is true:


   The current writer concludes its transaction, OR
   The number of open read-transactions on the shared-cache drops to zero.



  Failed attempts to open new read-transactions return SQLITE_LOCKED to the
  caller. If the caller then calls sqlite3_unlock_notify() to register for
  an unlock-notify callback, the blocking connection is the connection that
  currently has an open write-transaction on the shared-cache. This prevents
  writer-starvation since if no new read-transactions may be opened and 
  assuming all existing read-transactions are eventually concluded, the 
  writer will eventually have an opportunity to obtain the required
  write-lock.

The pthreads API

   By the time sqlite3_unlock_notify() is invoked by
      wait_for_unlock_notify(), it is possible that the blocking connection
      that prevented the sqlite3_step() or sqlite3_prepare_v2() call from 
      succeeding has already finished its transaction. In this case, the 
      unlock-notify callback is invoked immediately, before 
      sqlite3_unlock_notify() returns. Or, it is possible that the 
      unlock-notify callback is invoked by a second thread after 
      sqlite3_unlock_notify() is called but before the thread starts waiting 
      to be asynchronously signaled. 

   Exactly how such a potential race-condition is handled depends on the
      threads and synchronization primitives interface used by the application.
      This example uses pthreads, the interface provided by modern UNIX-like 
      systems, including Linux.

   The pthreads interface provides the pthread_cond_wait() function. 
      This function allows the caller to simultaneously release a mutex
      and start waiting for an asynchronous signal. Using this function,
      a "fired" flag and a mutex, the race-condition described above may
      be eliminated as follows:

   When the unlock-notify callback is invoked, which may be before the
      thread that called sqlite3_unlock_notify() begins waiting for the 
      asynchronous signal, it does the following:

    
       Obtains the mutex.
       Sets the "fired" flag to true.
       Attempts to signal a waiting thread.
       Releases the mutex.
    

   When the wait_for_unlock_notify() thread is ready to begin waiting for
      the unlock-notify callback to arrive, it:

    
       Obtains the mutex.
       Checks if the "fired" flag has been set. If so, the unlock-notify
           callback has already been invoked. Release the mutex and continue.
       Atomically releases the mutex and begins waiting for the
           asynchronous signal. When the signal arrives, continue.
    

   This way, it doesn't matter if the unlock-notify callback has already
      been invoked, or is being invoked, when the wait_for_unlock_notify() 
      thread begins blocking.

Possible Enhancements

   The code in this article could be improved in at least two ways:

  
     It could manage thread priorities.
     It could handle a special case of SQLITE_LOCKED that can occur
         when dropping a table or index.
  

  
    Even though the sqlite3_unlock_notify() function only allows the caller
    to specify a single user-context pointer, an unlock-notify callback 
    is passed an array of such context pointers. This is because if when
    a blocking connection concludes its transaction, if there is more
    than one unlock-notify registered to call the same C function, the
    context-pointers are marshaled into an array and a single callback
    issued. If each thread were assigned a priority, then instead of just
    signaling the threads in arbitrary order as this implementation does,
    higher priority threads could be signaled before lower priority threads.

  
    If a "DROP TABLE" or "DROP INDEX" SQL command is executed, and the
    same database connection currently has one or more actively
    executing SELECT statements, then SQLITE_LOCKED is returned. If
    sqlite3_unlock_notify() is called in this case, then the specified
    callback will be invoked immediately. Re-attempting the "DROP
    TABLE" or "DROP INDEX" statement will return another SQLITE_LOCKED 
    error. In the implementation of sqlite3_blocking_step() shown to the
    left, this could cause an infinite loop.

  
    The caller could distinguish between this special "DROP TABLE|INDEX" 
    case and other cases by using extended error codes. When it is appropriate
    to call sqlite3_unlock_notify(), the extended error code is
    SQLITE_LOCKED_SHAREDCACHE. Otherwise, in the "DROP TABLE|INDEX" case,
    it is just plain SQLITE_LOCKED. Another solution might be to limit
    the number of times that any single query could be reattempted (to say 
    100). Although this might be less efficient than one might wish, the 
    situation in question is not likely to occur often.


This page last modified on  2020-06-19 13:05:24 UTC


c2: Defense Against The Dark Arts
c5: security.html
c4:
1. SQLite Always Validates Its Inputs


SQLite should never crash, overflow a buffer, leak memory,
or exhibit any other harmful behavior, even when presented with
maliciously malformed SQL inputs or database files.  SQLite should
always detect erroneous inputs and raise an error, not crash or
corrupt memory.
Any malfunction caused by an SQL input or database file
is considered a serious bug and will be promptly addressed when
brought to the attention of the SQLite developers.  SQLite is
extensively fuzz-tested to help ensure that it is resistant
to these kinds of errors.


Nevertheless, bugs happen.
If you are writing an application that sends untrusted SQL inputs
or database files to SQLite, there are additional steps you can take
to help reduce the attack surface and
prevent zero-day exploits caused by undetected bugs.

1.1. Untrusted SQL Inputs

Applications that accept untrusted SQL inputs should take the following
precautions:



Set the SQLITE_DBCONFIG_DEFENSIVE flag.
This prevents ordinary SQL statements from deliberately corrupting the 
database file.  SQLite should be proof against attacks that involve both
malicious SQL inputs and a maliciously corrupted database file at the
same time.  Nevertheless, denying a script-only attacker access to 
corrupt database inputs provides an extra layer of defense.


Reduce the limits that SQLite imposes on inputs.  This can help prevent
denial of service attacks and other kinds of mischief that can occur
as a result of unusually large inputs.  You can do this either at compile-time
using -DSQLITE_MAX_... options, or at run-time using the
sqlite3_limit() interface.  Most applications can reduce limits
dramatically without impacting functionality.  The table below
provides some suggestions, though exact values will vary depending
on the application:


Limit SettingDefault ValueHigh-security Value
LIMIT_LENGTH1,000,000,0001,000,000
LIMIT_SQL_LENGTH1,000,000,000100,000
LIMIT_COLUMN2,000100
LIMIT_EXPR_DEPTH1,00010
LIMIT_COMPOUND_SELECT5003
LIMIT_VDBE_OP250,000,00025,000
LIMIT_FUNCTION_ARG1278
LIMIT_ATTACH100
LIMIT_LIKE_PATTERN_LENGTH50,00050
LIMIT_VARIABLE_NUMBER99910
LIMIT_TRIGGER_DEPTH1,00010



Consider using the sqlite3_set_authorizer() interface to limit
the scope of SQL that will be processed.  For example, an application
that does not need to change the database schema might add an
sqlite3_set_authorizer() callback that causes any CREATE or DROP
statement to fail.


The SQL language is very powerful, and so it is always possible for
malicious SQL inputs (or erroneous SQL inputs caused by an application
bug) to submit SQL that runs for a very long time.  To prevent this
from becoming a denial-of-service attack, consider using the
sqlite3_progress_handler() interface to invoke a callback periodically
as each SQL statement runs, and have that callback return non-zero to
abort the statement if the statement runs for too long.  Alternatively,
set a timer in a separate thread and invoke sqlite3_interrupt() when
the timer goes off to prevent the SQL statement from running forever.


Limit the maximum amount of memory that SQLite will allocate using
the sqlite3_hard_heap_limit64() interface.  This helps prevent
denial-of-service attacks.  To find out how much heap space an
application actually needs, run the it against typical inputs and
then measure the maximum instantaneous memory usage with the 
sqlite3_memory_highwater() interface.  Set the hard heap limit
to the maximum observed instantaneous memory usage plus some margin.


Consider setting the SQLITE_MAX_ALLOCATION_SIZE compile-time option
to something smaller than its default value of 2147483391 (0x7ffffeff).
A value of 100000000 (100 million) or even smaller would not be unreasonable,
depending on the application.


For embedded systems, consider compiling SQLite with the
-DSQLITE_ENABLE_MEMSYS5 option and then providing SQLite with
a fixed chunk of memory to use as its heap via the
sqlite3_config(SQLITE_CONFIG_HEAP) interface. This will
prevent malicious SQL from executing a denial-of-service attack
by using an excessive amount of memory. If (say) 5 MB of memory
is provided for SQLite to use, once that much has been consumed,
SQLite will start returning SQLITE_NOMEM errors rather than
soaking up memory needed by other parts of the application.
This also sandboxes SQLite's memory so that a write-after-free
error in some other part of the application will not cause
problems for SQLite, or vice versa.



To control memory usage in the printf() SQL function, compile
with "-DSQLITE_PRINTF_PRECISION_LIMIT=100000" or some similarly
reasonable value.
This #define limits the width and precision for %-substitutions in the
printf() function, and thus prevents a hostile SQL statement from
consuming large amounts of RAM via constructs such as
"printf('%1000000000s','hi')".

Note that SQLite uses its built-in printf() internally to help it
format the sql column in the sqlite_schema table.  For that reason,
no table, index, view, or trigger definition can be much larger than the
precision limit.  You can set a precision limit of less than 100000,
but be careful that whatever precision limit you use is at least as
long as the longest CREATE statement in your schema.






1.2. Untrusted SQLite Database Files

Applications that read or write SQLite database files of uncertain
provenance should take precautions enumerated below.

Even if the application does not deliberately accept database files 
from untrusted sources, beware of attacks in which a local 
database file is altered.  For best security, any database file which 
might have ever been writable by an agent in a different security domain
should be treated as suspect.



If the application includes any custom SQL functions or 
custom virtual tables that have side effects or that might leak
privileged information, then the application should use one or more
of the techniques below to prevent a maliciously crafted database
schema from surreptitiously running those SQL functions and/or
virtual tables for nefarious purposes:

 Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
     on each database connection as soon as it is opened.
 Run the PRAGMA trusted_schema=OFF statement on each database connection
     as soon as it is opened.
 Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
 Disable the surreptitious use of custom SQL functions and virtual tables
     by setting the SQLITE_DIRECTONLY flag on all custom SQL functions and
     the SQLITE_VTAB_DIRECTONLY flag on all custom virtual tables.



If the application does not use triggers or views, consider disabling the
unused capabilities with:

sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_TRIGGER,0,0);
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_VIEW,0,0);





For reading database files that are unusually high-risk, such as database
files that are received from remote machines, and possibly from anonymous
contributors, the following extra precautions
might be justified.  These added defenses come with performance costs,
however, and so may not be appropriate in every situation:



Run PRAGMA integrity_check or PRAGMA quick_check on the database
as the first SQL statement after opening the database files and
prior to running any other SQL statements.  Reject and refuse to
process any database file containing errors.


Enable the PRAGMA cell_size_check=ON setting.



Do not enable memory-mapped I/O.
In other words, make sure that PRAGMA mmap_size=0.



2. Summary


The precautions above are not required in order to use SQLite safely
with potentially hostile inputs.
However, they do provide an extra layer of defense against zero-day
exploits and are encouraged for applications that pass data from
untrusted sources into SQLite.
This page last modified on  2022-11-07 14:28:05 UTC


