diff options
Diffstat (limited to 'ext/misc')
55 files changed, 30431 insertions, 0 deletions
diff --git a/ext/misc/README.md b/ext/misc/README.md new file mode 100644 index 0000000..69cb230 --- /dev/null +++ b/ext/misc/README.md @@ -0,0 +1,60 @@ +## Miscellaneous Extensions + +This folder contains a collection of smaller loadable extensions. +See <https://www.sqlite.org/loadext.html> for instructions on how +to compile and use loadable extensions. +Each extension in this folder is implemented in a single file of C code. + +Each source file contains a description in its header comment. See the +header comments for details about each extension. Additional notes are +as follows: + + * **carray.c** — This module implements the + [carray](https://www.sqlite.org/carray.html) table-valued function. + It is a good example of how to go about implementing a custom + [table-valued function](https://www.sqlite.org/vtab.html#tabfunc2). + + * **csv.c** — A [virtual table](https://sqlite.org/vtab.html) + for reading + [Comma-Separated-Value (CSV) files](https://en.wikipedia.org/wiki/Comma-separated_values). + + * **dbdump.c** — This is not actually a loadable extension, but + rather a library that implements an approximate equivalent to the + ".dump" command of the + [command-line shell](https://www.sqlite.org/cli.html). + + * **json1.c** — Various SQL functions and table-valued functions + for processing JSON. This extension is already built into the + [SQLite amalgamation](https://sqlite.org/amalgamation.html). See + <https://sqlite.org/json1.html> for additional information. + + * **memvfs.c** — This file implements a custom + [VFS](https://www.sqlite.org/vfs.html) that stores an entire database + file in a single block of RAM. It serves as a good example of how + to implement a simple custom VFS. + + * **rot13.c** — This file implements the very simple rot13() + substitution function. This file makes a good template for implementing + new custom SQL functions for SQLite. + + * **series.c** — This is an implementation of the + "generate_series" [virtual table](https://www.sqlite.org/vtab.html). + It can make a good template for new custom virtual table implementations. + + * **shathree.c** — An implementation of the sha3() and + sha3_query() SQL functions. The file is named "shathree.c" instead + of "sha3.c" because the default entry point names in SQLite are based + on the source filename with digits removed, so if we used the name + "sha3.c" then the entry point would conflict with the prior "sha1.c" + extension. + + * **unionvtab.c** — Implementation of the unionvtab and + [swarmvtab](https://sqlite.org/swarmvtab.html) virtual tables. + These virtual tables allow a single + large table to be spread out across multiple database files. In the + case of swarmvtab, the individual database files can be attached on + demand. + + * **zipfile.c** — A [virtual table](https://sqlite.org/vtab.html) + that can read and write a + [ZIP archive](https://en.wikipedia.org/wiki/Zip_%28file_format%29). diff --git a/ext/misc/amatch.c b/ext/misc/amatch.c new file mode 100644 index 0000000..bafa432 --- /dev/null +++ b/ext/misc/amatch.c @@ -0,0 +1,1501 @@ +/* +** 2013-03-14 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file contains code for a demonstration virtual table that finds +** "approximate matches" - strings from a finite set that are nearly the +** same as a single input string. The virtual table is called "amatch". +** +** A amatch virtual table is created like this: +** +** CREATE VIRTUAL TABLE f USING approximate_match( +** vocabulary_table=<tablename>, -- V +** vocabulary_word=<columnname>, -- W +** vocabulary_language=<columnname>, -- L +** edit_distances=<edit-cost-table> +** ); +** +** When it is created, the new amatch table must be supplied with the +** the name of a table V and columns V.W and V.L such that +** +** SELECT W FROM V WHERE L=$language +** +** returns the allowed vocabulary for the match. If the "vocabulary_language" +** or L columnname is left unspecified or is an empty string, then no +** filtering of the vocabulary by language is performed. +** +** For efficiency, it is essential that the vocabulary table be indexed: +** +** CREATE vocab_index ON V(W) +** +** A separate edit-cost-table provides scoring information that defines +** what it means for one string to be "close" to another. +** +** The edit-cost-table must contain exactly four columns (more precisely, +** the statement "SELECT * FROM <edit-cost-table>" must return records +** that consist of four columns). It does not matter what the columns are +** named. +** +** Each row in the edit-cost-table represents a single character +** transformation going from user input to the vocabulary. The leftmost +** column of the row (column 0) contains an integer identifier of the +** language to which the transformation rule belongs (see "MULTIPLE LANGUAGES" +** below). The second column of the row (column 1) contains the input +** character or characters - the characters of user input. The third +** column contains characters as they appear in the vocabulary table. +** And the fourth column contains the integer cost of making the +** transformation. For example: +** +** CREATE TABLE f_data(iLang, cFrom, cTo, Cost); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, '', 'a', 100); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, 'b', '', 87); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, 'o', 'oe', 38); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, 'oe', 'o', 40); +** +** The first row inserted into the edit-cost-table by the SQL script +** above indicates that the cost of having an extra 'a' in the vocabulary +** table that is missing in the user input 100. (All costs are integers. +** Overall cost must not exceed 16777216.) The second INSERT statement +** creates a rule saying that the cost of having a single letter 'b' in +** user input which is missing in the vocabulary table is 87. The third +** INSERT statement mean that the cost of matching an 'o' in user input +** against an 'oe' in the vocabulary table is 38. And so forth. +** +** The following rules are special: +** +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, '?', '', 97); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, '', '?', 98); +** INSERT INTO f_data(iLang, cFrom, cTo, Cost) VALUES(0, '?', '?', 99); +** +** The '?' to '' rule is the cost of having any single character in the input +** that is not found in the vocabular. The '' to '?' rule is the cost of +** having a character in the vocabulary table that is missing from input. +** And the '?' to '?' rule is the cost of doing an arbitrary character +** substitution. These three generic rules apply across all languages. +** In other words, the iLang field is ignored for the generic substitution +** rules. If more than one cost is given for a generic substitution rule, +** then the lowest cost is used. +** +** Once it has been created, the amatch virtual table can be queried +** as follows: +** +** SELECT word, distance FROM f +** WHERE word MATCH 'abcdefg' +** AND distance<200; +** +** This query outputs the strings contained in the T(F) field that +** are close to "abcdefg" and in order of increasing distance. No string +** is output more than once. If there are multiple ways to transform the +** target string ("abcdefg") into a string in the vocabulary table then +** the lowest cost transform is the one that is returned. In this example, +** the search is limited to strings with a total distance of less than 200. +** +** For efficiency, it is important to put tight bounds on the distance. +** The time and memory space needed to perform this query is exponential +** in the maximum distance. A good rule of thumb is to limit the distance +** to no more than 1.5 or 2 times the maximum cost of any rule in the +** edit-cost-table. +** +** The amatch is a read-only table. Any attempt to DELETE, INSERT, or +** UPDATE on a amatch table will throw an error. +** +** It is important to put some kind of a limit on the amatch output. This +** can be either in the form of a LIMIT clause at the end of the query, +** or better, a "distance<NNN" constraint where NNN is some number. The +** running time and memory requirement is exponential in the value of NNN +** so you want to make sure that NNN is not too big. A value of NNN that +** is about twice the average transformation cost seems to give good results. +** +** The amatch table can be useful for tasks such as spelling correction. +** Suppose all allowed words are in table vocabulary(w). Then one would create +** an amatch virtual table like this: +** +** CREATE VIRTUAL TABLE ex1 USING amatch( +** vocabtable=vocabulary, +** vocabcolumn=w, +** edit_distances=ec1 +** ); +** +** Then given an input word $word, look up close spellings this way: +** +** SELECT word, distance FROM ex1 +** WHERE word MATCH $word AND distance<200; +** +** MULTIPLE LANGUAGES +** +** Normally, the "iLang" value associated with all character transformations +** in the edit-cost-table is zero. However, if required, the amatch +** virtual table allows multiple languages to be defined. Each query uses +** only a single iLang value. This allows, for example, a single +** amatch table to support multiple languages. +** +** By default, only the rules with iLang=0 are used. To specify an +** alternative language, a "language = ?" expression must be added to the +** WHERE clause of a SELECT, where ? is the integer identifier of the desired +** language. For example: +** +** SELECT word, distance FROM ex1 +** WHERE word MATCH $word +** AND distance<=200 +** AND language=1 -- Specify use language 1 instead of 0 +** +** If no "language = ?" constraint is specified in the WHERE clause, language +** 0 is used. +** +** LIMITS +** +** The maximum language number is 2147483647. The maximum length of either +** of the strings in the second or third column of the amatch data table +** is 50 bytes. The maximum cost on a rule is 1000. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <stdio.h> +#include <ctype.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Forward declaration of objects used by this implementation +*/ +typedef struct amatch_vtab amatch_vtab; +typedef struct amatch_cursor amatch_cursor; +typedef struct amatch_rule amatch_rule; +typedef struct amatch_word amatch_word; +typedef struct amatch_avl amatch_avl; + + +/***************************************************************************** +** AVL Tree implementation +*/ +/* +** Objects that want to be members of the AVL tree should embedded an +** instance of this structure. +*/ +struct amatch_avl { + amatch_word *pWord; /* Points to the object being stored in the tree */ + char *zKey; /* Key. zero-terminated string. Must be unique */ + amatch_avl *pBefore; /* Other elements less than zKey */ + amatch_avl *pAfter; /* Other elements greater than zKey */ + amatch_avl *pUp; /* Parent element */ + short int height; /* Height of this node. Leaf==1 */ + short int imbalance; /* Height difference between pBefore and pAfter */ +}; + +/* Recompute the amatch_avl.height and amatch_avl.imbalance fields for p. +** Assume that the children of p have correct heights. +*/ +static void amatchAvlRecomputeHeight(amatch_avl *p){ + short int hBefore = p->pBefore ? p->pBefore->height : 0; + short int hAfter = p->pAfter ? p->pAfter->height : 0; + p->imbalance = hBefore - hAfter; /* -: pAfter higher. +: pBefore higher */ + p->height = (hBefore>hAfter ? hBefore : hAfter)+1; +} + +/* +** P B +** / \ / \ +** B Z ==> X P +** / \ / \ +** X Y Y Z +** +*/ +static amatch_avl *amatchAvlRotateBefore(amatch_avl *pP){ + amatch_avl *pB = pP->pBefore; + amatch_avl *pY = pB->pAfter; + pB->pUp = pP->pUp; + pB->pAfter = pP; + pP->pUp = pB; + pP->pBefore = pY; + if( pY ) pY->pUp = pP; + amatchAvlRecomputeHeight(pP); + amatchAvlRecomputeHeight(pB); + return pB; +} + +/* +** P A +** / \ / \ +** X A ==> P Z +** / \ / \ +** Y Z X Y +** +*/ +static amatch_avl *amatchAvlRotateAfter(amatch_avl *pP){ + amatch_avl *pA = pP->pAfter; + amatch_avl *pY = pA->pBefore; + pA->pUp = pP->pUp; + pA->pBefore = pP; + pP->pUp = pA; + pP->pAfter = pY; + if( pY ) pY->pUp = pP; + amatchAvlRecomputeHeight(pP); + amatchAvlRecomputeHeight(pA); + return pA; +} + +/* +** Return a pointer to the pBefore or pAfter pointer in the parent +** of p that points to p. Or if p is the root node, return pp. +*/ +static amatch_avl **amatchAvlFromPtr(amatch_avl *p, amatch_avl **pp){ + amatch_avl *pUp = p->pUp; + if( pUp==0 ) return pp; + if( pUp->pAfter==p ) return &pUp->pAfter; + return &pUp->pBefore; +} + +/* +** Rebalance all nodes starting with p and working up to the root. +** Return the new root. +*/ +static amatch_avl *amatchAvlBalance(amatch_avl *p){ + amatch_avl *pTop = p; + amatch_avl **pp; + while( p ){ + amatchAvlRecomputeHeight(p); + if( p->imbalance>=2 ){ + amatch_avl *pB = p->pBefore; + if( pB->imbalance<0 ) p->pBefore = amatchAvlRotateAfter(pB); + pp = amatchAvlFromPtr(p,&p); + p = *pp = amatchAvlRotateBefore(p); + }else if( p->imbalance<=(-2) ){ + amatch_avl *pA = p->pAfter; + if( pA->imbalance>0 ) p->pAfter = amatchAvlRotateBefore(pA); + pp = amatchAvlFromPtr(p,&p); + p = *pp = amatchAvlRotateAfter(p); + } + pTop = p; + p = p->pUp; + } + return pTop; +} + +/* Search the tree rooted at p for an entry with zKey. Return a pointer +** to the entry or return NULL. +*/ +static amatch_avl *amatchAvlSearch(amatch_avl *p, const char *zKey){ + int c; + while( p && (c = strcmp(zKey, p->zKey))!=0 ){ + p = (c<0) ? p->pBefore : p->pAfter; + } + return p; +} + +/* Find the first node (the one with the smallest key). +*/ +static amatch_avl *amatchAvlFirst(amatch_avl *p){ + if( p ) while( p->pBefore ) p = p->pBefore; + return p; +} + +#if 0 /* NOT USED */ +/* Return the node with the next larger key after p. +*/ +static amatch_avl *amatchAvlNext(amatch_avl *p){ + amatch_avl *pPrev = 0; + while( p && p->pAfter==pPrev ){ + pPrev = p; + p = p->pUp; + } + if( p && pPrev==0 ){ + p = amatchAvlFirst(p->pAfter); + } + return p; +} +#endif + +#if 0 /* NOT USED */ +/* Verify AVL tree integrity +*/ +static int amatchAvlIntegrity(amatch_avl *pHead){ + amatch_avl *p; + if( pHead==0 ) return 1; + if( (p = pHead->pBefore)!=0 ){ + assert( p->pUp==pHead ); + assert( amatchAvlIntegrity(p) ); + assert( strcmp(p->zKey, pHead->zKey)<0 ); + while( p->pAfter ) p = p->pAfter; + assert( strcmp(p->zKey, pHead->zKey)<0 ); + } + if( (p = pHead->pAfter)!=0 ){ + assert( p->pUp==pHead ); + assert( amatchAvlIntegrity(p) ); + assert( strcmp(p->zKey, pHead->zKey)>0 ); + p = amatchAvlFirst(p); + assert( strcmp(p->zKey, pHead->zKey)>0 ); + } + return 1; +} +static int amatchAvlIntegrity2(amatch_avl *pHead){ + amatch_avl *p, *pNext; + for(p=amatchAvlFirst(pHead); p; p=pNext){ + pNext = amatchAvlNext(p); + if( pNext==0 ) break; + assert( strcmp(p->zKey, pNext->zKey)<0 ); + } + return 1; +} +#endif + +/* Insert a new node pNew. Return NULL on success. If the key is not +** unique, then do not perform the insert but instead leave pNew unchanged +** and return a pointer to an existing node with the same key. +*/ +static amatch_avl *amatchAvlInsert(amatch_avl **ppHead, amatch_avl *pNew){ + int c; + amatch_avl *p = *ppHead; + if( p==0 ){ + p = pNew; + pNew->pUp = 0; + }else{ + while( p ){ + c = strcmp(pNew->zKey, p->zKey); + if( c<0 ){ + if( p->pBefore ){ + p = p->pBefore; + }else{ + p->pBefore = pNew; + pNew->pUp = p; + break; + } + }else if( c>0 ){ + if( p->pAfter ){ + p = p->pAfter; + }else{ + p->pAfter = pNew; + pNew->pUp = p; + break; + } + }else{ + return p; + } + } + } + pNew->pBefore = 0; + pNew->pAfter = 0; + pNew->height = 1; + pNew->imbalance = 0; + *ppHead = amatchAvlBalance(p); + /* assert( amatchAvlIntegrity(*ppHead) ); */ + /* assert( amatchAvlIntegrity2(*ppHead) ); */ + return 0; +} + +/* Remove node pOld from the tree. pOld must be an element of the tree or +** the AVL tree will become corrupt. +*/ +static void amatchAvlRemove(amatch_avl **ppHead, amatch_avl *pOld){ + amatch_avl **ppParent; + amatch_avl *pBalance = 0; + /* assert( amatchAvlSearch(*ppHead, pOld->zKey)==pOld ); */ + ppParent = amatchAvlFromPtr(pOld, ppHead); + if( pOld->pBefore==0 && pOld->pAfter==0 ){ + *ppParent = 0; + pBalance = pOld->pUp; + }else if( pOld->pBefore && pOld->pAfter ){ + amatch_avl *pX, *pY; + pX = amatchAvlFirst(pOld->pAfter); + *amatchAvlFromPtr(pX, 0) = pX->pAfter; + if( pX->pAfter ) pX->pAfter->pUp = pX->pUp; + pBalance = pX->pUp; + pX->pAfter = pOld->pAfter; + if( pX->pAfter ){ + pX->pAfter->pUp = pX; + }else{ + assert( pBalance==pOld ); + pBalance = pX; + } + pX->pBefore = pY = pOld->pBefore; + if( pY ) pY->pUp = pX; + pX->pUp = pOld->pUp; + *ppParent = pX; + }else if( pOld->pBefore==0 ){ + *ppParent = pBalance = pOld->pAfter; + pBalance->pUp = pOld->pUp; + }else if( pOld->pAfter==0 ){ + *ppParent = pBalance = pOld->pBefore; + pBalance->pUp = pOld->pUp; + } + *ppHead = amatchAvlBalance(pBalance); + pOld->pUp = 0; + pOld->pBefore = 0; + pOld->pAfter = 0; + /* assert( amatchAvlIntegrity(*ppHead) ); */ + /* assert( amatchAvlIntegrity2(*ppHead) ); */ +} +/* +** End of the AVL Tree implementation +******************************************************************************/ + + +/* +** Various types. +** +** amatch_cost is the "cost" of an edit operation. +** +** amatch_len is the length of a matching string. +** +** amatch_langid is an ruleset identifier. +*/ +typedef int amatch_cost; +typedef signed char amatch_len; +typedef int amatch_langid; + +/* +** Limits +*/ +#define AMATCH_MX_LENGTH 50 /* Maximum length of a rule string */ +#define AMATCH_MX_LANGID 2147483647 /* Maximum rule ID */ +#define AMATCH_MX_COST 1000 /* Maximum single-rule cost */ + +/* +** A match or partial match +*/ +struct amatch_word { + amatch_word *pNext; /* Next on a list of all amatch_words */ + amatch_avl sCost; /* Linkage of this node into the cost tree */ + amatch_avl sWord; /* Linkage of this node into the word tree */ + amatch_cost rCost; /* Cost of the match so far */ + int iSeq; /* Sequence number */ + char zCost[10]; /* Cost key (text rendering of rCost) */ + short int nMatch; /* Input characters matched */ + char zWord[4]; /* Text of the word. Extra space appended as needed */ +}; + +/* +** Each transformation rule is stored as an instance of this object. +** All rules are kept on a linked list sorted by rCost. +*/ +struct amatch_rule { + amatch_rule *pNext; /* Next rule in order of increasing rCost */ + char *zFrom; /* Transform from (a string from user input) */ + amatch_cost rCost; /* Cost of this transformation */ + amatch_langid iLang; /* The langauge to which this rule belongs */ + amatch_len nFrom, nTo; /* Length of the zFrom and zTo strings */ + char zTo[4]; /* Tranform to V.W value (extra space appended) */ +}; + +/* +** A amatch virtual-table object +*/ +struct amatch_vtab { + sqlite3_vtab base; /* Base class - must be first */ + char *zClassName; /* Name of this class. Default: "amatch" */ + char *zDb; /* Name of database. (ex: "main") */ + char *zSelf; /* Name of this virtual table */ + char *zCostTab; /* Name of edit-cost-table */ + char *zVocabTab; /* Name of vocabulary table */ + char *zVocabWord; /* Name of vocabulary table word column */ + char *zVocabLang; /* Name of vocabulary table language column */ + amatch_rule *pRule; /* All active rules in this amatch */ + amatch_cost rIns; /* Generic insertion cost '' -> ? */ + amatch_cost rDel; /* Generic deletion cost ? -> '' */ + amatch_cost rSub; /* Generic substitution cost ? -> ? */ + sqlite3 *db; /* The database connection */ + sqlite3_stmt *pVCheck; /* Query to check zVocabTab */ + int nCursor; /* Number of active cursors */ +}; + +/* A amatch cursor object */ +struct amatch_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iRowid; /* The rowid of the current word */ + amatch_langid iLang; /* Use this language ID */ + amatch_cost rLimit; /* Maximum cost of any term */ + int nBuf; /* Space allocated for zBuf */ + int oomErr; /* True following an OOM error */ + int nWord; /* Number of amatch_word objects */ + char *zBuf; /* Temp-use buffer space */ + char *zInput; /* Input word to match against */ + amatch_vtab *pVtab; /* The virtual table this cursor belongs to */ + amatch_word *pAllWords; /* List of all amatch_word objects */ + amatch_word *pCurrent; /* Most recent solution */ + amatch_avl *pCost; /* amatch_word objects keyed by iCost */ + amatch_avl *pWord; /* amatch_word objects keyed by zWord */ +}; + +/* +** The two input rule lists are both sorted in order of increasing +** cost. Merge them together into a single list, sorted by cost, and +** return a pointer to the head of that list. +*/ +static amatch_rule *amatchMergeRules(amatch_rule *pA, amatch_rule *pB){ + amatch_rule head; + amatch_rule *pTail; + + pTail = &head; + while( pA && pB ){ + if( pA->rCost<=pB->rCost ){ + pTail->pNext = pA; + pTail = pA; + pA = pA->pNext; + }else{ + pTail->pNext = pB; + pTail = pB; + pB = pB->pNext; + } + } + if( pA==0 ){ + pTail->pNext = pB; + }else{ + pTail->pNext = pA; + } + return head.pNext; +} + +/* +** Statement pStmt currently points to a row in the amatch data table. This +** function allocates and populates a amatch_rule structure according to +** the content of the row. +** +** If successful, *ppRule is set to point to the new object and SQLITE_OK +** is returned. Otherwise, *ppRule is zeroed, *pzErr may be set to point +** to an error message and an SQLite error code returned. +*/ +static int amatchLoadOneRule( + amatch_vtab *p, /* Fuzzer virtual table handle */ + sqlite3_stmt *pStmt, /* Base rule on statements current row */ + amatch_rule **ppRule, /* OUT: New rule object */ + char **pzErr /* OUT: Error message */ +){ + sqlite3_int64 iLang = sqlite3_column_int64(pStmt, 0); + const char *zFrom = (const char *)sqlite3_column_text(pStmt, 1); + const char *zTo = (const char *)sqlite3_column_text(pStmt, 2); + amatch_cost rCost = sqlite3_column_int(pStmt, 3); + + int rc = SQLITE_OK; /* Return code */ + int nFrom; /* Size of string zFrom, in bytes */ + int nTo; /* Size of string zTo, in bytes */ + amatch_rule *pRule = 0; /* New rule object to return */ + + if( zFrom==0 ) zFrom = ""; + if( zTo==0 ) zTo = ""; + nFrom = (int)strlen(zFrom); + nTo = (int)strlen(zTo); + + /* Silently ignore null transformations */ + if( strcmp(zFrom, zTo)==0 ){ + if( zFrom[0]=='?' && zFrom[1]==0 ){ + if( p->rSub==0 || p->rSub>rCost ) p->rSub = rCost; + } + *ppRule = 0; + return SQLITE_OK; + } + + if( rCost<=0 || rCost>AMATCH_MX_COST ){ + *pzErr = sqlite3_mprintf("%s: cost must be between 1 and %d", + p->zClassName, AMATCH_MX_COST + ); + rc = SQLITE_ERROR; + }else + if( nFrom>AMATCH_MX_LENGTH || nTo>AMATCH_MX_LENGTH ){ + *pzErr = sqlite3_mprintf("%s: maximum string length is %d", + p->zClassName, AMATCH_MX_LENGTH + ); + rc = SQLITE_ERROR; + }else + if( iLang<0 || iLang>AMATCH_MX_LANGID ){ + *pzErr = sqlite3_mprintf("%s: iLang must be between 0 and %d", + p->zClassName, AMATCH_MX_LANGID + ); + rc = SQLITE_ERROR; + }else + if( strcmp(zFrom,"")==0 && strcmp(zTo,"?")==0 ){ + if( p->rIns==0 || p->rIns>rCost ) p->rIns = rCost; + }else + if( strcmp(zFrom,"?")==0 && strcmp(zTo,"")==0 ){ + if( p->rDel==0 || p->rDel>rCost ) p->rDel = rCost; + }else + { + pRule = sqlite3_malloc64( sizeof(*pRule) + nFrom + nTo ); + if( pRule==0 ){ + rc = SQLITE_NOMEM; + }else{ + memset(pRule, 0, sizeof(*pRule)); + pRule->zFrom = &pRule->zTo[nTo+1]; + pRule->nFrom = (amatch_len)nFrom; + memcpy(pRule->zFrom, zFrom, nFrom+1); + memcpy(pRule->zTo, zTo, nTo+1); + pRule->nTo = (amatch_len)nTo; + pRule->rCost = rCost; + pRule->iLang = (int)iLang; + } + } + + *ppRule = pRule; + return rc; +} + +/* +** Free all the content in the edit-cost-table +*/ +static void amatchFreeRules(amatch_vtab *p){ + while( p->pRule ){ + amatch_rule *pRule = p->pRule; + p->pRule = pRule->pNext; + sqlite3_free(pRule); + } + p->pRule = 0; +} + +/* +** Load the content of the amatch data table into memory. +*/ +static int amatchLoadRules( + sqlite3 *db, /* Database handle */ + amatch_vtab *p, /* Virtual amatch table to configure */ + char **pzErr /* OUT: Error message */ +){ + int rc = SQLITE_OK; /* Return code */ + char *zSql; /* SELECT used to read from rules table */ + amatch_rule *pHead = 0; + + zSql = sqlite3_mprintf("SELECT * FROM %Q.%Q", p->zDb, p->zCostTab); + if( zSql==0 ){ + rc = SQLITE_NOMEM; + }else{ + int rc2; /* finalize() return code */ + sqlite3_stmt *pStmt = 0; + rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0); + if( rc!=SQLITE_OK ){ + *pzErr = sqlite3_mprintf("%s: %s", p->zClassName, sqlite3_errmsg(db)); + }else if( sqlite3_column_count(pStmt)!=4 ){ + *pzErr = sqlite3_mprintf("%s: %s has %d columns, expected 4", + p->zClassName, p->zCostTab, sqlite3_column_count(pStmt) + ); + rc = SQLITE_ERROR; + }else{ + while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ + amatch_rule *pRule = 0; + rc = amatchLoadOneRule(p, pStmt, &pRule, pzErr); + if( pRule ){ + pRule->pNext = pHead; + pHead = pRule; + } + } + } + rc2 = sqlite3_finalize(pStmt); + if( rc==SQLITE_OK ) rc = rc2; + } + sqlite3_free(zSql); + + /* All rules are now in a singly linked list starting at pHead. This + ** block sorts them by cost and then sets amatch_vtab.pRule to point to + ** point to the head of the sorted list. + */ + if( rc==SQLITE_OK ){ + unsigned int i; + amatch_rule *pX; + amatch_rule *a[15]; + for(i=0; i<sizeof(a)/sizeof(a[0]); i++) a[i] = 0; + while( (pX = pHead)!=0 ){ + pHead = pX->pNext; + pX->pNext = 0; + for(i=0; a[i] && i<sizeof(a)/sizeof(a[0])-1; i++){ + pX = amatchMergeRules(a[i], pX); + a[i] = 0; + } + a[i] = amatchMergeRules(a[i], pX); + } + for(pX=a[0], i=1; i<sizeof(a)/sizeof(a[0]); i++){ + pX = amatchMergeRules(a[i], pX); + } + p->pRule = amatchMergeRules(p->pRule, pX); + }else{ + /* An error has occurred. Setting p->pRule to point to the head of the + ** allocated list ensures that the list will be cleaned up in this case. + */ + assert( p->pRule==0 ); + p->pRule = pHead; + } + + return rc; +} + +/* +** This function converts an SQL quoted string into an unquoted string +** and returns a pointer to a buffer allocated using sqlite3_malloc() +** containing the result. The caller should eventually free this buffer +** using sqlite3_free. +** +** Examples: +** +** "abc" becomes abc +** 'xyz' becomes xyz +** [pqr] becomes pqr +** `mno` becomes mno +*/ +static char *amatchDequote(const char *zIn){ + sqlite3_int64 nIn; /* Size of input string, in bytes */ + char *zOut; /* Output (dequoted) string */ + + nIn = strlen(zIn); + zOut = sqlite3_malloc64(nIn+1); + if( zOut ){ + char q = zIn[0]; /* Quote character (if any ) */ + + if( q!='[' && q!= '\'' && q!='"' && q!='`' ){ + memcpy(zOut, zIn, (size_t)(nIn+1)); + }else{ + int iOut = 0; /* Index of next byte to write to output */ + int iIn; /* Index of next byte to read from input */ + + if( q=='[' ) q = ']'; + for(iIn=1; iIn<nIn; iIn++){ + if( zIn[iIn]==q ) iIn++; + zOut[iOut++] = zIn[iIn]; + } + } + assert( (int)strlen(zOut)<=nIn ); + } + return zOut; +} + +/* +** Deallocate the pVCheck prepared statement. +*/ +static void amatchVCheckClear(amatch_vtab *p){ + if( p->pVCheck ){ + sqlite3_finalize(p->pVCheck); + p->pVCheck = 0; + } +} + +/* +** Deallocate an amatch_vtab object +*/ +static void amatchFree(amatch_vtab *p){ + if( p ){ + amatchFreeRules(p); + amatchVCheckClear(p); + sqlite3_free(p->zClassName); + sqlite3_free(p->zDb); + sqlite3_free(p->zCostTab); + sqlite3_free(p->zVocabTab); + sqlite3_free(p->zVocabWord); + sqlite3_free(p->zVocabLang); + sqlite3_free(p->zSelf); + memset(p, 0, sizeof(*p)); + sqlite3_free(p); + } +} + +/* +** xDisconnect/xDestroy method for the amatch module. +*/ +static int amatchDisconnect(sqlite3_vtab *pVtab){ + amatch_vtab *p = (amatch_vtab*)pVtab; + assert( p->nCursor==0 ); + amatchFree(p); + return SQLITE_OK; +} + +/* +** Check to see if the argument is of the form: +** +** KEY = VALUE +** +** If it is, return a pointer to the first character of VALUE. +** If not, return NULL. Spaces around the = are ignored. +*/ +static const char *amatchValueOfKey(const char *zKey, const char *zStr){ + int nKey = (int)strlen(zKey); + int nStr = (int)strlen(zStr); + int i; + if( nStr<nKey+1 ) return 0; + if( memcmp(zStr, zKey, nKey)!=0 ) return 0; + for(i=nKey; isspace((unsigned char)zStr[i]); i++){} + if( zStr[i]!='=' ) return 0; + i++; + while( isspace((unsigned char)zStr[i]) ){ i++; } + return zStr+i; +} + +/* +** xConnect/xCreate method for the amatch module. Arguments are: +** +** argv[0] -> module name ("approximate_match") +** argv[1] -> database name +** argv[2] -> table name +** argv[3...] -> arguments +*/ +static int amatchConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + int rc = SQLITE_OK; /* Return code */ + amatch_vtab *pNew = 0; /* New virtual table */ + const char *zModule = argv[0]; + const char *zDb = argv[1]; + const char *zVal; + int i; + + (void)pAux; + *ppVtab = 0; + pNew = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + rc = SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + pNew->zClassName = sqlite3_mprintf("%s", zModule); + if( pNew->zClassName==0 ) goto amatchConnectError; + pNew->zDb = sqlite3_mprintf("%s", zDb); + if( pNew->zDb==0 ) goto amatchConnectError; + pNew->zSelf = sqlite3_mprintf("%s", argv[2]); + if( pNew->zSelf==0 ) goto amatchConnectError; + for(i=3; i<argc; i++){ + zVal = amatchValueOfKey("vocabulary_table", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zVocabTab); + pNew->zVocabTab = amatchDequote(zVal); + if( pNew->zVocabTab==0 ) goto amatchConnectError; + continue; + } + zVal = amatchValueOfKey("vocabulary_word", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zVocabWord); + pNew->zVocabWord = amatchDequote(zVal); + if( pNew->zVocabWord==0 ) goto amatchConnectError; + continue; + } + zVal = amatchValueOfKey("vocabulary_language", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zVocabLang); + pNew->zVocabLang = amatchDequote(zVal); + if( pNew->zVocabLang==0 ) goto amatchConnectError; + continue; + } + zVal = amatchValueOfKey("edit_distances", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zCostTab); + pNew->zCostTab = amatchDequote(zVal); + if( pNew->zCostTab==0 ) goto amatchConnectError; + continue; + } + *pzErr = sqlite3_mprintf("unrecognized argument: [%s]\n", argv[i]); + amatchFree(pNew); + *ppVtab = 0; + return SQLITE_ERROR; + } + rc = SQLITE_OK; + if( pNew->zCostTab==0 ){ + *pzErr = sqlite3_mprintf("no edit_distances table specified"); + rc = SQLITE_ERROR; + }else{ + rc = amatchLoadRules(db, pNew, pzErr); + } + if( rc==SQLITE_OK ){ + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(word,distance,language," + "command HIDDEN,nword HIDDEN)" + ); +#define AMATCH_COL_WORD 0 +#define AMATCH_COL_DISTANCE 1 +#define AMATCH_COL_LANGUAGE 2 +#define AMATCH_COL_COMMAND 3 +#define AMATCH_COL_NWORD 4 + } + if( rc!=SQLITE_OK ){ + amatchFree(pNew); + } + *ppVtab = &pNew->base; + return rc; + +amatchConnectError: + amatchFree(pNew); + return rc; +} + +/* +** Open a new amatch cursor. +*/ +static int amatchOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ + amatch_vtab *p = (amatch_vtab*)pVTab; + amatch_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->pVtab = p; + *ppCursor = &pCur->base; + p->nCursor++; + return SQLITE_OK; +} + +/* +** Free up all the memory allocated by a cursor. Set it rLimit to 0 +** to indicate that it is at EOF. +*/ +static void amatchClearCursor(amatch_cursor *pCur){ + amatch_word *pWord, *pNextWord; + for(pWord=pCur->pAllWords; pWord; pWord=pNextWord){ + pNextWord = pWord->pNext; + sqlite3_free(pWord); + } + pCur->pAllWords = 0; + sqlite3_free(pCur->zInput); + pCur->zInput = 0; + sqlite3_free(pCur->zBuf); + pCur->zBuf = 0; + pCur->nBuf = 0; + pCur->pCost = 0; + pCur->pWord = 0; + pCur->pCurrent = 0; + pCur->rLimit = 1000000; + pCur->iLang = 0; + pCur->nWord = 0; +} + +/* +** Close a amatch cursor. +*/ +static int amatchClose(sqlite3_vtab_cursor *cur){ + amatch_cursor *pCur = (amatch_cursor *)cur; + amatchClearCursor(pCur); + pCur->pVtab->nCursor--; + sqlite3_free(pCur); + return SQLITE_OK; +} + +/* +** Render a 24-bit unsigned integer as a 4-byte base-64 number. +*/ +static void amatchEncodeInt(int x, char *z){ + static const char a[] = + "0123456789" + "ABCDEFGHIJ" + "KLMNOPQRST" + "UVWXYZ^abc" + "defghijklm" + "nopqrstuvw" + "xyz~"; + z[0] = a[(x>>18)&0x3f]; + z[1] = a[(x>>12)&0x3f]; + z[2] = a[(x>>6)&0x3f]; + z[3] = a[x&0x3f]; +} + +/* +** Write the zCost[] field for a amatch_word object +*/ +static void amatchWriteCost(amatch_word *pWord){ + amatchEncodeInt(pWord->rCost, pWord->zCost); + amatchEncodeInt(pWord->iSeq, pWord->zCost+4); + pWord->zCost[8] = 0; +} + +/* Circumvent compiler warnings about the use of strcpy() by supplying +** our own implementation. +*/ +static void amatchStrcpy(char *dest, const char *src){ + while( (*(dest++) = *(src++))!=0 ){} +} +static void amatchStrcat(char *dest, const char *src){ + while( *dest ) dest++; + amatchStrcpy(dest, src); +} + +/* +** Add a new amatch_word object to the queue. +** +** If a prior amatch_word object with the same zWord, and nMatch +** already exists, update its rCost (if the new rCost is less) but +** otherwise leave it unchanged. Do not add a duplicate. +** +** Do nothing if the cost exceeds threshold. +*/ +static void amatchAddWord( + amatch_cursor *pCur, + amatch_cost rCost, + int nMatch, + const char *zWordBase, + const char *zWordTail +){ + amatch_word *pWord; + amatch_avl *pNode; + amatch_avl *pOther; + int nBase, nTail; + char zBuf[4]; + + if( rCost>pCur->rLimit ){ + return; + } + nBase = (int)strlen(zWordBase); + nTail = (int)strlen(zWordTail); + if( nBase+nTail+3>pCur->nBuf ){ + pCur->nBuf = nBase+nTail+100; + pCur->zBuf = sqlite3_realloc(pCur->zBuf, pCur->nBuf); + if( pCur->zBuf==0 ){ + pCur->nBuf = 0; + return; + } + } + amatchEncodeInt(nMatch, zBuf); + memcpy(pCur->zBuf, zBuf+2, 2); + memcpy(pCur->zBuf+2, zWordBase, nBase); + memcpy(pCur->zBuf+2+nBase, zWordTail, nTail+1); + pNode = amatchAvlSearch(pCur->pWord, pCur->zBuf); + if( pNode ){ + pWord = pNode->pWord; + if( pWord->rCost>rCost ){ +#ifdef AMATCH_TRACE_1 + printf("UPDATE [%s][%.*s^%s] %d (\"%s\" \"%s\")\n", + pWord->zWord+2, pWord->nMatch, pCur->zInput, pCur->zInput, + pWord->rCost, pWord->zWord, pWord->zCost); +#endif + amatchAvlRemove(&pCur->pCost, &pWord->sCost); + pWord->rCost = rCost; + amatchWriteCost(pWord); +#ifdef AMATCH_TRACE_1 + printf(" ---> %d (\"%s\" \"%s\")\n", + pWord->rCost, pWord->zWord, pWord->zCost); +#endif + pOther = amatchAvlInsert(&pCur->pCost, &pWord->sCost); + assert( pOther==0 ); (void)pOther; + } + return; + } + pWord = sqlite3_malloc64( sizeof(*pWord) + nBase + nTail - 1 ); + if( pWord==0 ) return; + memset(pWord, 0, sizeof(*pWord)); + pWord->rCost = rCost; + pWord->iSeq = pCur->nWord++; + amatchWriteCost(pWord); + pWord->nMatch = (short)nMatch; + pWord->pNext = pCur->pAllWords; + pCur->pAllWords = pWord; + pWord->sCost.zKey = pWord->zCost; + pWord->sCost.pWord = pWord; + pOther = amatchAvlInsert(&pCur->pCost, &pWord->sCost); + assert( pOther==0 ); (void)pOther; + pWord->sWord.zKey = pWord->zWord; + pWord->sWord.pWord = pWord; + amatchStrcpy(pWord->zWord, pCur->zBuf); + pOther = amatchAvlInsert(&pCur->pWord, &pWord->sWord); + assert( pOther==0 ); (void)pOther; +#ifdef AMATCH_TRACE_1 + printf("INSERT [%s][%.*s^%s] %d (\"%s\" \"%s\")\n", pWord->zWord+2, + pWord->nMatch, pCur->zInput, pCur->zInput+pWord->nMatch, rCost, + pWord->zWord, pWord->zCost); +#endif +} + + +/* +** Advance a cursor to its next row of output +*/ +static int amatchNext(sqlite3_vtab_cursor *cur){ + amatch_cursor *pCur = (amatch_cursor*)cur; + amatch_word *pWord = 0; + amatch_avl *pNode; + int isMatch = 0; + amatch_vtab *p = pCur->pVtab; + int nWord; + int rc; + int i; + const char *zW; + amatch_rule *pRule; + char *zBuf = 0; + char nBuf = 0; + char zNext[8]; + char zNextIn[8]; + int nNextIn; + + if( p->pVCheck==0 ){ + char *zSql; + if( p->zVocabLang && p->zVocabLang[0] ){ + zSql = sqlite3_mprintf( + "SELECT \"%w\" FROM \"%w\"", + " WHERE \"%w\">=?1 AND \"%w\"=?2" + " ORDER BY 1", + p->zVocabWord, p->zVocabTab, + p->zVocabWord, p->zVocabLang + ); + }else{ + zSql = sqlite3_mprintf( + "SELECT \"%w\" FROM \"%w\"" + " WHERE \"%w\">=?1" + " ORDER BY 1", + p->zVocabWord, p->zVocabTab, + p->zVocabWord + ); + } + rc = sqlite3_prepare_v2(p->db, zSql, -1, &p->pVCheck, 0); + sqlite3_free(zSql); + if( rc ) return rc; + } + sqlite3_bind_int(p->pVCheck, 2, pCur->iLang); + + do{ + pNode = amatchAvlFirst(pCur->pCost); + if( pNode==0 ){ + pWord = 0; + break; + } + pWord = pNode->pWord; + amatchAvlRemove(&pCur->pCost, &pWord->sCost); + +#ifdef AMATCH_TRACE_1 + printf("PROCESS [%s][%.*s^%s] %d (\"%s\" \"%s\")\n", + pWord->zWord+2, pWord->nMatch, pCur->zInput, pCur->zInput+pWord->nMatch, + pWord->rCost, pWord->zWord, pWord->zCost); +#endif + nWord = (int)strlen(pWord->zWord+2); + if( nWord+20>nBuf ){ + nBuf = (char)(nWord+100); + zBuf = sqlite3_realloc(zBuf, nBuf); + if( zBuf==0 ) return SQLITE_NOMEM; + } + amatchStrcpy(zBuf, pWord->zWord+2); + zNext[0] = 0; + zNextIn[0] = pCur->zInput[pWord->nMatch]; + if( zNextIn[0] ){ + for(i=1; i<=4 && (pCur->zInput[pWord->nMatch+i]&0xc0)==0x80; i++){ + zNextIn[i] = pCur->zInput[pWord->nMatch+i]; + } + zNextIn[i] = 0; + nNextIn = i; + }else{ + nNextIn = 0; + } + + if( zNextIn[0] && zNextIn[0]!='*' ){ + sqlite3_reset(p->pVCheck); + amatchStrcat(zBuf, zNextIn); + sqlite3_bind_text(p->pVCheck, 1, zBuf, nWord+nNextIn, SQLITE_STATIC); + rc = sqlite3_step(p->pVCheck); + if( rc==SQLITE_ROW ){ + zW = (const char*)sqlite3_column_text(p->pVCheck, 0); + if( strncmp(zBuf, zW, nWord+nNextIn)==0 ){ + amatchAddWord(pCur, pWord->rCost, pWord->nMatch+nNextIn, zBuf, ""); + } + } + zBuf[nWord] = 0; + } + + while( 1 ){ + amatchStrcpy(zBuf+nWord, zNext); + sqlite3_reset(p->pVCheck); + sqlite3_bind_text(p->pVCheck, 1, zBuf, -1, SQLITE_TRANSIENT); + rc = sqlite3_step(p->pVCheck); + if( rc!=SQLITE_ROW ) break; + zW = (const char*)sqlite3_column_text(p->pVCheck, 0); + amatchStrcpy(zBuf+nWord, zNext); + if( strncmp(zW, zBuf, nWord)!=0 ) break; + if( (zNextIn[0]=='*' && zNextIn[1]==0) + || (zNextIn[0]==0 && zW[nWord]==0) + ){ + isMatch = 1; + zNextIn[0] = 0; + nNextIn = 0; + break; + } + zNext[0] = zW[nWord]; + for(i=1; i<=4 && (zW[nWord+i]&0xc0)==0x80; i++){ + zNext[i] = zW[nWord+i]; + } + zNext[i] = 0; + zBuf[nWord] = 0; + if( p->rIns>0 ){ + amatchAddWord(pCur, pWord->rCost+p->rIns, pWord->nMatch, + zBuf, zNext); + } + if( p->rSub>0 ){ + amatchAddWord(pCur, pWord->rCost+p->rSub, pWord->nMatch+nNextIn, + zBuf, zNext); + } + if( p->rIns<0 && p->rSub<0 ) break; + zNext[i-1]++; /* FIX ME */ + } + sqlite3_reset(p->pVCheck); + + if( p->rDel>0 ){ + zBuf[nWord] = 0; + amatchAddWord(pCur, pWord->rCost+p->rDel, pWord->nMatch+nNextIn, + zBuf, ""); + } + + for(pRule=p->pRule; pRule; pRule=pRule->pNext){ + if( pRule->iLang!=pCur->iLang ) continue; + if( strncmp(pRule->zFrom, pCur->zInput+pWord->nMatch, pRule->nFrom)==0 ){ + amatchAddWord(pCur, pWord->rCost+pRule->rCost, + pWord->nMatch+pRule->nFrom, pWord->zWord+2, pRule->zTo); + } + } + }while( !isMatch ); + pCur->pCurrent = pWord; + sqlite3_free(zBuf); + return SQLITE_OK; +} + +/* +** Called to "rewind" a cursor back to the beginning so that +** it starts its output over again. Always called at least once +** prior to any amatchColumn, amatchRowid, or amatchEof call. +*/ +static int amatchFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + amatch_cursor *pCur = (amatch_cursor *)pVtabCursor; + const char *zWord = "*"; + int idx; + + amatchClearCursor(pCur); + idx = 0; + if( idxNum & 1 ){ + zWord = (const char*)sqlite3_value_text(argv[0]); + idx++; + } + if( idxNum & 2 ){ + pCur->rLimit = (amatch_cost)sqlite3_value_int(argv[idx]); + idx++; + } + if( idxNum & 4 ){ + pCur->iLang = (amatch_cost)sqlite3_value_int(argv[idx]); + idx++; + } + pCur->zInput = sqlite3_mprintf("%s", zWord); + if( pCur->zInput==0 ) return SQLITE_NOMEM; + amatchAddWord(pCur, 0, 0, "", ""); + amatchNext(pVtabCursor); + + return SQLITE_OK; +} + +/* +** Only the word and distance columns have values. All other columns +** return NULL +*/ +static int amatchColumn(sqlite3_vtab_cursor *cur, sqlite3_context *ctx, int i){ + amatch_cursor *pCur = (amatch_cursor*)cur; + switch( i ){ + case AMATCH_COL_WORD: { + sqlite3_result_text(ctx, pCur->pCurrent->zWord+2, -1, SQLITE_STATIC); + break; + } + case AMATCH_COL_DISTANCE: { + sqlite3_result_int(ctx, pCur->pCurrent->rCost); + break; + } + case AMATCH_COL_LANGUAGE: { + sqlite3_result_int(ctx, pCur->iLang); + break; + } + case AMATCH_COL_NWORD: { + sqlite3_result_int(ctx, pCur->nWord); + break; + } + default: { + sqlite3_result_null(ctx); + break; + } + } + return SQLITE_OK; +} + +/* +** The rowid. +*/ +static int amatchRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + amatch_cursor *pCur = (amatch_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** EOF indicator +*/ +static int amatchEof(sqlite3_vtab_cursor *cur){ + amatch_cursor *pCur = (amatch_cursor*)cur; + return pCur->pCurrent==0; +} + +/* +** Search for terms of these forms: +** +** (A) word MATCH $str +** (B1) distance < $value +** (B2) distance <= $value +** (C) language == $language +** +** The distance< and distance<= are both treated as distance<=. +** The query plan number is a bit vector: +** +** bit 1: Term of the form (A) found +** bit 2: Term like (B1) or (B2) found +** bit 3: Term like (C) found +** +** If bit-1 is set, $str is always in filter.argv[0]. If bit-2 is set +** then $value is in filter.argv[0] if bit-1 is clear and is in +** filter.argv[1] if bit-1 is set. If bit-3 is set, then $ruleid is +** in filter.argv[0] if bit-1 and bit-2 are both zero, is in +** filter.argv[1] if exactly one of bit-1 and bit-2 are set, and is in +** filter.argv[2] if both bit-1 and bit-2 are set. +*/ +static int amatchBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int iPlan = 0; + int iDistTerm = -1; + int iLangTerm = -1; + int i; + const struct sqlite3_index_constraint *pConstraint; + + (void)tab; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + if( (iPlan & 1)==0 + && pConstraint->iColumn==0 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_MATCH + ){ + iPlan |= 1; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + } + if( (iPlan & 2)==0 + && pConstraint->iColumn==1 + && (pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT + || pConstraint->op==SQLITE_INDEX_CONSTRAINT_LE) + ){ + iPlan |= 2; + iDistTerm = i; + } + if( (iPlan & 4)==0 + && pConstraint->iColumn==2 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= 4; + pIdxInfo->aConstraintUsage[i].omit = 1; + iLangTerm = i; + } + } + if( iPlan & 2 ){ + pIdxInfo->aConstraintUsage[iDistTerm].argvIndex = 1+((iPlan&1)!=0); + } + if( iPlan & 4 ){ + int idx = 1; + if( iPlan & 1 ) idx++; + if( iPlan & 2 ) idx++; + pIdxInfo->aConstraintUsage[iLangTerm].argvIndex = idx; + } + pIdxInfo->idxNum = iPlan; + if( pIdxInfo->nOrderBy==1 + && pIdxInfo->aOrderBy[0].iColumn==1 + && pIdxInfo->aOrderBy[0].desc==0 + ){ + pIdxInfo->orderByConsumed = 1; + } + pIdxInfo->estimatedCost = (double)10000; + + return SQLITE_OK; +} + +/* +** The xUpdate() method. +** +** This implementation disallows DELETE and UPDATE. The only thing +** allowed is INSERT into the "command" column. +*/ +static int amatchUpdate( + sqlite3_vtab *pVTab, + int argc, + sqlite3_value **argv, + sqlite_int64 *pRowid +){ + amatch_vtab *p = (amatch_vtab*)pVTab; + const unsigned char *zCmd; + (void)pRowid; + if( argc==1 ){ + pVTab->zErrMsg = sqlite3_mprintf("DELETE from %s is not allowed", + p->zSelf); + return SQLITE_ERROR; + } + if( sqlite3_value_type(argv[0])!=SQLITE_NULL ){ + pVTab->zErrMsg = sqlite3_mprintf("UPDATE of %s is not allowed", + p->zSelf); + return SQLITE_ERROR; + } + if( sqlite3_value_type(argv[2+AMATCH_COL_WORD])!=SQLITE_NULL + || sqlite3_value_type(argv[2+AMATCH_COL_DISTANCE])!=SQLITE_NULL + || sqlite3_value_type(argv[2+AMATCH_COL_LANGUAGE])!=SQLITE_NULL + ){ + pVTab->zErrMsg = sqlite3_mprintf( + "INSERT INTO %s allowed for column [command] only", p->zSelf); + return SQLITE_ERROR; + } + zCmd = sqlite3_value_text(argv[2+AMATCH_COL_COMMAND]); + if( zCmd==0 ) return SQLITE_OK; + + return SQLITE_OK; +} + +/* +** A virtual table module that implements the "approximate_match". +*/ +static sqlite3_module amatchModule = { + 0, /* iVersion */ + amatchConnect, /* xCreate */ + amatchConnect, /* xConnect */ + amatchBestIndex, /* xBestIndex */ + amatchDisconnect, /* xDisconnect */ + amatchDisconnect, /* xDestroy */ + amatchOpen, /* xOpen - open a cursor */ + amatchClose, /* xClose - close a cursor */ + amatchFilter, /* xFilter - configure scan constraints */ + amatchNext, /* xNext - advance a cursor */ + amatchEof, /* xEof - check for end of scan */ + amatchColumn, /* xColumn - read data */ + amatchRowid, /* xRowid - read data */ + amatchUpdate, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +/* +** Register the amatch virtual table +*/ +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_amatch_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Not used */ +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "approximate_match", &amatchModule, 0); +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + return rc; +} diff --git a/ext/misc/anycollseq.c b/ext/misc/anycollseq.c new file mode 100644 index 0000000..27b7049 --- /dev/null +++ b/ext/misc/anycollseq.c @@ -0,0 +1,58 @@ +/* +** 2017-04-16 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a run-time loadable extension to SQLite that +** registers a sqlite3_collation_needed() callback to register a fake +** collating function for any unknown collating sequence. The fake +** collating function works like BINARY. +** +** This extension can be used to load schemas that contain one or more +** unknown collating sequences. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <string.h> + +static int anyCollFunc( + void *NotUsed, + int nKey1, const void *pKey1, + int nKey2, const void *pKey2 +){ + int rc, n; + n = nKey1<nKey2 ? nKey1 : nKey2; + rc = memcmp(pKey1, pKey2, n); + if( rc==0 ) rc = nKey1 - nKey2; + return rc; +} + +static void anyCollNeeded( + void *NotUsed, + sqlite3 *db, + int eTextRep, + const char *zCollName +){ + sqlite3_create_collation(db, zCollName, eTextRep, 0, anyCollFunc); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_anycollseq_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_collation_needed(db, 0, anyCollNeeded); + return rc; +} diff --git a/ext/misc/appendvfs.c b/ext/misc/appendvfs.c new file mode 100644 index 0000000..eb9ed01 --- /dev/null +++ b/ext/misc/appendvfs.c @@ -0,0 +1,672 @@ +/* +** 2017-10-20 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file implements a VFS shim that allows an SQLite database to be +** appended onto the end of some other file, such as an executable. +** +** A special record must appear at the end of the file that identifies the +** file as an appended database and provides the offset to the first page +** of the exposed content. (Or, it is the length of the content prefix.) +** For best performance page 1 should be located at a disk page boundary, +** though that is not required. +** +** When opening a database using this VFS, the connection might treat +** the file as an ordinary SQLite database, or it might treat it as a +** database appended onto some other file. The decision is made by +** applying the following rules in order: +** +** (1) An empty file is an ordinary database. +** +** (2) If the file ends with the appendvfs trailer string +** "Start-Of-SQLite3-NNNNNNNN" that file is an appended database. +** +** (3) If the file begins with the standard SQLite prefix string +** "SQLite format 3", that file is an ordinary database. +** +** (4) If none of the above apply and the SQLITE_OPEN_CREATE flag is +** set, then a new database is appended to the already existing file. +** +** (5) Otherwise, SQLITE_CANTOPEN is returned. +** +** To avoid unnecessary complications with the PENDING_BYTE, the size of +** the file containing the database is limited to 1GiB. (1073741824 bytes) +** This VFS will not read or write past the 1GiB mark. This restriction +** might be lifted in future versions. For now, if you need a larger +** database, then keep it in a separate file. +** +** If the file being opened is a plain database (not an appended one), then +** this shim is a pass-through into the default underlying VFS. (rule 3) +**/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> + +/* The append mark at the end of the database is: +** +** Start-Of-SQLite3-NNNNNNNN +** 123456789 123456789 12345 +** +** The NNNNNNNN represents a 64-bit big-endian unsigned integer which is +** the offset to page 1, and also the length of the prefix content. +*/ +#define APND_MARK_PREFIX "Start-Of-SQLite3-" +#define APND_MARK_PREFIX_SZ 17 +#define APND_MARK_FOS_SZ 8 +#define APND_MARK_SIZE (APND_MARK_PREFIX_SZ+APND_MARK_FOS_SZ) + +/* +** Maximum size of the combined prefix + database + append-mark. This +** must be less than 0x40000000 to avoid locking issues on Windows. +*/ +#define APND_MAX_SIZE (0x40000000) + +/* +** Try to align the database to an even multiple of APND_ROUNDUP bytes. +*/ +#ifndef APND_ROUNDUP +#define APND_ROUNDUP 4096 +#endif +#define APND_ALIGN_MASK ((sqlite3_int64)(APND_ROUNDUP-1)) +#define APND_START_ROUNDUP(fsz) (((fsz)+APND_ALIGN_MASK) & ~APND_ALIGN_MASK) + +/* +** Forward declaration of objects used by this utility +*/ +typedef struct sqlite3_vfs ApndVfs; +typedef struct ApndFile ApndFile; + +/* Access to a lower-level VFS that (might) implement dynamic loading, +** access to randomness, etc. +*/ +#define ORIGVFS(p) ((sqlite3_vfs*)((p)->pAppData)) +#define ORIGFILE(p) ((sqlite3_file*)(((ApndFile*)(p))+1)) + +/* An open appendvfs file +** +** An instance of this structure describes the appended database file. +** A separate sqlite3_file object is always appended. The appended +** sqlite3_file object (which can be accessed using ORIGFILE()) describes +** the entire file, including the prefix, the database, and the +** append-mark. +** +** The structure of an AppendVFS database is like this: +** +** +-------------+---------+----------+-------------+ +** | prefix-file | padding | database | append-mark | +** +-------------+---------+----------+-------------+ +** ^ ^ +** | | +** iPgOne iMark +** +** +** "prefix file" - file onto which the database has been appended. +** "padding" - zero or more bytes inserted so that "database" +** starts on an APND_ROUNDUP boundary +** "database" - The SQLite database file +** "append-mark" - The 25-byte "Start-Of-SQLite3-NNNNNNNN" that indicates +** the offset from the start of prefix-file to the start +** of "database". +** +** The size of the database is iMark - iPgOne. +** +** The NNNNNNNN in the "Start-Of-SQLite3-NNNNNNNN" suffix is the value +** of iPgOne stored as a big-ending 64-bit integer. +** +** iMark will be the size of the underlying file minus 25 (APND_MARKSIZE). +** Or, iMark is -1 to indicate that it has not yet been written. +*/ +struct ApndFile { + sqlite3_file base; /* Subclass. MUST BE FIRST! */ + sqlite3_int64 iPgOne; /* Offset to the start of the database */ + sqlite3_int64 iMark; /* Offset of the append mark. -1 if unwritten */ + /* Always followed by another sqlite3_file that describes the whole file */ +}; + +/* +** Methods for ApndFile +*/ +static int apndClose(sqlite3_file*); +static int apndRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); +static int apndWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); +static int apndTruncate(sqlite3_file*, sqlite3_int64 size); +static int apndSync(sqlite3_file*, int flags); +static int apndFileSize(sqlite3_file*, sqlite3_int64 *pSize); +static int apndLock(sqlite3_file*, int); +static int apndUnlock(sqlite3_file*, int); +static int apndCheckReservedLock(sqlite3_file*, int *pResOut); +static int apndFileControl(sqlite3_file*, int op, void *pArg); +static int apndSectorSize(sqlite3_file*); +static int apndDeviceCharacteristics(sqlite3_file*); +static int apndShmMap(sqlite3_file*, int iPg, int pgsz, int, void volatile**); +static int apndShmLock(sqlite3_file*, int offset, int n, int flags); +static void apndShmBarrier(sqlite3_file*); +static int apndShmUnmap(sqlite3_file*, int deleteFlag); +static int apndFetch(sqlite3_file*, sqlite3_int64 iOfst, int iAmt, void **pp); +static int apndUnfetch(sqlite3_file*, sqlite3_int64 iOfst, void *p); + +/* +** Methods for ApndVfs +*/ +static int apndOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); +static int apndDelete(sqlite3_vfs*, const char *zName, int syncDir); +static int apndAccess(sqlite3_vfs*, const char *zName, int flags, int *); +static int apndFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); +static void *apndDlOpen(sqlite3_vfs*, const char *zFilename); +static void apndDlError(sqlite3_vfs*, int nByte, char *zErrMsg); +static void (*apndDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); +static void apndDlClose(sqlite3_vfs*, void*); +static int apndRandomness(sqlite3_vfs*, int nByte, char *zOut); +static int apndSleep(sqlite3_vfs*, int microseconds); +static int apndCurrentTime(sqlite3_vfs*, double*); +static int apndGetLastError(sqlite3_vfs*, int, char *); +static int apndCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); +static int apndSetSystemCall(sqlite3_vfs*, const char*,sqlite3_syscall_ptr); +static sqlite3_syscall_ptr apndGetSystemCall(sqlite3_vfs*, const char *z); +static const char *apndNextSystemCall(sqlite3_vfs*, const char *zName); + +static sqlite3_vfs apnd_vfs = { + 3, /* iVersion (set when registered) */ + 0, /* szOsFile (set when registered) */ + 1024, /* mxPathname */ + 0, /* pNext */ + "apndvfs", /* zName */ + 0, /* pAppData (set when registered) */ + apndOpen, /* xOpen */ + apndDelete, /* xDelete */ + apndAccess, /* xAccess */ + apndFullPathname, /* xFullPathname */ + apndDlOpen, /* xDlOpen */ + apndDlError, /* xDlError */ + apndDlSym, /* xDlSym */ + apndDlClose, /* xDlClose */ + apndRandomness, /* xRandomness */ + apndSleep, /* xSleep */ + apndCurrentTime, /* xCurrentTime */ + apndGetLastError, /* xGetLastError */ + apndCurrentTimeInt64, /* xCurrentTimeInt64 */ + apndSetSystemCall, /* xSetSystemCall */ + apndGetSystemCall, /* xGetSystemCall */ + apndNextSystemCall /* xNextSystemCall */ +}; + +static const sqlite3_io_methods apnd_io_methods = { + 3, /* iVersion */ + apndClose, /* xClose */ + apndRead, /* xRead */ + apndWrite, /* xWrite */ + apndTruncate, /* xTruncate */ + apndSync, /* xSync */ + apndFileSize, /* xFileSize */ + apndLock, /* xLock */ + apndUnlock, /* xUnlock */ + apndCheckReservedLock, /* xCheckReservedLock */ + apndFileControl, /* xFileControl */ + apndSectorSize, /* xSectorSize */ + apndDeviceCharacteristics, /* xDeviceCharacteristics */ + apndShmMap, /* xShmMap */ + apndShmLock, /* xShmLock */ + apndShmBarrier, /* xShmBarrier */ + apndShmUnmap, /* xShmUnmap */ + apndFetch, /* xFetch */ + apndUnfetch /* xUnfetch */ +}; + +/* +** Close an apnd-file. +*/ +static int apndClose(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xClose(pFile); +} + +/* +** Read data from an apnd-file. +*/ +static int apndRead( + sqlite3_file *pFile, + void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + ApndFile *paf = (ApndFile *)pFile; + pFile = ORIGFILE(pFile); + return pFile->pMethods->xRead(pFile, zBuf, iAmt, paf->iPgOne+iOfst); +} + +/* +** Add the append-mark onto what should become the end of the file. +* If and only if this succeeds, internal ApndFile.iMark is updated. +* Parameter iWriteEnd is the appendvfs-relative offset of the new mark. +*/ +static int apndWriteMark( + ApndFile *paf, + sqlite3_file *pFile, + sqlite_int64 iWriteEnd +){ + sqlite_int64 iPgOne = paf->iPgOne; + unsigned char a[APND_MARK_SIZE]; + int i = APND_MARK_FOS_SZ; + int rc; + assert(pFile == ORIGFILE(paf)); + memcpy(a, APND_MARK_PREFIX, APND_MARK_PREFIX_SZ); + while( --i >= 0 ){ + a[APND_MARK_PREFIX_SZ+i] = (unsigned char)(iPgOne & 0xff); + iPgOne >>= 8; + } + iWriteEnd += paf->iPgOne; + if( SQLITE_OK==(rc = pFile->pMethods->xWrite + (pFile, a, APND_MARK_SIZE, iWriteEnd)) ){ + paf->iMark = iWriteEnd; + } + return rc; +} + +/* +** Write data to an apnd-file. +*/ +static int apndWrite( + sqlite3_file *pFile, + const void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + ApndFile *paf = (ApndFile *)pFile; + sqlite_int64 iWriteEnd = iOfst + iAmt; + if( iWriteEnd>=APND_MAX_SIZE ) return SQLITE_FULL; + pFile = ORIGFILE(pFile); + /* If append-mark is absent or will be overwritten, write it. */ + if( paf->iMark < 0 || paf->iPgOne + iWriteEnd > paf->iMark ){ + int rc = apndWriteMark(paf, pFile, iWriteEnd); + if( SQLITE_OK!=rc ) return rc; + } + return pFile->pMethods->xWrite(pFile, zBuf, iAmt, paf->iPgOne+iOfst); +} + +/* +** Truncate an apnd-file. +*/ +static int apndTruncate(sqlite3_file *pFile, sqlite_int64 size){ + ApndFile *paf = (ApndFile *)pFile; + pFile = ORIGFILE(pFile); + /* The append mark goes out first so truncate failure does not lose it. */ + if( SQLITE_OK!=apndWriteMark(paf, pFile, size) ) return SQLITE_IOERR; + /* Truncate underlying file just past append mark */ + return pFile->pMethods->xTruncate(pFile, paf->iMark+APND_MARK_SIZE); +} + +/* +** Sync an apnd-file. +*/ +static int apndSync(sqlite3_file *pFile, int flags){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xSync(pFile, flags); +} + +/* +** Return the current file-size of an apnd-file. +** If the append mark is not yet there, the file-size is 0. +*/ +static int apndFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ + ApndFile *paf = (ApndFile *)pFile; + *pSize = ( paf->iMark >= 0 )? (paf->iMark - paf->iPgOne) : 0; + return SQLITE_OK; +} + +/* +** Lock an apnd-file. +*/ +static int apndLock(sqlite3_file *pFile, int eLock){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xLock(pFile, eLock); +} + +/* +** Unlock an apnd-file. +*/ +static int apndUnlock(sqlite3_file *pFile, int eLock){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xUnlock(pFile, eLock); +} + +/* +** Check if another file-handle holds a RESERVED lock on an apnd-file. +*/ +static int apndCheckReservedLock(sqlite3_file *pFile, int *pResOut){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xCheckReservedLock(pFile, pResOut); +} + +/* +** File control method. For custom operations on an apnd-file. +*/ +static int apndFileControl(sqlite3_file *pFile, int op, void *pArg){ + ApndFile *paf = (ApndFile *)pFile; + int rc; + pFile = ORIGFILE(pFile); + if( op==SQLITE_FCNTL_SIZE_HINT ) *(sqlite3_int64*)pArg += paf->iPgOne; + rc = pFile->pMethods->xFileControl(pFile, op, pArg); + if( rc==SQLITE_OK && op==SQLITE_FCNTL_VFSNAME ){ + *(char**)pArg = sqlite3_mprintf("apnd(%lld)/%z", paf->iPgOne,*(char**)pArg); + } + return rc; +} + +/* +** Return the sector-size in bytes for an apnd-file. +*/ +static int apndSectorSize(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xSectorSize(pFile); +} + +/* +** Return the device characteristic flags supported by an apnd-file. +*/ +static int apndDeviceCharacteristics(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xDeviceCharacteristics(pFile); +} + +/* Create a shared memory file mapping */ +static int apndShmMap( + sqlite3_file *pFile, + int iPg, + int pgsz, + int bExtend, + void volatile **pp +){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmMap(pFile,iPg,pgsz,bExtend,pp); +} + +/* Perform locking on a shared-memory segment */ +static int apndShmLock(sqlite3_file *pFile, int offset, int n, int flags){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmLock(pFile,offset,n,flags); +} + +/* Memory barrier operation on shared memory */ +static void apndShmBarrier(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + pFile->pMethods->xShmBarrier(pFile); +} + +/* Unmap a shared memory segment */ +static int apndShmUnmap(sqlite3_file *pFile, int deleteFlag){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmUnmap(pFile,deleteFlag); +} + +/* Fetch a page of a memory-mapped file */ +static int apndFetch( + sqlite3_file *pFile, + sqlite3_int64 iOfst, + int iAmt, + void **pp +){ + ApndFile *p = (ApndFile *)pFile; + if( p->iMark < 0 || iOfst+iAmt > p->iMark ){ + return SQLITE_IOERR; /* Cannot read what is not yet there. */ + } + pFile = ORIGFILE(pFile); + return pFile->pMethods->xFetch(pFile, iOfst+p->iPgOne, iAmt, pp); +} + +/* Release a memory-mapped page */ +static int apndUnfetch(sqlite3_file *pFile, sqlite3_int64 iOfst, void *pPage){ + ApndFile *p = (ApndFile *)pFile; + pFile = ORIGFILE(pFile); + return pFile->pMethods->xUnfetch(pFile, iOfst+p->iPgOne, pPage); +} + +/* +** Try to read the append-mark off the end of a file. Return the +** start of the appended database if the append-mark is present. +** If there is no valid append-mark, return -1; +** +** An append-mark is only valid if the NNNNNNNN start-of-database offset +** indicates that the appended database contains at least one page. The +** start-of-database value must be a multiple of 512. +*/ +static sqlite3_int64 apndReadMark(sqlite3_int64 sz, sqlite3_file *pFile){ + int rc, i; + sqlite3_int64 iMark; + int msbs = 8 * (APND_MARK_FOS_SZ-1); + unsigned char a[APND_MARK_SIZE]; + + if( APND_MARK_SIZE!=(sz & 0x1ff) ) return -1; + rc = pFile->pMethods->xRead(pFile, a, APND_MARK_SIZE, sz-APND_MARK_SIZE); + if( rc ) return -1; + if( memcmp(a, APND_MARK_PREFIX, APND_MARK_PREFIX_SZ)!=0 ) return -1; + iMark = ((sqlite3_int64)(a[APND_MARK_PREFIX_SZ] & 0x7f)) << msbs; + for(i=1; i<8; i++){ + msbs -= 8; + iMark |= (sqlite3_int64)a[APND_MARK_PREFIX_SZ+i]<<msbs; + } + if( iMark > (sz - APND_MARK_SIZE - 512) ) return -1; + if( iMark & 0x1ff ) return -1; + return iMark; +} + +static const char apvfsSqliteHdr[] = "SQLite format 3"; +/* +** Check to see if the file is an appendvfs SQLite database file. +** Return true iff it is such. Parameter sz is the file's size. +*/ +static int apndIsAppendvfsDatabase(sqlite3_int64 sz, sqlite3_file *pFile){ + int rc; + char zHdr[16]; + sqlite3_int64 iMark = apndReadMark(sz, pFile); + if( iMark>=0 ){ + /* If file has the correct end-marker, the expected odd size, and the + ** SQLite DB type marker where the end-marker puts it, then it + ** is an appendvfs database. + */ + rc = pFile->pMethods->xRead(pFile, zHdr, sizeof(zHdr), iMark); + if( SQLITE_OK==rc + && memcmp(zHdr, apvfsSqliteHdr, sizeof(zHdr))==0 + && (sz & 0x1ff) == APND_MARK_SIZE + && sz>=512+APND_MARK_SIZE + ){ + return 1; /* It's an appendvfs database */ + } + } + return 0; +} + +/* +** Check to see if the file is an ordinary SQLite database file. +** Return true iff so. Parameter sz is the file's size. +*/ +static int apndIsOrdinaryDatabaseFile(sqlite3_int64 sz, sqlite3_file *pFile){ + char zHdr[16]; + if( apndIsAppendvfsDatabase(sz, pFile) /* rule 2 */ + || (sz & 0x1ff) != 0 + || SQLITE_OK!=pFile->pMethods->xRead(pFile, zHdr, sizeof(zHdr), 0) + || memcmp(zHdr, apvfsSqliteHdr, sizeof(zHdr))!=0 + ){ + return 0; + }else{ + return 1; + } +} + +/* +** Open an apnd file handle. +*/ +static int apndOpen( + sqlite3_vfs *pApndVfs, + const char *zName, + sqlite3_file *pFile, + int flags, + int *pOutFlags +){ + ApndFile *pApndFile = (ApndFile*)pFile; + sqlite3_file *pBaseFile = ORIGFILE(pFile); + sqlite3_vfs *pBaseVfs = ORIGVFS(pApndVfs); + int rc; + sqlite3_int64 sz = 0; + if( (flags & SQLITE_OPEN_MAIN_DB)==0 ){ + /* The appendvfs is not to be used for transient or temporary databases. + ** Just use the base VFS open to initialize the given file object and + ** open the underlying file. (Appendvfs is then unused for this file.) + */ + return pBaseVfs->xOpen(pBaseVfs, zName, pFile, flags, pOutFlags); + } + memset(pApndFile, 0, sizeof(ApndFile)); + pFile->pMethods = &apnd_io_methods; + pApndFile->iMark = -1; /* Append mark not yet written */ + + rc = pBaseVfs->xOpen(pBaseVfs, zName, pBaseFile, flags, pOutFlags); + if( rc==SQLITE_OK ){ + rc = pBaseFile->pMethods->xFileSize(pBaseFile, &sz); + if( rc ){ + pBaseFile->pMethods->xClose(pBaseFile); + } + } + if( rc ){ + pFile->pMethods = 0; + return rc; + } + if( apndIsOrdinaryDatabaseFile(sz, pBaseFile) ){ + /* The file being opened appears to be just an ordinary DB. Copy + ** the base dispatch-table so this instance mimics the base VFS. + */ + memmove(pApndFile, pBaseFile, pBaseVfs->szOsFile); + return SQLITE_OK; + } + pApndFile->iPgOne = apndReadMark(sz, pFile); + if( pApndFile->iPgOne>=0 ){ + pApndFile->iMark = sz - APND_MARK_SIZE; /* Append mark found */ + return SQLITE_OK; + } + if( (flags & SQLITE_OPEN_CREATE)==0 ){ + pBaseFile->pMethods->xClose(pBaseFile); + rc = SQLITE_CANTOPEN; + pFile->pMethods = 0; + }else{ + /* Round newly added appendvfs location to #define'd page boundary. + ** Note that nothing has yet been written to the underlying file. + ** The append mark will be written along with first content write. + ** Until then, paf->iMark value indicates it is not yet written. + */ + pApndFile->iPgOne = APND_START_ROUNDUP(sz); + } + return rc; +} + +/* +** Delete an apnd file. +** For an appendvfs, this could mean delete the appendvfs portion, +** leaving the appendee as it was before it gained an appendvfs. +** For now, this code deletes the underlying file too. +*/ +static int apndDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ + return ORIGVFS(pVfs)->xDelete(ORIGVFS(pVfs), zPath, dirSync); +} + +/* +** All other VFS methods are pass-thrus. +*/ +static int apndAccess( + sqlite3_vfs *pVfs, + const char *zPath, + int flags, + int *pResOut +){ + return ORIGVFS(pVfs)->xAccess(ORIGVFS(pVfs), zPath, flags, pResOut); +} +static int apndFullPathname( + sqlite3_vfs *pVfs, + const char *zPath, + int nOut, + char *zOut +){ + return ORIGVFS(pVfs)->xFullPathname(ORIGVFS(pVfs),zPath,nOut,zOut); +} +static void *apndDlOpen(sqlite3_vfs *pVfs, const char *zPath){ + return ORIGVFS(pVfs)->xDlOpen(ORIGVFS(pVfs), zPath); +} +static void apndDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ + ORIGVFS(pVfs)->xDlError(ORIGVFS(pVfs), nByte, zErrMsg); +} +static void (*apndDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ + return ORIGVFS(pVfs)->xDlSym(ORIGVFS(pVfs), p, zSym); +} +static void apndDlClose(sqlite3_vfs *pVfs, void *pHandle){ + ORIGVFS(pVfs)->xDlClose(ORIGVFS(pVfs), pHandle); +} +static int apndRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ + return ORIGVFS(pVfs)->xRandomness(ORIGVFS(pVfs), nByte, zBufOut); +} +static int apndSleep(sqlite3_vfs *pVfs, int nMicro){ + return ORIGVFS(pVfs)->xSleep(ORIGVFS(pVfs), nMicro); +} +static int apndCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ + return ORIGVFS(pVfs)->xCurrentTime(ORIGVFS(pVfs), pTimeOut); +} +static int apndGetLastError(sqlite3_vfs *pVfs, int a, char *b){ + return ORIGVFS(pVfs)->xGetLastError(ORIGVFS(pVfs), a, b); +} +static int apndCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ + return ORIGVFS(pVfs)->xCurrentTimeInt64(ORIGVFS(pVfs), p); +} +static int apndSetSystemCall( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_syscall_ptr pCall +){ + return ORIGVFS(pVfs)->xSetSystemCall(ORIGVFS(pVfs),zName,pCall); +} +static sqlite3_syscall_ptr apndGetSystemCall( + sqlite3_vfs *pVfs, + const char *zName +){ + return ORIGVFS(pVfs)->xGetSystemCall(ORIGVFS(pVfs),zName); +} +static const char *apndNextSystemCall(sqlite3_vfs *pVfs, const char *zName){ + return ORIGVFS(pVfs)->xNextSystemCall(ORIGVFS(pVfs), zName); +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +/* +** This routine is called when the extension is loaded. +** Register the new VFS. +*/ +int sqlite3_appendvfs_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + sqlite3_vfs *pOrig; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; + (void)db; + pOrig = sqlite3_vfs_find(0); + if( pOrig==0 ) return SQLITE_ERROR; + apnd_vfs.iVersion = pOrig->iVersion; + apnd_vfs.pAppData = pOrig; + apnd_vfs.szOsFile = pOrig->szOsFile + sizeof(ApndFile); + rc = sqlite3_vfs_register(&apnd_vfs, 0); +#ifdef APPENDVFS_TEST + if( rc==SQLITE_OK ){ + rc = sqlite3_auto_extension((void(*)(void))apndvfsRegister); + } +#endif + if( rc==SQLITE_OK ) rc = SQLITE_OK_LOAD_PERMANENTLY; + return rc; +} diff --git a/ext/misc/blobio.c b/ext/misc/blobio.c new file mode 100644 index 0000000..3a1ee84 --- /dev/null +++ b/ext/misc/blobio.c @@ -0,0 +1,152 @@ +/* +** 2019-03-30 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** An SQL function that uses the incremental BLOB I/O mechanism of SQLite +** to read or write part of a blob. This is intended for debugging use +** in the CLI. +** +** readblob(SCHEMA,TABLE,COLUMN,ROWID,OFFSET,N) +** +** Returns N bytes of the blob starting at OFFSET. +** +** writeblob(SCHEMA,TABLE,COLUMN,ROWID,OFFSET,NEWDATA) +** +** NEWDATA must be a blob. The content of NEWDATA overwrites the +** existing BLOB data at SCHEMA.TABLE.COLUMN for row ROWID beginning +** at OFFSET bytes into the blob. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +static void readblobFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3_blob *pBlob = 0; + const char *zSchema; + const char *zTable; + const char *zColumn; + sqlite3_int64 iRowid; + int iOfst; + unsigned char *aData; + int nData; + sqlite3 *db; + int rc; + + zSchema = (const char*)sqlite3_value_text(argv[0]); + zTable = (const char*)sqlite3_value_text(argv[1]); + if( zTable==0 ){ + sqlite3_result_error(context, "bad table name", -1); + return; + } + zColumn = (const char*)sqlite3_value_text(argv[2]); + if( zTable==0 ){ + sqlite3_result_error(context, "bad column name", -1); + return; + } + iRowid = sqlite3_value_int64(argv[3]); + iOfst = sqlite3_value_int(argv[4]); + nData = sqlite3_value_int(argv[5]); + if( nData<=0 ) return; + aData = sqlite3_malloc64( nData+1 ); + if( aData==0 ){ + sqlite3_result_error_nomem(context); + return; + } + db = sqlite3_context_db_handle(context); + rc = sqlite3_blob_open(db, zSchema, zTable, zColumn, iRowid, 0, &pBlob); + if( rc ){ + sqlite3_free(aData); + sqlite3_result_error(context, "cannot open BLOB pointer", -1); + return; + } + rc = sqlite3_blob_read(pBlob, aData, nData, iOfst); + sqlite3_blob_close(pBlob); + if( rc ){ + sqlite3_free(aData); + sqlite3_result_error(context, "BLOB read failed", -1); + }else{ + sqlite3_result_blob(context, aData, nData, sqlite3_free); + } +} + +static void writeblobFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3_blob *pBlob = 0; + const char *zSchema; + const char *zTable; + const char *zColumn; + sqlite3_int64 iRowid; + int iOfst; + unsigned char *aData; + int nData; + sqlite3 *db; + int rc; + + zSchema = (const char*)sqlite3_value_text(argv[0]); + zTable = (const char*)sqlite3_value_text(argv[1]); + if( zTable==0 ){ + sqlite3_result_error(context, "bad table name", -1); + return; + } + zColumn = (const char*)sqlite3_value_text(argv[2]); + if( zTable==0 ){ + sqlite3_result_error(context, "bad column name", -1); + return; + } + iRowid = sqlite3_value_int64(argv[3]); + iOfst = sqlite3_value_int(argv[4]); + if( sqlite3_value_type(argv[5])!=SQLITE_BLOB ){ + sqlite3_result_error(context, "6th argument must be a BLOB", -1); + return; + } + nData = sqlite3_value_bytes(argv[5]); + aData = (unsigned char *)sqlite3_value_blob(argv[5]); + db = sqlite3_context_db_handle(context); + rc = sqlite3_blob_open(db, zSchema, zTable, zColumn, iRowid, 1, &pBlob); + if( rc ){ + sqlite3_result_error(context, "cannot open BLOB pointer", -1); + return; + } + rc = sqlite3_blob_write(pBlob, aData, nData, iOfst); + sqlite3_blob_close(pBlob); + if( rc ){ + sqlite3_result_error(context, "BLOB write failed", -1); + } +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_blobio_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "readblob", 6, SQLITE_UTF8, 0, + readblobFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "writeblob", 6, SQLITE_UTF8, 0, + writeblobFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/btreeinfo.c b/ext/misc/btreeinfo.c new file mode 100644 index 0000000..22f8268 --- /dev/null +++ b/ext/misc/btreeinfo.c @@ -0,0 +1,429 @@ +/* +** 2017-10-24 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains an implementation of the "sqlite_btreeinfo" virtual table. +** +** The sqlite_btreeinfo virtual table is a read-only eponymous-only virtual +** table that shows information about all btrees in an SQLite database file. +** The schema is like this: +** +** CREATE TABLE sqlite_btreeinfo( +** type TEXT, -- "table" or "index" +** name TEXT, -- Name of table or index for this btree. +** tbl_name TEXT, -- Associated table +** rootpage INT, -- The root page of the btree +** sql TEXT, -- SQL for this btree - from sqlite_schema +** hasRowid BOOLEAN, -- True if the btree has a rowid +** nEntry INT, -- Estimated number of entries +** nPage INT, -- Estimated number of pages +** depth INT, -- Depth of the btree +** szPage INT, -- Size of each page in bytes +** zSchema TEXT HIDDEN -- The schema to which this btree belongs +** ); +** +** The first 5 fields are taken directly from the sqlite_schema table. +** Considering only the first 5 fields, the only difference between +** this virtual table and the sqlite_schema table is that this virtual +** table omits all entries that have a 0 or NULL rowid - in other words +** it omits triggers and views. +** +** The value added by this table comes in the next 5 fields. +** +** Note that nEntry and nPage are *estimated*. They are computed doing +** a single search from the root to a leaf, counting the number of cells +** at each level, and assuming that unvisited pages have a similar number +** of cells. +** +** The sqlite_dbpage virtual table must be available for this virtual table +** to operate. +** +** USAGE EXAMPLES: +** +** Show the table btrees in a schema order with the tables with the most +** rows occuring first: +** +** SELECT name, nEntry +** FROM sqlite_btreeinfo +** WHERE type='table' +** ORDER BY nEntry DESC, name; +** +** Show the names of all WITHOUT ROWID tables: +** +** SELECT name FROM sqlite_btreeinfo +** WHERE type='table' AND NOT hasRowid; +*/ +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> + +/* Columns available in this virtual table */ +#define BINFO_COLUMN_TYPE 0 +#define BINFO_COLUMN_NAME 1 +#define BINFO_COLUMN_TBL_NAME 2 +#define BINFO_COLUMN_ROOTPAGE 3 +#define BINFO_COLUMN_SQL 4 +#define BINFO_COLUMN_HASROWID 5 +#define BINFO_COLUMN_NENTRY 6 +#define BINFO_COLUMN_NPAGE 7 +#define BINFO_COLUMN_DEPTH 8 +#define BINFO_COLUMN_SZPAGE 9 +#define BINFO_COLUMN_SCHEMA 10 + +/* Forward declarations */ +typedef struct BinfoTable BinfoTable; +typedef struct BinfoCursor BinfoCursor; + +/* A cursor for the sqlite_btreeinfo table */ +struct BinfoCursor { + sqlite3_vtab_cursor base; /* Base class. Must be first */ + sqlite3_stmt *pStmt; /* Query against sqlite_schema */ + int rc; /* Result of previous sqlite_step() call */ + int hasRowid; /* hasRowid value. Negative if unknown. */ + sqlite3_int64 nEntry; /* nEntry value */ + int nPage; /* nPage value */ + int depth; /* depth value */ + int szPage; /* size of a btree page. 0 if unknown */ + char *zSchema; /* Schema being interrogated */ +}; + +/* The sqlite_btreeinfo table */ +struct BinfoTable { + sqlite3_vtab base; /* Base class. Must be first */ + sqlite3 *db; /* The databse connection */ +}; + +/* +** Connect to the sqlite_btreeinfo virtual table. +*/ +static int binfoConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + BinfoTable *pTab = 0; + int rc = SQLITE_OK; + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(\n" + " type TEXT,\n" + " name TEXT,\n" + " tbl_name TEXT,\n" + " rootpage INT,\n" + " sql TEXT,\n" + " hasRowid BOOLEAN,\n" + " nEntry INT,\n" + " nPage INT,\n" + " depth INT,\n" + " szPage INT,\n" + " zSchema TEXT HIDDEN\n" + ")"); + if( rc==SQLITE_OK ){ + pTab = (BinfoTable *)sqlite3_malloc64(sizeof(BinfoTable)); + if( pTab==0 ) rc = SQLITE_NOMEM; + } + assert( rc==SQLITE_OK || pTab==0 ); + if( pTab ){ + pTab->db = db; + } + *ppVtab = (sqlite3_vtab*)pTab; + return rc; +} + +/* +** Disconnect from or destroy a btreeinfo virtual table. +*/ +static int binfoDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** idxNum: +** +** 0 Use "main" for the schema +** 1 Schema identified by parameter ?1 +*/ +static int binfoBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdxInfo){ + int i; + pIdxInfo->estimatedCost = 10000.0; /* Cost estimate */ + pIdxInfo->estimatedRows = 100; + for(i=0; i<pIdxInfo->nConstraint; i++){ + struct sqlite3_index_constraint *p = &pIdxInfo->aConstraint[i]; + if( p->usable + && p->iColumn==BINFO_COLUMN_SCHEMA + && p->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + pIdxInfo->estimatedCost = 1000.0; + pIdxInfo->idxNum = 1; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + break; + } + } + return SQLITE_OK; +} + +/* +** Open a new btreeinfo cursor. +*/ +static int binfoOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ + BinfoCursor *pCsr; + + pCsr = (BinfoCursor *)sqlite3_malloc64(sizeof(BinfoCursor)); + if( pCsr==0 ){ + return SQLITE_NOMEM; + }else{ + memset(pCsr, 0, sizeof(BinfoCursor)); + pCsr->base.pVtab = pVTab; + } + + *ppCursor = (sqlite3_vtab_cursor *)pCsr; + return SQLITE_OK; +} + +/* +** Close a btreeinfo cursor. +*/ +static int binfoClose(sqlite3_vtab_cursor *pCursor){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + sqlite3_finalize(pCsr->pStmt); + sqlite3_free(pCsr->zSchema); + sqlite3_free(pCsr); + return SQLITE_OK; +} + +/* +** Move a btreeinfo cursor to the next entry in the file. +*/ +static int binfoNext(sqlite3_vtab_cursor *pCursor){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + pCsr->rc = sqlite3_step(pCsr->pStmt); + pCsr->hasRowid = -1; + return pCsr->rc==SQLITE_ERROR ? SQLITE_ERROR : SQLITE_OK; +} + +/* We have reached EOF if previous sqlite3_step() returned +** anything other than SQLITE_ROW; +*/ +static int binfoEof(sqlite3_vtab_cursor *pCursor){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + return pCsr->rc!=SQLITE_ROW; +} + +/* Position a cursor back to the beginning. +*/ +static int binfoFilter( + sqlite3_vtab_cursor *pCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + BinfoTable *pTab = (BinfoTable *)pCursor->pVtab; + char *zSql; + int rc; + + sqlite3_free(pCsr->zSchema); + if( idxNum==1 && sqlite3_value_type(argv[0])!=SQLITE_NULL ){ + pCsr->zSchema = sqlite3_mprintf("%s", sqlite3_value_text(argv[0])); + }else{ + pCsr->zSchema = sqlite3_mprintf("main"); + } + zSql = sqlite3_mprintf( + "SELECT 0, 'table','sqlite_schema','sqlite_schema',1,NULL " + "UNION ALL " + "SELECT rowid, type, name, tbl_name, rootpage, sql" + " FROM \"%w\".sqlite_schema WHERE rootpage>=1", + pCsr->zSchema); + sqlite3_finalize(pCsr->pStmt); + pCsr->pStmt = 0; + pCsr->hasRowid = -1; + rc = sqlite3_prepare_v2(pTab->db, zSql, -1, &pCsr->pStmt, 0); + sqlite3_free(zSql); + if( rc==SQLITE_OK ){ + rc = binfoNext(pCursor); + } + return rc; +} + +/* Decode big-endian integers */ +static unsigned int get_uint16(unsigned char *a){ + return (a[0]<<8)|a[1]; +} +static unsigned int get_uint32(unsigned char *a){ + return (a[0]<<24)|(a[1]<<16)|(a[2]<<8)|a[3]; +} + +/* Examine the b-tree rooted at pgno and estimate its size. +** Return non-zero if anything goes wrong. +*/ +static int binfoCompute(sqlite3 *db, int pgno, BinfoCursor *pCsr){ + sqlite3_int64 nEntry = 1; + int nPage = 1; + unsigned char *aData; + sqlite3_stmt *pStmt = 0; + int rc = SQLITE_OK; + int pgsz = 0; + int nCell; + int iCell; + + rc = sqlite3_prepare_v2(db, + "SELECT data FROM sqlite_dbpage('main') WHERE pgno=?1", -1, + &pStmt, 0); + if( rc ) return rc; + pCsr->depth = 1; + while(1){ + sqlite3_bind_int(pStmt, 1, pgno); + rc = sqlite3_step(pStmt); + if( rc!=SQLITE_ROW ){ + rc = SQLITE_ERROR; + break; + } + pCsr->szPage = pgsz = sqlite3_column_bytes(pStmt, 0); + aData = (unsigned char*)sqlite3_column_blob(pStmt, 0); + if( aData==0 ){ + rc = SQLITE_NOMEM; + break; + } + if( pgno==1 ){ + aData += 100; + pgsz -= 100; + } + pCsr->hasRowid = aData[0]!=2 && aData[0]!=10; + nCell = get_uint16(aData+3); + nEntry *= (nCell+1); + if( aData[0]==10 || aData[0]==13 ) break; + nPage *= (nCell+1); + if( nCell<=1 ){ + pgno = get_uint32(aData+8); + }else{ + iCell = get_uint16(aData+12+2*(nCell/2)); + if( pgno==1 ) iCell -= 100; + if( iCell<=12 || iCell>=pgsz-4 ){ + rc = SQLITE_CORRUPT; + break; + } + pgno = get_uint32(aData+iCell); + } + pCsr->depth++; + sqlite3_reset(pStmt); + } + sqlite3_finalize(pStmt); + pCsr->nPage = nPage; + pCsr->nEntry = nEntry; + if( rc==SQLITE_ROW ) rc = SQLITE_OK; + return rc; +} + +/* Return a column for the sqlite_btreeinfo table */ +static int binfoColumn( + sqlite3_vtab_cursor *pCursor, + sqlite3_context *ctx, + int i +){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + if( i>=BINFO_COLUMN_HASROWID && i<=BINFO_COLUMN_SZPAGE && pCsr->hasRowid<0 ){ + int pgno = sqlite3_column_int(pCsr->pStmt, BINFO_COLUMN_ROOTPAGE+1); + sqlite3 *db = sqlite3_context_db_handle(ctx); + int rc = binfoCompute(db, pgno, pCsr); + if( rc ){ + pCursor->pVtab->zErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(db)); + return SQLITE_ERROR; + } + } + switch( i ){ + case BINFO_COLUMN_NAME: + case BINFO_COLUMN_TYPE: + case BINFO_COLUMN_TBL_NAME: + case BINFO_COLUMN_ROOTPAGE: + case BINFO_COLUMN_SQL: { + sqlite3_result_value(ctx, sqlite3_column_value(pCsr->pStmt, i+1)); + break; + } + case BINFO_COLUMN_HASROWID: { + sqlite3_result_int(ctx, pCsr->hasRowid); + break; + } + case BINFO_COLUMN_NENTRY: { + sqlite3_result_int64(ctx, pCsr->nEntry); + break; + } + case BINFO_COLUMN_NPAGE: { + sqlite3_result_int(ctx, pCsr->nPage); + break; + } + case BINFO_COLUMN_DEPTH: { + sqlite3_result_int(ctx, pCsr->depth); + break; + } + case BINFO_COLUMN_SCHEMA: { + sqlite3_result_text(ctx, pCsr->zSchema, -1, SQLITE_STATIC); + break; + } + } + return SQLITE_OK; +} + +/* Return the ROWID for the sqlite_btreeinfo table */ +static int binfoRowid(sqlite3_vtab_cursor *pCursor, sqlite_int64 *pRowid){ + BinfoCursor *pCsr = (BinfoCursor *)pCursor; + *pRowid = sqlite3_column_int64(pCsr->pStmt, 0); + return SQLITE_OK; +} + +/* +** Invoke this routine to register the "sqlite_btreeinfo" virtual table module +*/ +int sqlite3BinfoRegister(sqlite3 *db){ + static sqlite3_module binfo_module = { + 0, /* iVersion */ + 0, /* xCreate */ + binfoConnect, /* xConnect */ + binfoBestIndex, /* xBestIndex */ + binfoDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + binfoOpen, /* xOpen - open a cursor */ + binfoClose, /* xClose - close a cursor */ + binfoFilter, /* xFilter - configure scan constraints */ + binfoNext, /* xNext - advance a cursor */ + binfoEof, /* xEof - check for end of scan */ + binfoColumn, /* xColumn - read data */ + binfoRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ + }; + return sqlite3_create_module(db, "sqlite_btreeinfo", &binfo_module, 0); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_btreeinfo_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + SQLITE_EXTENSION_INIT2(pApi); + return sqlite3BinfoRegister(db); +} diff --git a/ext/misc/carray.c b/ext/misc/carray.c new file mode 100644 index 0000000..b6eb045 --- /dev/null +++ b/ext/misc/carray.c @@ -0,0 +1,525 @@ +/* +** 2016-06-29 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates how to create a table-valued-function that +** returns the values in a C-language array. +** Examples: +** +** SELECT * FROM carray($ptr,5) +** +** The query above returns 5 integers contained in a C-language array +** at the address $ptr. $ptr is a pointer to the array of integers. +** The pointer value must be assigned to $ptr using the +** sqlite3_bind_pointer() interface with a pointer type of "carray". +** For example: +** +** static int aX[] = { 53, 9, 17, 2231, 4, 99 }; +** int i = sqlite3_bind_parameter_index(pStmt, "$ptr"); +** sqlite3_bind_pointer(pStmt, i, aX, "carray", 0); +** +** There is an optional third parameter to determine the datatype of +** the C-language array. Allowed values of the third parameter are +** 'int32', 'int64', 'double', 'char*'. Example: +** +** SELECT * FROM carray($ptr,10,'char*'); +** +** The default value of the third parameter is 'int32'. +** +** HOW IT WORKS +** +** The carray "function" is really a virtual table with the +** following schema: +** +** CREATE TABLE carray( +** value, +** pointer HIDDEN, +** count HIDDEN, +** ctype TEXT HIDDEN +** ); +** +** If the hidden columns "pointer" and "count" are unconstrained, then +** the virtual table has no rows. Otherwise, the virtual table interprets +** the integer value of "pointer" as a pointer to the array and "count" +** as the number of elements in the array. The virtual table steps through +** the array, element by element. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* Allowed values for the mFlags parameter to sqlite3_carray_bind(). +** Must exactly match the definitions in carray.h. +*/ +#ifndef CARRAY_INT32 +# define CARRAY_INT32 0 /* Data is 32-bit signed integers */ +# define CARRAY_INT64 1 /* Data is 64-bit signed integers */ +# define CARRAY_DOUBLE 2 /* Data is doubles */ +# define CARRAY_TEXT 3 /* Data is char* */ +#endif + +#ifndef SQLITE_API +# ifdef _WIN32 +# define SQLITE_API __declspec(dllexport) +# else +# define SQLITE_API +# endif +#endif + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Names of allowed datatypes +*/ +static const char *azType[] = { "int32", "int64", "double", "char*" }; + +/* +** Structure used to hold the sqlite3_carray_bind() information +*/ +typedef struct carray_bind carray_bind; +struct carray_bind { + void *aData; /* The data */ + int nData; /* Number of elements */ + int mFlags; /* Control flags */ + void (*xDel)(void*); /* Destructor for aData */ +}; + + +/* carray_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct carray_cursor carray_cursor; +struct carray_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iRowid; /* The rowid */ + void *pPtr; /* Pointer to the array of values */ + sqlite3_int64 iCnt; /* Number of integers in the array */ + unsigned char eType; /* One of the CARRAY_type values */ +}; + +/* +** The carrayConnect() method is invoked to create a new +** carray_vtab that describes the carray virtual table. +** +** Think of this routine as the constructor for carray_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the carray_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against carray will look like. +*/ +static int carrayConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + sqlite3_vtab *pNew; + int rc; + +/* Column numbers */ +#define CARRAY_COLUMN_VALUE 0 +#define CARRAY_COLUMN_POINTER 1 +#define CARRAY_COLUMN_COUNT 2 +#define CARRAY_COLUMN_CTYPE 3 + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(value,pointer hidden,count hidden,ctype hidden)"); + if( rc==SQLITE_OK ){ + pNew = *ppVtab = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + } + return rc; +} + +/* +** This method is the destructor for carray_cursor objects. +*/ +static int carrayDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new carray_cursor object. +*/ +static int carrayOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + carray_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a carray_cursor. +*/ +static int carrayClose(sqlite3_vtab_cursor *cur){ + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a carray_cursor to its next row of output. +*/ +static int carrayNext(sqlite3_vtab_cursor *cur){ + carray_cursor *pCur = (carray_cursor*)cur; + pCur->iRowid++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the carray_cursor +** is currently pointing. +*/ +static int carrayColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + carray_cursor *pCur = (carray_cursor*)cur; + sqlite3_int64 x = 0; + switch( i ){ + case CARRAY_COLUMN_POINTER: return SQLITE_OK; + case CARRAY_COLUMN_COUNT: x = pCur->iCnt; break; + case CARRAY_COLUMN_CTYPE: { + sqlite3_result_text(ctx, azType[pCur->eType], -1, SQLITE_STATIC); + return SQLITE_OK; + } + default: { + switch( pCur->eType ){ + case CARRAY_INT32: { + int *p = (int*)pCur->pPtr; + sqlite3_result_int(ctx, p[pCur->iRowid-1]); + return SQLITE_OK; + } + case CARRAY_INT64: { + sqlite3_int64 *p = (sqlite3_int64*)pCur->pPtr; + sqlite3_result_int64(ctx, p[pCur->iRowid-1]); + return SQLITE_OK; + } + case CARRAY_DOUBLE: { + double *p = (double*)pCur->pPtr; + sqlite3_result_double(ctx, p[pCur->iRowid-1]); + return SQLITE_OK; + } + case CARRAY_TEXT: { + const char **p = (const char**)pCur->pPtr; + sqlite3_result_text(ctx, p[pCur->iRowid-1], -1, SQLITE_TRANSIENT); + return SQLITE_OK; + } + } + } + } + sqlite3_result_int64(ctx, x); + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int carrayRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + carray_cursor *pCur = (carray_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int carrayEof(sqlite3_vtab_cursor *cur){ + carray_cursor *pCur = (carray_cursor*)cur; + return pCur->iRowid>pCur->iCnt; +} + +/* +** This method is called to "rewind" the carray_cursor object back +** to the first row of output. +*/ +static int carrayFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + carray_cursor *pCur = (carray_cursor *)pVtabCursor; + pCur->pPtr = 0; + pCur->iCnt = 0; + switch( idxNum ){ + case 1: { + carray_bind *pBind = sqlite3_value_pointer(argv[0], "carray-bind"); + if( pBind==0 ) break; + pCur->pPtr = pBind->aData; + pCur->iCnt = pBind->nData; + pCur->eType = pBind->mFlags & 0x03; + break; + } + case 2: + case 3: { + pCur->pPtr = sqlite3_value_pointer(argv[0], "carray"); + pCur->iCnt = pCur->pPtr ? sqlite3_value_int64(argv[1]) : 0; + if( idxNum<3 ){ + pCur->eType = CARRAY_INT32; + }else{ + unsigned char i; + const char *zType = (const char*)sqlite3_value_text(argv[2]); + for(i=0; i<sizeof(azType)/sizeof(azType[0]); i++){ + if( sqlite3_stricmp(zType, azType[i])==0 ) break; + } + if( i>=sizeof(azType)/sizeof(azType[0]) ){ + pVtabCursor->pVtab->zErrMsg = sqlite3_mprintf( + "unknown datatype: %Q", zType); + return SQLITE_ERROR; + }else{ + pCur->eType = i; + } + } + break; + } + } + pCur->iRowid = 1; + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the carray virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +** +** In this implementation idxNum is used to represent the +** query plan. idxStr is unused. +** +** idxNum is: +** +** 1 If only the pointer= constraint exists. In this case, the +** parameter must be bound using sqlite3_carray_bind(). +** +** 2 if the pointer= and count= constraints exist. +** +** 3 if the ctype= constraint also exists. +** +** idxNum is 0 otherwise and carray becomes an empty table. +*/ +static int carrayBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; /* Loop over constraints */ + int ptrIdx = -1; /* Index of the pointer= constraint, or -1 if none */ + int cntIdx = -1; /* Index of the count= constraint, or -1 if none */ + int ctypeIdx = -1; /* Index of the ctype= constraint, or -1 if none */ + + const struct sqlite3_index_constraint *pConstraint; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + if( pConstraint->op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue; + switch( pConstraint->iColumn ){ + case CARRAY_COLUMN_POINTER: + ptrIdx = i; + break; + case CARRAY_COLUMN_COUNT: + cntIdx = i; + break; + case CARRAY_COLUMN_CTYPE: + ctypeIdx = i; + break; + } + } + if( ptrIdx>=0 ){ + pIdxInfo->aConstraintUsage[ptrIdx].argvIndex = 1; + pIdxInfo->aConstraintUsage[ptrIdx].omit = 1; + pIdxInfo->estimatedCost = (double)1; + pIdxInfo->estimatedRows = 100; + pIdxInfo->idxNum = 1; + if( cntIdx>=0 ){ + pIdxInfo->aConstraintUsage[cntIdx].argvIndex = 2; + pIdxInfo->aConstraintUsage[cntIdx].omit = 1; + pIdxInfo->idxNum = 2; + if( ctypeIdx>=0 ){ + pIdxInfo->aConstraintUsage[ctypeIdx].argvIndex = 3; + pIdxInfo->aConstraintUsage[ctypeIdx].omit = 1; + pIdxInfo->idxNum = 3; + } + } + }else{ + pIdxInfo->estimatedCost = (double)2147483647; + pIdxInfo->estimatedRows = 2147483647; + pIdxInfo->idxNum = 0; + } + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** carray virtual table. +*/ +static sqlite3_module carrayModule = { + 0, /* iVersion */ + 0, /* xCreate */ + carrayConnect, /* xConnect */ + carrayBestIndex, /* xBestIndex */ + carrayDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + carrayOpen, /* xOpen - open a cursor */ + carrayClose, /* xClose - close a cursor */ + carrayFilter, /* xFilter - configure scan constraints */ + carrayNext, /* xNext - advance a cursor */ + carrayEof, /* xEof - check for end of scan */ + carrayColumn, /* xColumn - read data */ + carrayRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; + +/* +** Destructor for the carray_bind object +*/ +static void carrayBindDel(void *pPtr){ + carray_bind *p = (carray_bind*)pPtr; + if( p->xDel!=SQLITE_STATIC ){ + p->xDel(p->aData); + } + sqlite3_free(p); +} + +/* +** Invoke this interface in order to bind to the single-argument +** version of CARRAY(). +*/ +SQLITE_API int sqlite3_carray_bind( + sqlite3_stmt *pStmt, + int idx, + void *aData, + int nData, + int mFlags, + void (*xDestroy)(void*) +){ + carray_bind *pNew; + int i; + pNew = sqlite3_malloc64(sizeof(*pNew)); + if( pNew==0 ){ + if( xDestroy!=SQLITE_STATIC && xDestroy!=SQLITE_TRANSIENT ){ + xDestroy(aData); + } + return SQLITE_NOMEM; + } + pNew->nData = nData; + pNew->mFlags = mFlags; + if( xDestroy==SQLITE_TRANSIENT ){ + sqlite3_int64 sz = nData; + switch( mFlags & 0x03 ){ + case CARRAY_INT32: sz *= 4; break; + case CARRAY_INT64: sz *= 8; break; + case CARRAY_DOUBLE: sz *= 8; break; + case CARRAY_TEXT: sz *= sizeof(char*); break; + } + if( (mFlags & 0x03)==CARRAY_TEXT ){ + for(i=0; i<nData; i++){ + const char *z = ((char**)aData)[i]; + if( z ) sz += strlen(z) + 1; + } + } + pNew->aData = sqlite3_malloc64( sz ); + if( pNew->aData==0 ){ + sqlite3_free(pNew); + return SQLITE_NOMEM; + } + if( (mFlags & 0x03)==CARRAY_TEXT ){ + char **az = (char**)pNew->aData; + char *z = (char*)&az[nData]; + for(i=0; i<nData; i++){ + const char *zData = ((char**)aData)[i]; + sqlite3_int64 n; + if( zData==0 ){ + az[i] = 0; + continue; + } + az[i] = z; + n = strlen(zData); + memcpy(z, zData, n+1); + z += n+1; + } + }else{ + memcpy(pNew->aData, aData, sz); + } + pNew->xDel = sqlite3_free; + }else{ + pNew->aData = aData; + pNew->xDel = xDestroy; + } + return sqlite3_bind_pointer(pStmt, idx, pNew, "carray-bind", carrayBindDel); +} + + +/* +** For testing purpose in the TCL test harness, we need a method for +** setting the pointer value. The inttoptr(X) SQL function accomplishes +** this. Tcl script will bind an integer to X and the inttoptr() SQL +** function will use sqlite3_result_pointer() to convert that integer into +** a pointer. +** +** This is for testing on TCL only. +*/ +#ifdef SQLITE_TEST +static void inttoptrFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + void *p; + sqlite3_int64 i64; + i64 = sqlite3_value_int64(argv[0]); + if( sizeof(i64)==sizeof(p) ){ + memcpy(&p, &i64, sizeof(p)); + }else{ + int i32 = i64 & 0xffffffff; + memcpy(&p, &i32, sizeof(p)); + } + sqlite3_result_pointer(context, p, "carray", 0); +} +#endif /* SQLITE_TEST */ + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +SQLITE_API int sqlite3_carray_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "carray", &carrayModule, 0); +#ifdef SQLITE_TEST + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "inttoptr", 1, SQLITE_UTF8, 0, + inttoptrFunc, 0, 0); + } +#endif /* SQLITE_TEST */ +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + return rc; +} diff --git a/ext/misc/carray.h b/ext/misc/carray.h new file mode 100644 index 0000000..63df006 --- /dev/null +++ b/ext/misc/carray.h @@ -0,0 +1,50 @@ +/* +** 2020-11-17 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** Interface definitions for the CARRAY table-valued function +** extension. +*/ + +#ifndef _CARRAY_H +#define _CARRAY_H + +#include "sqlite3.h" /* Required for error code definitions */ + +#ifdef __cplusplus +extern "C" { +#endif + +/* Use this interface to bind an array to the single-argument version +** of CARRAY(). +*/ +SQLITE_API int sqlite3_carray_bind( + sqlite3_stmt *pStmt, /* Statement to be bound */ + int i, /* Parameter index */ + void *aData, /* Pointer to array data */ + int nData, /* Number of data elements */ + int mFlags, /* CARRAY flags */ + void (*xDel)(void*) /* Destructgor for aData*/ +); + +/* Allowed values for the mFlags parameter to sqlite3_carray_bind(). +*/ +#define CARRAY_INT32 0 /* Data is 32-bit signed integers */ +#define CARRAY_INT64 1 /* Data is 64-bit signed integers */ +#define CARRAY_DOUBLE 2 /* Data is doubles */ +#define CARRAY_TEXT 3 /* Data is char* */ + +#ifdef __cplusplus +} /* end of the 'extern "C"' block */ +#endif + +#endif /* ifndef _CARRAY_H */ + diff --git a/ext/misc/cksumvfs.c b/ext/misc/cksumvfs.c new file mode 100644 index 0000000..e7c2c9d --- /dev/null +++ b/ext/misc/cksumvfs.c @@ -0,0 +1,880 @@ +/* +** 2020-04-20 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file implements a VFS shim that writes a checksum on each page +** of an SQLite database file. When reading pages, the checksum is verified +** and an error is raised if the checksum is incorrect. +** +** COMPILING +** +** This extension requires SQLite 3.32.0 or later. It uses the +** sqlite3_database_file_object() interface which was added in +** version 3.32.0, so it will not link with an earlier version of +** SQLite. +** +** To build this extension as a separately loaded shared library or +** DLL, use compiler command-lines similar to the following: +** +** (linux) gcc -fPIC -shared cksumvfs.c -o cksumvfs.so +** (mac) clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib +** (windows) cl cksumvfs.c -link -dll -out:cksumvfs.dll +** +** You may want to add additional compiler options, of course, +** according to the needs of your project. +** +** If you want to statically link this extension with your product, +** then compile it like any other C-language module but add the +** "-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that +** it is being statically linked rather than dynamically linked +** +** LOADING +** +** To load this extension as a shared library, you first have to +** bring up a dummy SQLite database connection to use as the argument +** to the sqlite3_load_extension() API call. Then you invoke the +** sqlite3_load_extension() API and shutdown the dummy database +** connection. All subsequent database connections that are opened +** will include this extension. For example: +** +** sqlite3 *db; +** sqlite3_open(":memory:", &db); +** sqlite3_load_extension(db, "./cksumvfs"); +** sqlite3_close(db); +** +** If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and +** statically linked against the application, initialize it using +** a single API call as follows: +** +** sqlite3_register_cksumvfs(); +** +** Cksumvfs is a VFS Shim. When loaded, "cksmvfs" becomes the new +** default VFS and it uses the prior default VFS as the next VFS +** down in the stack. This is normally what you want. However, in +** complex situations where multiple VFS shims are being loaded, +** it might be important to ensure that cksumvfs is loaded in the +** correct order so that it sequences itself into the default VFS +** Shim stack in the right order. +** +** USING +** +** Open database connections using the sqlite3_open() or +** sqlite3_open_v2() interfaces, as normal. Ordinary database files +** (without a checksum) will operate normally. Databases with +** checksums will return an SQLITE_IOERR_DATA error if a page is +** encountered that contains an invalid checksum. +** +** Checksumming only works on databases that have a reserve-bytes +** value of exactly 8. The default value for reserve-bytes is 0. +** Hence, newly created database files will omit the checksum by +** default. To create a database that includes a checksum, change +** the reserve-bytes value to 8 by runing: +** +** int n = 8; +** sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVE_BYTES, &n); +** +** If you do this immediately after creating a new database file, +** before anything else has been written into the file, then that +** might be all that you need to do. Otherwise, the API call +** above should be followed by: +** +** sqlite3_exec(db, "VACUUM", 0, 0, 0); +** +** It never hurts to run the VACUUM, even if you don't need it. +** If the database is in WAL mode, you should shutdown and +** reopen all database connections before continuing. +** +** From the CLI, use the ".filectrl reserve_bytes 8" command, +** followed by "VACUUM;". +** +** Note that SQLite allows the number of reserve-bytes to be +** increased but not decreased. So if a database file already +** has a reserve-bytes value greater than 8, there is no way to +** activate checksumming on that database, other than to dump +** and restore the database file. Note also that other extensions +** might also make use of the reserve-bytes. Checksumming will +** be incompatible with those other extensions. +** +** VERIFICATION OF CHECKSUMS +** +** If any checksum is incorrect, the "PRAGMA quick_check" command +** will find it. To verify that checksums are actually enabled +** and running, use the following query: +** +** SELECT count(*), verify_checksum(data) +** FROM sqlite_dbpage +** GROUP BY 2; +** +** There are three possible outputs form the verify_checksum() +** function: 1, 0, and NULL. 1 is returned if the checksum is +** correct. 0 is returned if the checksum is incorrect. NULL +** is returned if the page is unreadable. If checksumming is +** enabled, the read will fail if the checksum is wrong, so the +** usual result from verify_checksum() on a bad checksum is NULL. +** +** If everything is OK, the query above should return a single +** row where the second column is 1. Any other result indicates +** either that there is a checksum error, or checksum validation +** is disabled. +** +** CONTROLLING CHECKSUM VERIFICATION +** +** The cksumvfs extension implements a new PRAGMA statement that can +** be used to disable, re-enable, or query the status of checksum +** verification: +** +** PRAGMA checksum_verification; -- query status +** PRAGMA checksum_verification=OFF; -- disable verification +** PRAGMA checksum_verification=ON; -- re-enable verification +** +** The "checksum_verification" pragma will return "1" (true) or "0" +** (false) if checksum verification is enabled or disabled, respectively. +** "Verification" in this context means the feature that causes +** SQLITE_IOERR_DATA errors if a checksum mismatch is detected while +** reading. Checksums are always kept up-to-date as long as the +** reserve-bytes value of the database is 8, regardless of the setting +** of this pragma. Checksum verification can be disabled (for example) +** to do forensic analysis of a database that has previously reported +** a checksum error. +** +** The "checksum_verification" pragma will always respond with "0" if +** the database file does not have a reserve-bytes value of 8. The +** pragma will return no rows at all if the cksumvfs extension is +** not loaded. +** +** IMPLEMENTATION NOTES +** +** The checksum is stored in the last 8 bytes of each page. This +** module only operates if the "bytes of reserved space on each page" +** value at offset 20 the SQLite database header is exactly 8. If +** the reserved-space value is not 8, this module is a no-op. +*/ +#if defined(SQLITE_AMALGAMATION) && !defined(SQLITE_CKSUMVFS_STATIC) +# define SQLITE_CKSUMVFS_STATIC +#endif +#ifdef SQLITE_CKSUMVFS_STATIC +# include "sqlite3.h" +#else +# include "sqlite3ext.h" + SQLITE_EXTENSION_INIT1 +#endif +#include <string.h> +#include <assert.h> + + +/* +** Forward declaration of objects used by this utility +*/ +typedef struct sqlite3_vfs CksmVfs; +typedef struct CksmFile CksmFile; + +/* +** Useful datatype abbreviations +*/ +#if !defined(SQLITE_AMALGAMATION) + typedef unsigned char u8; + typedef unsigned int u32; +#endif + +/* Access to a lower-level VFS that (might) implement dynamic loading, +** access to randomness, etc. +*/ +#define ORIGVFS(p) ((sqlite3_vfs*)((p)->pAppData)) +#define ORIGFILE(p) ((sqlite3_file*)(((CksmFile*)(p))+1)) + +/* An open file */ +struct CksmFile { + sqlite3_file base; /* IO methods */ + const char *zFName; /* Original name of the file */ + char computeCksm; /* True to compute checksums. + ** Always true if reserve size is 8. */ + char verifyCksm; /* True to verify checksums */ + char isWal; /* True if processing a WAL file */ + char inCkpt; /* Currently doing a checkpoint */ + CksmFile *pPartner; /* Ptr from WAL to main-db, or from main-db to WAL */ +}; + +/* +** Methods for CksmFile +*/ +static int cksmClose(sqlite3_file*); +static int cksmRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); +static int cksmWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); +static int cksmTruncate(sqlite3_file*, sqlite3_int64 size); +static int cksmSync(sqlite3_file*, int flags); +static int cksmFileSize(sqlite3_file*, sqlite3_int64 *pSize); +static int cksmLock(sqlite3_file*, int); +static int cksmUnlock(sqlite3_file*, int); +static int cksmCheckReservedLock(sqlite3_file*, int *pResOut); +static int cksmFileControl(sqlite3_file*, int op, void *pArg); +static int cksmSectorSize(sqlite3_file*); +static int cksmDeviceCharacteristics(sqlite3_file*); +static int cksmShmMap(sqlite3_file*, int iPg, int pgsz, int, void volatile**); +static int cksmShmLock(sqlite3_file*, int offset, int n, int flags); +static void cksmShmBarrier(sqlite3_file*); +static int cksmShmUnmap(sqlite3_file*, int deleteFlag); +static int cksmFetch(sqlite3_file*, sqlite3_int64 iOfst, int iAmt, void **pp); +static int cksmUnfetch(sqlite3_file*, sqlite3_int64 iOfst, void *p); + +/* +** Methods for CksmVfs +*/ +static int cksmOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); +static int cksmDelete(sqlite3_vfs*, const char *zName, int syncDir); +static int cksmAccess(sqlite3_vfs*, const char *zName, int flags, int *); +static int cksmFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); +static void *cksmDlOpen(sqlite3_vfs*, const char *zFilename); +static void cksmDlError(sqlite3_vfs*, int nByte, char *zErrMsg); +static void (*cksmDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); +static void cksmDlClose(sqlite3_vfs*, void*); +static int cksmRandomness(sqlite3_vfs*, int nByte, char *zOut); +static int cksmSleep(sqlite3_vfs*, int microseconds); +static int cksmCurrentTime(sqlite3_vfs*, double*); +static int cksmGetLastError(sqlite3_vfs*, int, char *); +static int cksmCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); +static int cksmSetSystemCall(sqlite3_vfs*, const char*,sqlite3_syscall_ptr); +static sqlite3_syscall_ptr cksmGetSystemCall(sqlite3_vfs*, const char *z); +static const char *cksmNextSystemCall(sqlite3_vfs*, const char *zName); + +static sqlite3_vfs cksm_vfs = { + 3, /* iVersion (set when registered) */ + 0, /* szOsFile (set when registered) */ + 1024, /* mxPathname */ + 0, /* pNext */ + "cksmvfs", /* zName */ + 0, /* pAppData (set when registered) */ + cksmOpen, /* xOpen */ + cksmDelete, /* xDelete */ + cksmAccess, /* xAccess */ + cksmFullPathname, /* xFullPathname */ + cksmDlOpen, /* xDlOpen */ + cksmDlError, /* xDlError */ + cksmDlSym, /* xDlSym */ + cksmDlClose, /* xDlClose */ + cksmRandomness, /* xRandomness */ + cksmSleep, /* xSleep */ + cksmCurrentTime, /* xCurrentTime */ + cksmGetLastError, /* xGetLastError */ + cksmCurrentTimeInt64, /* xCurrentTimeInt64 */ + cksmSetSystemCall, /* xSetSystemCall */ + cksmGetSystemCall, /* xGetSystemCall */ + cksmNextSystemCall /* xNextSystemCall */ +}; + +static const sqlite3_io_methods cksm_io_methods = { + 3, /* iVersion */ + cksmClose, /* xClose */ + cksmRead, /* xRead */ + cksmWrite, /* xWrite */ + cksmTruncate, /* xTruncate */ + cksmSync, /* xSync */ + cksmFileSize, /* xFileSize */ + cksmLock, /* xLock */ + cksmUnlock, /* xUnlock */ + cksmCheckReservedLock, /* xCheckReservedLock */ + cksmFileControl, /* xFileControl */ + cksmSectorSize, /* xSectorSize */ + cksmDeviceCharacteristics, /* xDeviceCharacteristics */ + cksmShmMap, /* xShmMap */ + cksmShmLock, /* xShmLock */ + cksmShmBarrier, /* xShmBarrier */ + cksmShmUnmap, /* xShmUnmap */ + cksmFetch, /* xFetch */ + cksmUnfetch /* xUnfetch */ +}; + +/* Do byte swapping on a unsigned 32-bit integer */ +#define BYTESWAP32(x) ( \ + (((x)&0x000000FF)<<24) + (((x)&0x0000FF00)<<8) \ + + (((x)&0x00FF0000)>>8) + (((x)&0xFF000000)>>24) \ +) + +/* Compute a checksum on a buffer */ +static void cksmCompute( + u8 *a, /* Content to be checksummed */ + int nByte, /* Bytes of content in a[]. Must be a multiple of 8. */ + u8 *aOut /* OUT: Final 8-byte checksum value output */ +){ + u32 s1 = 0, s2 = 0; + u32 *aData = (u32*)a; + u32 *aEnd = (u32*)&a[nByte]; + u32 x = 1; + + assert( nByte>=8 ); + assert( (nByte&0x00000007)==0 ); + assert( nByte<=65536 ); + + if( 1 == *(u8*)&x ){ + /* Little-endian */ + do { + s1 += *aData++ + s2; + s2 += *aData++ + s1; + }while( aData<aEnd ); + }else{ + /* Big-endian */ + do { + s1 += BYTESWAP32(aData[0]) + s2; + s2 += BYTESWAP32(aData[1]) + s1; + aData += 2; + }while( aData<aEnd ); + s1 = BYTESWAP32(s1); + s2 = BYTESWAP32(s2); + } + memcpy(aOut, &s1, 4); + memcpy(aOut+4, &s2, 4); +} + +/* +** SQL function: verify_checksum(BLOB) +** +** Return 0 or 1 if the checksum is invalid or valid. Or return +** NULL if the input is not a BLOB that is the right size for a +** database page. +*/ +static void cksmVerifyFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int nByte; + u8 *data; + u8 cksum[8]; + data = (u8*)sqlite3_value_blob(argv[0]); + if( data==0 ) return; + if( sqlite3_value_type(argv[0])!=SQLITE_BLOB ) return; + nByte = sqlite3_value_bytes(argv[0]); + if( nByte<512 || nByte>65536 || (nByte & (nByte-1))!=0 ) return; + cksmCompute(data, nByte-8, cksum); + sqlite3_result_int(context, memcmp(data+nByte-8,cksum,8)==0); +} + +#ifdef SQLITE_CKSUMVFS_INIT_FUNCNAME +/* +** SQL function: initialize_cksumvfs(SCHEMANAME) +** +** This SQL functions (whose name is actually determined at compile-time +** by the value of the SQLITE_CKSUMVFS_INIT_FUNCNAME macro) invokes: +** +** sqlite3_file_control(db, SCHEMANAME, SQLITE_FCNTL_RESERVE_BYTE, &n); +** +** In order to set the reserve bytes value to 8, so that cksumvfs will +** operation. This feature is provided (if and only if the +** SQLITE_CKSUMVFS_INIT_FUNCNAME compile-time option is set to a string +** which is the name of the SQL function) so as to provide the ability +** to invoke the file-control in programming languages that lack +** direct access to the sqlite3_file_control() interface (ex: Java). +** +** This interface is undocumented, apart from this comment. Usage +** example: +** +** 1. Compile with -DSQLITE_CKSUMVFS_INIT_FUNCNAME="ckvfs_init" +** 2. Run: "SELECT cksum_init('main'); VACUUM;" +*/ +static void cksmInitFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int nByte = 8; + const char *zSchemaName = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + sqlite3_file_control(db, zSchemaName, SQLITE_FCNTL_RESERVE_BYTES, &nByte); + /* Return NULL */ +} +#endif /* SQLITE_CKSUMBFS_INIT_FUNCNAME */ + +/* +** Close a cksm-file. +*/ +static int cksmClose(sqlite3_file *pFile){ + CksmFile *p = (CksmFile *)pFile; + if( p->pPartner ){ + assert( p->pPartner->pPartner==p ); + p->pPartner->pPartner = 0; + p->pPartner = 0; + } + pFile = ORIGFILE(pFile); + return pFile->pMethods->xClose(pFile); +} + +/* +** Set the computeCkSm and verifyCksm flags, if they need to be +** changed. +*/ +static void cksmSetFlags(CksmFile *p, int hasCorrectReserveSize){ + if( hasCorrectReserveSize!=p->computeCksm ){ + p->computeCksm = p->verifyCksm = hasCorrectReserveSize; + if( p->pPartner ){ + p->pPartner->verifyCksm = hasCorrectReserveSize; + p->pPartner->computeCksm = hasCorrectReserveSize; + } + } +} + +/* +** Read data from a cksm-file. +*/ +static int cksmRead( + sqlite3_file *pFile, + void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + int rc; + CksmFile *p = (CksmFile *)pFile; + pFile = ORIGFILE(pFile); + rc = pFile->pMethods->xRead(pFile, zBuf, iAmt, iOfst); + if( rc==SQLITE_OK ){ + if( iOfst==0 && iAmt>=100 && ( + memcmp(zBuf,"SQLite format 3",16)==0 || memcmp(zBuf,"ZV-",3)==0 + )){ + u8 *d = (u8*)zBuf; + char hasCorrectReserveSize = (d[20]==8); + cksmSetFlags(p, hasCorrectReserveSize); + } + /* Verify the checksum if + ** (1) the size indicates that we are dealing with a complete + ** database page + ** (2) checksum verification is enabled + ** (3) we are not in the middle of checkpoint + */ + if( iAmt>=512 /* (1) */ + && p->verifyCksm /* (2) */ + && !p->inCkpt /* (3) */ + ){ + u8 cksum[8]; + cksmCompute((u8*)zBuf, iAmt-8, cksum); + if( memcmp((u8*)zBuf+iAmt-8, cksum, 8)!=0 ){ + sqlite3_log(SQLITE_IOERR_DATA, + "checksum fault offset %lld of \"%s\"", + iOfst, p->zFName); + rc = SQLITE_IOERR_DATA; + } + } + } + return rc; +} + +/* +** Write data to a cksm-file. +*/ +static int cksmWrite( + sqlite3_file *pFile, + const void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + CksmFile *p = (CksmFile *)pFile; + pFile = ORIGFILE(pFile); + if( iOfst==0 && iAmt>=100 && ( + memcmp(zBuf,"SQLite format 3",16)==0 || memcmp(zBuf,"ZV-",3)==0 + )){ + u8 *d = (u8*)zBuf; + char hasCorrectReserveSize = (d[20]==8); + cksmSetFlags(p, hasCorrectReserveSize); + } + /* If the write size is appropriate for a database page and if + ** checksums where ever enabled, then it will be safe to compute + ** the checksums. The reserve byte size might have increased, but + ** it will never decrease. And because it cannot decrease, the + ** checksum will not overwrite anything. + */ + if( iAmt>=512 + && p->computeCksm + && !p->inCkpt + ){ + cksmCompute((u8*)zBuf, iAmt-8, ((u8*)zBuf)+iAmt-8); + } + return pFile->pMethods->xWrite(pFile, zBuf, iAmt, iOfst); +} + +/* +** Truncate a cksm-file. +*/ +static int cksmTruncate(sqlite3_file *pFile, sqlite_int64 size){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xTruncate(pFile, size); +} + +/* +** Sync a cksm-file. +*/ +static int cksmSync(sqlite3_file *pFile, int flags){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xSync(pFile, flags); +} + +/* +** Return the current file-size of a cksm-file. +*/ +static int cksmFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ + CksmFile *p = (CksmFile *)pFile; + pFile = ORIGFILE(p); + return pFile->pMethods->xFileSize(pFile, pSize); +} + +/* +** Lock a cksm-file. +*/ +static int cksmLock(sqlite3_file *pFile, int eLock){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xLock(pFile, eLock); +} + +/* +** Unlock a cksm-file. +*/ +static int cksmUnlock(sqlite3_file *pFile, int eLock){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xUnlock(pFile, eLock); +} + +/* +** Check if another file-handle holds a RESERVED lock on a cksm-file. +*/ +static int cksmCheckReservedLock(sqlite3_file *pFile, int *pResOut){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xCheckReservedLock(pFile, pResOut); +} + +/* +** File control method. For custom operations on a cksm-file. +*/ +static int cksmFileControl(sqlite3_file *pFile, int op, void *pArg){ + int rc; + CksmFile *p = (CksmFile*)pFile; + pFile = ORIGFILE(pFile); + if( op==SQLITE_FCNTL_PRAGMA ){ + char **azArg = (char**)pArg; + assert( azArg[1]!=0 ); + if( sqlite3_stricmp(azArg[1],"checksum_verification")==0 ){ + char *zArg = azArg[2]; + if( zArg!=0 ){ + if( (zArg[0]>='1' && zArg[0]<='9') + || sqlite3_strlike("enable%",zArg,0)==0 + || sqlite3_stricmp("yes",zArg)==0 + || sqlite3_stricmp("on",zArg)==0 + ){ + p->verifyCksm = p->computeCksm; + }else{ + p->verifyCksm = 0; + } + if( p->pPartner ) p->pPartner->verifyCksm = p->verifyCksm; + } + azArg[0] = sqlite3_mprintf("%d",p->verifyCksm); + return SQLITE_OK; + }else if( p->computeCksm && azArg[2]!=0 + && sqlite3_stricmp(azArg[1], "page_size")==0 ){ + /* Do not allow page size changes on a checksum database */ + return SQLITE_OK; + } + }else if( op==SQLITE_FCNTL_CKPT_START || op==SQLITE_FCNTL_CKPT_DONE ){ + p->inCkpt = op==SQLITE_FCNTL_CKPT_START; + if( p->pPartner ) p->pPartner->inCkpt = p->inCkpt; + }else if( op==SQLITE_FCNTL_CKSM_FILE ){ + /* This VFS needs to obtain a pointer to the corresponding database + ** file handle from within xOpen() calls to open wal files. To do this, + ** it uses the sqlite3_database_file_object() API to obtain a pointer + ** to the file-handle used by SQLite to access the db file. This is + ** fine if cksmvfs happens to be the top-level VFS, but not if there + ** are one or more wrapper VFS. To handle this case, this file-control + ** is used to extract the cksmvfs file-handle from any wrapper file + ** handle. */ + sqlite3_file **ppFile = (sqlite3_file**)pArg; + *ppFile = (sqlite3_file*)p; + return SQLITE_OK; + } + rc = pFile->pMethods->xFileControl(pFile, op, pArg); + if( rc==SQLITE_OK && op==SQLITE_FCNTL_VFSNAME ){ + *(char**)pArg = sqlite3_mprintf("cksm/%z", *(char**)pArg); + } + return rc; +} + +/* +** Return the sector-size in bytes for a cksm-file. +*/ +static int cksmSectorSize(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xSectorSize(pFile); +} + +/* +** Return the device characteristic flags supported by a cksm-file. +*/ +static int cksmDeviceCharacteristics(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xDeviceCharacteristics(pFile); +} + +/* Create a shared memory file mapping */ +static int cksmShmMap( + sqlite3_file *pFile, + int iPg, + int pgsz, + int bExtend, + void volatile **pp +){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmMap(pFile,iPg,pgsz,bExtend,pp); +} + +/* Perform locking on a shared-memory segment */ +static int cksmShmLock(sqlite3_file *pFile, int offset, int n, int flags){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmLock(pFile,offset,n,flags); +} + +/* Memory barrier operation on shared memory */ +static void cksmShmBarrier(sqlite3_file *pFile){ + pFile = ORIGFILE(pFile); + pFile->pMethods->xShmBarrier(pFile); +} + +/* Unmap a shared memory segment */ +static int cksmShmUnmap(sqlite3_file *pFile, int deleteFlag){ + pFile = ORIGFILE(pFile); + return pFile->pMethods->xShmUnmap(pFile,deleteFlag); +} + +/* Fetch a page of a memory-mapped file */ +static int cksmFetch( + sqlite3_file *pFile, + sqlite3_int64 iOfst, + int iAmt, + void **pp +){ + CksmFile *p = (CksmFile *)pFile; + if( p->computeCksm ){ + *pp = 0; + return SQLITE_OK; + } + pFile = ORIGFILE(pFile); + if( pFile->pMethods->iVersion>2 && pFile->pMethods->xFetch ){ + return pFile->pMethods->xFetch(pFile, iOfst, iAmt, pp); + } + *pp = 0; + return SQLITE_OK; +} + +/* Release a memory-mapped page */ +static int cksmUnfetch(sqlite3_file *pFile, sqlite3_int64 iOfst, void *pPage){ + pFile = ORIGFILE(pFile); + if( pFile->pMethods->iVersion>2 && pFile->pMethods->xUnfetch ){ + return pFile->pMethods->xUnfetch(pFile, iOfst, pPage); + } + return SQLITE_OK; +} + +/* +** Open a cksm file handle. +*/ +static int cksmOpen( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_file *pFile, + int flags, + int *pOutFlags +){ + CksmFile *p; + sqlite3_file *pSubFile; + sqlite3_vfs *pSubVfs; + int rc; + pSubVfs = ORIGVFS(pVfs); + if( (flags & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_WAL))==0 ){ + return pSubVfs->xOpen(pSubVfs, zName, pFile, flags, pOutFlags); + } + p = (CksmFile*)pFile; + memset(p, 0, sizeof(*p)); + pSubFile = ORIGFILE(pFile); + pFile->pMethods = &cksm_io_methods; + rc = pSubVfs->xOpen(pSubVfs, zName, pSubFile, flags, pOutFlags); + if( rc ) goto cksm_open_done; + if( flags & SQLITE_OPEN_WAL ){ + sqlite3_file *pDb = sqlite3_database_file_object(zName); + rc = pDb->pMethods->xFileControl(pDb, SQLITE_FCNTL_CKSM_FILE, (void*)&pDb); + assert( rc==SQLITE_OK ); + p->pPartner = (CksmFile*)pDb; + assert( p->pPartner->pPartner==0 ); + p->pPartner->pPartner = p; + p->isWal = 1; + p->computeCksm = p->pPartner->computeCksm; + }else{ + p->isWal = 0; + p->computeCksm = 0; + } + p->zFName = zName; +cksm_open_done: + if( rc ) pFile->pMethods = 0; + return rc; +} + +/* +** All other VFS methods are pass-thrus. +*/ +static int cksmDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ + return ORIGVFS(pVfs)->xDelete(ORIGVFS(pVfs), zPath, dirSync); +} +static int cksmAccess( + sqlite3_vfs *pVfs, + const char *zPath, + int flags, + int *pResOut +){ + return ORIGVFS(pVfs)->xAccess(ORIGVFS(pVfs), zPath, flags, pResOut); +} +static int cksmFullPathname( + sqlite3_vfs *pVfs, + const char *zPath, + int nOut, + char *zOut +){ + return ORIGVFS(pVfs)->xFullPathname(ORIGVFS(pVfs),zPath,nOut,zOut); +} +static void *cksmDlOpen(sqlite3_vfs *pVfs, const char *zPath){ + return ORIGVFS(pVfs)->xDlOpen(ORIGVFS(pVfs), zPath); +} +static void cksmDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ + ORIGVFS(pVfs)->xDlError(ORIGVFS(pVfs), nByte, zErrMsg); +} +static void (*cksmDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ + return ORIGVFS(pVfs)->xDlSym(ORIGVFS(pVfs), p, zSym); +} +static void cksmDlClose(sqlite3_vfs *pVfs, void *pHandle){ + ORIGVFS(pVfs)->xDlClose(ORIGVFS(pVfs), pHandle); +} +static int cksmRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ + return ORIGVFS(pVfs)->xRandomness(ORIGVFS(pVfs), nByte, zBufOut); +} +static int cksmSleep(sqlite3_vfs *pVfs, int nMicro){ + return ORIGVFS(pVfs)->xSleep(ORIGVFS(pVfs), nMicro); +} +static int cksmCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ + return ORIGVFS(pVfs)->xCurrentTime(ORIGVFS(pVfs), pTimeOut); +} +static int cksmGetLastError(sqlite3_vfs *pVfs, int a, char *b){ + return ORIGVFS(pVfs)->xGetLastError(ORIGVFS(pVfs), a, b); +} +static int cksmCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ + sqlite3_vfs *pOrig = ORIGVFS(pVfs); + int rc; + assert( pOrig->iVersion>=2 ); + if( pOrig->xCurrentTimeInt64 ){ + rc = pOrig->xCurrentTimeInt64(pOrig, p); + }else{ + double r; + rc = pOrig->xCurrentTime(pOrig, &r); + *p = (sqlite3_int64)(r*86400000.0); + } + return rc; +} +static int cksmSetSystemCall( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_syscall_ptr pCall +){ + return ORIGVFS(pVfs)->xSetSystemCall(ORIGVFS(pVfs),zName,pCall); +} +static sqlite3_syscall_ptr cksmGetSystemCall( + sqlite3_vfs *pVfs, + const char *zName +){ + return ORIGVFS(pVfs)->xGetSystemCall(ORIGVFS(pVfs),zName); +} +static const char *cksmNextSystemCall(sqlite3_vfs *pVfs, const char *zName){ + return ORIGVFS(pVfs)->xNextSystemCall(ORIGVFS(pVfs), zName); +} + +/* Register the verify_checksum() SQL function. +*/ +static int cksmRegisterFunc( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc; + if( db==0 ) return SQLITE_OK; + rc = sqlite3_create_function(db, "verify_checksum", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, cksmVerifyFunc, 0, 0); +#ifdef SQLITE_CKSUMVFS_INIT_FUNCNAME + (void)sqlite3_create_function(db, SQLITE_CKSUMVFS_INIT_FUNCNAME, 1, + SQLITE_UTF8|SQLITE_DIRECTONLY, + 0, cksmInitFunc, 0, 0); +#endif + return rc; +} + +/* +** Register the cksum VFS as the default VFS for the system. +** Also make arrangements to automatically register the "verify_checksum()" +** SQL function on each new database connection. +*/ +static int cksmRegisterVfs(void){ + int rc = SQLITE_OK; + sqlite3_vfs *pOrig; + if( sqlite3_vfs_find("cksmvfs")!=0 ) return SQLITE_OK; + pOrig = sqlite3_vfs_find(0); + if( pOrig==0 ) return SQLITE_ERROR; + cksm_vfs.iVersion = pOrig->iVersion; + cksm_vfs.pAppData = pOrig; + cksm_vfs.szOsFile = pOrig->szOsFile + sizeof(CksmFile); + rc = sqlite3_vfs_register(&cksm_vfs, 1); + if( rc==SQLITE_OK ){ + rc = sqlite3_auto_extension((void(*)(void))cksmRegisterFunc); + } + return rc; +} + +#if defined(SQLITE_CKSUMVFS_STATIC) +/* This variant of the initializer runs when the extension is +** statically linked. +*/ +int sqlite3_register_cksumvfs(const char *NotUsed){ + (void)NotUsed; + return cksmRegisterVfs(); +} +int sqlite3_unregister_cksumvfs(void){ + if( sqlite3_vfs_find("cksmvfs") ){ + sqlite3_vfs_unregister(&cksm_vfs); + sqlite3_cancel_auto_extension((void(*)(void))cksmRegisterFunc); + } + return SQLITE_OK; +} +#endif /* defined(SQLITE_CKSUMVFS_STATIC */ + +#if !defined(SQLITE_CKSUMVFS_STATIC) +/* This variant of the initializer function is used when the +** extension is shared library to be loaded at run-time. +*/ +#ifdef _WIN32 +__declspec(dllexport) +#endif +/* +** This routine is called by sqlite3_load_extension() when the +** extension is first loaded. +***/ +int sqlite3_cksumvfs_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* not used */ + rc = cksmRegisterFunc(db, 0, 0); + if( rc==SQLITE_OK ){ + rc = cksmRegisterVfs(); + } + if( rc==SQLITE_OK ) rc = SQLITE_OK_LOAD_PERMANENTLY; + return rc; +} +#endif /* !defined(SQLITE_CKSUMVFS_STATIC) */ diff --git a/ext/misc/closure.c b/ext/misc/closure.c new file mode 100644 index 0000000..db9b2b7 --- /dev/null +++ b/ext/misc/closure.c @@ -0,0 +1,965 @@ +/* +** 2013-04-16 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file contains code for a virtual table that finds the transitive +** closure of a parent/child relationship in a real table. The virtual +** table is called "transitive_closure". +** +** A transitive_closure virtual table is created like this: +** +** CREATE VIRTUAL TABLE x USING transitive_closure( +** tablename=<tablename>, -- T +** idcolumn=<columnname>, -- X +** parentcolumn=<columnname> -- P +** ); +** +** When it is created, the new transitive_closure table may be supplied +** with default values for the name of a table T and columns T.X and T.P. +** The T.X and T.P columns must contain integers. The ideal case is for +** T.X to be the INTEGER PRIMARY KEY. The T.P column should reference +** the T.X column. The row referenced by T.P is the parent of the current row. +** +** The tablename, idcolumn, and parentcolumn supplied by the CREATE VIRTUAL +** TABLE statement may be overridden in individual queries by including +** terms like tablename='newtable', idcolumn='id2', or +** parentcolumn='parent3' in the WHERE clause of the query. +** +** For efficiency, it is essential that there be an index on the P column: +** +** CREATE Tidx1 ON T(P) +** +** Suppose a specific instance of the closure table is as follows: +** +** CREATE VIRTUAL TABLE ct1 USING transitive_closure( +** tablename='group', +** idcolumn='groupId', +** parentcolumn='parentId' +** ); +** +** Such an instance of the transitive_closure virtual table would be +** appropriate for walking a tree defined using a table like this, for example: +** +** CREATE TABLE group( +** groupId INTEGER PRIMARY KEY, +** parentId INTEGER REFERENCES group +** ); +** CREATE INDEX group_idx1 ON group(parentId); +** +** The group table above would presumably have other application-specific +** fields. The key point here is that rows of the group table form a +** tree. The purpose of the ct1 virtual table is to easily extract +** branches of that tree. +** +** Once it has been created, the ct1 virtual table can be queried +** as follows: +** +** SELECT * FROM element +** WHERE element.groupId IN (SELECT id FROM ct1 WHERE root=?1); +** +** The above query will return all elements that are part of group ?1 +** or children of group ?1 or grand-children of ?1 and so forth for all +** descendents of group ?1. The same query can be formulated as a join: +** +** SELECT element.* FROM element, ct1 +** WHERE element.groupid=ct1.id +** AND ct1.root=?1; +** +** The depth of the transitive_closure (the number of generations of +** parent/child relations to follow) can be limited by setting "depth" +** column in the WHERE clause. So, for example, the following query +** finds only children and grandchildren but no further descendents: +** +** SELECT element.* FROM element, ct1 +** WHERE element.groupid=ct1.id +** AND ct1.root=?1 +** AND ct1.depth<=2; +** +** The "ct1.depth<=2" term could be a strict equality "ct1.depth=2" in +** order to find only the grandchildren of ?1, not ?1 itself or the +** children of ?1. +** +** The root=?1 term must be supplied in WHERE clause or else the query +** of the ct1 virtual table will return an empty set. The tablename, +** idcolumn, and parentcolumn attributes can be overridden in the WHERE +** clause if desired. So, for example, the ct1 table could be repurposed +** to find ancestors rather than descendents by inverting the roles of +** the idcolumn and parentcolumn: +** +** SELECT element.* FROM element, ct1 +** WHERE element.groupid=ct1.id +** AND ct1.root=?1 +** AND ct1.idcolumn='parentId' +** AND ct1.parentcolumn='groupId'; +** +** Multiple calls to ct1 could be combined. For example, the following +** query finds all elements that "cousins" of groupId ?1. That is to say +** elements where the groupId is a grandchild of the grandparent of ?1. +** (This definition of "cousins" also includes siblings and self.) +** +** SELECT element.* FROM element, ct1 +** WHERE element.groupId=ct1.id +** AND ct1.depth=2 +** AND ct1.root IN (SELECT id FROM ct1 +** WHERE root=?1 +** AND depth=2 +** AND idcolumn='parentId' +** AND parentcolumn='groupId'); +** +** In our example, the group.groupId column is unique and thus the +** subquery will return exactly one row. For that reason, the IN +** operator could be replaced by "=" to get the same result. But +** in the general case where the idcolumn is not unique, an IN operator +** would be required for this kind of query. +** +** Note that because the tablename, idcolumn, and parentcolumn can +** all be specified in the query, it is possible for an application +** to define a single transitive_closure virtual table for use on lots +** of different hierarchy tables. One might say: +** +** CREATE VIRTUAL TABLE temp.closure USING transitive_closure; +** +** As each database connection is being opened. Then the application +** would always have a "closure" virtual table handy to use for querying. +** +** SELECT element.* FROM element, closure +** WHERE element.groupid=ct1.id +** AND closure.root=?1 +** AND closure.tablename='group' +** AND closure.idname='groupId' +** AND closure.parentname='parentId'; +** +** See the documentation at http://www.sqlite.org/loadext.html for information +** on how to compile and use loadable extensions such as this one. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <stdio.h> +#include <ctype.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Forward declaration of objects used by this implementation +*/ +typedef struct closure_vtab closure_vtab; +typedef struct closure_cursor closure_cursor; +typedef struct closure_queue closure_queue; +typedef struct closure_avl closure_avl; + +/***************************************************************************** +** AVL Tree implementation +*/ +/* +** Objects that want to be members of the AVL tree should embedded an +** instance of this structure. +*/ +struct closure_avl { + sqlite3_int64 id; /* Id of this entry in the table */ + int iGeneration; /* Which generation is this entry part of */ + closure_avl *pList; /* A linked list of nodes */ + closure_avl *pBefore; /* Other elements less than id */ + closure_avl *pAfter; /* Other elements greater than id */ + closure_avl *pUp; /* Parent element */ + short int height; /* Height of this node. Leaf==1 */ + short int imbalance; /* Height difference between pBefore and pAfter */ +}; + +/* Recompute the closure_avl.height and closure_avl.imbalance fields for p. +** Assume that the children of p have correct heights. +*/ +static void closureAvlRecomputeHeight(closure_avl *p){ + short int hBefore = p->pBefore ? p->pBefore->height : 0; + short int hAfter = p->pAfter ? p->pAfter->height : 0; + p->imbalance = hBefore - hAfter; /* -: pAfter higher. +: pBefore higher */ + p->height = (hBefore>hAfter ? hBefore : hAfter)+1; +} + +/* +** P B +** / \ / \ +** B Z ==> X P +** / \ / \ +** X Y Y Z +** +*/ +static closure_avl *closureAvlRotateBefore(closure_avl *pP){ + closure_avl *pB = pP->pBefore; + closure_avl *pY = pB->pAfter; + pB->pUp = pP->pUp; + pB->pAfter = pP; + pP->pUp = pB; + pP->pBefore = pY; + if( pY ) pY->pUp = pP; + closureAvlRecomputeHeight(pP); + closureAvlRecomputeHeight(pB); + return pB; +} + +/* +** P A +** / \ / \ +** X A ==> P Z +** / \ / \ +** Y Z X Y +** +*/ +static closure_avl *closureAvlRotateAfter(closure_avl *pP){ + closure_avl *pA = pP->pAfter; + closure_avl *pY = pA->pBefore; + pA->pUp = pP->pUp; + pA->pBefore = pP; + pP->pUp = pA; + pP->pAfter = pY; + if( pY ) pY->pUp = pP; + closureAvlRecomputeHeight(pP); + closureAvlRecomputeHeight(pA); + return pA; +} + +/* +** Return a pointer to the pBefore or pAfter pointer in the parent +** of p that points to p. Or if p is the root node, return pp. +*/ +static closure_avl **closureAvlFromPtr(closure_avl *p, closure_avl **pp){ + closure_avl *pUp = p->pUp; + if( pUp==0 ) return pp; + if( pUp->pAfter==p ) return &pUp->pAfter; + return &pUp->pBefore; +} + +/* +** Rebalance all nodes starting with p and working up to the root. +** Return the new root. +*/ +static closure_avl *closureAvlBalance(closure_avl *p){ + closure_avl *pTop = p; + closure_avl **pp; + while( p ){ + closureAvlRecomputeHeight(p); + if( p->imbalance>=2 ){ + closure_avl *pB = p->pBefore; + if( pB->imbalance<0 ) p->pBefore = closureAvlRotateAfter(pB); + pp = closureAvlFromPtr(p,&p); + p = *pp = closureAvlRotateBefore(p); + }else if( p->imbalance<=(-2) ){ + closure_avl *pA = p->pAfter; + if( pA->imbalance>0 ) p->pAfter = closureAvlRotateBefore(pA); + pp = closureAvlFromPtr(p,&p); + p = *pp = closureAvlRotateAfter(p); + } + pTop = p; + p = p->pUp; + } + return pTop; +} + +/* Search the tree rooted at p for an entry with id. Return a pointer +** to the entry or return NULL. +*/ +static closure_avl *closureAvlSearch(closure_avl *p, sqlite3_int64 id){ + while( p && id!=p->id ){ + p = (id<p->id) ? p->pBefore : p->pAfter; + } + return p; +} + +/* Find the first node (the one with the smallest key). +*/ +static closure_avl *closureAvlFirst(closure_avl *p){ + if( p ) while( p->pBefore ) p = p->pBefore; + return p; +} + +/* Return the node with the next larger key after p. +*/ +closure_avl *closureAvlNext(closure_avl *p){ + closure_avl *pPrev = 0; + while( p && p->pAfter==pPrev ){ + pPrev = p; + p = p->pUp; + } + if( p && pPrev==0 ){ + p = closureAvlFirst(p->pAfter); + } + return p; +} + +/* Insert a new node pNew. Return NULL on success. If the key is not +** unique, then do not perform the insert but instead leave pNew unchanged +** and return a pointer to an existing node with the same key. +*/ +static closure_avl *closureAvlInsert( + closure_avl **ppHead, /* Head of the tree */ + closure_avl *pNew /* New node to be inserted */ +){ + closure_avl *p = *ppHead; + if( p==0 ){ + p = pNew; + pNew->pUp = 0; + }else{ + while( p ){ + if( pNew->id<p->id ){ + if( p->pBefore ){ + p = p->pBefore; + }else{ + p->pBefore = pNew; + pNew->pUp = p; + break; + } + }else if( pNew->id>p->id ){ + if( p->pAfter ){ + p = p->pAfter; + }else{ + p->pAfter = pNew; + pNew->pUp = p; + break; + } + }else{ + return p; + } + } + } + pNew->pBefore = 0; + pNew->pAfter = 0; + pNew->height = 1; + pNew->imbalance = 0; + *ppHead = closureAvlBalance(p); + return 0; +} + +/* Walk the tree can call xDestroy on each node +*/ +static void closureAvlDestroy(closure_avl *p, void (*xDestroy)(closure_avl*)){ + if( p ){ + closureAvlDestroy(p->pBefore, xDestroy); + closureAvlDestroy(p->pAfter, xDestroy); + xDestroy(p); + } +} +/* +** End of the AVL Tree implementation +******************************************************************************/ + +/* +** A closure virtual-table object +*/ +struct closure_vtab { + sqlite3_vtab base; /* Base class - must be first */ + char *zDb; /* Name of database. (ex: "main") */ + char *zSelf; /* Name of this virtual table */ + char *zTableName; /* Name of table holding parent/child relation */ + char *zIdColumn; /* Name of ID column of zTableName */ + char *zParentColumn; /* Name of PARENT column in zTableName */ + sqlite3 *db; /* The database connection */ + int nCursor; /* Number of pending cursors */ +}; + +/* A closure cursor object */ +struct closure_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + closure_vtab *pVtab; /* The virtual table this cursor belongs to */ + char *zTableName; /* Name of table holding parent/child relation */ + char *zIdColumn; /* Name of ID column of zTableName */ + char *zParentColumn; /* Name of PARENT column in zTableName */ + closure_avl *pCurrent; /* Current element of output */ + closure_avl *pClosure; /* The complete closure tree */ +}; + +/* A queue of AVL nodes */ +struct closure_queue { + closure_avl *pFirst; /* Oldest node on the queue */ + closure_avl *pLast; /* Youngest node on the queue */ +}; + +/* +** Add a node to the end of the queue +*/ +static void queuePush(closure_queue *pQueue, closure_avl *pNode){ + pNode->pList = 0; + if( pQueue->pLast ){ + pQueue->pLast->pList = pNode; + }else{ + pQueue->pFirst = pNode; + } + pQueue->pLast = pNode; +} + +/* +** Extract the oldest element (the front element) from the queue. +*/ +static closure_avl *queuePull(closure_queue *pQueue){ + closure_avl *p = pQueue->pFirst; + if( p ){ + pQueue->pFirst = p->pList; + if( pQueue->pFirst==0 ) pQueue->pLast = 0; + } + return p; +} + +/* +** This function converts an SQL quoted string into an unquoted string +** and returns a pointer to a buffer allocated using sqlite3_malloc() +** containing the result. The caller should eventually free this buffer +** using sqlite3_free. +** +** Examples: +** +** "abc" becomes abc +** 'xyz' becomes xyz +** [pqr] becomes pqr +** `mno` becomes mno +*/ +static char *closureDequote(const char *zIn){ + sqlite3_int64 nIn; /* Size of input string, in bytes */ + char *zOut; /* Output (dequoted) string */ + + nIn = strlen(zIn); + zOut = sqlite3_malloc64(nIn+1); + if( zOut ){ + char q = zIn[0]; /* Quote character (if any ) */ + + if( q!='[' && q!= '\'' && q!='"' && q!='`' ){ + memcpy(zOut, zIn, (size_t)(nIn+1)); + }else{ + int iOut = 0; /* Index of next byte to write to output */ + int iIn; /* Index of next byte to read from input */ + + if( q=='[' ) q = ']'; + for(iIn=1; iIn<nIn; iIn++){ + if( zIn[iIn]==q ) iIn++; + zOut[iOut++] = zIn[iIn]; + } + } + assert( (int)strlen(zOut)<=nIn ); + } + return zOut; +} + +/* +** Deallocate an closure_vtab object +*/ +static void closureFree(closure_vtab *p){ + if( p ){ + sqlite3_free(p->zDb); + sqlite3_free(p->zSelf); + sqlite3_free(p->zTableName); + sqlite3_free(p->zIdColumn); + sqlite3_free(p->zParentColumn); + memset(p, 0, sizeof(*p)); + sqlite3_free(p); + } +} + +/* +** xDisconnect/xDestroy method for the closure module. +*/ +static int closureDisconnect(sqlite3_vtab *pVtab){ + closure_vtab *p = (closure_vtab*)pVtab; + assert( p->nCursor==0 ); + closureFree(p); + return SQLITE_OK; +} + +/* +** Check to see if the argument is of the form: +** +** KEY = VALUE +** +** If it is, return a pointer to the first character of VALUE. +** If not, return NULL. Spaces around the = are ignored. +*/ +static const char *closureValueOfKey(const char *zKey, const char *zStr){ + int nKey = (int)strlen(zKey); + int nStr = (int)strlen(zStr); + int i; + if( nStr<nKey+1 ) return 0; + if( memcmp(zStr, zKey, nKey)!=0 ) return 0; + for(i=nKey; isspace((unsigned char)zStr[i]); i++){} + if( zStr[i]!='=' ) return 0; + i++; + while( isspace((unsigned char)zStr[i]) ){ i++; } + return zStr+i; +} + +/* +** xConnect/xCreate method for the closure module. Arguments are: +** +** argv[0] -> module name ("transitive_closure") +** argv[1] -> database name +** argv[2] -> table name +** argv[3...] -> arguments +*/ +static int closureConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + int rc = SQLITE_OK; /* Return code */ + closure_vtab *pNew = 0; /* New virtual table */ + const char *zDb = argv[1]; + const char *zVal; + int i; + + (void)pAux; + *ppVtab = 0; + pNew = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + rc = SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + pNew->zDb = sqlite3_mprintf("%s", zDb); + if( pNew->zDb==0 ) goto closureConnectError; + pNew->zSelf = sqlite3_mprintf("%s", argv[2]); + if( pNew->zSelf==0 ) goto closureConnectError; + for(i=3; i<argc; i++){ + zVal = closureValueOfKey("tablename", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zTableName); + pNew->zTableName = closureDequote(zVal); + if( pNew->zTableName==0 ) goto closureConnectError; + continue; + } + zVal = closureValueOfKey("idcolumn", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zIdColumn); + pNew->zIdColumn = closureDequote(zVal); + if( pNew->zIdColumn==0 ) goto closureConnectError; + continue; + } + zVal = closureValueOfKey("parentcolumn", argv[i]); + if( zVal ){ + sqlite3_free(pNew->zParentColumn); + pNew->zParentColumn = closureDequote(zVal); + if( pNew->zParentColumn==0 ) goto closureConnectError; + continue; + } + *pzErr = sqlite3_mprintf("unrecognized argument: [%s]\n", argv[i]); + closureFree(pNew); + *ppVtab = 0; + return SQLITE_ERROR; + } + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(id,depth,root HIDDEN,tablename HIDDEN," + "idcolumn HIDDEN,parentcolumn HIDDEN)" + ); +#define CLOSURE_COL_ID 0 +#define CLOSURE_COL_DEPTH 1 +#define CLOSURE_COL_ROOT 2 +#define CLOSURE_COL_TABLENAME 3 +#define CLOSURE_COL_IDCOLUMN 4 +#define CLOSURE_COL_PARENTCOLUMN 5 + if( rc!=SQLITE_OK ){ + closureFree(pNew); + } + *ppVtab = &pNew->base; + return rc; + +closureConnectError: + closureFree(pNew); + return rc; +} + +/* +** Open a new closure cursor. +*/ +static int closureOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ + closure_vtab *p = (closure_vtab*)pVTab; + closure_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->pVtab = p; + *ppCursor = &pCur->base; + p->nCursor++; + return SQLITE_OK; +} + +/* +** Free up all the memory allocated by a cursor. Set it rLimit to 0 +** to indicate that it is at EOF. +*/ +static void closureClearCursor(closure_cursor *pCur){ + closureAvlDestroy(pCur->pClosure, (void(*)(closure_avl*))sqlite3_free); + sqlite3_free(pCur->zTableName); + sqlite3_free(pCur->zIdColumn); + sqlite3_free(pCur->zParentColumn); + pCur->zTableName = 0; + pCur->zIdColumn = 0; + pCur->zParentColumn = 0; + pCur->pCurrent = 0; + pCur->pClosure = 0; +} + +/* +** Close a closure cursor. +*/ +static int closureClose(sqlite3_vtab_cursor *cur){ + closure_cursor *pCur = (closure_cursor *)cur; + closureClearCursor(pCur); + pCur->pVtab->nCursor--; + sqlite3_free(pCur); + return SQLITE_OK; +} + +/* +** Advance a cursor to its next row of output +*/ +static int closureNext(sqlite3_vtab_cursor *cur){ + closure_cursor *pCur = (closure_cursor*)cur; + pCur->pCurrent = closureAvlNext(pCur->pCurrent); + return SQLITE_OK; +} + +/* +** Allocate and insert a node +*/ +static int closureInsertNode( + closure_queue *pQueue, /* Add new node to this queue */ + closure_cursor *pCur, /* The cursor into which to add the node */ + sqlite3_int64 id, /* The node ID */ + int iGeneration /* The generation number for this node */ +){ + closure_avl *pNew = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->id = id; + pNew->iGeneration = iGeneration; + closureAvlInsert(&pCur->pClosure, pNew); + queuePush(pQueue, pNew); + return SQLITE_OK; +} + +/* +** Called to "rewind" a cursor back to the beginning so that +** it starts its output over again. Always called at least once +** prior to any closureColumn, closureRowid, or closureEof call. +** +** This routine actually computes the closure. +** +** See the comment at the beginning of closureBestIndex() for a +** description of the meaning of idxNum. The idxStr parameter is +** not used. +*/ +static int closureFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + closure_cursor *pCur = (closure_cursor *)pVtabCursor; + closure_vtab *pVtab = pCur->pVtab; + sqlite3_int64 iRoot; + int mxGen = 999999999; + char *zSql; + sqlite3_stmt *pStmt; + closure_avl *pAvl; + int rc = SQLITE_OK; + const char *zTableName = pVtab->zTableName; + const char *zIdColumn = pVtab->zIdColumn; + const char *zParentColumn = pVtab->zParentColumn; + closure_queue sQueue; + + (void)idxStr; /* Unused parameter */ + (void)argc; /* Unused parameter */ + closureClearCursor(pCur); + memset(&sQueue, 0, sizeof(sQueue)); + if( (idxNum & 1)==0 ){ + /* No root=$root in the WHERE clause. Return an empty set */ + return SQLITE_OK; + } + iRoot = sqlite3_value_int64(argv[0]); + if( (idxNum & 0x000f0)!=0 ){ + mxGen = sqlite3_value_int(argv[(idxNum>>4)&0x0f]); + if( (idxNum & 0x00002)!=0 ) mxGen--; + } + if( (idxNum & 0x00f00)!=0 ){ + zTableName = (const char*)sqlite3_value_text(argv[(idxNum>>8)&0x0f]); + pCur->zTableName = sqlite3_mprintf("%s", zTableName); + } + if( (idxNum & 0x0f000)!=0 ){ + zIdColumn = (const char*)sqlite3_value_text(argv[(idxNum>>12)&0x0f]); + pCur->zIdColumn = sqlite3_mprintf("%s", zIdColumn); + } + if( (idxNum & 0x0f0000)!=0 ){ + zParentColumn = (const char*)sqlite3_value_text(argv[(idxNum>>16)&0x0f]); + pCur->zParentColumn = sqlite3_mprintf("%s", zParentColumn); + } + + zSql = sqlite3_mprintf( + "SELECT \"%w\".\"%w\" FROM \"%w\" WHERE \"%w\".\"%w\"=?1", + zTableName, zIdColumn, zTableName, zTableName, zParentColumn); + if( zSql==0 ){ + return SQLITE_NOMEM; + }else{ + rc = sqlite3_prepare_v2(pVtab->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ){ + sqlite3_free(pVtab->base.zErrMsg); + pVtab->base.zErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(pVtab->db)); + return rc; + } + } + if( rc==SQLITE_OK ){ + rc = closureInsertNode(&sQueue, pCur, iRoot, 0); + } + while( (pAvl = queuePull(&sQueue))!=0 ){ + if( pAvl->iGeneration>=mxGen ) continue; + sqlite3_bind_int64(pStmt, 1, pAvl->id); + while( rc==SQLITE_OK && sqlite3_step(pStmt)==SQLITE_ROW ){ + if( sqlite3_column_type(pStmt,0)==SQLITE_INTEGER ){ + sqlite3_int64 iNew = sqlite3_column_int64(pStmt, 0); + if( closureAvlSearch(pCur->pClosure, iNew)==0 ){ + rc = closureInsertNode(&sQueue, pCur, iNew, pAvl->iGeneration+1); + } + } + } + sqlite3_reset(pStmt); + } + sqlite3_finalize(pStmt); + if( rc==SQLITE_OK ){ + pCur->pCurrent = closureAvlFirst(pCur->pClosure); + } + + return rc; +} + +/* +** Only the word and distance columns have values. All other columns +** return NULL +*/ +static int closureColumn(sqlite3_vtab_cursor *cur, sqlite3_context *ctx, int i){ + closure_cursor *pCur = (closure_cursor*)cur; + switch( i ){ + case CLOSURE_COL_ID: { + sqlite3_result_int64(ctx, pCur->pCurrent->id); + break; + } + case CLOSURE_COL_DEPTH: { + sqlite3_result_int(ctx, pCur->pCurrent->iGeneration); + break; + } + case CLOSURE_COL_ROOT: { + sqlite3_result_null(ctx); + break; + } + case CLOSURE_COL_TABLENAME: { + sqlite3_result_text(ctx, + pCur->zTableName ? pCur->zTableName : pCur->pVtab->zTableName, + -1, SQLITE_TRANSIENT); + break; + } + case CLOSURE_COL_IDCOLUMN: { + sqlite3_result_text(ctx, + pCur->zIdColumn ? pCur->zIdColumn : pCur->pVtab->zIdColumn, + -1, SQLITE_TRANSIENT); + break; + } + case CLOSURE_COL_PARENTCOLUMN: { + sqlite3_result_text(ctx, + pCur->zParentColumn ? pCur->zParentColumn : pCur->pVtab->zParentColumn, + -1, SQLITE_TRANSIENT); + break; + } + } + return SQLITE_OK; +} + +/* +** The rowid. For the closure table, this is the same as the "id" column. +*/ +static int closureRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + closure_cursor *pCur = (closure_cursor*)cur; + *pRowid = pCur->pCurrent->id; + return SQLITE_OK; +} + +/* +** EOF indicator +*/ +static int closureEof(sqlite3_vtab_cursor *cur){ + closure_cursor *pCur = (closure_cursor*)cur; + return pCur->pCurrent==0; +} + +/* +** Search for terms of these forms: +** +** (A) root = $root +** (B1) depth < $depth +** (B2) depth <= $depth +** (B3) depth = $depth +** (C) tablename = $tablename +** (D) idcolumn = $idcolumn +** (E) parentcolumn = $parentcolumn +** +** +** +** idxNum meaning +** ---------- ------------------------------------------------------ +** 0x00000001 Term of the form (A) found +** 0x00000002 The term of bit-2 is like (B1) +** 0x000000f0 Index in filter.argv[] of $depth. 0 if not used. +** 0x00000f00 Index in filter.argv[] of $tablename. 0 if not used. +** 0x0000f000 Index in filter.argv[] of $idcolumn. 0 if not used +** 0x000f0000 Index in filter.argv[] of $parentcolumn. 0 if not used. +** +** There must be a term of type (A). If there is not, then the index type +** is 0 and the query will return an empty set. +*/ +static int closureBestIndex( + sqlite3_vtab *pTab, /* The virtual table */ + sqlite3_index_info *pIdxInfo /* Information about the query */ +){ + int iPlan = 0; + int i; + int idx = 1; + const struct sqlite3_index_constraint *pConstraint; + closure_vtab *pVtab = (closure_vtab*)pTab; + double rCost = 10000000.0; + + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + if( (iPlan & 1)==0 + && pConstraint->iColumn==CLOSURE_COL_ROOT + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= 1; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + rCost /= 100.0; + } + if( (iPlan & 0x0000f0)==0 + && pConstraint->iColumn==CLOSURE_COL_DEPTH + && (pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT + || pConstraint->op==SQLITE_INDEX_CONSTRAINT_LE + || pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ) + ){ + iPlan |= idx<<4; + pIdxInfo->aConstraintUsage[i].argvIndex = ++idx; + if( pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT ) iPlan |= 0x000002; + rCost /= 5.0; + } + if( (iPlan & 0x000f00)==0 + && pConstraint->iColumn==CLOSURE_COL_TABLENAME + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= idx<<8; + pIdxInfo->aConstraintUsage[i].argvIndex = ++idx; + pIdxInfo->aConstraintUsage[i].omit = 1; + rCost /= 5.0; + } + if( (iPlan & 0x00f000)==0 + && pConstraint->iColumn==CLOSURE_COL_IDCOLUMN + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= idx<<12; + pIdxInfo->aConstraintUsage[i].argvIndex = ++idx; + pIdxInfo->aConstraintUsage[i].omit = 1; + } + if( (iPlan & 0x0f0000)==0 + && pConstraint->iColumn==CLOSURE_COL_PARENTCOLUMN + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= idx<<16; + pIdxInfo->aConstraintUsage[i].argvIndex = ++idx; + pIdxInfo->aConstraintUsage[i].omit = 1; + } + } + if( (pVtab->zTableName==0 && (iPlan & 0x000f00)==0) + || (pVtab->zIdColumn==0 && (iPlan & 0x00f000)==0) + || (pVtab->zParentColumn==0 && (iPlan & 0x0f0000)==0) + ){ + /* All of tablename, idcolumn, and parentcolumn must be specified + ** in either the CREATE VIRTUAL TABLE or in the WHERE clause constraints + ** or else the result is an empty set. */ + iPlan = 0; + } + if( (iPlan&1)==0 ){ + /* If there is no usable "root=?" term, then set the index-type to 0. + ** Also clear any argvIndex variables already set. This is necessary + ** to prevent the core from throwing an "xBestIndex malfunction error" + ** error (because the argvIndex values are not contiguously assigned + ** starting from 1). */ + rCost *= 1e30; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + pIdxInfo->aConstraintUsage[i].argvIndex = 0; + } + iPlan = 0; + } + pIdxInfo->idxNum = iPlan; + if( pIdxInfo->nOrderBy==1 + && pIdxInfo->aOrderBy[0].iColumn==CLOSURE_COL_ID + && pIdxInfo->aOrderBy[0].desc==0 + ){ + pIdxInfo->orderByConsumed = 1; + } + pIdxInfo->estimatedCost = rCost; + + return SQLITE_OK; +} + +/* +** A virtual table module that implements the "transitive_closure". +*/ +static sqlite3_module closureModule = { + 0, /* iVersion */ + closureConnect, /* xCreate */ + closureConnect, /* xConnect */ + closureBestIndex, /* xBestIndex */ + closureDisconnect, /* xDisconnect */ + closureDisconnect, /* xDestroy */ + closureOpen, /* xOpen - open a cursor */ + closureClose, /* xClose - close a cursor */ + closureFilter, /* xFilter - configure scan constraints */ + closureNext, /* xNext - advance a cursor */ + closureEof, /* xEof - check for end of scan */ + closureColumn, /* xColumn - read data */ + closureRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +/* +** Register the closure virtual table +*/ +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_closure_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "transitive_closure", &closureModule, 0); +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + return rc; +} diff --git a/ext/misc/completion.c b/ext/misc/completion.c new file mode 100644 index 0000000..d9e7b85 --- /dev/null +++ b/ext/misc/completion.c @@ -0,0 +1,501 @@ +/* +** 2017-07-10 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements an eponymous virtual table that returns suggested +** completions for a partial SQL input. +** +** Suggested usage: +** +** SELECT DISTINCT candidate COLLATE nocase +** FROM completion($prefix,$wholeline) +** ORDER BY 1; +** +** The two query parameters are optional. $prefix is the text of the +** current word being typed and that is to be completed. $wholeline is +** the complete input line, used for context. +** +** The raw completion() table might return the same candidate multiple +** times, for example if the same column name is used to two or more +** tables. And the candidates are returned in an arbitrary order. Hence, +** the DISTINCT and ORDER BY are recommended. +** +** This virtual table operates at the speed of human typing, and so there +** is no attempt to make it fast. Even a slow implementation will be much +** faster than any human can type. +** +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <ctype.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* completion_vtab is a subclass of sqlite3_vtab which will +** serve as the underlying representation of a completion virtual table +*/ +typedef struct completion_vtab completion_vtab; +struct completion_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this completion vtab */ +}; + +/* completion_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct completion_cursor completion_cursor; +struct completion_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this cursor */ + int nPrefix, nLine; /* Number of bytes in zPrefix and zLine */ + char *zPrefix; /* The prefix for the word we want to complete */ + char *zLine; /* The whole that we want to complete */ + const char *zCurrentRow; /* Current output row */ + int szRow; /* Length of the zCurrentRow string */ + sqlite3_stmt *pStmt; /* Current statement */ + sqlite3_int64 iRowid; /* The rowid */ + int ePhase; /* Current phase */ + int j; /* inter-phase counter */ +}; + +/* Values for ePhase: +*/ +#define COMPLETION_FIRST_PHASE 1 +#define COMPLETION_KEYWORDS 1 +#define COMPLETION_PRAGMAS 2 +#define COMPLETION_FUNCTIONS 3 +#define COMPLETION_COLLATIONS 4 +#define COMPLETION_INDEXES 5 +#define COMPLETION_TRIGGERS 6 +#define COMPLETION_DATABASES 7 +#define COMPLETION_TABLES 8 /* Also VIEWs and TRIGGERs */ +#define COMPLETION_COLUMNS 9 +#define COMPLETION_MODULES 10 +#define COMPLETION_EOF 11 + +/* +** The completionConnect() method is invoked to create a new +** completion_vtab that describes the completion virtual table. +** +** Think of this routine as the constructor for completion_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the completion_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against completion will look like. +*/ +static int completionConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + completion_vtab *pNew; + int rc; + + (void)(pAux); /* Unused parameter */ + (void)(argc); /* Unused parameter */ + (void)(argv); /* Unused parameter */ + (void)(pzErr); /* Unused parameter */ + +/* Column numbers */ +#define COMPLETION_COLUMN_CANDIDATE 0 /* Suggested completion of the input */ +#define COMPLETION_COLUMN_PREFIX 1 /* Prefix of the word to be completed */ +#define COMPLETION_COLUMN_WHOLELINE 2 /* Entire line seen so far */ +#define COMPLETION_COLUMN_PHASE 3 /* ePhase - used for debugging only */ + + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(" + " candidate TEXT," + " prefix TEXT HIDDEN," + " wholeline TEXT HIDDEN," + " phase INT HIDDEN" /* Used for debugging only */ + ")"); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + } + return rc; +} + +/* +** This method is the destructor for completion_cursor objects. +*/ +static int completionDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new completion_cursor object. +*/ +static int completionOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + completion_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->db = ((completion_vtab*)p)->db; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Reset the completion_cursor. +*/ +static void completionCursorReset(completion_cursor *pCur){ + sqlite3_free(pCur->zPrefix); pCur->zPrefix = 0; pCur->nPrefix = 0; + sqlite3_free(pCur->zLine); pCur->zLine = 0; pCur->nLine = 0; + sqlite3_finalize(pCur->pStmt); pCur->pStmt = 0; + pCur->j = 0; +} + +/* +** Destructor for a completion_cursor. +*/ +static int completionClose(sqlite3_vtab_cursor *cur){ + completionCursorReset((completion_cursor*)cur); + sqlite3_free(cur); + return SQLITE_OK; +} + +/* +** Advance a completion_cursor to its next row of output. +** +** The ->ePhase, ->j, and ->pStmt fields of the completion_cursor object +** record the current state of the scan. This routine sets ->zCurrentRow +** to the current row of output and then returns. If no more rows remain, +** then ->ePhase is set to COMPLETION_EOF which will signal the virtual +** table that has reached the end of its scan. +** +** The current implementation just lists potential identifiers and +** keywords and filters them by zPrefix. Future enhancements should +** take zLine into account to try to restrict the set of identifiers and +** keywords based on what would be legal at the current point of input. +*/ +static int completionNext(sqlite3_vtab_cursor *cur){ + completion_cursor *pCur = (completion_cursor*)cur; + int eNextPhase = 0; /* Next phase to try if current phase reaches end */ + int iCol = -1; /* If >=0, step pCur->pStmt and use the i-th column */ + pCur->iRowid++; + while( pCur->ePhase!=COMPLETION_EOF ){ + switch( pCur->ePhase ){ + case COMPLETION_KEYWORDS: { + if( pCur->j >= sqlite3_keyword_count() ){ + pCur->zCurrentRow = 0; + pCur->ePhase = COMPLETION_DATABASES; + }else{ + sqlite3_keyword_name(pCur->j++, &pCur->zCurrentRow, &pCur->szRow); + } + iCol = -1; + break; + } + case COMPLETION_DATABASES: { + if( pCur->pStmt==0 ){ + sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, + &pCur->pStmt, 0); + } + iCol = 1; + eNextPhase = COMPLETION_TABLES; + break; + } + case COMPLETION_TABLES: { + if( pCur->pStmt==0 ){ + sqlite3_stmt *pS2; + char *zSql = 0; + const char *zSep = ""; + sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, &pS2, 0); + while( sqlite3_step(pS2)==SQLITE_ROW ){ + const char *zDb = (const char*)sqlite3_column_text(pS2, 1); + zSql = sqlite3_mprintf( + "%z%s" + "SELECT name FROM \"%w\".sqlite_schema", + zSql, zSep, zDb + ); + if( zSql==0 ) return SQLITE_NOMEM; + zSep = " UNION "; + } + sqlite3_finalize(pS2); + sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0); + sqlite3_free(zSql); + } + iCol = 0; + eNextPhase = COMPLETION_COLUMNS; + break; + } + case COMPLETION_COLUMNS: { + if( pCur->pStmt==0 ){ + sqlite3_stmt *pS2; + char *zSql = 0; + const char *zSep = ""; + sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, &pS2, 0); + while( sqlite3_step(pS2)==SQLITE_ROW ){ + const char *zDb = (const char*)sqlite3_column_text(pS2, 1); + zSql = sqlite3_mprintf( + "%z%s" + "SELECT pti.name FROM \"%w\".sqlite_schema AS sm" + " JOIN pragma_table_info(sm.name,%Q) AS pti" + " WHERE sm.type='table'", + zSql, zSep, zDb, zDb + ); + if( zSql==0 ) return SQLITE_NOMEM; + zSep = " UNION "; + } + sqlite3_finalize(pS2); + sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pStmt, 0); + sqlite3_free(zSql); + } + iCol = 0; + eNextPhase = COMPLETION_EOF; + break; + } + } + if( iCol<0 ){ + /* This case is when the phase presets zCurrentRow */ + if( pCur->zCurrentRow==0 ) continue; + }else{ + if( sqlite3_step(pCur->pStmt)==SQLITE_ROW ){ + /* Extract the next row of content */ + pCur->zCurrentRow = (const char*)sqlite3_column_text(pCur->pStmt, iCol); + pCur->szRow = sqlite3_column_bytes(pCur->pStmt, iCol); + }else{ + /* When all rows are finished, advance to the next phase */ + sqlite3_finalize(pCur->pStmt); + pCur->pStmt = 0; + pCur->ePhase = eNextPhase; + continue; + } + } + if( pCur->nPrefix==0 ) break; + if( pCur->nPrefix<=pCur->szRow + && sqlite3_strnicmp(pCur->zPrefix, pCur->zCurrentRow, pCur->nPrefix)==0 + ){ + break; + } + } + + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the completion_cursor +** is currently pointing. +*/ +static int completionColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + completion_cursor *pCur = (completion_cursor*)cur; + switch( i ){ + case COMPLETION_COLUMN_CANDIDATE: { + sqlite3_result_text(ctx, pCur->zCurrentRow, pCur->szRow,SQLITE_TRANSIENT); + break; + } + case COMPLETION_COLUMN_PREFIX: { + sqlite3_result_text(ctx, pCur->zPrefix, -1, SQLITE_TRANSIENT); + break; + } + case COMPLETION_COLUMN_WHOLELINE: { + sqlite3_result_text(ctx, pCur->zLine, -1, SQLITE_TRANSIENT); + break; + } + case COMPLETION_COLUMN_PHASE: { + sqlite3_result_int(ctx, pCur->ePhase); + break; + } + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int completionRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + completion_cursor *pCur = (completion_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int completionEof(sqlite3_vtab_cursor *cur){ + completion_cursor *pCur = (completion_cursor*)cur; + return pCur->ePhase >= COMPLETION_EOF; +} + +/* +** This method is called to "rewind" the completion_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to completionColumn() or completionRowid() or +** completionEof(). +*/ +static int completionFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + completion_cursor *pCur = (completion_cursor *)pVtabCursor; + int iArg = 0; + (void)(idxStr); /* Unused parameter */ + (void)(argc); /* Unused parameter */ + completionCursorReset(pCur); + if( idxNum & 1 ){ + pCur->nPrefix = sqlite3_value_bytes(argv[iArg]); + if( pCur->nPrefix>0 ){ + pCur->zPrefix = sqlite3_mprintf("%s", sqlite3_value_text(argv[iArg])); + if( pCur->zPrefix==0 ) return SQLITE_NOMEM; + } + iArg = 1; + } + if( idxNum & 2 ){ + pCur->nLine = sqlite3_value_bytes(argv[iArg]); + if( pCur->nLine>0 ){ + pCur->zLine = sqlite3_mprintf("%s", sqlite3_value_text(argv[iArg])); + if( pCur->zLine==0 ) return SQLITE_NOMEM; + } + } + if( pCur->zLine!=0 && pCur->zPrefix==0 ){ + int i = pCur->nLine; + while( i>0 && (isalnum(pCur->zLine[i-1]) || pCur->zLine[i-1]=='_') ){ + i--; + } + pCur->nPrefix = pCur->nLine - i; + if( pCur->nPrefix>0 ){ + pCur->zPrefix = sqlite3_mprintf("%.*s", pCur->nPrefix, pCur->zLine + i); + if( pCur->zPrefix==0 ) return SQLITE_NOMEM; + } + } + pCur->iRowid = 0; + pCur->ePhase = COMPLETION_FIRST_PHASE; + return completionNext(pVtabCursor); +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the completion virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +** +** There are two hidden parameters that act as arguments to the table-valued +** function: "prefix" and "wholeline". Bit 0 of idxNum is set if "prefix" +** is available and bit 1 is set if "wholeline" is available. +*/ +static int completionBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; /* Loop over constraints */ + int idxNum = 0; /* The query plan bitmask */ + int prefixIdx = -1; /* Index of the start= constraint, or -1 if none */ + int wholelineIdx = -1; /* Index of the stop= constraint, or -1 if none */ + int nArg = 0; /* Number of arguments that completeFilter() expects */ + const struct sqlite3_index_constraint *pConstraint; + + (void)(tab); /* Unused parameter */ + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + if( pConstraint->op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue; + switch( pConstraint->iColumn ){ + case COMPLETION_COLUMN_PREFIX: + prefixIdx = i; + idxNum |= 1; + break; + case COMPLETION_COLUMN_WHOLELINE: + wholelineIdx = i; + idxNum |= 2; + break; + } + } + if( prefixIdx>=0 ){ + pIdxInfo->aConstraintUsage[prefixIdx].argvIndex = ++nArg; + pIdxInfo->aConstraintUsage[prefixIdx].omit = 1; + } + if( wholelineIdx>=0 ){ + pIdxInfo->aConstraintUsage[wholelineIdx].argvIndex = ++nArg; + pIdxInfo->aConstraintUsage[wholelineIdx].omit = 1; + } + pIdxInfo->idxNum = idxNum; + pIdxInfo->estimatedCost = (double)5000 - 1000*nArg; + pIdxInfo->estimatedRows = 500 - 100*nArg; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** completion virtual table. +*/ +static sqlite3_module completionModule = { + 0, /* iVersion */ + 0, /* xCreate */ + completionConnect, /* xConnect */ + completionBestIndex, /* xBestIndex */ + completionDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + completionOpen, /* xOpen - open a cursor */ + completionClose, /* xClose - close a cursor */ + completionFilter, /* xFilter - configure scan constraints */ + completionNext, /* xNext - advance a cursor */ + completionEof, /* xEof - check for end of scan */ + completionColumn, /* xColumn - read data */ + completionRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +int sqlite3CompletionVtabInit(sqlite3 *db){ + int rc = SQLITE_OK; +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "completion", &completionModule, 0); +#endif + return rc; +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_completion_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)(pzErrMsg); /* Unused parameter */ +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3CompletionVtabInit(db); +#endif + return rc; +} diff --git a/ext/misc/compress.c b/ext/misc/compress.c new file mode 100644 index 0000000..2e7a316 --- /dev/null +++ b/ext/misc/compress.c @@ -0,0 +1,131 @@ +/* +** 2014-06-13 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements SQL compression functions +** compress() and uncompress() using ZLIB. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <zlib.h> + +/* +** Implementation of the "compress(X)" SQL function. The input X is +** compressed using zLib and the output is returned. +** +** The output is a BLOB that begins with a variable-length integer that +** is the input size in bytes (the size of X before compression). The +** variable-length integer is implemented as 1 to 5 bytes. There are +** seven bits per integer stored in the lower seven bits of each byte. +** More significant bits occur first. The most significant bit (0x80) +** is a flag to indicate the end of the integer. +** +** This function, SQLAR, and ZIP all use the same "deflate" compression +** algorithm, but each is subtly different: +** +** * ZIP uses raw deflate. +** +** * SQLAR uses the "zlib format" which is raw deflate with a two-byte +** algorithm-identification header and a four-byte checksum at the end. +** +** * This utility uses the "zlib format" like SQLAR, but adds the variable- +** length integer uncompressed size value at the beginning. +** +** This function might be extended in the future to support compression +** formats other than deflate, by providing a different algorithm-id +** mark following the variable-length integer size parameter. +*/ +static void compressFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *pIn; + unsigned char *pOut; + unsigned int nIn; + unsigned long int nOut; + unsigned char x[8]; + int rc; + int i, j; + + pIn = sqlite3_value_blob(argv[0]); + nIn = sqlite3_value_bytes(argv[0]); + nOut = 13 + nIn + (nIn+999)/1000; + pOut = sqlite3_malloc( nOut+5 ); + for(i=4; i>=0; i--){ + x[i] = (nIn >> (7*(4-i)))&0x7f; + } + for(i=0; i<4 && x[i]==0; i++){} + for(j=0; i<=4; i++, j++) pOut[j] = x[i]; + pOut[j-1] |= 0x80; + rc = compress(&pOut[j], &nOut, pIn, nIn); + if( rc==Z_OK ){ + sqlite3_result_blob(context, pOut, nOut+j, sqlite3_free); + }else{ + sqlite3_free(pOut); + } +} + +/* +** Implementation of the "uncompress(X)" SQL function. The argument X +** is a blob which was obtained from compress(Y). The output will be +** the value Y. +*/ +static void uncompressFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *pIn; + unsigned char *pOut; + unsigned int nIn; + unsigned long int nOut; + int rc; + int i; + + pIn = sqlite3_value_blob(argv[0]); + nIn = sqlite3_value_bytes(argv[0]); + nOut = 0; + for(i=0; i<nIn && i<5; i++){ + nOut = (nOut<<7) | (pIn[i]&0x7f); + if( (pIn[i]&0x80)!=0 ){ i++; break; } + } + pOut = sqlite3_malloc( nOut+1 ); + rc = uncompress(pOut, &nOut, &pIn[i], nIn-i); + if( rc==Z_OK ){ + sqlite3_result_blob(context, pOut, nOut, sqlite3_free); + }else{ + sqlite3_free(pOut); + } +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_compress_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "compress", 1, + SQLITE_UTF8 | SQLITE_INNOCUOUS, + 0, compressFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "uncompress", 1, + SQLITE_UTF8 | SQLITE_INNOCUOUS | SQLITE_DETERMINISTIC, + 0, uncompressFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/csv.c b/ext/misc/csv.c new file mode 100644 index 0000000..870a0cf --- /dev/null +++ b/ext/misc/csv.c @@ -0,0 +1,964 @@ +/* +** 2016-05-28 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains the implementation of an SQLite virtual table for +** reading CSV files. +** +** Usage: +** +** .load ./csv +** CREATE VIRTUAL TABLE temp.csv USING csv(filename=FILENAME); +** SELECT * FROM csv; +** +** The columns are named "c1", "c2", "c3", ... by default. Or the +** application can define its own CREATE TABLE statement using the +** schema= parameter, like this: +** +** CREATE VIRTUAL TABLE temp.csv2 USING csv( +** filename = "../http.log", +** schema = "CREATE TABLE x(date,ipaddr,url,referrer,userAgent)" +** ); +** +** Instead of specifying a file, the text of the CSV can be loaded using +** the data= parameter. +** +** If the columns=N parameter is supplied, then the CSV file is assumed to have +** N columns. If both the columns= and schema= parameters are omitted, then +** the number and names of the columns is determined by the first line of +** the CSV input. +** +** Some extra debugging features (used for testing virtual tables) are available +** if this module is compiled with -DSQLITE_TEST. +*/ +#include <sqlite3ext.h> +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <stdlib.h> +#include <assert.h> +#include <stdarg.h> +#include <ctype.h> +#include <stdio.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** A macro to hint to the compiler that a function should not be +** inlined. +*/ +#if defined(__GNUC__) +# define CSV_NOINLINE __attribute__((noinline)) +#elif defined(_MSC_VER) && _MSC_VER>=1310 +# define CSV_NOINLINE __declspec(noinline) +#else +# define CSV_NOINLINE +#endif + + +/* Max size of the error message in a CsvReader */ +#define CSV_MXERR 200 + +/* Size of the CsvReader input buffer */ +#define CSV_INBUFSZ 1024 + +/* A context object used when read a CSV file. */ +typedef struct CsvReader CsvReader; +struct CsvReader { + FILE *in; /* Read the CSV text from this input stream */ + char *z; /* Accumulated text for a field */ + int n; /* Number of bytes in z */ + int nAlloc; /* Space allocated for z[] */ + int nLine; /* Current line number */ + int bNotFirst; /* True if prior text has been seen */ + int cTerm; /* Character that terminated the most recent field */ + size_t iIn; /* Next unread character in the input buffer */ + size_t nIn; /* Number of characters in the input buffer */ + char *zIn; /* The input buffer */ + char zErr[CSV_MXERR]; /* Error message */ +}; + +/* Initialize a CsvReader object */ +static void csv_reader_init(CsvReader *p){ + p->in = 0; + p->z = 0; + p->n = 0; + p->nAlloc = 0; + p->nLine = 0; + p->bNotFirst = 0; + p->nIn = 0; + p->zIn = 0; + p->zErr[0] = 0; +} + +/* Close and reset a CsvReader object */ +static void csv_reader_reset(CsvReader *p){ + if( p->in ){ + fclose(p->in); + sqlite3_free(p->zIn); + } + sqlite3_free(p->z); + csv_reader_init(p); +} + +/* Report an error on a CsvReader */ +static void csv_errmsg(CsvReader *p, const char *zFormat, ...){ + va_list ap; + va_start(ap, zFormat); + sqlite3_vsnprintf(CSV_MXERR, p->zErr, zFormat, ap); + va_end(ap); +} + +/* Open the file associated with a CsvReader +** Return the number of errors. +*/ +static int csv_reader_open( + CsvReader *p, /* The reader to open */ + const char *zFilename, /* Read from this filename */ + const char *zData /* ... or use this data */ +){ + if( zFilename ){ + p->zIn = sqlite3_malloc( CSV_INBUFSZ ); + if( p->zIn==0 ){ + csv_errmsg(p, "out of memory"); + return 1; + } + p->in = fopen(zFilename, "rb"); + if( p->in==0 ){ + sqlite3_free(p->zIn); + csv_reader_reset(p); + csv_errmsg(p, "cannot open '%s' for reading", zFilename); + return 1; + } + }else{ + assert( p->in==0 ); + p->zIn = (char*)zData; + p->nIn = strlen(zData); + } + return 0; +} + +/* The input buffer has overflowed. Refill the input buffer, then +** return the next character +*/ +static CSV_NOINLINE int csv_getc_refill(CsvReader *p){ + size_t got; + + assert( p->iIn>=p->nIn ); /* Only called on an empty input buffer */ + assert( p->in!=0 ); /* Only called if reading froma file */ + + got = fread(p->zIn, 1, CSV_INBUFSZ, p->in); + if( got==0 ) return EOF; + p->nIn = got; + p->iIn = 1; + return p->zIn[0]; +} + +/* Return the next character of input. Return EOF at end of input. */ +static int csv_getc(CsvReader *p){ + if( p->iIn >= p->nIn ){ + if( p->in!=0 ) return csv_getc_refill(p); + return EOF; + } + return ((unsigned char*)p->zIn)[p->iIn++]; +} + +/* Increase the size of p->z and append character c to the end. +** Return 0 on success and non-zero if there is an OOM error */ +static CSV_NOINLINE int csv_resize_and_append(CsvReader *p, char c){ + char *zNew; + int nNew = p->nAlloc*2 + 100; + zNew = sqlite3_realloc64(p->z, nNew); + if( zNew ){ + p->z = zNew; + p->nAlloc = nNew; + p->z[p->n++] = c; + return 0; + }else{ + csv_errmsg(p, "out of memory"); + return 1; + } +} + +/* Append a single character to the CsvReader.z[] array. +** Return 0 on success and non-zero if there is an OOM error */ +static int csv_append(CsvReader *p, char c){ + if( p->n>=p->nAlloc-1 ) return csv_resize_and_append(p, c); + p->z[p->n++] = c; + return 0; +} + +/* Read a single field of CSV text. Compatible with rfc4180 and extended +** with the option of having a separator other than ",". +** +** + Input comes from p->in. +** + Store results in p->z of length p->n. Space to hold p->z comes +** from sqlite3_malloc64(). +** + Keep track of the line number in p->nLine. +** + Store the character that terminates the field in p->cTerm. Store +** EOF on end-of-file. +** +** Return 0 at EOF or on OOM. On EOF, the p->cTerm character will have +** been set to EOF. +*/ +static char *csv_read_one_field(CsvReader *p){ + int c; + p->n = 0; + c = csv_getc(p); + if( c==EOF ){ + p->cTerm = EOF; + return 0; + } + if( c=='"' ){ + int pc, ppc; + int startLine = p->nLine; + pc = ppc = 0; + while( 1 ){ + c = csv_getc(p); + if( c<='"' || pc=='"' ){ + if( c=='\n' ) p->nLine++; + if( c=='"' ){ + if( pc=='"' ){ + pc = 0; + continue; + } + } + if( (c==',' && pc=='"') + || (c=='\n' && pc=='"') + || (c=='\n' && pc=='\r' && ppc=='"') + || (c==EOF && pc=='"') + ){ + do{ p->n--; }while( p->z[p->n]!='"' ); + p->cTerm = (char)c; + break; + } + if( pc=='"' && c!='\r' ){ + csv_errmsg(p, "line %d: unescaped %c character", p->nLine, '"'); + break; + } + if( c==EOF ){ + csv_errmsg(p, "line %d: unterminated %c-quoted field\n", + startLine, '"'); + p->cTerm = (char)c; + break; + } + } + if( csv_append(p, (char)c) ) return 0; + ppc = pc; + pc = c; + } + }else{ + /* If this is the first field being parsed and it begins with the + ** UTF-8 BOM (0xEF BB BF) then skip the BOM */ + if( (c&0xff)==0xef && p->bNotFirst==0 ){ + csv_append(p, (char)c); + c = csv_getc(p); + if( (c&0xff)==0xbb ){ + csv_append(p, (char)c); + c = csv_getc(p); + if( (c&0xff)==0xbf ){ + p->bNotFirst = 1; + p->n = 0; + return csv_read_one_field(p); + } + } + } + while( c>',' || (c!=EOF && c!=',' && c!='\n') ){ + if( csv_append(p, (char)c) ) return 0; + c = csv_getc(p); + } + if( c=='\n' ){ + p->nLine++; + if( p->n>0 && p->z[p->n-1]=='\r' ) p->n--; + } + p->cTerm = (char)c; + } + assert( p->z==0 || p->n<p->nAlloc ); + if( p->z ) p->z[p->n] = 0; + p->bNotFirst = 1; + return p->z; +} + + +/* Forward references to the various virtual table methods implemented +** in this file. */ +static int csvtabCreate(sqlite3*, void*, int, const char*const*, + sqlite3_vtab**,char**); +static int csvtabConnect(sqlite3*, void*, int, const char*const*, + sqlite3_vtab**,char**); +static int csvtabBestIndex(sqlite3_vtab*,sqlite3_index_info*); +static int csvtabDisconnect(sqlite3_vtab*); +static int csvtabOpen(sqlite3_vtab*, sqlite3_vtab_cursor**); +static int csvtabClose(sqlite3_vtab_cursor*); +static int csvtabFilter(sqlite3_vtab_cursor*, int idxNum, const char *idxStr, + int argc, sqlite3_value **argv); +static int csvtabNext(sqlite3_vtab_cursor*); +static int csvtabEof(sqlite3_vtab_cursor*); +static int csvtabColumn(sqlite3_vtab_cursor*,sqlite3_context*,int); +static int csvtabRowid(sqlite3_vtab_cursor*,sqlite3_int64*); + +/* An instance of the CSV virtual table */ +typedef struct CsvTable { + sqlite3_vtab base; /* Base class. Must be first */ + char *zFilename; /* Name of the CSV file */ + char *zData; /* Raw CSV data in lieu of zFilename */ + long iStart; /* Offset to start of data in zFilename */ + int nCol; /* Number of columns in the CSV file */ + unsigned int tstFlags; /* Bit values used for testing */ +} CsvTable; + +/* Allowed values for tstFlags */ +#define CSVTEST_FIDX 0x0001 /* Pretend that constrained searchs cost less*/ + +/* A cursor for the CSV virtual table */ +typedef struct CsvCursor { + sqlite3_vtab_cursor base; /* Base class. Must be first */ + CsvReader rdr; /* The CsvReader object */ + char **azVal; /* Value of the current row */ + int *aLen; /* Length of each entry */ + sqlite3_int64 iRowid; /* The current rowid. Negative for EOF */ +} CsvCursor; + +/* Transfer error message text from a reader into a CsvTable */ +static void csv_xfer_error(CsvTable *pTab, CsvReader *pRdr){ + sqlite3_free(pTab->base.zErrMsg); + pTab->base.zErrMsg = sqlite3_mprintf("%s", pRdr->zErr); +} + +/* +** This method is the destructor fo a CsvTable object. +*/ +static int csvtabDisconnect(sqlite3_vtab *pVtab){ + CsvTable *p = (CsvTable*)pVtab; + sqlite3_free(p->zFilename); + sqlite3_free(p->zData); + sqlite3_free(p); + return SQLITE_OK; +} + +/* Skip leading whitespace. Return a pointer to the first non-whitespace +** character, or to the zero terminator if the string has only whitespace */ +static const char *csv_skip_whitespace(const char *z){ + while( isspace((unsigned char)z[0]) ) z++; + return z; +} + +/* Remove trailing whitespace from the end of string z[] */ +static void csv_trim_whitespace(char *z){ + size_t n = strlen(z); + while( n>0 && isspace((unsigned char)z[n]) ) n--; + z[n] = 0; +} + +/* Dequote the string */ +static void csv_dequote(char *z){ + int j; + char cQuote = z[0]; + size_t i, n; + + if( cQuote!='\'' && cQuote!='"' ) return; + n = strlen(z); + if( n<2 || z[n-1]!=z[0] ) return; + for(i=1, j=0; i<n-1; i++){ + if( z[i]==cQuote && z[i+1]==cQuote ) i++; + z[j++] = z[i]; + } + z[j] = 0; +} + +/* Check to see if the string is of the form: "TAG = VALUE" with optional +** whitespace before and around tokens. If it is, return a pointer to the +** first character of VALUE. If it is not, return NULL. +*/ +static const char *csv_parameter(const char *zTag, int nTag, const char *z){ + z = csv_skip_whitespace(z); + if( strncmp(zTag, z, nTag)!=0 ) return 0; + z = csv_skip_whitespace(z+nTag); + if( z[0]!='=' ) return 0; + return csv_skip_whitespace(z+1); +} + +/* Decode a parameter that requires a dequoted string. +** +** Return 1 if the parameter is seen, or 0 if not. 1 is returned +** even if there is an error. If an error occurs, then an error message +** is left in p->zErr. If there are no errors, p->zErr[0]==0. +*/ +static int csv_string_parameter( + CsvReader *p, /* Leave the error message here, if there is one */ + const char *zParam, /* Parameter we are checking for */ + const char *zArg, /* Raw text of the virtual table argment */ + char **pzVal /* Write the dequoted string value here */ +){ + const char *zValue; + zValue = csv_parameter(zParam,(int)strlen(zParam),zArg); + if( zValue==0 ) return 0; + p->zErr[0] = 0; + if( *pzVal ){ + csv_errmsg(p, "more than one '%s' parameter", zParam); + return 1; + } + *pzVal = sqlite3_mprintf("%s", zValue); + if( *pzVal==0 ){ + csv_errmsg(p, "out of memory"); + return 1; + } + csv_trim_whitespace(*pzVal); + csv_dequote(*pzVal); + return 1; +} + + +/* Return 0 if the argument is false and 1 if it is true. Return -1 if +** we cannot really tell. +*/ +static int csv_boolean(const char *z){ + if( sqlite3_stricmp("yes",z)==0 + || sqlite3_stricmp("on",z)==0 + || sqlite3_stricmp("true",z)==0 + || (z[0]=='1' && z[1]==0) + ){ + return 1; + } + if( sqlite3_stricmp("no",z)==0 + || sqlite3_stricmp("off",z)==0 + || sqlite3_stricmp("false",z)==0 + || (z[0]=='0' && z[1]==0) + ){ + return 0; + } + return -1; +} + +/* Check to see if the string is of the form: "TAG = BOOLEAN" or just "TAG". +** If it is, set *pValue to be the value of the boolean ("true" if there is +** not "= BOOLEAN" component) and return non-zero. If the input string +** does not begin with TAG, return zero. +*/ +static int csv_boolean_parameter( + const char *zTag, /* Tag we are looking for */ + int nTag, /* Size of the tag in bytes */ + const char *z, /* Input parameter */ + int *pValue /* Write boolean value here */ +){ + int b; + z = csv_skip_whitespace(z); + if( strncmp(zTag, z, nTag)!=0 ) return 0; + z = csv_skip_whitespace(z + nTag); + if( z[0]==0 ){ + *pValue = 1; + return 1; + } + if( z[0]!='=' ) return 0; + z = csv_skip_whitespace(z+1); + b = csv_boolean(z); + if( b>=0 ){ + *pValue = b; + return 1; + } + return 0; +} + +/* +** Parameters: +** filename=FILENAME Name of file containing CSV content +** data=TEXT Direct CSV content. +** schema=SCHEMA Alternative CSV schema. +** header=YES|NO First row of CSV defines the names of +** columns if "yes". Default "no". +** columns=N Assume the CSV file contains N columns. +** +** Only available if compiled with SQLITE_TEST: +** +** testflags=N Bitmask of test flags. Optional +** +** If schema= is omitted, then the columns are named "c0", "c1", "c2", +** and so forth. If columns=N is omitted, then the file is opened and +** the number of columns in the first row is counted to determine the +** column count. If header=YES, then the first row is skipped. +*/ +static int csvtabConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + CsvTable *pNew = 0; /* The CsvTable object to construct */ + int bHeader = -1; /* header= flags. -1 means not seen yet */ + int rc = SQLITE_OK; /* Result code from this routine */ + int i, j; /* Loop counters */ +#ifdef SQLITE_TEST + int tstFlags = 0; /* Value for testflags=N parameter */ +#endif + int b; /* Value of a boolean parameter */ + int nCol = -99; /* Value of the columns= parameter */ + CsvReader sRdr; /* A CSV file reader used to store an error + ** message and/or to count the number of columns */ + static const char *azParam[] = { + "filename", "data", "schema", + }; + char *azPValue[3]; /* Parameter values */ +# define CSV_FILENAME (azPValue[0]) +# define CSV_DATA (azPValue[1]) +# define CSV_SCHEMA (azPValue[2]) + + + assert( sizeof(azPValue)==sizeof(azParam) ); + memset(&sRdr, 0, sizeof(sRdr)); + memset(azPValue, 0, sizeof(azPValue)); + for(i=3; i<argc; i++){ + const char *z = argv[i]; + const char *zValue; + for(j=0; j<sizeof(azParam)/sizeof(azParam[0]); j++){ + if( csv_string_parameter(&sRdr, azParam[j], z, &azPValue[j]) ) break; + } + if( j<sizeof(azParam)/sizeof(azParam[0]) ){ + if( sRdr.zErr[0] ) goto csvtab_connect_error; + }else + if( csv_boolean_parameter("header",6,z,&b) ){ + if( bHeader>=0 ){ + csv_errmsg(&sRdr, "more than one 'header' parameter"); + goto csvtab_connect_error; + } + bHeader = b; + }else +#ifdef SQLITE_TEST + if( (zValue = csv_parameter("testflags",9,z))!=0 ){ + tstFlags = (unsigned int)atoi(zValue); + }else +#endif + if( (zValue = csv_parameter("columns",7,z))!=0 ){ + if( nCol>0 ){ + csv_errmsg(&sRdr, "more than one 'columns' parameter"); + goto csvtab_connect_error; + } + nCol = atoi(zValue); + if( nCol<=0 ){ + csv_errmsg(&sRdr, "column= value must be positive"); + goto csvtab_connect_error; + } + }else + { + csv_errmsg(&sRdr, "bad parameter: '%s'", z); + goto csvtab_connect_error; + } + } + if( (CSV_FILENAME==0)==(CSV_DATA==0) ){ + csv_errmsg(&sRdr, "must specify either filename= or data= but not both"); + goto csvtab_connect_error; + } + + if( (nCol<=0 || bHeader==1) + && csv_reader_open(&sRdr, CSV_FILENAME, CSV_DATA) + ){ + goto csvtab_connect_error; + } + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) goto csvtab_connect_oom; + memset(pNew, 0, sizeof(*pNew)); + if( CSV_SCHEMA==0 ){ + sqlite3_str *pStr = sqlite3_str_new(0); + char *zSep = ""; + int iCol = 0; + sqlite3_str_appendf(pStr, "CREATE TABLE x("); + if( nCol<0 && bHeader<1 ){ + nCol = 0; + do{ + csv_read_one_field(&sRdr); + nCol++; + }while( sRdr.cTerm==',' ); + } + if( nCol>0 && bHeader<1 ){ + for(iCol=0; iCol<nCol; iCol++){ + sqlite3_str_appendf(pStr, "%sc%d TEXT", zSep, iCol); + zSep = ","; + } + }else{ + do{ + char *z = csv_read_one_field(&sRdr); + if( (nCol>0 && iCol<nCol) || (nCol<0 && bHeader) ){ + sqlite3_str_appendf(pStr,"%s\"%w\" TEXT", zSep, z); + zSep = ","; + iCol++; + } + }while( sRdr.cTerm==',' ); + if( nCol<0 ){ + nCol = iCol; + }else{ + while( iCol<nCol ){ + sqlite3_str_appendf(pStr,"%sc%d TEXT", zSep, ++iCol); + zSep = ","; + } + } + } + pNew->nCol = nCol; + sqlite3_str_appendf(pStr, ")"); + CSV_SCHEMA = sqlite3_str_finish(pStr); + if( CSV_SCHEMA==0 ) goto csvtab_connect_oom; + }else if( nCol<0 ){ + do{ + csv_read_one_field(&sRdr); + pNew->nCol++; + }while( sRdr.cTerm==',' ); + }else{ + pNew->nCol = nCol; + } + pNew->zFilename = CSV_FILENAME; CSV_FILENAME = 0; + pNew->zData = CSV_DATA; CSV_DATA = 0; +#ifdef SQLITE_TEST + pNew->tstFlags = tstFlags; +#endif + if( bHeader!=1 ){ + pNew->iStart = 0; + }else if( pNew->zData ){ + pNew->iStart = (int)sRdr.iIn; + }else{ + pNew->iStart = (int)(ftell(sRdr.in) - sRdr.nIn + sRdr.iIn); + } + csv_reader_reset(&sRdr); + rc = sqlite3_declare_vtab(db, CSV_SCHEMA); + if( rc ){ + csv_errmsg(&sRdr, "bad schema: '%s' - %s", CSV_SCHEMA, sqlite3_errmsg(db)); + goto csvtab_connect_error; + } + for(i=0; i<sizeof(azPValue)/sizeof(azPValue[0]); i++){ + sqlite3_free(azPValue[i]); + } + /* Rationale for DIRECTONLY: + ** An attacker who controls a database schema could use this vtab + ** to exfiltrate sensitive data from other files in the filesystem. + ** And, recommended practice is to put all CSV virtual tables in the + ** TEMP namespace, so they should still be usable from within TEMP + ** views, so there shouldn't be a serious loss of functionality by + ** prohibiting the use of this vtab from persistent triggers and views. + */ + sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY); + return SQLITE_OK; + +csvtab_connect_oom: + rc = SQLITE_NOMEM; + csv_errmsg(&sRdr, "out of memory"); + +csvtab_connect_error: + if( pNew ) csvtabDisconnect(&pNew->base); + for(i=0; i<sizeof(azPValue)/sizeof(azPValue[0]); i++){ + sqlite3_free(azPValue[i]); + } + if( sRdr.zErr[0] ){ + sqlite3_free(*pzErr); + *pzErr = sqlite3_mprintf("%s", sRdr.zErr); + } + csv_reader_reset(&sRdr); + if( rc==SQLITE_OK ) rc = SQLITE_ERROR; + return rc; +} + +/* +** Reset the current row content held by a CsvCursor. +*/ +static void csvtabCursorRowReset(CsvCursor *pCur){ + CsvTable *pTab = (CsvTable*)pCur->base.pVtab; + int i; + for(i=0; i<pTab->nCol; i++){ + sqlite3_free(pCur->azVal[i]); + pCur->azVal[i] = 0; + pCur->aLen[i] = 0; + } +} + +/* +** The xConnect and xCreate methods do the same thing, but they must be +** different so that the virtual table is not an eponymous virtual table. +*/ +static int csvtabCreate( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + return csvtabConnect(db, pAux, argc, argv, ppVtab, pzErr); +} + +/* +** Destructor for a CsvCursor. +*/ +static int csvtabClose(sqlite3_vtab_cursor *cur){ + CsvCursor *pCur = (CsvCursor*)cur; + csvtabCursorRowReset(pCur); + csv_reader_reset(&pCur->rdr); + sqlite3_free(cur); + return SQLITE_OK; +} + +/* +** Constructor for a new CsvTable cursor object. +*/ +static int csvtabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + CsvTable *pTab = (CsvTable*)p; + CsvCursor *pCur; + size_t nByte; + nByte = sizeof(*pCur) + (sizeof(char*)+sizeof(int))*pTab->nCol; + pCur = sqlite3_malloc64( nByte ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, nByte); + pCur->azVal = (char**)&pCur[1]; + pCur->aLen = (int*)&pCur->azVal[pTab->nCol]; + *ppCursor = &pCur->base; + if( csv_reader_open(&pCur->rdr, pTab->zFilename, pTab->zData) ){ + csv_xfer_error(pTab, &pCur->rdr); + return SQLITE_ERROR; + } + return SQLITE_OK; +} + + +/* +** Advance a CsvCursor to its next row of input. +** Set the EOF marker if we reach the end of input. +*/ +static int csvtabNext(sqlite3_vtab_cursor *cur){ + CsvCursor *pCur = (CsvCursor*)cur; + CsvTable *pTab = (CsvTable*)cur->pVtab; + int i = 0; + char *z; + do{ + z = csv_read_one_field(&pCur->rdr); + if( z==0 ){ + break; + } + if( i<pTab->nCol ){ + if( pCur->aLen[i] < pCur->rdr.n+1 ){ + char *zNew = sqlite3_realloc64(pCur->azVal[i], pCur->rdr.n+1); + if( zNew==0 ){ + csv_errmsg(&pCur->rdr, "out of memory"); + csv_xfer_error(pTab, &pCur->rdr); + break; + } + pCur->azVal[i] = zNew; + pCur->aLen[i] = pCur->rdr.n+1; + } + memcpy(pCur->azVal[i], z, pCur->rdr.n+1); + i++; + } + }while( pCur->rdr.cTerm==',' ); + if( z==0 && i==0 ){ + pCur->iRowid = -1; + }else{ + pCur->iRowid++; + while( i<pTab->nCol ){ + sqlite3_free(pCur->azVal[i]); + pCur->azVal[i] = 0; + pCur->aLen[i] = 0; + i++; + } + } + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the CsvCursor +** is currently pointing. +*/ +static int csvtabColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + CsvCursor *pCur = (CsvCursor*)cur; + CsvTable *pTab = (CsvTable*)cur->pVtab; + if( i>=0 && i<pTab->nCol && pCur->azVal[i]!=0 ){ + sqlite3_result_text(ctx, pCur->azVal[i], -1, SQLITE_TRANSIENT); + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. +*/ +static int csvtabRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + CsvCursor *pCur = (CsvCursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int csvtabEof(sqlite3_vtab_cursor *cur){ + CsvCursor *pCur = (CsvCursor*)cur; + return pCur->iRowid<0; +} + +/* +** Only a full table scan is supported. So xFilter simply rewinds to +** the beginning. +*/ +static int csvtabFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + CsvCursor *pCur = (CsvCursor*)pVtabCursor; + CsvTable *pTab = (CsvTable*)pVtabCursor->pVtab; + pCur->iRowid = 0; + + /* Ensure the field buffer is always allocated. Otherwise, if the + ** first field is zero bytes in size, this may be mistaken for an OOM + ** error in csvtabNext(). */ + if( csv_append(&pCur->rdr, 0) ) return SQLITE_NOMEM; + + if( pCur->rdr.in==0 ){ + assert( pCur->rdr.zIn==pTab->zData ); + assert( pTab->iStart>=0 ); + assert( (size_t)pTab->iStart<=pCur->rdr.nIn ); + pCur->rdr.iIn = pTab->iStart; + }else{ + fseek(pCur->rdr.in, pTab->iStart, SEEK_SET); + pCur->rdr.iIn = 0; + pCur->rdr.nIn = 0; + } + return csvtabNext(pVtabCursor); +} + +/* +** Only a forward full table scan is supported. xBestIndex is mostly +** a no-op. If CSVTEST_FIDX is set, then the presence of equality +** constraints lowers the estimated cost, which is fiction, but is useful +** for testing certain kinds of virtual table behavior. +*/ +static int csvtabBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + pIdxInfo->estimatedCost = 1000000; +#ifdef SQLITE_TEST + if( (((CsvTable*)tab)->tstFlags & CSVTEST_FIDX)!=0 ){ + /* The usual (and sensible) case is to always do a full table scan. + ** The code in this branch only runs when testflags=1. This code + ** generates an artifical and unrealistic plan which is useful + ** for testing virtual table logic but is not helpful to real applications. + ** + ** Any ==, LIKE, or GLOB constraint is marked as usable by the virtual + ** table (even though it is not) and the cost of running the virtual table + ** is reduced from 1 million to just 10. The constraints are *not* marked + ** as omittable, however, so the query planner should still generate a + ** plan that gives a correct answer, even if they plan is not optimal. + */ + int i; + int nConst = 0; + for(i=0; i<pIdxInfo->nConstraint; i++){ + unsigned char op; + if( pIdxInfo->aConstraint[i].usable==0 ) continue; + op = pIdxInfo->aConstraint[i].op; + if( op==SQLITE_INDEX_CONSTRAINT_EQ + || op==SQLITE_INDEX_CONSTRAINT_LIKE + || op==SQLITE_INDEX_CONSTRAINT_GLOB + ){ + pIdxInfo->estimatedCost = 10; + pIdxInfo->aConstraintUsage[nConst].argvIndex = nConst+1; + nConst++; + } + } + } +#endif + return SQLITE_OK; +} + + +static sqlite3_module CsvModule = { + 0, /* iVersion */ + csvtabCreate, /* xCreate */ + csvtabConnect, /* xConnect */ + csvtabBestIndex, /* xBestIndex */ + csvtabDisconnect, /* xDisconnect */ + csvtabDisconnect, /* xDestroy */ + csvtabOpen, /* xOpen - open a cursor */ + csvtabClose, /* xClose - close a cursor */ + csvtabFilter, /* xFilter - configure scan constraints */ + csvtabNext, /* xNext - advance a cursor */ + csvtabEof, /* xEof - check for end of scan */ + csvtabColumn, /* xColumn - read data */ + csvtabRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; + +#ifdef SQLITE_TEST +/* +** For virtual table testing, make a version of the CSV virtual table +** available that has an xUpdate function. But the xUpdate always returns +** SQLITE_READONLY since the CSV file is not really writable. +*/ +static int csvtabUpdate(sqlite3_vtab *p,int n,sqlite3_value**v,sqlite3_int64*x){ + return SQLITE_READONLY; +} +static sqlite3_module CsvModuleFauxWrite = { + 0, /* iVersion */ + csvtabCreate, /* xCreate */ + csvtabConnect, /* xConnect */ + csvtabBestIndex, /* xBestIndex */ + csvtabDisconnect, /* xDisconnect */ + csvtabDisconnect, /* xDestroy */ + csvtabOpen, /* xOpen - open a cursor */ + csvtabClose, /* xClose - close a cursor */ + csvtabFilter, /* xFilter - configure scan constraints */ + csvtabNext, /* xNext - advance a cursor */ + csvtabEof, /* xEof - check for end of scan */ + csvtabColumn, /* xColumn - read data */ + csvtabRowid, /* xRowid - read data */ + csvtabUpdate, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; +#endif /* SQLITE_TEST */ + +#endif /* !defined(SQLITE_OMIT_VIRTUALTABLE) */ + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +/* +** This routine is called when the extension is loaded. The new +** CSV virtual table module is registered with the calling database +** connection. +*/ +int sqlite3_csv_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ +#ifndef SQLITE_OMIT_VIRTUALTABLE + int rc; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_create_module(db, "csv", &CsvModule, 0); +#ifdef SQLITE_TEST + if( rc==SQLITE_OK ){ + rc = sqlite3_create_module(db, "csv_wr", &CsvModuleFauxWrite, 0); + } +#endif + return rc; +#else + return SQLITE_OK; +#endif +} diff --git a/ext/misc/dbdump.c b/ext/misc/dbdump.c new file mode 100644 index 0000000..ecf7d81 --- /dev/null +++ b/ext/misc/dbdump.c @@ -0,0 +1,724 @@ +/* +** 2016-03-13 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file implements a C-language subroutine that converts the content +** of an SQLite database into UTF-8 text SQL statements that can be used +** to exactly recreate the original database. ROWID values are preserved. +** +** A prototype of the implemented subroutine is this: +** +** int sqlite3_db_dump( +** sqlite3 *db, +** const char *zSchema, +** const char *zTable, +** void (*xCallback)(void*, const char*), +** void *pArg +** ); +** +** The db parameter is the database connection. zSchema is the schema within +** that database which is to be dumped. Usually the zSchema is "main" but +** can also be "temp" or any ATTACH-ed database. If zTable is not NULL, then +** only the content of that one table is dumped. If zTable is NULL, then all +** tables are dumped. +** +** The generate text is passed to xCallback() in multiple calls. The second +** argument to xCallback() is a copy of the pArg parameter. The first +** argument is some of the output text that this routine generates. The +** signature to xCallback() is designed to make it compatible with fputs(). +** +** The sqlite3_db_dump() subroutine returns SQLITE_OK on success or some error +** code if it encounters a problem. +** +** If this file is compiled with -DDBDUMP_STANDALONE then a "main()" routine +** is included so that this routine becomes a command-line utility. The +** command-line utility takes two or three arguments which are the name +** of the database file, the schema, and optionally the table, forming the +** first three arguments of a single call to the library routine. +*/ +#include "sqlite3.h" +#include <stdarg.h> +#include <string.h> +#include <ctype.h> + +/* +** The state of the dump process. +*/ +typedef struct DState DState; +struct DState { + sqlite3 *db; /* The database connection */ + int nErr; /* Number of errors seen so far */ + int rc; /* Error code */ + int writableSchema; /* True if in writable_schema mode */ + int (*xCallback)(const char*,void*); /* Send output here */ + void *pArg; /* Argument to xCallback() */ +}; + +/* +** A variable length string to which one can append text. +*/ +typedef struct DText DText; +struct DText { + char *z; /* The text */ + int n; /* Number of bytes of content in z[] */ + int nAlloc; /* Number of bytes allocated to z[] */ +}; + +/* +** Initialize and destroy a DText object +*/ +static void initText(DText *p){ + memset(p, 0, sizeof(*p)); +} +static void freeText(DText *p){ + sqlite3_free(p->z); + initText(p); +} + +/* zIn is either a pointer to a NULL-terminated string in memory obtained +** from malloc(), or a NULL pointer. The string pointed to by zAppend is +** added to zIn, and the result returned in memory obtained from malloc(). +** zIn, if it was not NULL, is freed. +** +** If the third argument, quote, is not '\0', then it is used as a +** quote character for zAppend. +*/ +static void appendText(DText *p, char const *zAppend, char quote){ + int len; + int i; + int nAppend = (int)(strlen(zAppend) & 0x3fffffff); + + len = nAppend+p->n+1; + if( quote ){ + len += 2; + for(i=0; i<nAppend; i++){ + if( zAppend[i]==quote ) len++; + } + } + + if( p->n+len>=p->nAlloc ){ + char *zNew; + p->nAlloc = p->nAlloc*2 + len + 20; + zNew = sqlite3_realloc(p->z, p->nAlloc); + if( zNew==0 ){ + freeText(p); + return; + } + p->z = zNew; + } + + if( quote ){ + char *zCsr = p->z+p->n; + *zCsr++ = quote; + for(i=0; i<nAppend; i++){ + *zCsr++ = zAppend[i]; + if( zAppend[i]==quote ) *zCsr++ = quote; + } + *zCsr++ = quote; + p->n = (int)(zCsr - p->z); + *zCsr = '\0'; + }else{ + memcpy(p->z+p->n, zAppend, nAppend); + p->n += nAppend; + p->z[p->n] = '\0'; + } +} + +/* +** Attempt to determine if identifier zName needs to be quoted, either +** because it contains non-alphanumeric characters, or because it is an +** SQLite keyword. Be conservative in this estimate: When in doubt assume +** that quoting is required. +** +** Return '"' if quoting is required. Return 0 if no quoting is required. +*/ +static char quoteChar(const char *zName){ + int i; + if( !isalpha((unsigned char)zName[0]) && zName[0]!='_' ) return '"'; + for(i=0; zName[i]; i++){ + if( !isalnum((unsigned char)zName[i]) && zName[i]!='_' ) return '"'; + } + return sqlite3_keyword_check(zName, i) ? '"' : 0; +} + + +/* +** Release memory previously allocated by tableColumnList(). +*/ +static void freeColumnList(char **azCol){ + int i; + for(i=1; azCol[i]; i++){ + sqlite3_free(azCol[i]); + } + /* azCol[0] is a static string */ + sqlite3_free(azCol); +} + +/* +** Return a list of pointers to strings which are the names of all +** columns in table zTab. The memory to hold the names is dynamically +** allocated and must be released by the caller using a subsequent call +** to freeColumnList(). +** +** The azCol[0] entry is usually NULL. However, if zTab contains a rowid +** value that needs to be preserved, then azCol[0] is filled in with the +** name of the rowid column. +** +** The first regular column in the table is azCol[1]. The list is terminated +** by an entry with azCol[i]==0. +*/ +static char **tableColumnList(DState *p, const char *zTab){ + char **azCol = 0; + sqlite3_stmt *pStmt = 0; + char *zSql; + int nCol = 0; + int nAlloc = 0; + int nPK = 0; /* Number of PRIMARY KEY columns seen */ + int isIPK = 0; /* True if one PRIMARY KEY column of type INTEGER */ + int preserveRowid = 1; + int rc; + + zSql = sqlite3_mprintf("PRAGMA table_info=%Q", zTab); + if( zSql==0 ) return 0; + rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ) return 0; + while( sqlite3_step(pStmt)==SQLITE_ROW ){ + if( nCol>=nAlloc-2 ){ + char **azNew; + nAlloc = nAlloc*2 + nCol + 10; + azNew = sqlite3_realloc64(azCol, nAlloc*sizeof(azCol[0])); + if( azNew==0 ) goto col_oom; + azCol = azNew; + azCol[0] = 0; + } + azCol[++nCol] = sqlite3_mprintf("%s", sqlite3_column_text(pStmt, 1)); + if( azCol[nCol]==0 ) goto col_oom; + if( sqlite3_column_int(pStmt, 5) ){ + nPK++; + if( nPK==1 + && sqlite3_stricmp((const char*)sqlite3_column_text(pStmt,2), + "INTEGER")==0 + ){ + isIPK = 1; + }else{ + isIPK = 0; + } + } + } + sqlite3_finalize(pStmt); + pStmt = 0; + azCol[nCol+1] = 0; + + /* The decision of whether or not a rowid really needs to be preserved + ** is tricky. We never need to preserve a rowid for a WITHOUT ROWID table + ** or a table with an INTEGER PRIMARY KEY. We are unable to preserve + ** rowids on tables where the rowid is inaccessible because there are other + ** columns in the table named "rowid", "_rowid_", and "oid". + */ + if( isIPK ){ + /* If a single PRIMARY KEY column with type INTEGER was seen, then it + ** might be an alise for the ROWID. But it might also be a WITHOUT ROWID + ** table or a INTEGER PRIMARY KEY DESC column, neither of which are + ** ROWID aliases. To distinguish these cases, check to see if + ** there is a "pk" entry in "PRAGMA index_list". There will be + ** no "pk" index if the PRIMARY KEY really is an alias for the ROWID. + */ + zSql = sqlite3_mprintf("SELECT 1 FROM pragma_index_list(%Q)" + " WHERE origin='pk'", zTab); + if( zSql==0 ) goto col_oom; + rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ){ + freeColumnList(azCol); + return 0; + } + rc = sqlite3_step(pStmt); + sqlite3_finalize(pStmt); + pStmt = 0; + preserveRowid = rc==SQLITE_ROW; + } + if( preserveRowid ){ + /* Only preserve the rowid if we can find a name to use for the + ** rowid */ + static char *azRowid[] = { "rowid", "_rowid_", "oid" }; + int i, j; + for(j=0; j<3; j++){ + for(i=1; i<=nCol; i++){ + if( sqlite3_stricmp(azRowid[j],azCol[i])==0 ) break; + } + if( i>nCol ){ + /* At this point, we know that azRowid[j] is not the name of any + ** ordinary column in the table. Verify that azRowid[j] is a valid + ** name for the rowid before adding it to azCol[0]. WITHOUT ROWID + ** tables will fail this last check */ + rc = sqlite3_table_column_metadata(p->db,0,zTab,azRowid[j],0,0,0,0,0); + if( rc==SQLITE_OK ) azCol[0] = azRowid[j]; + break; + } + } + } + return azCol; + +col_oom: + sqlite3_finalize(pStmt); + freeColumnList(azCol); + p->nErr++; + p->rc = SQLITE_NOMEM; + return 0; +} + +/* +** Send mprintf-formatted content to the output callback. +*/ +static void output_formatted(DState *p, const char *zFormat, ...){ + va_list ap; + char *z; + va_start(ap, zFormat); + z = sqlite3_vmprintf(zFormat, ap); + va_end(ap); + p->xCallback(z, p->pArg); + sqlite3_free(z); +} + +/* +** Find a string that is not found anywhere in z[]. Return a pointer +** to that string. +** +** Try to use zA and zB first. If both of those are already found in z[] +** then make up some string and store it in the buffer zBuf. +*/ +static const char *unused_string( + const char *z, /* Result must not appear anywhere in z */ + const char *zA, const char *zB, /* Try these first */ + char *zBuf /* Space to store a generated string */ +){ + unsigned i = 0; + if( strstr(z, zA)==0 ) return zA; + if( strstr(z, zB)==0 ) return zB; + do{ + sqlite3_snprintf(20,zBuf,"(%s%u)", zA, i++); + }while( strstr(z,zBuf)!=0 ); + return zBuf; +} + +/* +** Output the given string as a quoted string using SQL quoting conventions. +** Additionallly , escape the "\n" and "\r" characters so that they do not +** get corrupted by end-of-line translation facilities in some operating +** systems. +*/ +static void output_quoted_escaped_string(DState *p, const char *z){ + int i; + char c; + for(i=0; (c = z[i])!=0 && c!='\'' && c!='\n' && c!='\r'; i++){} + if( c==0 ){ + output_formatted(p,"'%s'",z); + }else{ + const char *zNL = 0; + const char *zCR = 0; + int nNL = 0; + int nCR = 0; + char zBuf1[20], zBuf2[20]; + for(i=0; z[i]; i++){ + if( z[i]=='\n' ) nNL++; + if( z[i]=='\r' ) nCR++; + } + if( nNL ){ + p->xCallback("replace(", p->pArg); + zNL = unused_string(z, "\\n", "\\012", zBuf1); + } + if( nCR ){ + p->xCallback("replace(", p->pArg); + zCR = unused_string(z, "\\r", "\\015", zBuf2); + } + p->xCallback("'", p->pArg); + while( *z ){ + for(i=0; (c = z[i])!=0 && c!='\n' && c!='\r' && c!='\''; i++){} + if( c=='\'' ) i++; + if( i ){ + output_formatted(p, "%.*s", i, z); + z += i; + } + if( c=='\'' ){ + p->xCallback("'", p->pArg); + continue; + } + if( c==0 ){ + break; + } + z++; + if( c=='\n' ){ + p->xCallback(zNL, p->pArg); + continue; + } + p->xCallback(zCR, p->pArg); + } + p->xCallback("'", p->pArg); + if( nCR ){ + output_formatted(p, ",'%s',char(13))", zCR); + } + if( nNL ){ + output_formatted(p, ",'%s',char(10))", zNL); + } + } +} + +/* +** This is an sqlite3_exec callback routine used for dumping the database. +** Each row received by this callback consists of a table name, +** the table type ("index" or "table") and SQL to create the table. +** This routine should print text sufficient to recreate the table. +*/ +static int dump_callback(void *pArg, int nArg, char **azArg, char **azCol){ + int rc; + const char *zTable; + const char *zType; + const char *zSql; + DState *p = (DState*)pArg; + sqlite3_stmt *pStmt; + + (void)azCol; + if( nArg!=3 ) return 1; + zTable = azArg[0]; + zType = azArg[1]; + zSql = azArg[2]; + + if( strcmp(zTable, "sqlite_sequence")==0 ){ + p->xCallback("DELETE FROM sqlite_sequence;\n", p->pArg); + }else if( sqlite3_strglob("sqlite_stat?", zTable)==0 ){ + p->xCallback("ANALYZE sqlite_schema;\n", p->pArg); + }else if( strncmp(zTable, "sqlite_", 7)==0 ){ + return 0; + }else if( strncmp(zSql, "CREATE VIRTUAL TABLE", 20)==0 ){ + if( !p->writableSchema ){ + p->xCallback("PRAGMA writable_schema=ON;\n", p->pArg); + p->writableSchema = 1; + } + output_formatted(p, + "INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)" + "VALUES('table','%q','%q',0,'%q');", + zTable, zTable, zSql); + return 0; + }else{ + if( sqlite3_strglob("CREATE TABLE ['\"]*", zSql)==0 ){ + p->xCallback("CREATE TABLE IF NOT EXISTS ", p->pArg); + p->xCallback(zSql+13, p->pArg); + }else{ + p->xCallback(zSql, p->pArg); + } + p->xCallback(";\n", p->pArg); + } + + if( strcmp(zType, "table")==0 ){ + DText sSelect; + DText sTable; + char **azTCol; + int i; + int nCol; + + azTCol = tableColumnList(p, zTable); + if( azTCol==0 ) return 0; + + initText(&sTable); + appendText(&sTable, "INSERT INTO ", 0); + + /* Always quote the table name, even if it appears to be pure ascii, + ** in case it is a keyword. Ex: INSERT INTO "table" ... */ + appendText(&sTable, zTable, quoteChar(zTable)); + + /* If preserving the rowid, add a column list after the table name. + ** In other words: "INSERT INTO tab(rowid,a,b,c,...) VALUES(...)" + ** instead of the usual "INSERT INTO tab VALUES(...)". + */ + if( azTCol[0] ){ + appendText(&sTable, "(", 0); + appendText(&sTable, azTCol[0], 0); + for(i=1; azTCol[i]; i++){ + appendText(&sTable, ",", 0); + appendText(&sTable, azTCol[i], quoteChar(azTCol[i])); + } + appendText(&sTable, ")", 0); + } + appendText(&sTable, " VALUES(", 0); + + /* Build an appropriate SELECT statement */ + initText(&sSelect); + appendText(&sSelect, "SELECT ", 0); + if( azTCol[0] ){ + appendText(&sSelect, azTCol[0], 0); + appendText(&sSelect, ",", 0); + } + for(i=1; azTCol[i]; i++){ + appendText(&sSelect, azTCol[i], quoteChar(azTCol[i])); + if( azTCol[i+1] ){ + appendText(&sSelect, ",", 0); + } + } + nCol = i; + if( azTCol[0]==0 ) nCol--; + freeColumnList(azTCol); + appendText(&sSelect, " FROM ", 0); + appendText(&sSelect, zTable, quoteChar(zTable)); + + rc = sqlite3_prepare_v2(p->db, sSelect.z, -1, &pStmt, 0); + if( rc!=SQLITE_OK ){ + p->nErr++; + if( p->rc==SQLITE_OK ) p->rc = rc; + }else{ + while( SQLITE_ROW==sqlite3_step(pStmt) ){ + p->xCallback(sTable.z, p->pArg); + for(i=0; i<nCol; i++){ + if( i ) p->xCallback(",", p->pArg); + switch( sqlite3_column_type(pStmt,i) ){ + case SQLITE_INTEGER: { + output_formatted(p, "%lld", sqlite3_column_int64(pStmt,i)); + break; + } + case SQLITE_FLOAT: { + double r = sqlite3_column_double(pStmt,i); + sqlite3_uint64 ur; + memcpy(&ur,&r,sizeof(r)); + if( ur==0x7ff0000000000000LL ){ + p->xCallback("1e999", p->pArg); + }else if( ur==0xfff0000000000000LL ){ + p->xCallback("-1e999", p->pArg); + }else{ + output_formatted(p, "%!.20g", r); + } + break; + } + case SQLITE_NULL: { + p->xCallback("NULL", p->pArg); + break; + } + case SQLITE_TEXT: { + output_quoted_escaped_string(p, + (const char*)sqlite3_column_text(pStmt,i)); + break; + } + case SQLITE_BLOB: { + int nByte = sqlite3_column_bytes(pStmt,i); + unsigned char *a = (unsigned char*)sqlite3_column_blob(pStmt,i); + int j; + p->xCallback("x'", p->pArg); + for(j=0; j<nByte; j++){ + char zWord[3]; + zWord[0] = "0123456789abcdef"[(a[j]>>4)&15]; + zWord[1] = "0123456789abcdef"[a[j]&15]; + zWord[2] = 0; + p->xCallback(zWord, p->pArg); + } + p->xCallback("'", p->pArg); + break; + } + } + } + p->xCallback(");\n", p->pArg); + } + } + sqlite3_finalize(pStmt); + freeText(&sTable); + freeText(&sSelect); + } + return 0; +} + + +/* +** Execute a query statement that will generate SQL output. Print +** the result columns, comma-separated, on a line and then add a +** semicolon terminator to the end of that line. +** +** If the number of columns is 1 and that column contains text "--" +** then write the semicolon on a separate line. That way, if a +** "--" comment occurs at the end of the statement, the comment +** won't consume the semicolon terminator. +*/ +static void output_sql_from_query( + DState *p, /* Query context */ + const char *zSelect, /* SELECT statement to extract content */ + ... +){ + sqlite3_stmt *pSelect; + int rc; + int nResult; + int i; + const char *z; + char *zSql; + va_list ap; + va_start(ap, zSelect); + zSql = sqlite3_vmprintf(zSelect, ap); + va_end(ap); + if( zSql==0 ){ + p->rc = SQLITE_NOMEM; + p->nErr++; + return; + } + rc = sqlite3_prepare_v2(p->db, zSql, -1, &pSelect, 0); + sqlite3_free(zSql); + if( rc!=SQLITE_OK || !pSelect ){ + output_formatted(p, "/**** ERROR: (%d) %s *****/\n", rc, + sqlite3_errmsg(p->db)); + p->nErr++; + return; + } + rc = sqlite3_step(pSelect); + nResult = sqlite3_column_count(pSelect); + while( rc==SQLITE_ROW ){ + z = (const char*)sqlite3_column_text(pSelect, 0); + p->xCallback(z, p->pArg); + for(i=1; i<nResult; i++){ + p->xCallback(",", p->pArg); + p->xCallback((const char*)sqlite3_column_text(pSelect,i), p->pArg); + } + if( z==0 ) z = ""; + while( z[0] && (z[0]!='-' || z[1]!='-') ) z++; + if( z[0] ){ + p->xCallback("\n;\n", p->pArg); + }else{ + p->xCallback(";\n", p->pArg); + } + rc = sqlite3_step(pSelect); + } + rc = sqlite3_finalize(pSelect); + if( rc!=SQLITE_OK ){ + output_formatted(p, "/**** ERROR: (%d) %s *****/\n", rc, + sqlite3_errmsg(p->db)); + if( (rc&0xff)!=SQLITE_CORRUPT ) p->nErr++; + } +} + +/* +** Run zQuery. Use dump_callback() as the callback routine so that +** the contents of the query are output as SQL statements. +** +** If we get a SQLITE_CORRUPT error, rerun the query after appending +** "ORDER BY rowid DESC" to the end. +*/ +static void run_schema_dump_query( + DState *p, + const char *zQuery, + ... +){ + char *zErr = 0; + char *z; + va_list ap; + va_start(ap, zQuery); + z = sqlite3_vmprintf(zQuery, ap); + va_end(ap); + sqlite3_exec(p->db, z, dump_callback, p, &zErr); + sqlite3_free(z); + if( zErr ){ + output_formatted(p, "/****** %s ******/\n", zErr); + sqlite3_free(zErr); + p->nErr++; + zErr = 0; + } +} + +/* +** Convert an SQLite database into SQL statements that will recreate that +** database. +*/ +int sqlite3_db_dump( + sqlite3 *db, /* The database connection */ + const char *zSchema, /* Which schema to dump. Usually "main". */ + const char *zTable, /* Which table to dump. NULL means everything. */ + int (*xCallback)(const char*,void*), /* Output sent to this callback */ + void *pArg /* Second argument of the callback */ +){ + DState x; + memset(&x, 0, sizeof(x)); + x.rc = sqlite3_exec(db, "BEGIN", 0, 0, 0); + if( x.rc ) return x.rc; + x.db = db; + x.xCallback = xCallback; + x.pArg = pArg; + xCallback("PRAGMA foreign_keys=OFF;\nBEGIN TRANSACTION;\n", pArg); + if( zTable==0 ){ + run_schema_dump_query(&x, + "SELECT name, type, sql FROM \"%w\".sqlite_schema " + "WHERE sql NOT NULL AND type=='table' AND name!='sqlite_sequence'", + zSchema + ); + run_schema_dump_query(&x, + "SELECT name, type, sql FROM \"%w\".sqlite_schema " + "WHERE name=='sqlite_sequence'", zSchema + ); + output_sql_from_query(&x, + "SELECT sql FROM sqlite_schema " + "WHERE sql NOT NULL AND type IN ('index','trigger','view')", 0 + ); + }else{ + run_schema_dump_query(&x, + "SELECT name, type, sql FROM \"%w\".sqlite_schema " + "WHERE tbl_name=%Q COLLATE nocase AND type=='table'" + " AND sql NOT NULL", + zSchema, zTable + ); + output_sql_from_query(&x, + "SELECT sql FROM \"%w\".sqlite_schema " + "WHERE sql NOT NULL" + " AND type IN ('index','trigger','view')" + " AND tbl_name=%Q COLLATE nocase", + zSchema, zTable + ); + } + if( x.writableSchema ){ + xCallback("PRAGMA writable_schema=OFF;\n", pArg); + } + xCallback(x.nErr ? "ROLLBACK; -- due to errors\n" : "COMMIT;\n", pArg); + sqlite3_exec(db, "COMMIT", 0, 0, 0); + return x.rc; +} + + + +/* The generic subroutine is above. The code the follows implements +** the command-line interface. +*/ +#ifdef DBDUMP_STANDALONE +#include <stdio.h> + +/* +** Command-line interface +*/ +int main(int argc, char **argv){ + sqlite3 *db; + const char *zDb; + const char *zSchema; + const char *zTable = 0; + int rc; + + if( argc<2 || argc>4 ){ + fprintf(stderr, "Usage: %s DATABASE ?SCHEMA? ?TABLE?\n", argv[0]); + return 1; + } + zDb = argv[1]; + zSchema = argc>=3 ? argv[2] : "main"; + zTable = argc==4 ? argv[3] : 0; + + rc = sqlite3_open(zDb, &db); + if( rc ){ + fprintf(stderr, "Cannot open \"%s\": %s\n", zDb, sqlite3_errmsg(db)); + sqlite3_close(db); + return 1; + } + rc = sqlite3_db_dump(db, zSchema, zTable, + (int(*)(const char*,void*))fputs, (void*)stdout); + if( rc ){ + fprintf(stderr, "Error: sqlite3_db_dump() returns %d\n", rc); + } + sqlite3_close(db); + return rc!=SQLITE_OK; +} +#endif /* DBDUMP_STANDALONE */ diff --git a/ext/misc/decimal.c b/ext/misc/decimal.c new file mode 100644 index 0000000..37c6c2f --- /dev/null +++ b/ext/misc/decimal.c @@ -0,0 +1,635 @@ +/* +** 2020-06-22 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** Routines to implement arbitrary-precision decimal math. +** +** The focus here is on simplicity and correctness, not performance. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <ctype.h> +#include <stdlib.h> + +/* Mark a function parameter as unused, to suppress nuisance compiler +** warnings. */ +#ifndef UNUSED_PARAMETER +# define UNUSED_PARAMETER(X) (void)(X) +#endif + + +/* A decimal object */ +typedef struct Decimal Decimal; +struct Decimal { + char sign; /* 0 for positive, 1 for negative */ + char oom; /* True if an OOM is encountered */ + char isNull; /* True if holds a NULL rather than a number */ + char isInit; /* True upon initialization */ + int nDigit; /* Total number of digits */ + int nFrac; /* Number of digits to the right of the decimal point */ + signed char *a; /* Array of digits. Most significant first. */ +}; + +/* +** Release memory held by a Decimal, but do not free the object itself. +*/ +static void decimal_clear(Decimal *p){ + sqlite3_free(p->a); +} + +/* +** Destroy a Decimal object +*/ +static void decimal_free(Decimal *p){ + if( p ){ + decimal_clear(p); + sqlite3_free(p); + } +} + +/* +** Allocate a new Decimal object. Initialize it to the number given +** by the input string. +*/ +static Decimal *decimal_new( + sqlite3_context *pCtx, + sqlite3_value *pIn, + int nAlt, + const unsigned char *zAlt +){ + Decimal *p; + int n, i; + const unsigned char *zIn; + int iExp = 0; + p = sqlite3_malloc( sizeof(*p) ); + if( p==0 ) goto new_no_mem; + p->sign = 0; + p->oom = 0; + p->isInit = 1; + p->isNull = 0; + p->nDigit = 0; + p->nFrac = 0; + if( zAlt ){ + n = nAlt, + zIn = zAlt; + }else{ + if( sqlite3_value_type(pIn)==SQLITE_NULL ){ + p->a = 0; + p->isNull = 1; + return p; + } + n = sqlite3_value_bytes(pIn); + zIn = sqlite3_value_text(pIn); + } + p->a = sqlite3_malloc64( n+1 ); + if( p->a==0 ) goto new_no_mem; + for(i=0; isspace(zIn[i]); i++){} + if( zIn[i]=='-' ){ + p->sign = 1; + i++; + }else if( zIn[i]=='+' ){ + i++; + } + while( i<n && zIn[i]=='0' ) i++; + while( i<n ){ + char c = zIn[i]; + if( c>='0' && c<='9' ){ + p->a[p->nDigit++] = c - '0'; + }else if( c=='.' ){ + p->nFrac = p->nDigit + 1; + }else if( c=='e' || c=='E' ){ + int j = i+1; + int neg = 0; + if( j>=n ) break; + if( zIn[j]=='-' ){ + neg = 1; + j++; + }else if( zIn[j]=='+' ){ + j++; + } + while( j<n && iExp<1000000 ){ + if( zIn[j]>='0' && zIn[j]<='9' ){ + iExp = iExp*10 + zIn[j] - '0'; + } + j++; + } + if( neg ) iExp = -iExp; + break; + } + i++; + } + if( p->nFrac ){ + p->nFrac = p->nDigit - (p->nFrac - 1); + } + if( iExp>0 ){ + if( p->nFrac>0 ){ + if( iExp<=p->nFrac ){ + p->nFrac -= iExp; + iExp = 0; + }else{ + iExp -= p->nFrac; + p->nFrac = 0; + } + } + if( iExp>0 ){ + p->a = sqlite3_realloc64(p->a, p->nDigit + iExp + 1 ); + if( p->a==0 ) goto new_no_mem; + memset(p->a+p->nDigit, 0, iExp); + p->nDigit += iExp; + } + }else if( iExp<0 ){ + int nExtra; + iExp = -iExp; + nExtra = p->nDigit - p->nFrac - 1; + if( nExtra ){ + if( nExtra>=iExp ){ + p->nFrac += iExp; + iExp = 0; + }else{ + iExp -= nExtra; + p->nFrac = p->nDigit - 1; + } + } + if( iExp>0 ){ + p->a = sqlite3_realloc64(p->a, p->nDigit + iExp + 1 ); + if( p->a==0 ) goto new_no_mem; + memmove(p->a+iExp, p->a, p->nDigit); + memset(p->a, 0, iExp); + p->nDigit += iExp; + p->nFrac += iExp; + } + } + return p; + +new_no_mem: + if( pCtx ) sqlite3_result_error_nomem(pCtx); + sqlite3_free(p); + return 0; +} + +/* +** Make the given Decimal the result. +*/ +static void decimal_result(sqlite3_context *pCtx, Decimal *p){ + char *z; + int i, j; + int n; + if( p==0 || p->oom ){ + sqlite3_result_error_nomem(pCtx); + return; + } + if( p->isNull ){ + sqlite3_result_null(pCtx); + return; + } + z = sqlite3_malloc( p->nDigit+4 ); + if( z==0 ){ + sqlite3_result_error_nomem(pCtx); + return; + } + i = 0; + if( p->nDigit==0 || (p->nDigit==1 && p->a[0]==0) ){ + p->sign = 0; + } + if( p->sign ){ + z[0] = '-'; + i = 1; + } + n = p->nDigit - p->nFrac; + if( n<=0 ){ + z[i++] = '0'; + } + j = 0; + while( n>1 && p->a[j]==0 ){ + j++; + n--; + } + while( n>0 ){ + z[i++] = p->a[j] + '0'; + j++; + n--; + } + if( p->nFrac ){ + z[i++] = '.'; + do{ + z[i++] = p->a[j] + '0'; + j++; + }while( j<p->nDigit ); + } + z[i] = 0; + sqlite3_result_text(pCtx, z, i, sqlite3_free); +} + +/* +** SQL Function: decimal(X) +** +** Convert input X into decimal and then back into text +*/ +static void decimalFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *p = decimal_new(context, argv[0], 0, 0); + UNUSED_PARAMETER(argc); + decimal_result(context, p); + decimal_free(p); +} + +/* +** Compare to Decimal objects. Return negative, 0, or positive if the +** first object is less than, equal to, or greater than the second. +** +** Preconditions for this routine: +** +** pA!=0 +** pA->isNull==0 +** pB!=0 +** pB->isNull==0 +*/ +static int decimal_cmp(const Decimal *pA, const Decimal *pB){ + int nASig, nBSig, rc, n; + if( pA->sign!=pB->sign ){ + return pA->sign ? -1 : +1; + } + if( pA->sign ){ + const Decimal *pTemp = pA; + pA = pB; + pB = pTemp; + } + nASig = pA->nDigit - pA->nFrac; + nBSig = pB->nDigit - pB->nFrac; + if( nASig!=nBSig ){ + return nASig - nBSig; + } + n = pA->nDigit; + if( n>pB->nDigit ) n = pB->nDigit; + rc = memcmp(pA->a, pB->a, n); + if( rc==0 ){ + rc = pA->nDigit - pB->nDigit; + } + return rc; +} + +/* +** SQL Function: decimal_cmp(X, Y) +** +** Return negative, zero, or positive if X is less then, equal to, or +** greater than Y. +*/ +static void decimalCmpFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *pA = 0, *pB = 0; + int rc; + + UNUSED_PARAMETER(argc); + pA = decimal_new(context, argv[0], 0, 0); + if( pA==0 || pA->isNull ) goto cmp_done; + pB = decimal_new(context, argv[1], 0, 0); + if( pB==0 || pB->isNull ) goto cmp_done; + rc = decimal_cmp(pA, pB); + if( rc<0 ) rc = -1; + else if( rc>0 ) rc = +1; + sqlite3_result_int(context, rc); +cmp_done: + decimal_free(pA); + decimal_free(pB); +} + +/* +** Expand the Decimal so that it has a least nDigit digits and nFrac +** digits to the right of the decimal point. +*/ +static void decimal_expand(Decimal *p, int nDigit, int nFrac){ + int nAddSig; + int nAddFrac; + if( p==0 ) return; + nAddFrac = nFrac - p->nFrac; + nAddSig = (nDigit - p->nDigit) - nAddFrac; + if( nAddFrac==0 && nAddSig==0 ) return; + p->a = sqlite3_realloc64(p->a, nDigit+1); + if( p->a==0 ){ + p->oom = 1; + return; + } + if( nAddSig ){ + memmove(p->a+nAddSig, p->a, p->nDigit); + memset(p->a, 0, nAddSig); + p->nDigit += nAddSig; + } + if( nAddFrac ){ + memset(p->a+p->nDigit, 0, nAddFrac); + p->nDigit += nAddFrac; + p->nFrac += nAddFrac; + } +} + +/* +** Add the value pB into pA. +** +** Both pA and pB might become denormalized by this routine. +*/ +static void decimal_add(Decimal *pA, Decimal *pB){ + int nSig, nFrac, nDigit; + int i, rc; + if( pA==0 ){ + return; + } + if( pA->oom || pB==0 || pB->oom ){ + pA->oom = 1; + return; + } + if( pA->isNull || pB->isNull ){ + pA->isNull = 1; + return; + } + nSig = pA->nDigit - pA->nFrac; + if( nSig && pA->a[0]==0 ) nSig--; + if( nSig<pB->nDigit-pB->nFrac ){ + nSig = pB->nDigit - pB->nFrac; + } + nFrac = pA->nFrac; + if( nFrac<pB->nFrac ) nFrac = pB->nFrac; + nDigit = nSig + nFrac + 1; + decimal_expand(pA, nDigit, nFrac); + decimal_expand(pB, nDigit, nFrac); + if( pA->oom || pB->oom ){ + pA->oom = 1; + }else{ + if( pA->sign==pB->sign ){ + int carry = 0; + for(i=nDigit-1; i>=0; i--){ + int x = pA->a[i] + pB->a[i] + carry; + if( x>=10 ){ + carry = 1; + pA->a[i] = x - 10; + }else{ + carry = 0; + pA->a[i] = x; + } + } + }else{ + signed char *aA, *aB; + int borrow = 0; + rc = memcmp(pA->a, pB->a, nDigit); + if( rc<0 ){ + aA = pB->a; + aB = pA->a; + pA->sign = !pA->sign; + }else{ + aA = pA->a; + aB = pB->a; + } + for(i=nDigit-1; i>=0; i--){ + int x = aA[i] - aB[i] - borrow; + if( x<0 ){ + pA->a[i] = x+10; + borrow = 1; + }else{ + pA->a[i] = x; + borrow = 0; + } + } + } + } +} + +/* +** Compare text in decimal order. +*/ +static int decimalCollFunc( + void *notUsed, + int nKey1, const void *pKey1, + int nKey2, const void *pKey2 +){ + const unsigned char *zA = (const unsigned char*)pKey1; + const unsigned char *zB = (const unsigned char*)pKey2; + Decimal *pA = decimal_new(0, 0, nKey1, zA); + Decimal *pB = decimal_new(0, 0, nKey2, zB); + int rc; + UNUSED_PARAMETER(notUsed); + if( pA==0 || pB==0 ){ + rc = 0; + }else{ + rc = decimal_cmp(pA, pB); + } + decimal_free(pA); + decimal_free(pB); + return rc; +} + + +/* +** SQL Function: decimal_add(X, Y) +** decimal_sub(X, Y) +** +** Return the sum or difference of X and Y. +*/ +static void decimalAddFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *pA = decimal_new(context, argv[0], 0, 0); + Decimal *pB = decimal_new(context, argv[1], 0, 0); + UNUSED_PARAMETER(argc); + decimal_add(pA, pB); + decimal_result(context, pA); + decimal_free(pA); + decimal_free(pB); +} +static void decimalSubFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *pA = decimal_new(context, argv[0], 0, 0); + Decimal *pB = decimal_new(context, argv[1], 0, 0); + UNUSED_PARAMETER(argc); + if( pB ){ + pB->sign = !pB->sign; + decimal_add(pA, pB); + decimal_result(context, pA); + } + decimal_free(pA); + decimal_free(pB); +} + +/* Aggregate funcion: decimal_sum(X) +** +** Works like sum() except that it uses decimal arithmetic for unlimited +** precision. +*/ +static void decimalSumStep( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *p; + Decimal *pArg; + UNUSED_PARAMETER(argc); + p = sqlite3_aggregate_context(context, sizeof(*p)); + if( p==0 ) return; + if( !p->isInit ){ + p->isInit = 1; + p->a = sqlite3_malloc(2); + if( p->a==0 ){ + p->oom = 1; + }else{ + p->a[0] = 0; + } + p->nDigit = 1; + p->nFrac = 0; + } + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + pArg = decimal_new(context, argv[0], 0, 0); + decimal_add(p, pArg); + decimal_free(pArg); +} +static void decimalSumInverse( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *p; + Decimal *pArg; + UNUSED_PARAMETER(argc); + p = sqlite3_aggregate_context(context, sizeof(*p)); + if( p==0 ) return; + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + pArg = decimal_new(context, argv[0], 0, 0); + if( pArg ) pArg->sign = !pArg->sign; + decimal_add(p, pArg); + decimal_free(pArg); +} +static void decimalSumValue(sqlite3_context *context){ + Decimal *p = sqlite3_aggregate_context(context, 0); + if( p==0 ) return; + decimal_result(context, p); +} +static void decimalSumFinalize(sqlite3_context *context){ + Decimal *p = sqlite3_aggregate_context(context, 0); + if( p==0 ) return; + decimal_result(context, p); + decimal_clear(p); +} + +/* +** SQL Function: decimal_mul(X, Y) +** +** Return the product of X and Y. +** +** All significant digits after the decimal point are retained. +** Trailing zeros after the decimal point are omitted as long as +** the number of digits after the decimal point is no less than +** either the number of digits in either input. +*/ +static void decimalMulFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + Decimal *pA = decimal_new(context, argv[0], 0, 0); + Decimal *pB = decimal_new(context, argv[1], 0, 0); + signed char *acc = 0; + int i, j, k; + int minFrac; + UNUSED_PARAMETER(argc); + if( pA==0 || pA->oom || pA->isNull + || pB==0 || pB->oom || pB->isNull + ){ + goto mul_end; + } + acc = sqlite3_malloc64( pA->nDigit + pB->nDigit + 2 ); + if( acc==0 ){ + sqlite3_result_error_nomem(context); + goto mul_end; + } + memset(acc, 0, pA->nDigit + pB->nDigit + 2); + minFrac = pA->nFrac; + if( pB->nFrac<minFrac ) minFrac = pB->nFrac; + for(i=pA->nDigit-1; i>=0; i--){ + signed char f = pA->a[i]; + int carry = 0, x; + for(j=pB->nDigit-1, k=i+j+3; j>=0; j--, k--){ + x = acc[k] + f*pB->a[j] + carry; + acc[k] = x%10; + carry = x/10; + } + x = acc[k] + carry; + acc[k] = x%10; + acc[k-1] += x/10; + } + sqlite3_free(pA->a); + pA->a = acc; + acc = 0; + pA->nDigit += pB->nDigit + 2; + pA->nFrac += pB->nFrac; + pA->sign ^= pB->sign; + while( pA->nFrac>minFrac && pA->a[pA->nDigit-1]==0 ){ + pA->nFrac--; + pA->nDigit--; + } + decimal_result(context, pA); + +mul_end: + sqlite3_free(acc); + decimal_free(pA); + decimal_free(pB); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_decimal_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + static const struct { + const char *zFuncName; + int nArg; + void (*xFunc)(sqlite3_context*,int,sqlite3_value**); + } aFunc[] = { + { "decimal", 1, decimalFunc }, + { "decimal_cmp", 2, decimalCmpFunc }, + { "decimal_add", 2, decimalAddFunc }, + { "decimal_sub", 2, decimalSubFunc }, + { "decimal_mul", 2, decimalMulFunc }, + }; + unsigned int i; + (void)pzErrMsg; /* Unused parameter */ + + SQLITE_EXTENSION_INIT2(pApi); + + for(i=0; i<sizeof(aFunc)/sizeof(aFunc[0]) && rc==SQLITE_OK; i++){ + rc = sqlite3_create_function(db, aFunc[i].zFuncName, aFunc[i].nArg, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, aFunc[i].xFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_window_function(db, "decimal_sum", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, 0, + decimalSumStep, decimalSumFinalize, + decimalSumValue, decimalSumInverse, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_collation(db, "decimal", SQLITE_UTF8, + 0, decimalCollFunc); + } + return rc; +} diff --git a/ext/misc/eval.c b/ext/misc/eval.c new file mode 100644 index 0000000..d3849d6 --- /dev/null +++ b/ext/misc/eval.c @@ -0,0 +1,125 @@ +/* +** 2014-11-10 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements SQL function eval() which runs +** SQL statements recursively. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <string.h> + +/* +** Structure used to accumulate the output +*/ +struct EvalResult { + char *z; /* Accumulated output */ + const char *zSep; /* Separator */ + int szSep; /* Size of the separator string */ + sqlite3_int64 nAlloc; /* Number of bytes allocated for z[] */ + sqlite3_int64 nUsed; /* Number of bytes of z[] actually used */ +}; + +/* +** Callback from sqlite_exec() for the eval() function. +*/ +static int callback(void *pCtx, int argc, char **argv, char **colnames){ + struct EvalResult *p = (struct EvalResult*)pCtx; + int i; + if( argv==0 ) return 0; + for(i=0; i<argc; i++){ + const char *z = argv[i] ? argv[i] : ""; + size_t sz = strlen(z); + if( (sqlite3_int64)sz+p->nUsed+p->szSep+1 > p->nAlloc ){ + char *zNew; + p->nAlloc = p->nAlloc*2 + sz + p->szSep + 1; + /* Using sqlite3_realloc64() would be better, but it is a recent + ** addition and will cause a segfault if loaded by an older version + ** of SQLite. */ + zNew = p->nAlloc<=0x7fffffff ? sqlite3_realloc64(p->z, p->nAlloc) : 0; + if( zNew==0 ){ + sqlite3_free(p->z); + memset(p, 0, sizeof(*p)); + return 1; + } + p->z = zNew; + } + if( p->nUsed>0 ){ + memcpy(&p->z[p->nUsed], p->zSep, p->szSep); + p->nUsed += p->szSep; + } + memcpy(&p->z[p->nUsed], z, sz); + p->nUsed += sz; + } + return 0; +} + +/* +** Implementation of the eval(X) and eval(X,Y) SQL functions. +** +** Evaluate the SQL text in X. Return the results, using string +** Y as the separator. If Y is omitted, use a single space character. +*/ +static void sqlEvalFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSql; + sqlite3 *db; + char *zErr = 0; + int rc; + struct EvalResult x; + + memset(&x, 0, sizeof(x)); + x.zSep = " "; + zSql = (const char*)sqlite3_value_text(argv[0]); + if( zSql==0 ) return; + if( argc>1 ){ + x.zSep = (const char*)sqlite3_value_text(argv[1]); + if( x.zSep==0 ) return; + } + x.szSep = (int)strlen(x.zSep); + db = sqlite3_context_db_handle(context); + rc = sqlite3_exec(db, zSql, callback, &x, &zErr); + if( rc!=SQLITE_OK ){ + sqlite3_result_error(context, zErr, -1); + sqlite3_free(zErr); + }else if( x.zSep==0 ){ + sqlite3_result_error_nomem(context); + sqlite3_free(x.z); + }else{ + sqlite3_result_text(context, x.z, (int)x.nUsed, sqlite3_free); + } +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_eval_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "eval", 1, + SQLITE_UTF8|SQLITE_DIRECTONLY, 0, + sqlEvalFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "eval", 2, + SQLITE_UTF8|SQLITE_DIRECTONLY, 0, + sqlEvalFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/explain.c b/ext/misc/explain.c new file mode 100644 index 0000000..0095194 --- /dev/null +++ b/ext/misc/explain.c @@ -0,0 +1,322 @@ +/* +** 2018-09-16 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates an eponymous virtual table that returns the +** EXPLAIN output from an SQL statement. +** +** Usage example: +** +** .load ./explain +** SELECT p2 FROM explain('SELECT * FROM sqlite_schema') +** WHERE opcode='OpenRead'; +** +** This module was originally written to help simplify SQLite testing, +** by providing an easier means of verifying certain patterns in the +** generated bytecode. +*/ +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* explain_vtab is a subclass of sqlite3_vtab which will +** serve as the underlying representation of a explain virtual table +*/ +typedef struct explain_vtab explain_vtab; +struct explain_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this explain vtab */ +}; + +/* explain_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result from an EXPLAIN operation. +*/ +typedef struct explain_cursor explain_cursor; +struct explain_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this cursor */ + char *zSql; /* Value for the EXPLN_COLUMN_SQL column */ + sqlite3_stmt *pExplain; /* Statement being explained */ + int rc; /* Result of last sqlite3_step() on pExplain */ +}; + +/* +** The explainConnect() method is invoked to create a new +** explain_vtab that describes the explain virtual table. +** +** Think of this routine as the constructor for explain_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the explain_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against explain will look like. +*/ +static int explainConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + explain_vtab *pNew; + int rc; + +/* Column numbers */ +#define EXPLN_COLUMN_ADDR 0 /* Instruction address */ +#define EXPLN_COLUMN_OPCODE 1 /* Opcode */ +#define EXPLN_COLUMN_P1 2 /* Operand 1 */ +#define EXPLN_COLUMN_P2 3 /* Operand 2 */ +#define EXPLN_COLUMN_P3 4 /* Operand 3 */ +#define EXPLN_COLUMN_P4 5 /* Operand 4 */ +#define EXPLN_COLUMN_P5 6 /* Operand 5 */ +#define EXPLN_COLUMN_COMMENT 7 /* Comment */ +#define EXPLN_COLUMN_SQL 8 /* SQL that is being explained */ + + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(addr,opcode,p1,p2,p3,p4,p5,comment,sql HIDDEN)"); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + } + return rc; +} + +/* +** This method is the destructor for explain_cursor objects. +*/ +static int explainDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new explain_cursor object. +*/ +static int explainOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + explain_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->db = ((explain_vtab*)p)->db; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a explain_cursor. +*/ +static int explainClose(sqlite3_vtab_cursor *cur){ + explain_cursor *pCur = (explain_cursor*)cur; + sqlite3_finalize(pCur->pExplain); + sqlite3_free(pCur->zSql); + sqlite3_free(pCur); + return SQLITE_OK; +} + + +/* +** Advance a explain_cursor to its next row of output. +*/ +static int explainNext(sqlite3_vtab_cursor *cur){ + explain_cursor *pCur = (explain_cursor*)cur; + pCur->rc = sqlite3_step(pCur->pExplain); + if( pCur->rc!=SQLITE_DONE && pCur->rc!=SQLITE_ROW ) return pCur->rc; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the explain_cursor +** is currently pointing. +*/ +static int explainColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + explain_cursor *pCur = (explain_cursor*)cur; + if( i==EXPLN_COLUMN_SQL ){ + sqlite3_result_text(ctx, pCur->zSql, -1, SQLITE_TRANSIENT); + }else{ + sqlite3_result_value(ctx, sqlite3_column_value(pCur->pExplain, i)); + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int explainRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + explain_cursor *pCur = (explain_cursor*)cur; + *pRowid = sqlite3_column_int64(pCur->pExplain, 0); + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int explainEof(sqlite3_vtab_cursor *cur){ + explain_cursor *pCur = (explain_cursor*)cur; + return pCur->rc!=SQLITE_ROW; +} + +/* +** This method is called to "rewind" the explain_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to explainColumn() or explainRowid() or +** explainEof(). +** +** The argv[0] is the SQL statement that is to be explained. +*/ +static int explainFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + explain_cursor *pCur = (explain_cursor *)pVtabCursor; + char *zSql = 0; + int rc; + sqlite3_finalize(pCur->pExplain); + pCur->pExplain = 0; + if( sqlite3_value_type(argv[0])!=SQLITE_TEXT ){ + pCur->rc = SQLITE_DONE; + return SQLITE_OK; + } + sqlite3_free(pCur->zSql); + pCur->zSql = sqlite3_mprintf("%s", sqlite3_value_text(argv[0])); + if( pCur->zSql ){ + zSql = sqlite3_mprintf("EXPLAIN %s", pCur->zSql); + } + if( zSql==0 ){ + rc = SQLITE_NOMEM; + }else{ + rc = sqlite3_prepare_v2(pCur->db, zSql, -1, &pCur->pExplain, 0); + sqlite3_free(zSql); + } + if( rc ){ + sqlite3_finalize(pCur->pExplain); + pCur->pExplain = 0; + sqlite3_free(pCur->zSql); + pCur->zSql = 0; + }else{ + pCur->rc = sqlite3_step(pCur->pExplain); + rc = (pCur->rc==SQLITE_DONE || pCur->rc==SQLITE_ROW) ? SQLITE_OK : pCur->rc; + } + return rc; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the explain virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int explainBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; /* Loop counter */ + int idx = -1; /* Index of a usable == constraint against SQL */ + int unusable = 0; /* True if there are unusable constraints on SQL */ + + pIdxInfo->estimatedRows = 500; + for(i=0; i<pIdxInfo->nConstraint; i++){ + struct sqlite3_index_constraint *p = &pIdxInfo->aConstraint[i]; + if( p->iColumn!=EXPLN_COLUMN_SQL ) continue; + if( !p->usable ){ + unusable = 1; + }else if( p->op==SQLITE_INDEX_CONSTRAINT_EQ ){ + idx = i; + } + } + if( idx>=0 ){ + /* There exists a usable == constraint against the SQL column */ + pIdxInfo->estimatedCost = 10.0; + pIdxInfo->idxNum = 1; + pIdxInfo->aConstraintUsage[idx].argvIndex = 1; + pIdxInfo->aConstraintUsage[idx].omit = 1; + }else if( unusable ){ + /* There are unusable constraints against the SQL column. Do not allow + ** this plan to continue forward. */ + return SQLITE_CONSTRAINT; + } + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** explain virtual table. +*/ +static sqlite3_module explainModule = { + 0, /* iVersion */ + 0, /* xCreate */ + explainConnect, /* xConnect */ + explainBestIndex, /* xBestIndex */ + explainDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + explainOpen, /* xOpen - open a cursor */ + explainClose, /* xClose - close a cursor */ + explainFilter, /* xFilter - configure scan constraints */ + explainNext, /* xNext - advance a cursor */ + explainEof, /* xEof - check for end of scan */ + explainColumn, /* xColumn - read data */ + explainRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0, /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +int sqlite3ExplainVtabInit(sqlite3 *db){ + int rc = SQLITE_OK; +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "explain", &explainModule, 0); +#endif + return rc; +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_explain_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3ExplainVtabInit(db); +#endif + return rc; +} diff --git a/ext/misc/fileio.c b/ext/misc/fileio.c new file mode 100644 index 0000000..7cdbd59 --- /dev/null +++ b/ext/misc/fileio.c @@ -0,0 +1,1030 @@ +/* +** 2014-06-13 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements SQL functions readfile() and +** writefile(), and eponymous virtual type "fsdir". +** +** WRITEFILE(FILE, DATA [, MODE [, MTIME]]): +** +** If neither of the optional arguments is present, then this UDF +** function writes blob DATA to file FILE. If successful, the number +** of bytes written is returned. If an error occurs, NULL is returned. +** +** If the first option argument - MODE - is present, then it must +** be passed an integer value that corresponds to a POSIX mode +** value (file type + permissions, as returned in the stat.st_mode +** field by the stat() system call). Three types of files may +** be written/created: +** +** regular files: (mode & 0170000)==0100000 +** symbolic links: (mode & 0170000)==0120000 +** directories: (mode & 0170000)==0040000 +** +** For a directory, the DATA is ignored. For a symbolic link, it is +** interpreted as text and used as the target of the link. For a +** regular file, it is interpreted as a blob and written into the +** named file. Regardless of the type of file, its permissions are +** set to (mode & 0777) before returning. +** +** If the optional MTIME argument is present, then it is interpreted +** as an integer - the number of seconds since the unix epoch. The +** modification-time of the target file is set to this value before +** returning. +** +** If three or more arguments are passed to this function and an +** error is encountered, an exception is raised. +** +** READFILE(FILE): +** +** Read and return the contents of file FILE (type blob) from disk. +** +** FSDIR: +** +** Used as follows: +** +** SELECT * FROM fsdir($path [, $dir]); +** +** Parameter $path is an absolute or relative pathname. If the file that it +** refers to does not exist, it is an error. If the path refers to a regular +** file or symbolic link, it returns a single row. Or, if the path refers +** to a directory, it returns one row for the directory, and one row for each +** file within the hierarchy rooted at $path. +** +** Each row has the following columns: +** +** name: Path to file or directory (text value). +** mode: Value of stat.st_mode for directory entry (an integer). +** mtime: Value of stat.st_mtime for directory entry (an integer). +** data: For a regular file, a blob containing the file data. For a +** symlink, a text value containing the text of the link. For a +** directory, NULL. +** +** If a non-NULL value is specified for the optional $dir parameter and +** $path is a relative path, then $path is interpreted relative to $dir. +** And the paths returned in the "name" column of the table are also +** relative to directory $dir. +** +** Notes on building this extension for Windows: +** Unless linked statically with the SQLite library, a preprocessor +** symbol, FILEIO_WIN32_DLL, must be #define'd to create a stand-alone +** DLL form of this extension for WIN32. See its use below for details. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdio.h> +#include <string.h> +#include <assert.h> + +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#if !defined(_WIN32) && !defined(WIN32) +# include <unistd.h> +# include <dirent.h> +# include <utime.h> +# include <sys/time.h> +#else +# include "windows.h" +# include <io.h> +# include <direct.h> +# include "test_windirent.h" +# define dirent DIRENT +# ifndef chmod +# define chmod _chmod +# endif +# ifndef stat +# define stat _stat +# endif +# define mkdir(path,mode) _mkdir(path) +# define lstat(path,buf) stat(path,buf) +#endif +#include <time.h> +#include <errno.h> + + +/* +** Structure of the fsdir() table-valued function +*/ + /* 0 1 2 3 4 5 */ +#define FSDIR_SCHEMA "(name,mode,mtime,data,path HIDDEN,dir HIDDEN)" +#define FSDIR_COLUMN_NAME 0 /* Name of the file */ +#define FSDIR_COLUMN_MODE 1 /* Access mode */ +#define FSDIR_COLUMN_MTIME 2 /* Last modification time */ +#define FSDIR_COLUMN_DATA 3 /* File content */ +#define FSDIR_COLUMN_PATH 4 /* Path to top of search */ +#define FSDIR_COLUMN_DIR 5 /* Path is relative to this directory */ + + +/* +** Set the result stored by context ctx to a blob containing the +** contents of file zName. Or, leave the result unchanged (NULL) +** if the file does not exist or is unreadable. +** +** If the file exceeds the SQLite blob size limit, through an +** SQLITE_TOOBIG error. +** +** Throw an SQLITE_IOERR if there are difficulties pulling the file +** off of disk. +*/ +static void readFileContents(sqlite3_context *ctx, const char *zName){ + FILE *in; + sqlite3_int64 nIn; + void *pBuf; + sqlite3 *db; + int mxBlob; + + in = fopen(zName, "rb"); + if( in==0 ){ + /* File does not exist or is unreadable. Leave the result set to NULL. */ + return; + } + fseek(in, 0, SEEK_END); + nIn = ftell(in); + rewind(in); + db = sqlite3_context_db_handle(ctx); + mxBlob = sqlite3_limit(db, SQLITE_LIMIT_LENGTH, -1); + if( nIn>mxBlob ){ + sqlite3_result_error_code(ctx, SQLITE_TOOBIG); + fclose(in); + return; + } + pBuf = sqlite3_malloc64( nIn ? nIn : 1 ); + if( pBuf==0 ){ + sqlite3_result_error_nomem(ctx); + fclose(in); + return; + } + if( nIn==(sqlite3_int64)fread(pBuf, 1, (size_t)nIn, in) ){ + sqlite3_result_blob64(ctx, pBuf, nIn, sqlite3_free); + }else{ + sqlite3_result_error_code(ctx, SQLITE_IOERR); + sqlite3_free(pBuf); + } + fclose(in); +} + +/* +** Implementation of the "readfile(X)" SQL function. The entire content +** of the file named X is read and returned as a BLOB. NULL is returned +** if the file does not exist or is unreadable. +*/ +static void readfileFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zName; + (void)(argc); /* Unused parameter */ + zName = (const char*)sqlite3_value_text(argv[0]); + if( zName==0 ) return; + readFileContents(context, zName); +} + +/* +** Set the error message contained in context ctx to the results of +** vprintf(zFmt, ...). +*/ +static void ctxErrorMsg(sqlite3_context *ctx, const char *zFmt, ...){ + char *zMsg = 0; + va_list ap; + va_start(ap, zFmt); + zMsg = sqlite3_vmprintf(zFmt, ap); + sqlite3_result_error(ctx, zMsg, -1); + sqlite3_free(zMsg); + va_end(ap); +} + +#if defined(_WIN32) +/* +** This function is designed to convert a Win32 FILETIME structure into the +** number of seconds since the Unix Epoch (1970-01-01 00:00:00 UTC). +*/ +static sqlite3_uint64 fileTimeToUnixTime( + LPFILETIME pFileTime +){ + SYSTEMTIME epochSystemTime; + ULARGE_INTEGER epochIntervals; + FILETIME epochFileTime; + ULARGE_INTEGER fileIntervals; + + memset(&epochSystemTime, 0, sizeof(SYSTEMTIME)); + epochSystemTime.wYear = 1970; + epochSystemTime.wMonth = 1; + epochSystemTime.wDay = 1; + SystemTimeToFileTime(&epochSystemTime, &epochFileTime); + epochIntervals.LowPart = epochFileTime.dwLowDateTime; + epochIntervals.HighPart = epochFileTime.dwHighDateTime; + + fileIntervals.LowPart = pFileTime->dwLowDateTime; + fileIntervals.HighPart = pFileTime->dwHighDateTime; + + return (fileIntervals.QuadPart - epochIntervals.QuadPart) / 10000000; +} + + +#if defined(FILEIO_WIN32_DLL) && (defined(_WIN32) || defined(WIN32)) +# /* To allow a standalone DLL, use this next replacement function: */ +# undef sqlite3_win32_utf8_to_unicode +# define sqlite3_win32_utf8_to_unicode utf8_to_utf16 +# +LPWSTR utf8_to_utf16(const char *z){ + int nAllot = MultiByteToWideChar(CP_UTF8, 0, z, -1, NULL, 0); + LPWSTR rv = sqlite3_malloc(nAllot * sizeof(WCHAR)); + if( rv!=0 && 0 < MultiByteToWideChar(CP_UTF8, 0, z, -1, rv, nAllot) ) + return rv; + sqlite3_free(rv); + return 0; +} +#endif + +/* +** This function attempts to normalize the time values found in the stat() +** buffer to UTC. This is necessary on Win32, where the runtime library +** appears to return these values as local times. +*/ +static void statTimesToUtc( + const char *zPath, + struct stat *pStatBuf +){ + HANDLE hFindFile; + WIN32_FIND_DATAW fd; + LPWSTR zUnicodeName; + extern LPWSTR sqlite3_win32_utf8_to_unicode(const char*); + zUnicodeName = sqlite3_win32_utf8_to_unicode(zPath); + if( zUnicodeName ){ + memset(&fd, 0, sizeof(WIN32_FIND_DATAW)); + hFindFile = FindFirstFileW(zUnicodeName, &fd); + if( hFindFile!=NULL ){ + pStatBuf->st_ctime = (time_t)fileTimeToUnixTime(&fd.ftCreationTime); + pStatBuf->st_atime = (time_t)fileTimeToUnixTime(&fd.ftLastAccessTime); + pStatBuf->st_mtime = (time_t)fileTimeToUnixTime(&fd.ftLastWriteTime); + FindClose(hFindFile); + } + sqlite3_free(zUnicodeName); + } +} +#endif + +/* +** This function is used in place of stat(). On Windows, special handling +** is required in order for the included time to be returned as UTC. On all +** other systems, this function simply calls stat(). +*/ +static int fileStat( + const char *zPath, + struct stat *pStatBuf +){ +#if defined(_WIN32) + int rc = stat(zPath, pStatBuf); + if( rc==0 ) statTimesToUtc(zPath, pStatBuf); + return rc; +#else + return stat(zPath, pStatBuf); +#endif +} + +/* +** This function is used in place of lstat(). On Windows, special handling +** is required in order for the included time to be returned as UTC. On all +** other systems, this function simply calls lstat(). +*/ +static int fileLinkStat( + const char *zPath, + struct stat *pStatBuf +){ +#if defined(_WIN32) + int rc = lstat(zPath, pStatBuf); + if( rc==0 ) statTimesToUtc(zPath, pStatBuf); + return rc; +#else + return lstat(zPath, pStatBuf); +#endif +} + +/* +** Argument zFile is the name of a file that will be created and/or written +** by SQL function writefile(). This function ensures that the directory +** zFile will be written to exists, creating it if required. The permissions +** for any path components created by this function are set in accordance +** with the current umask. +** +** If an OOM condition is encountered, SQLITE_NOMEM is returned. Otherwise, +** SQLITE_OK is returned if the directory is successfully created, or +** SQLITE_ERROR otherwise. +*/ +static int makeDirectory( + const char *zFile +){ + char *zCopy = sqlite3_mprintf("%s", zFile); + int rc = SQLITE_OK; + + if( zCopy==0 ){ + rc = SQLITE_NOMEM; + }else{ + int nCopy = (int)strlen(zCopy); + int i = 1; + + while( rc==SQLITE_OK ){ + struct stat sStat; + int rc2; + + for(; zCopy[i]!='/' && i<nCopy; i++); + if( i==nCopy ) break; + zCopy[i] = '\0'; + + rc2 = fileStat(zCopy, &sStat); + if( rc2!=0 ){ + if( mkdir(zCopy, 0777) ) rc = SQLITE_ERROR; + }else{ + if( !S_ISDIR(sStat.st_mode) ) rc = SQLITE_ERROR; + } + zCopy[i] = '/'; + i++; + } + + sqlite3_free(zCopy); + } + + return rc; +} + +/* +** This function does the work for the writefile() UDF. Refer to +** header comments at the top of this file for details. +*/ +static int writeFile( + sqlite3_context *pCtx, /* Context to return bytes written in */ + const char *zFile, /* File to write */ + sqlite3_value *pData, /* Data to write */ + mode_t mode, /* MODE parameter passed to writefile() */ + sqlite3_int64 mtime /* MTIME parameter (or -1 to not set time) */ +){ + if( zFile==0 ) return 1; +#if !defined(_WIN32) && !defined(WIN32) + if( S_ISLNK(mode) ){ + const char *zTo = (const char*)sqlite3_value_text(pData); + if( zTo==0 || symlink(zTo, zFile)<0 ) return 1; + }else +#endif + { + if( S_ISDIR(mode) ){ + if( mkdir(zFile, mode) ){ + /* The mkdir() call to create the directory failed. This might not + ** be an error though - if there is already a directory at the same + ** path and either the permissions already match or can be changed + ** to do so using chmod(), it is not an error. */ + struct stat sStat; + if( errno!=EEXIST + || 0!=fileStat(zFile, &sStat) + || !S_ISDIR(sStat.st_mode) + || ((sStat.st_mode&0777)!=(mode&0777) && 0!=chmod(zFile, mode&0777)) + ){ + return 1; + } + } + }else{ + sqlite3_int64 nWrite = 0; + const char *z; + int rc = 0; + FILE *out = fopen(zFile, "wb"); + if( out==0 ) return 1; + z = (const char*)sqlite3_value_blob(pData); + if( z ){ + sqlite3_int64 n = fwrite(z, 1, sqlite3_value_bytes(pData), out); + nWrite = sqlite3_value_bytes(pData); + if( nWrite!=n ){ + rc = 1; + } + } + fclose(out); + if( rc==0 && mode && chmod(zFile, mode & 0777) ){ + rc = 1; + } + if( rc ) return 2; + sqlite3_result_int64(pCtx, nWrite); + } + } + + if( mtime>=0 ){ +#if defined(_WIN32) +#if !SQLITE_OS_WINRT + /* Windows */ + FILETIME lastAccess; + FILETIME lastWrite; + SYSTEMTIME currentTime; + LONGLONG intervals; + HANDLE hFile; + LPWSTR zUnicodeName; + extern LPWSTR sqlite3_win32_utf8_to_unicode(const char*); + + GetSystemTime(¤tTime); + SystemTimeToFileTime(¤tTime, &lastAccess); + intervals = Int32x32To64(mtime, 10000000) + 116444736000000000; + lastWrite.dwLowDateTime = (DWORD)intervals; + lastWrite.dwHighDateTime = intervals >> 32; + zUnicodeName = sqlite3_win32_utf8_to_unicode(zFile); + if( zUnicodeName==0 ){ + return 1; + } + hFile = CreateFileW( + zUnicodeName, FILE_WRITE_ATTRIBUTES, 0, NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS, NULL + ); + sqlite3_free(zUnicodeName); + if( hFile!=INVALID_HANDLE_VALUE ){ + BOOL bResult = SetFileTime(hFile, NULL, &lastAccess, &lastWrite); + CloseHandle(hFile); + return !bResult; + }else{ + return 1; + } +#endif +#elif defined(AT_FDCWD) && 0 /* utimensat() is not universally available */ + /* Recent unix */ + struct timespec times[2]; + times[0].tv_nsec = times[1].tv_nsec = 0; + times[0].tv_sec = time(0); + times[1].tv_sec = mtime; + if( utimensat(AT_FDCWD, zFile, times, AT_SYMLINK_NOFOLLOW) ){ + return 1; + } +#else + /* Legacy unix */ + struct timeval times[2]; + times[0].tv_usec = times[1].tv_usec = 0; + times[0].tv_sec = time(0); + times[1].tv_sec = mtime; + if( utimes(zFile, times) ){ + return 1; + } +#endif + } + + return 0; +} + +/* +** Implementation of the "writefile(W,X[,Y[,Z]]])" SQL function. +** Refer to header comments at the top of this file for details. +*/ +static void writefileFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zFile; + mode_t mode = 0; + int res; + sqlite3_int64 mtime = -1; + + if( argc<2 || argc>4 ){ + sqlite3_result_error(context, + "wrong number of arguments to function writefile()", -1 + ); + return; + } + + zFile = (const char*)sqlite3_value_text(argv[0]); + if( zFile==0 ) return; + if( argc>=3 ){ + mode = (mode_t)sqlite3_value_int(argv[2]); + } + if( argc==4 ){ + mtime = sqlite3_value_int64(argv[3]); + } + + res = writeFile(context, zFile, argv[1], mode, mtime); + if( res==1 && errno==ENOENT ){ + if( makeDirectory(zFile)==SQLITE_OK ){ + res = writeFile(context, zFile, argv[1], mode, mtime); + } + } + + if( argc>2 && res!=0 ){ + if( S_ISLNK(mode) ){ + ctxErrorMsg(context, "failed to create symlink: %s", zFile); + }else if( S_ISDIR(mode) ){ + ctxErrorMsg(context, "failed to create directory: %s", zFile); + }else{ + ctxErrorMsg(context, "failed to write file: %s", zFile); + } + } +} + +/* +** SQL function: lsmode(MODE) +** +** Given a numberic st_mode from stat(), convert it into a human-readable +** text string in the style of "ls -l". +*/ +static void lsModeFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int i; + int iMode = sqlite3_value_int(argv[0]); + char z[16]; + (void)argc; + if( S_ISLNK(iMode) ){ + z[0] = 'l'; + }else if( S_ISREG(iMode) ){ + z[0] = '-'; + }else if( S_ISDIR(iMode) ){ + z[0] = 'd'; + }else{ + z[0] = '?'; + } + for(i=0; i<3; i++){ + int m = (iMode >> ((2-i)*3)); + char *a = &z[1 + i*3]; + a[0] = (m & 0x4) ? 'r' : '-'; + a[1] = (m & 0x2) ? 'w' : '-'; + a[2] = (m & 0x1) ? 'x' : '-'; + } + z[10] = '\0'; + sqlite3_result_text(context, z, -1, SQLITE_TRANSIENT); +} + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Cursor type for recursively iterating through a directory structure. +*/ +typedef struct fsdir_cursor fsdir_cursor; +typedef struct FsdirLevel FsdirLevel; + +struct FsdirLevel { + DIR *pDir; /* From opendir() */ + char *zDir; /* Name of directory (nul-terminated) */ +}; + +struct fsdir_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + + int nLvl; /* Number of entries in aLvl[] array */ + int iLvl; /* Index of current entry */ + FsdirLevel *aLvl; /* Hierarchy of directories being traversed */ + + const char *zBase; + int nBase; + + struct stat sStat; /* Current lstat() results */ + char *zPath; /* Path to current entry */ + sqlite3_int64 iRowid; /* Current rowid */ +}; + +typedef struct fsdir_tab fsdir_tab; +struct fsdir_tab { + sqlite3_vtab base; /* Base class - must be first */ +}; + +/* +** Construct a new fsdir virtual table object. +*/ +static int fsdirConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + fsdir_tab *pNew = 0; + int rc; + (void)pAux; + (void)argc; + (void)argv; + (void)pzErr; + rc = sqlite3_declare_vtab(db, "CREATE TABLE x" FSDIR_SCHEMA); + if( rc==SQLITE_OK ){ + pNew = (fsdir_tab*)sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY); + } + *ppVtab = (sqlite3_vtab*)pNew; + return rc; +} + +/* +** This method is the destructor for fsdir vtab objects. +*/ +static int fsdirDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new fsdir_cursor object. +*/ +static int fsdirOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + fsdir_cursor *pCur; + (void)p; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->iLvl = -1; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Reset a cursor back to the state it was in when first returned +** by fsdirOpen(). +*/ +static void fsdirResetCursor(fsdir_cursor *pCur){ + int i; + for(i=0; i<=pCur->iLvl; i++){ + FsdirLevel *pLvl = &pCur->aLvl[i]; + if( pLvl->pDir ) closedir(pLvl->pDir); + sqlite3_free(pLvl->zDir); + } + sqlite3_free(pCur->zPath); + sqlite3_free(pCur->aLvl); + pCur->aLvl = 0; + pCur->zPath = 0; + pCur->zBase = 0; + pCur->nBase = 0; + pCur->nLvl = 0; + pCur->iLvl = -1; + pCur->iRowid = 1; +} + +/* +** Destructor for an fsdir_cursor. +*/ +static int fsdirClose(sqlite3_vtab_cursor *cur){ + fsdir_cursor *pCur = (fsdir_cursor*)cur; + + fsdirResetCursor(pCur); + sqlite3_free(pCur); + return SQLITE_OK; +} + +/* +** Set the error message for the virtual table associated with cursor +** pCur to the results of vprintf(zFmt, ...). +*/ +static void fsdirSetErrmsg(fsdir_cursor *pCur, const char *zFmt, ...){ + va_list ap; + va_start(ap, zFmt); + pCur->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap); + va_end(ap); +} + + +/* +** Advance an fsdir_cursor to its next row of output. +*/ +static int fsdirNext(sqlite3_vtab_cursor *cur){ + fsdir_cursor *pCur = (fsdir_cursor*)cur; + mode_t m = pCur->sStat.st_mode; + + pCur->iRowid++; + if( S_ISDIR(m) ){ + /* Descend into this directory */ + int iNew = pCur->iLvl + 1; + FsdirLevel *pLvl; + if( iNew>=pCur->nLvl ){ + int nNew = iNew+1; + sqlite3_int64 nByte = nNew*sizeof(FsdirLevel); + FsdirLevel *aNew = (FsdirLevel*)sqlite3_realloc64(pCur->aLvl, nByte); + if( aNew==0 ) return SQLITE_NOMEM; + memset(&aNew[pCur->nLvl], 0, sizeof(FsdirLevel)*(nNew-pCur->nLvl)); + pCur->aLvl = aNew; + pCur->nLvl = nNew; + } + pCur->iLvl = iNew; + pLvl = &pCur->aLvl[iNew]; + + pLvl->zDir = pCur->zPath; + pCur->zPath = 0; + pLvl->pDir = opendir(pLvl->zDir); + if( pLvl->pDir==0 ){ + fsdirSetErrmsg(pCur, "cannot read directory: %s", pCur->zPath); + return SQLITE_ERROR; + } + } + + while( pCur->iLvl>=0 ){ + FsdirLevel *pLvl = &pCur->aLvl[pCur->iLvl]; + struct dirent *pEntry = readdir(pLvl->pDir); + if( pEntry ){ + if( pEntry->d_name[0]=='.' ){ + if( pEntry->d_name[1]=='.' && pEntry->d_name[2]=='\0' ) continue; + if( pEntry->d_name[1]=='\0' ) continue; + } + sqlite3_free(pCur->zPath); + pCur->zPath = sqlite3_mprintf("%s/%s", pLvl->zDir, pEntry->d_name); + if( pCur->zPath==0 ) return SQLITE_NOMEM; + if( fileLinkStat(pCur->zPath, &pCur->sStat) ){ + fsdirSetErrmsg(pCur, "cannot stat file: %s", pCur->zPath); + return SQLITE_ERROR; + } + return SQLITE_OK; + } + closedir(pLvl->pDir); + sqlite3_free(pLvl->zDir); + pLvl->pDir = 0; + pLvl->zDir = 0; + pCur->iLvl--; + } + + /* EOF */ + sqlite3_free(pCur->zPath); + pCur->zPath = 0; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the series_cursor +** is currently pointing. +*/ +static int fsdirColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + fsdir_cursor *pCur = (fsdir_cursor*)cur; + switch( i ){ + case FSDIR_COLUMN_NAME: { + sqlite3_result_text(ctx, &pCur->zPath[pCur->nBase], -1, SQLITE_TRANSIENT); + break; + } + + case FSDIR_COLUMN_MODE: + sqlite3_result_int64(ctx, pCur->sStat.st_mode); + break; + + case FSDIR_COLUMN_MTIME: + sqlite3_result_int64(ctx, pCur->sStat.st_mtime); + break; + + case FSDIR_COLUMN_DATA: { + mode_t m = pCur->sStat.st_mode; + if( S_ISDIR(m) ){ + sqlite3_result_null(ctx); +#if !defined(_WIN32) && !defined(WIN32) + }else if( S_ISLNK(m) ){ + char aStatic[64]; + char *aBuf = aStatic; + sqlite3_int64 nBuf = 64; + int n; + + while( 1 ){ + n = readlink(pCur->zPath, aBuf, nBuf); + if( n<nBuf ) break; + if( aBuf!=aStatic ) sqlite3_free(aBuf); + nBuf = nBuf*2; + aBuf = sqlite3_malloc64(nBuf); + if( aBuf==0 ){ + sqlite3_result_error_nomem(ctx); + return SQLITE_NOMEM; + } + } + + sqlite3_result_text(ctx, aBuf, n, SQLITE_TRANSIENT); + if( aBuf!=aStatic ) sqlite3_free(aBuf); +#endif + }else{ + readFileContents(ctx, pCur->zPath); + } + } + case FSDIR_COLUMN_PATH: + default: { + /* The FSDIR_COLUMN_PATH and FSDIR_COLUMN_DIR are input parameters. + ** always return their values as NULL */ + break; + } + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** first row returned is assigned rowid value 1, and each subsequent +** row a value 1 more than that of the previous. +*/ +static int fsdirRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + fsdir_cursor *pCur = (fsdir_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int fsdirEof(sqlite3_vtab_cursor *cur){ + fsdir_cursor *pCur = (fsdir_cursor*)cur; + return (pCur->zPath==0); +} + +/* +** xFilter callback. +** +** idxNum==1 PATH parameter only +** idxNum==2 Both PATH and DIR supplied +*/ +static int fsdirFilter( + sqlite3_vtab_cursor *cur, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + const char *zDir = 0; + fsdir_cursor *pCur = (fsdir_cursor*)cur; + (void)idxStr; + fsdirResetCursor(pCur); + + if( idxNum==0 ){ + fsdirSetErrmsg(pCur, "table function fsdir requires an argument"); + return SQLITE_ERROR; + } + + assert( argc==idxNum && (argc==1 || argc==2) ); + zDir = (const char*)sqlite3_value_text(argv[0]); + if( zDir==0 ){ + fsdirSetErrmsg(pCur, "table function fsdir requires a non-NULL argument"); + return SQLITE_ERROR; + } + if( argc==2 ){ + pCur->zBase = (const char*)sqlite3_value_text(argv[1]); + } + if( pCur->zBase ){ + pCur->nBase = (int)strlen(pCur->zBase)+1; + pCur->zPath = sqlite3_mprintf("%s/%s", pCur->zBase, zDir); + }else{ + pCur->zPath = sqlite3_mprintf("%s", zDir); + } + + if( pCur->zPath==0 ){ + return SQLITE_NOMEM; + } + if( fileLinkStat(pCur->zPath, &pCur->sStat) ){ + fsdirSetErrmsg(pCur, "cannot stat file: %s", pCur->zPath); + return SQLITE_ERROR; + } + + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the generate_series virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +** +** In this implementation idxNum is used to represent the +** query plan. idxStr is unused. +** +** The query plan is represented by values of idxNum: +** +** (1) The path value is supplied by argv[0] +** (2) Path is in argv[0] and dir is in argv[1] +*/ +static int fsdirBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; /* Loop over constraints */ + int idxPath = -1; /* Index in pIdxInfo->aConstraint of PATH= */ + int idxDir = -1; /* Index in pIdxInfo->aConstraint of DIR= */ + int seenPath = 0; /* True if an unusable PATH= constraint is seen */ + int seenDir = 0; /* True if an unusable DIR= constraint is seen */ + const struct sqlite3_index_constraint *pConstraint; + + (void)tab; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue; + switch( pConstraint->iColumn ){ + case FSDIR_COLUMN_PATH: { + if( pConstraint->usable ){ + idxPath = i; + seenPath = 0; + }else if( idxPath<0 ){ + seenPath = 1; + } + break; + } + case FSDIR_COLUMN_DIR: { + if( pConstraint->usable ){ + idxDir = i; + seenDir = 0; + }else if( idxDir<0 ){ + seenDir = 1; + } + break; + } + } + } + if( seenPath || seenDir ){ + /* If input parameters are unusable, disallow this plan */ + return SQLITE_CONSTRAINT; + } + + if( idxPath<0 ){ + pIdxInfo->idxNum = 0; + /* The pIdxInfo->estimatedCost should have been initialized to a huge + ** number. Leave it unchanged. */ + pIdxInfo->estimatedRows = 0x7fffffff; + }else{ + pIdxInfo->aConstraintUsage[idxPath].omit = 1; + pIdxInfo->aConstraintUsage[idxPath].argvIndex = 1; + if( idxDir>=0 ){ + pIdxInfo->aConstraintUsage[idxDir].omit = 1; + pIdxInfo->aConstraintUsage[idxDir].argvIndex = 2; + pIdxInfo->idxNum = 2; + pIdxInfo->estimatedCost = 10.0; + }else{ + pIdxInfo->idxNum = 1; + pIdxInfo->estimatedCost = 100.0; + } + } + + return SQLITE_OK; +} + +/* +** Register the "fsdir" virtual table. +*/ +static int fsdirRegister(sqlite3 *db){ + static sqlite3_module fsdirModule = { + 0, /* iVersion */ + 0, /* xCreate */ + fsdirConnect, /* xConnect */ + fsdirBestIndex, /* xBestIndex */ + fsdirDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + fsdirOpen, /* xOpen - open a cursor */ + fsdirClose, /* xClose - close a cursor */ + fsdirFilter, /* xFilter - configure scan constraints */ + fsdirNext, /* xNext - advance a cursor */ + fsdirEof, /* xEof - check for end of scan */ + fsdirColumn, /* xColumn - read data */ + fsdirRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0, /* xShadowName */ + }; + + int rc = sqlite3_create_module(db, "fsdir", &fsdirModule, 0); + return rc; +} +#else /* SQLITE_OMIT_VIRTUALTABLE */ +# define fsdirRegister(x) SQLITE_OK +#endif + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_fileio_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "readfile", 1, + SQLITE_UTF8|SQLITE_DIRECTONLY, 0, + readfileFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "writefile", -1, + SQLITE_UTF8|SQLITE_DIRECTONLY, 0, + writefileFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "lsmode", 1, SQLITE_UTF8, 0, + lsModeFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = fsdirRegister(db); + } + return rc; +} + +#if defined(FILEIO_WIN32_DLL) && (defined(_WIN32) || defined(WIN32)) +/* To allow a standalone DLL, make test_windirent.c use the same + * redefined SQLite API calls as the above extension code does. + * Just pull in this .c to accomplish this. As a beneficial side + * effect, this extension becomes a single translation unit. */ +# include "test_windirent.c" +#endif diff --git a/ext/misc/fossildelta.c b/ext/misc/fossildelta.c new file mode 100644 index 0000000..6a597e0 --- /dev/null +++ b/ext/misc/fossildelta.c @@ -0,0 +1,1092 @@ +/* +** 2019-02-19 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements the delta functions used by the RBU +** extension. Three scalar functions and one table-valued function are +** implemented here: +** +** delta_apply(X,D) -- apply delta D to file X and return the result +** delta_create(X,Y) -- compute and return a delta that carries X into Y +** delta_output_size(D) -- blob size in bytes output from applying delta D +** delta_parse(D) -- returns rows describing delta D +** +** The delta format is the Fossil delta format, described in a comment +** on the delete_create() function implementation below, and also at +** +** https://www.fossil-scm.org/fossil/doc/trunk/www/delta_format.wiki +** +** This delta format is used by the RBU extension, which is the main +** reason that these routines are included in the extension library. +** RBU does not use this extension directly. Rather, this extension is +** provided as a convenience to developers who want to analyze RBU files +** that contain deltas. +*/ +#include <string.h> +#include <assert.h> +#include <stdlib.h> +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 + +#ifndef SQLITE_AMALGAMATION +/* +** The "u32" type must be an unsigned 32-bit integer. Adjust this +*/ +typedef unsigned int u32; + +/* +** Must be a 16-bit value +*/ +typedef short int s16; +typedef unsigned short int u16; + +#endif /* SQLITE_AMALGAMATION */ + + +/* +** The width of a hash window in bytes. The algorithm only works if this +** is a power of 2. +*/ +#define NHASH 16 + +/* +** The current state of the rolling hash. +** +** z[] holds the values that have been hashed. z[] is a circular buffer. +** z[i] is the first entry and z[(i+NHASH-1)%NHASH] is the last entry of +** the window. +** +** Hash.a is the sum of all elements of hash.z[]. Hash.b is a weighted +** sum. Hash.b is z[i]*NHASH + z[i+1]*(NHASH-1) + ... + z[i+NHASH-1]*1. +** (Each index for z[] should be module NHASH, of course. The %NHASH operator +** is omitted in the prior expression for brevity.) +*/ +typedef struct hash hash; +struct hash { + u16 a, b; /* Hash values */ + u16 i; /* Start of the hash window */ + char z[NHASH]; /* The values that have been hashed */ +}; + +/* +** Initialize the rolling hash using the first NHASH characters of z[] +*/ +static void hash_init(hash *pHash, const char *z){ + u16 a, b, i; + a = b = z[0]; + for(i=1; i<NHASH; i++){ + a += z[i]; + b += a; + } + memcpy(pHash->z, z, NHASH); + pHash->a = a & 0xffff; + pHash->b = b & 0xffff; + pHash->i = 0; +} + +/* +** Advance the rolling hash by a single character "c" +*/ +static void hash_next(hash *pHash, int c){ + u16 old = pHash->z[pHash->i]; + pHash->z[pHash->i] = c; + pHash->i = (pHash->i+1)&(NHASH-1); + pHash->a = pHash->a - old + c; + pHash->b = pHash->b - NHASH*old + pHash->a; +} + +/* +** Return a 32-bit hash value +*/ +static u32 hash_32bit(hash *pHash){ + return (pHash->a & 0xffff) | (((u32)(pHash->b & 0xffff))<<16); +} + +/* +** Compute a hash on NHASH bytes. +** +** This routine is intended to be equivalent to: +** hash h; +** hash_init(&h, zInput); +** return hash_32bit(&h); +*/ +static u32 hash_once(const char *z){ + u16 a, b, i; + a = b = z[0]; + for(i=1; i<NHASH; i++){ + a += z[i]; + b += a; + } + return a | (((u32)b)<<16); +} + +/* +** Write an base-64 integer into the given buffer. +*/ +static void putInt(unsigned int v, char **pz){ + static const char zDigits[] = + "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"; + /* 123456789 123456789 123456789 123456789 123456789 123456789 123 */ + int i, j; + char zBuf[20]; + if( v==0 ){ + *(*pz)++ = '0'; + return; + } + for(i=0; v>0; i++, v>>=6){ + zBuf[i] = zDigits[v&0x3f]; + } + for(j=i-1; j>=0; j--){ + *(*pz)++ = zBuf[j]; + } +} + +/* +** Read bytes from *pz and convert them into a positive integer. When +** finished, leave *pz pointing to the first character past the end of +** the integer. The *pLen parameter holds the length of the string +** in *pz and is decremented once for each character in the integer. +*/ +static unsigned int deltaGetInt(const char **pz, int *pLen){ + static const signed char zValue[] = { + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, + 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, -1, -1, -1, -1, 36, + -1, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, + 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, -1, -1, -1, 63, -1, + }; + unsigned int v = 0; + int c; + unsigned char *z = (unsigned char*)*pz; + unsigned char *zStart = z; + while( (c = zValue[0x7f&*(z++)])>=0 ){ + v = (v<<6) + c; + } + z--; + *pLen -= z - zStart; + *pz = (char*)z; + return v; +} + +/* +** Return the number digits in the base-64 representation of a positive integer +*/ +static int digit_count(int v){ + unsigned int i, x; + for(i=1, x=64; v>=x; i++, x <<= 6){} + return i; +} + +#ifdef __GNUC__ +# define GCC_VERSION (__GNUC__*1000000+__GNUC_MINOR__*1000+__GNUC_PATCHLEVEL__) +#else +# define GCC_VERSION 0 +#endif + +/* +** Compute a 32-bit big-endian checksum on the N-byte buffer. If the +** buffer is not a multiple of 4 bytes length, compute the sum that would +** have occurred if the buffer was padded with zeros to the next multiple +** of four bytes. +*/ +static unsigned int checksum(const char *zIn, size_t N){ + static const int byteOrderTest = 1; + const unsigned char *z = (const unsigned char *)zIn; + const unsigned char *zEnd = (const unsigned char*)&zIn[N&~3]; + unsigned sum = 0; + assert( (z - (const unsigned char*)0)%4==0 ); /* Four-byte alignment */ + if( 0==*(char*)&byteOrderTest ){ + /* This is a big-endian machine */ + while( z<zEnd ){ + sum += *(unsigned*)z; + z += 4; + } + }else{ + /* A little-endian machine */ +#if GCC_VERSION>=4003000 + while( z<zEnd ){ + sum += __builtin_bswap32(*(unsigned*)z); + z += 4; + } +#elif defined(_MSC_VER) && _MSC_VER>=1300 + while( z<zEnd ){ + sum += _byteswap_ulong(*(unsigned*)z); + z += 4; + } +#else + unsigned sum0 = 0; + unsigned sum1 = 0; + unsigned sum2 = 0; + while(N >= 16){ + sum0 += ((unsigned)z[0] + z[4] + z[8] + z[12]); + sum1 += ((unsigned)z[1] + z[5] + z[9] + z[13]); + sum2 += ((unsigned)z[2] + z[6] + z[10]+ z[14]); + sum += ((unsigned)z[3] + z[7] + z[11]+ z[15]); + z += 16; + N -= 16; + } + while(N >= 4){ + sum0 += z[0]; + sum1 += z[1]; + sum2 += z[2]; + sum += z[3]; + z += 4; + N -= 4; + } + sum += (sum2 << 8) + (sum1 << 16) + (sum0 << 24); +#endif + } + switch(N&3){ + case 3: sum += (z[2] << 8); + case 2: sum += (z[1] << 16); + case 1: sum += (z[0] << 24); + default: ; + } + return sum; +} + +/* +** Create a new delta. +** +** The delta is written into a preallocated buffer, zDelta, which +** should be at least 60 bytes longer than the target file, zOut. +** The delta string will be NUL-terminated, but it might also contain +** embedded NUL characters if either the zSrc or zOut files are +** binary. This function returns the length of the delta string +** in bytes, excluding the final NUL terminator character. +** +** Output Format: +** +** The delta begins with a base64 number followed by a newline. This +** number is the number of bytes in the TARGET file. Thus, given a +** delta file z, a program can compute the size of the output file +** simply by reading the first line and decoding the base-64 number +** found there. The delta_output_size() routine does exactly this. +** +** After the initial size number, the delta consists of a series of +** literal text segments and commands to copy from the SOURCE file. +** A copy command looks like this: +** +** NNN@MMM, +** +** where NNN is the number of bytes to be copied and MMM is the offset +** into the source file of the first byte (both base-64). If NNN is 0 +** it means copy the rest of the input file. Literal text is like this: +** +** NNN:TTTTT +** +** where NNN is the number of bytes of text (base-64) and TTTTT is the text. +** +** The last term is of the form +** +** NNN; +** +** In this case, NNN is a 32-bit bigendian checksum of the output file +** that can be used to verify that the delta applied correctly. All +** numbers are in base-64. +** +** Pure text files generate a pure text delta. Binary files generate a +** delta that may contain some binary data. +** +** Algorithm: +** +** The encoder first builds a hash table to help it find matching +** patterns in the source file. 16-byte chunks of the source file +** sampled at evenly spaced intervals are used to populate the hash +** table. +** +** Next we begin scanning the target file using a sliding 16-byte +** window. The hash of the 16-byte window in the target is used to +** search for a matching section in the source file. When a match +** is found, a copy command is added to the delta. An effort is +** made to extend the matching section to regions that come before +** and after the 16-byte hash window. A copy command is only issued +** if the result would use less space that just quoting the text +** literally. Literal text is added to the delta for sections that +** do not match or which can not be encoded efficiently using copy +** commands. +*/ +static int delta_create( + const char *zSrc, /* The source or pattern file */ + unsigned int lenSrc, /* Length of the source file */ + const char *zOut, /* The target file */ + unsigned int lenOut, /* Length of the target file */ + char *zDelta /* Write the delta into this buffer */ +){ + int i, base; + char *zOrigDelta = zDelta; + hash h; + int nHash; /* Number of hash table entries */ + int *landmark; /* Primary hash table */ + int *collide; /* Collision chain */ + int lastRead = -1; /* Last byte of zSrc read by a COPY command */ + + /* Add the target file size to the beginning of the delta + */ + putInt(lenOut, &zDelta); + *(zDelta++) = '\n'; + + /* If the source file is very small, it means that we have no + ** chance of ever doing a copy command. Just output a single + ** literal segment for the entire target and exit. + */ + if( lenSrc<=NHASH ){ + putInt(lenOut, &zDelta); + *(zDelta++) = ':'; + memcpy(zDelta, zOut, lenOut); + zDelta += lenOut; + putInt(checksum(zOut, lenOut), &zDelta); + *(zDelta++) = ';'; + return zDelta - zOrigDelta; + } + + /* Compute the hash table used to locate matching sections in the + ** source file. + */ + nHash = lenSrc/NHASH; + collide = sqlite3_malloc64( (sqlite3_int64)nHash*2*sizeof(int) ); + memset(collide, -1, nHash*2*sizeof(int)); + landmark = &collide[nHash]; + for(i=0; i<lenSrc-NHASH; i+=NHASH){ + int hv = hash_once(&zSrc[i]) % nHash; + collide[i/NHASH] = landmark[hv]; + landmark[hv] = i/NHASH; + } + + /* Begin scanning the target file and generating copy commands and + ** literal sections of the delta. + */ + base = 0; /* We have already generated everything before zOut[base] */ + while( base+NHASH<lenOut ){ + int iSrc, iBlock; + unsigned int bestCnt, bestOfst=0, bestLitsz=0; + hash_init(&h, &zOut[base]); + i = 0; /* Trying to match a landmark against zOut[base+i] */ + bestCnt = 0; + while( 1 ){ + int hv; + int limit = 250; + + hv = hash_32bit(&h) % nHash; + iBlock = landmark[hv]; + while( iBlock>=0 && (limit--)>0 ){ + /* + ** The hash window has identified a potential match against + ** landmark block iBlock. But we need to investigate further. + ** + ** Look for a region in zOut that matches zSrc. Anchor the search + ** at zSrc[iSrc] and zOut[base+i]. Do not include anything prior to + ** zOut[base] or after zOut[outLen] nor anything after zSrc[srcLen]. + ** + ** Set cnt equal to the length of the match and set ofst so that + ** zSrc[ofst] is the first element of the match. litsz is the number + ** of characters between zOut[base] and the beginning of the match. + ** sz will be the overhead (in bytes) needed to encode the copy + ** command. Only generate copy command if the overhead of the + ** copy command is less than the amount of literal text to be copied. + */ + int cnt, ofst, litsz; + int j, k, x, y; + int sz; + int limitX; + + /* Beginning at iSrc, match forwards as far as we can. j counts + ** the number of characters that match */ + iSrc = iBlock*NHASH; + y = base+i; + limitX = ( lenSrc-iSrc <= lenOut-y ) ? lenSrc : iSrc + lenOut - y; + for(x=iSrc; x<limitX; x++, y++){ + if( zSrc[x]!=zOut[y] ) break; + } + j = x - iSrc - 1; + + /* Beginning at iSrc-1, match backwards as far as we can. k counts + ** the number of characters that match */ + for(k=1; k<iSrc && k<=i; k++){ + if( zSrc[iSrc-k]!=zOut[base+i-k] ) break; + } + k--; + + /* Compute the offset and size of the matching region */ + ofst = iSrc-k; + cnt = j+k+1; + litsz = i-k; /* Number of bytes of literal text before the copy */ + /* sz will hold the number of bytes needed to encode the "insert" + ** command and the copy command, not counting the "insert" text */ + sz = digit_count(i-k)+digit_count(cnt)+digit_count(ofst)+3; + if( cnt>=sz && cnt>bestCnt ){ + /* Remember this match only if it is the best so far and it + ** does not increase the file size */ + bestCnt = cnt; + bestOfst = iSrc-k; + bestLitsz = litsz; + } + + /* Check the next matching block */ + iBlock = collide[iBlock]; + } + + /* We have a copy command that does not cause the delta to be larger + ** than a literal insert. So add the copy command to the delta. + */ + if( bestCnt>0 ){ + if( bestLitsz>0 ){ + /* Add an insert command before the copy */ + putInt(bestLitsz,&zDelta); + *(zDelta++) = ':'; + memcpy(zDelta, &zOut[base], bestLitsz); + zDelta += bestLitsz; + base += bestLitsz; + } + base += bestCnt; + putInt(bestCnt, &zDelta); + *(zDelta++) = '@'; + putInt(bestOfst, &zDelta); + *(zDelta++) = ','; + if( bestOfst + bestCnt -1 > lastRead ){ + lastRead = bestOfst + bestCnt - 1; + } + bestCnt = 0; + break; + } + + /* If we reach this point, it means no match is found so far */ + if( base+i+NHASH>=lenOut ){ + /* We have reached the end of the file and have not found any + ** matches. Do an "insert" for everything that does not match */ + putInt(lenOut-base, &zDelta); + *(zDelta++) = ':'; + memcpy(zDelta, &zOut[base], lenOut-base); + zDelta += lenOut-base; + base = lenOut; + break; + } + + /* Advance the hash by one character. Keep looking for a match */ + hash_next(&h, zOut[base+i+NHASH]); + i++; + } + } + /* Output a final "insert" record to get all the text at the end of + ** the file that does not match anything in the source file. + */ + if( base<lenOut ){ + putInt(lenOut-base, &zDelta); + *(zDelta++) = ':'; + memcpy(zDelta, &zOut[base], lenOut-base); + zDelta += lenOut-base; + } + /* Output the final checksum record. */ + putInt(checksum(zOut, lenOut), &zDelta); + *(zDelta++) = ';'; + sqlite3_free(collide); + return zDelta - zOrigDelta; +} + +/* +** Return the size (in bytes) of the output from applying +** a delta. +** +** This routine is provided so that an procedure that is able +** to call delta_apply() can learn how much space is required +** for the output and hence allocate nor more space that is really +** needed. +*/ +static int delta_output_size(const char *zDelta, int lenDelta){ + int size; + size = deltaGetInt(&zDelta, &lenDelta); + if( *zDelta!='\n' ){ + /* ERROR: size integer not terminated by "\n" */ + return -1; + } + return size; +} + + +/* +** Apply a delta. +** +** The output buffer should be big enough to hold the whole output +** file and a NUL terminator at the end. The delta_output_size() +** routine will determine this size for you. +** +** The delta string should be null-terminated. But the delta string +** may contain embedded NUL characters (if the input and output are +** binary files) so we also have to pass in the length of the delta in +** the lenDelta parameter. +** +** This function returns the size of the output file in bytes (excluding +** the final NUL terminator character). Except, if the delta string is +** malformed or intended for use with a source file other than zSrc, +** then this routine returns -1. +** +** Refer to the delta_create() documentation above for a description +** of the delta file format. +*/ +static int delta_apply( + const char *zSrc, /* The source or pattern file */ + int lenSrc, /* Length of the source file */ + const char *zDelta, /* Delta to apply to the pattern */ + int lenDelta, /* Length of the delta */ + char *zOut /* Write the output into this preallocated buffer */ +){ + unsigned int limit; + unsigned int total = 0; +#ifdef FOSSIL_ENABLE_DELTA_CKSUM_TEST + char *zOrigOut = zOut; +#endif + + limit = deltaGetInt(&zDelta, &lenDelta); + if( *zDelta!='\n' ){ + /* ERROR: size integer not terminated by "\n" */ + return -1; + } + zDelta++; lenDelta--; + while( *zDelta && lenDelta>0 ){ + unsigned int cnt, ofst; + cnt = deltaGetInt(&zDelta, &lenDelta); + switch( zDelta[0] ){ + case '@': { + zDelta++; lenDelta--; + ofst = deltaGetInt(&zDelta, &lenDelta); + if( lenDelta>0 && zDelta[0]!=',' ){ + /* ERROR: copy command not terminated by ',' */ + return -1; + } + zDelta++; lenDelta--; + total += cnt; + if( total>limit ){ + /* ERROR: copy exceeds output file size */ + return -1; + } + if( ofst+cnt > lenSrc ){ + /* ERROR: copy extends past end of input */ + return -1; + } + memcpy(zOut, &zSrc[ofst], cnt); + zOut += cnt; + break; + } + case ':': { + zDelta++; lenDelta--; + total += cnt; + if( total>limit ){ + /* ERROR: insert command gives an output larger than predicted */ + return -1; + } + if( cnt>lenDelta ){ + /* ERROR: insert count exceeds size of delta */ + return -1; + } + memcpy(zOut, zDelta, cnt); + zOut += cnt; + zDelta += cnt; + lenDelta -= cnt; + break; + } + case ';': { + zDelta++; lenDelta--; + zOut[0] = 0; +#ifdef FOSSIL_ENABLE_DELTA_CKSUM_TEST + if( cnt!=checksum(zOrigOut, total) ){ + /* ERROR: bad checksum */ + return -1; + } +#endif + if( total!=limit ){ + /* ERROR: generated size does not match predicted size */ + return -1; + } + return total; + } + default: { + /* ERROR: unknown delta operator */ + return -1; + } + } + } + /* ERROR: unterminated delta */ + return -1; +} + +/* +** SQL functions: delta_create(X,Y) +** +** Return a delta for carrying X into Y. +*/ +static void deltaCreateFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *aOrig; int nOrig; /* old blob */ + const char *aNew; int nNew; /* new blob */ + char *aOut; int nOut; /* output delta */ + + assert( argc==2 ); + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + if( sqlite3_value_type(argv[1])==SQLITE_NULL ) return; + nOrig = sqlite3_value_bytes(argv[0]); + aOrig = (const char*)sqlite3_value_blob(argv[0]); + nNew = sqlite3_value_bytes(argv[1]); + aNew = (const char*)sqlite3_value_blob(argv[1]); + aOut = sqlite3_malloc64(nNew+70); + if( aOut==0 ){ + sqlite3_result_error_nomem(context); + }else{ + nOut = delta_create(aOrig, nOrig, aNew, nNew, aOut); + if( nOut<0 ){ + sqlite3_free(aOut); + sqlite3_result_error(context, "cannot create fossil delta", -1); + }else{ + sqlite3_result_blob(context, aOut, nOut, sqlite3_free); + } + } +} + +/* +** SQL functions: delta_apply(X,D) +** +** Return the result of applying delta D to input X. +*/ +static void deltaApplyFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *aOrig; int nOrig; /* The X input */ + const char *aDelta; int nDelta; /* The input delta (D) */ + char *aOut; int nOut, nOut2; /* The output */ + + assert( argc==2 ); + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + if( sqlite3_value_type(argv[1])==SQLITE_NULL ) return; + nOrig = sqlite3_value_bytes(argv[0]); + aOrig = (const char*)sqlite3_value_blob(argv[0]); + nDelta = sqlite3_value_bytes(argv[1]); + aDelta = (const char*)sqlite3_value_blob(argv[1]); + + /* Figure out the size of the output */ + nOut = delta_output_size(aDelta, nDelta); + if( nOut<0 ){ + sqlite3_result_error(context, "corrupt fossil delta", -1); + return; + } + aOut = sqlite3_malloc64((sqlite3_int64)nOut+1); + if( aOut==0 ){ + sqlite3_result_error_nomem(context); + }else{ + nOut2 = delta_apply(aOrig, nOrig, aDelta, nDelta, aOut); + if( nOut2!=nOut ){ + sqlite3_free(aOut); + sqlite3_result_error(context, "corrupt fossil delta", -1); + }else{ + sqlite3_result_blob(context, aOut, nOut, sqlite3_free); + } + } +} + + +/* +** SQL functions: delta_output_size(D) +** +** Return the size of the output that results from applying delta D. +*/ +static void deltaOutputSizeFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *aDelta; int nDelta; /* The input delta (D) */ + int nOut; /* Size of output */ + assert( argc==1 ); + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + nDelta = sqlite3_value_bytes(argv[0]); + aDelta = (const char*)sqlite3_value_blob(argv[0]); + + /* Figure out the size of the output */ + nOut = delta_output_size(aDelta, nDelta); + if( nOut<0 ){ + sqlite3_result_error(context, "corrupt fossil delta", -1); + return; + }else{ + sqlite3_result_int(context, nOut); + } +} + +/***************************************************************************** +** Table-valued SQL function: delta_parse(DELTA) +** +** Schema: +** +** CREATE TABLE delta_parse( +** op TEXT, +** a1 INT, +** a2 ANY, +** delta HIDDEN BLOB +** ); +** +** Given an input DELTA, this function parses the delta and returns +** rows for each entry in the delta. The op column has one of the +** values SIZE, COPY, INSERT, CHECKSUM, ERROR. +** +** Assuming no errors, the first row has op='SIZE'. a1 is the size of +** the output in bytes and a2 is NULL. +** +** After the initial SIZE row, there are zero or more 'COPY' and/or 'INSERT' +** rows. A COPY row means content is copied from the source into the +** output. Column a1 is the number of bytes to copy and a2 is the offset +** into source from which to begin copying. An INSERT row means to +** insert text into the output stream. Column a1 is the number of bytes +** to insert and column is a BLOB that contains the text to be inserted. +** +** The last row of a well-formed delta will have an op value of 'CHECKSUM'. +** The a1 column will be the value of the checksum and a2 will be NULL. +** +** If the input delta is not well-formed, then a row with an op value +** of 'ERROR' is returned. The a1 value of the ERROR row is the offset +** into the delta where the error was encountered and a2 is NULL. +*/ +typedef struct deltaparsevtab_vtab deltaparsevtab_vtab; +typedef struct deltaparsevtab_cursor deltaparsevtab_cursor; +struct deltaparsevtab_vtab { + sqlite3_vtab base; /* Base class - must be first */ + /* No additional information needed */ +}; +struct deltaparsevtab_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + char *aDelta; /* The delta being parsed */ + int nDelta; /* Number of bytes in the delta */ + int iCursor; /* Current cursor location */ + int eOp; /* Name of current operator */ + unsigned int a1, a2; /* Arguments to current operator */ + int iNext; /* Next cursor value */ +}; + +/* Operator names: +*/ +static const char *azOp[] = { + "SIZE", "COPY", "INSERT", "CHECKSUM", "ERROR", "EOF" +}; +#define DELTAPARSE_OP_SIZE 0 +#define DELTAPARSE_OP_COPY 1 +#define DELTAPARSE_OP_INSERT 2 +#define DELTAPARSE_OP_CHECKSUM 3 +#define DELTAPARSE_OP_ERROR 4 +#define DELTAPARSE_OP_EOF 5 + +/* +** The deltaparsevtabConnect() method is invoked to create a new +** deltaparse virtual table. +** +** Think of this routine as the constructor for deltaparsevtab_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the deltaparsevtab_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against the virtual table will look like. +*/ +static int deltaparsevtabConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + deltaparsevtab_vtab *pNew; + int rc; + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(op,a1,a2,delta HIDDEN)" + ); + /* For convenience, define symbolic names for the index to each column. */ +#define DELTAPARSEVTAB_OP 0 +#define DELTAPARSEVTAB_A1 1 +#define DELTAPARSEVTAB_A2 2 +#define DELTAPARSEVTAB_DELTA 3 + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc64( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + } + return rc; +} + +/* +** This method is the destructor for deltaparsevtab_vtab objects. +*/ +static int deltaparsevtabDisconnect(sqlite3_vtab *pVtab){ + deltaparsevtab_vtab *p = (deltaparsevtab_vtab*)pVtab; + sqlite3_free(p); + return SQLITE_OK; +} + +/* +** Constructor for a new deltaparsevtab_cursor object. +*/ +static int deltaparsevtabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + deltaparsevtab_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a deltaparsevtab_cursor. +*/ +static int deltaparsevtabClose(sqlite3_vtab_cursor *cur){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor*)cur; + sqlite3_free(pCur->aDelta); + sqlite3_free(pCur); + return SQLITE_OK; +} + + +/* +** Advance a deltaparsevtab_cursor to its next row of output. +*/ +static int deltaparsevtabNext(sqlite3_vtab_cursor *cur){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor*)cur; + const char *z; + int i = 0; + + pCur->iCursor = pCur->iNext; + z = pCur->aDelta + pCur->iCursor; + pCur->a1 = deltaGetInt(&z, &i); + switch( z[0] ){ + case '@': { + z++; + pCur->a2 = deltaGetInt(&z, &i); + pCur->eOp = DELTAPARSE_OP_COPY; + pCur->iNext = (int)(&z[1] - pCur->aDelta); + break; + } + case ':': { + z++; + pCur->a2 = (unsigned int)(z - pCur->aDelta); + pCur->eOp = DELTAPARSE_OP_INSERT; + pCur->iNext = (int)(&z[pCur->a1] - pCur->aDelta); + break; + } + case ';': { + pCur->eOp = DELTAPARSE_OP_CHECKSUM; + pCur->iNext = pCur->nDelta; + break; + } + default: { + if( pCur->iNext==pCur->nDelta ){ + pCur->eOp = DELTAPARSE_OP_EOF; + }else{ + pCur->eOp = DELTAPARSE_OP_ERROR; + pCur->iNext = pCur->nDelta; + } + break; + } + } + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the deltaparsevtab_cursor +** is currently pointing. +*/ +static int deltaparsevtabColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor*)cur; + switch( i ){ + case DELTAPARSEVTAB_OP: { + sqlite3_result_text(ctx, azOp[pCur->eOp], -1, SQLITE_STATIC); + break; + } + case DELTAPARSEVTAB_A1: { + sqlite3_result_int(ctx, pCur->a1); + break; + } + case DELTAPARSEVTAB_A2: { + if( pCur->eOp==DELTAPARSE_OP_COPY ){ + sqlite3_result_int(ctx, pCur->a2); + }else if( pCur->eOp==DELTAPARSE_OP_INSERT ){ + sqlite3_result_blob(ctx, pCur->aDelta+pCur->a2, pCur->a1, + SQLITE_TRANSIENT); + } + break; + } + case DELTAPARSEVTAB_DELTA: { + sqlite3_result_blob(ctx, pCur->aDelta, pCur->nDelta, SQLITE_TRANSIENT); + break; + } + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int deltaparsevtabRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor*)cur; + *pRowid = pCur->iCursor; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int deltaparsevtabEof(sqlite3_vtab_cursor *cur){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor*)cur; + return pCur->eOp==DELTAPARSE_OP_EOF; +} + +/* +** This method is called to "rewind" the deltaparsevtab_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to deltaparsevtabColumn() or deltaparsevtabRowid() or +** deltaparsevtabEof(). +*/ +static int deltaparsevtabFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + deltaparsevtab_cursor *pCur = (deltaparsevtab_cursor *)pVtabCursor; + const char *a; + int i = 0; + pCur->eOp = DELTAPARSE_OP_ERROR; + if( idxNum!=1 ){ + return SQLITE_OK; + } + pCur->nDelta = sqlite3_value_bytes(argv[0]); + a = (const char*)sqlite3_value_blob(argv[0]); + if( pCur->nDelta==0 || a==0 ){ + return SQLITE_OK; + } + pCur->aDelta = sqlite3_malloc64( pCur->nDelta+1 ); + if( pCur->aDelta==0 ){ + pCur->nDelta = 0; + return SQLITE_NOMEM; + } + memcpy(pCur->aDelta, a, pCur->nDelta); + pCur->aDelta[pCur->nDelta] = 0; + a = pCur->aDelta; + pCur->eOp = DELTAPARSE_OP_SIZE; + pCur->a1 = deltaGetInt(&a, &i); + if( a[0]!='\n' ){ + pCur->eOp = DELTAPARSE_OP_ERROR; + pCur->a1 = pCur->a2 = 0; + pCur->iNext = pCur->nDelta; + return SQLITE_OK; + } + a++; + pCur->iNext = (unsigned int)(a - pCur->aDelta); + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int deltaparsevtabBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; + for(i=0; i<pIdxInfo->nConstraint; i++){ + if( pIdxInfo->aConstraint[i].iColumn != DELTAPARSEVTAB_DELTA ) continue; + if( pIdxInfo->aConstraint[i].usable==0 ) continue; + if( pIdxInfo->aConstraint[i].op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + pIdxInfo->estimatedCost = (double)1; + pIdxInfo->estimatedRows = 10; + pIdxInfo->idxNum = 1; + return SQLITE_OK; + } + pIdxInfo->idxNum = 0; + pIdxInfo->estimatedCost = (double)0x7fffffff; + pIdxInfo->estimatedRows = 0x7fffffff; + return SQLITE_CONSTRAINT; +} + +/* +** This following structure defines all the methods for the +** virtual table. +*/ +static sqlite3_module deltaparsevtabModule = { + /* iVersion */ 0, + /* xCreate */ 0, + /* xConnect */ deltaparsevtabConnect, + /* xBestIndex */ deltaparsevtabBestIndex, + /* xDisconnect */ deltaparsevtabDisconnect, + /* xDestroy */ 0, + /* xOpen */ deltaparsevtabOpen, + /* xClose */ deltaparsevtabClose, + /* xFilter */ deltaparsevtabFilter, + /* xNext */ deltaparsevtabNext, + /* xEof */ deltaparsevtabEof, + /* xColumn */ deltaparsevtabColumn, + /* xRowid */ deltaparsevtabRowid, + /* xUpdate */ 0, + /* xBegin */ 0, + /* xSync */ 0, + /* xCommit */ 0, + /* xRollback */ 0, + /* xFindMethod */ 0, + /* xRename */ 0, + /* xSavepoint */ 0, + /* xRelease */ 0, + /* xRollbackTo */ 0, + /* xShadowName */ 0 +}; + + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_fossildelta_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + static const int enc = SQLITE_UTF8|SQLITE_INNOCUOUS; + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "delta_create", 2, enc, 0, + deltaCreateFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "delta_apply", 2, enc, 0, + deltaApplyFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "delta_output_size", 1, enc, 0, + deltaOutputSizeFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_module(db, "delta_parse", &deltaparsevtabModule, 0); + } + return rc; +} diff --git a/ext/misc/fuzzer.c b/ext/misc/fuzzer.c new file mode 100644 index 0000000..65d9d8d --- /dev/null +++ b/ext/misc/fuzzer.c @@ -0,0 +1,1187 @@ +/* +** 2011 March 24 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** Code for a demonstration virtual table that generates variations +** on an input word at increasing edit distances from the original. +** +** A fuzzer virtual table is created like this: +** +** CREATE VIRTUAL TABLE f USING fuzzer(<fuzzer-data-table>); +** +** When it is created, the new fuzzer table must be supplied with the +** name of a "fuzzer data table", which must reside in the same database +** file as the new fuzzer table. The fuzzer data table contains the various +** transformations and their costs that the fuzzer logic uses to generate +** variations. +** +** The fuzzer data table must contain exactly four columns (more precisely, +** the statement "SELECT * FROM <fuzzer_data_table>" must return records +** that consist of four columns). It does not matter what the columns are +** named. +** +** Each row in the fuzzer data table represents a single character +** transformation. The left most column of the row (column 0) contains an +** integer value - the identifier of the ruleset to which the transformation +** rule belongs (see "MULTIPLE RULE SETS" below). The second column of the +** row (column 0) contains the input character or characters. The third +** column contains the output character or characters. And the fourth column +** contains the integer cost of making the transformation. For example: +** +** CREATE TABLE f_data(ruleset, cFrom, cTo, Cost); +** INSERT INTO f_data(ruleset, cFrom, cTo, Cost) VALUES(0, '', 'a', 100); +** INSERT INTO f_data(ruleset, cFrom, cTo, Cost) VALUES(0, 'b', '', 87); +** INSERT INTO f_data(ruleset, cFrom, cTo, Cost) VALUES(0, 'o', 'oe', 38); +** INSERT INTO f_data(ruleset, cFrom, cTo, Cost) VALUES(0, 'oe', 'o', 40); +** +** The first row inserted into the fuzzer data table by the SQL script +** above indicates that the cost of inserting a letter 'a' is 100. (All +** costs are integers. We recommend that costs be scaled so that the +** average cost is around 100.) The second INSERT statement creates a rule +** saying that the cost of deleting a single letter 'b' is 87. The third +** and fourth INSERT statements mean that the cost of transforming a +** single letter "o" into the two-letter sequence "oe" is 38 and that the +** cost of transforming "oe" back into "o" is 40. +** +** The contents of the fuzzer data table are loaded into main memory when +** a fuzzer table is first created, and may be internally reloaded by the +** system at any subsequent time. Therefore, the fuzzer data table should be +** populated before the fuzzer table is created and not modified thereafter. +** If you do need to modify the contents of the fuzzer data table, it is +** recommended that the associated fuzzer table be dropped, the fuzzer data +** table edited, and the fuzzer table recreated within a single transaction. +** Alternatively, the fuzzer data table can be edited then the database +** connection can be closed and reopened. +** +** Once it has been created, the fuzzer table can be queried as follows: +** +** SELECT word, distance FROM f +** WHERE word MATCH 'abcdefg' +** AND distance<200; +** +** This first query outputs the string "abcdefg" and all strings that +** can be derived from that string by appling the specified transformations. +** The strings are output together with their total transformation cost +** (called "distance") and appear in order of increasing cost. No string +** is output more than once. If there are multiple ways to transform the +** target string into the output string then the lowest cost transform is +** the one that is returned. In the example, the search is limited to +** strings with a total distance of less than 200. +** +** The fuzzer is a read-only table. Any attempt to DELETE, INSERT, or +** UPDATE on a fuzzer table will throw an error. +** +** It is important to put some kind of a limit on the fuzzer output. This +** can be either in the form of a LIMIT clause at the end of the query, +** or better, a "distance<NNN" constraint where NNN is some number. The +** running time and memory requirement is exponential in the value of NNN +** so you want to make sure that NNN is not too big. A value of NNN that +** is about twice the average transformation cost seems to give good results. +** +** The fuzzer table can be useful for tasks such as spelling correction. +** Suppose there is a second table vocabulary(w) where the w column contains +** all correctly spelled words. Let $word be a word you want to look up. +** +** SELECT vocabulary.w FROM f, vocabulary +** WHERE f.word MATCH $word +** AND f.distance<=200 +** AND f.word=vocabulary.w +** LIMIT 20 +** +** The query above gives the 20 closest words to the $word being tested. +** (Note that for good performance, the vocubulary.w column should be +** indexed.) +** +** A similar query can be used to find all words in the dictionary that +** begin with some prefix $prefix: +** +** SELECT vocabulary.w FROM f, vocabulary +** WHERE f.word MATCH $prefix +** AND f.distance<=200 +** AND vocabulary.w BETWEEN f.word AND (f.word || x'F7BFBFBF') +** LIMIT 50 +** +** This last query will show up to 50 words out of the vocabulary that +** match or nearly match the $prefix. +** +** MULTIPLE RULE SETS +** +** Normally, the "ruleset" value associated with all character transformations +** in the fuzzer data table is zero. However, if required, the fuzzer table +** allows multiple rulesets to be defined. Each query uses only a single +** ruleset. This allows, for example, a single fuzzer table to support +** multiple languages. +** +** By default, only the rules from ruleset 0 are used. To specify an +** alternative ruleset, a "ruleset = ?" expression must be added to the +** WHERE clause of a SELECT, where ? is the identifier of the desired +** ruleset. For example: +** +** SELECT vocabulary.w FROM f, vocabulary +** WHERE f.word MATCH $word +** AND f.distance<=200 +** AND f.word=vocabulary.w +** AND f.ruleset=1 -- Specify the ruleset to use here +** LIMIT 20 +** +** If no "ruleset = ?" constraint is specified in the WHERE clause, ruleset +** 0 is used. +** +** LIMITS +** +** The maximum ruleset number is 2147483647. The maximum length of either +** of the strings in the second or third column of the fuzzer data table +** is 50 bytes. The maximum cost on a rule is 1000. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 + +/* If SQLITE_DEBUG is not defined, disable assert statements. */ +#if !defined(NDEBUG) && !defined(SQLITE_DEBUG) +# define NDEBUG +#endif + +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <stdio.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Forward declaration of objects used by this implementation +*/ +typedef struct fuzzer_vtab fuzzer_vtab; +typedef struct fuzzer_cursor fuzzer_cursor; +typedef struct fuzzer_rule fuzzer_rule; +typedef struct fuzzer_seen fuzzer_seen; +typedef struct fuzzer_stem fuzzer_stem; + +/* +** Various types. +** +** fuzzer_cost is the "cost" of an edit operation. +** +** fuzzer_len is the length of a matching string. +** +** fuzzer_ruleid is an ruleset identifier. +*/ +typedef int fuzzer_cost; +typedef signed char fuzzer_len; +typedef int fuzzer_ruleid; + +/* +** Limits +*/ +#define FUZZER_MX_LENGTH 50 /* Maximum length of a rule string */ +#define FUZZER_MX_RULEID 2147483647 /* Maximum rule ID */ +#define FUZZER_MX_COST 1000 /* Maximum single-rule cost */ +#define FUZZER_MX_OUTPUT_LENGTH 100 /* Maximum length of an output string */ + + +/* +** Each transformation rule is stored as an instance of this object. +** All rules are kept on a linked list sorted by rCost. +*/ +struct fuzzer_rule { + fuzzer_rule *pNext; /* Next rule in order of increasing rCost */ + char *zFrom; /* Transform from */ + fuzzer_cost rCost; /* Cost of this transformation */ + fuzzer_len nFrom, nTo; /* Length of the zFrom and zTo strings */ + fuzzer_ruleid iRuleset; /* The rule set to which this rule belongs */ + char zTo[4]; /* Transform to (extra space appended) */ +}; + +/* +** A stem object is used to generate variants. It is also used to record +** previously generated outputs. +** +** Every stem is added to a hash table as it is output. Generation of +** duplicate stems is suppressed. +** +** Active stems (those that might generate new outputs) are kepts on a linked +** list sorted by increasing cost. The cost is the sum of rBaseCost and +** pRule->rCost. +*/ +struct fuzzer_stem { + char *zBasis; /* Word being fuzzed */ + const fuzzer_rule *pRule; /* Current rule to apply */ + fuzzer_stem *pNext; /* Next stem in rCost order */ + fuzzer_stem *pHash; /* Next stem with same hash on zBasis */ + fuzzer_cost rBaseCost; /* Base cost of getting to zBasis */ + fuzzer_cost rCostX; /* Precomputed rBaseCost + pRule->rCost */ + fuzzer_len nBasis; /* Length of the zBasis string */ + fuzzer_len n; /* Apply pRule at this character offset */ +}; + +/* +** A fuzzer virtual-table object +*/ +struct fuzzer_vtab { + sqlite3_vtab base; /* Base class - must be first */ + char *zClassName; /* Name of this class. Default: "fuzzer" */ + fuzzer_rule *pRule; /* All active rules in this fuzzer */ + int nCursor; /* Number of active cursors */ +}; + +#define FUZZER_HASH 4001 /* Hash table size */ +#define FUZZER_NQUEUE 20 /* Number of slots on the stem queue */ + +/* A fuzzer cursor object */ +struct fuzzer_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iRowid; /* The rowid of the current word */ + fuzzer_vtab *pVtab; /* The virtual table this cursor belongs to */ + fuzzer_cost rLimit; /* Maximum cost of any term */ + fuzzer_stem *pStem; /* Stem with smallest rCostX */ + fuzzer_stem *pDone; /* Stems already processed to completion */ + fuzzer_stem *aQueue[FUZZER_NQUEUE]; /* Queue of stems with higher rCostX */ + int mxQueue; /* Largest used index in aQueue[] */ + char *zBuf; /* Temporary use buffer */ + int nBuf; /* Bytes allocated for zBuf */ + int nStem; /* Number of stems allocated */ + int iRuleset; /* Only process rules from this ruleset */ + fuzzer_rule nullRule; /* Null rule used first */ + fuzzer_stem *apHash[FUZZER_HASH]; /* Hash of previously generated terms */ +}; + +/* +** The two input rule lists are both sorted in order of increasing +** cost. Merge them together into a single list, sorted by cost, and +** return a pointer to the head of that list. +*/ +static fuzzer_rule *fuzzerMergeRules(fuzzer_rule *pA, fuzzer_rule *pB){ + fuzzer_rule head; + fuzzer_rule *pTail; + + pTail = &head; + while( pA && pB ){ + if( pA->rCost<=pB->rCost ){ + pTail->pNext = pA; + pTail = pA; + pA = pA->pNext; + }else{ + pTail->pNext = pB; + pTail = pB; + pB = pB->pNext; + } + } + if( pA==0 ){ + pTail->pNext = pB; + }else{ + pTail->pNext = pA; + } + return head.pNext; +} + +/* +** Statement pStmt currently points to a row in the fuzzer data table. This +** function allocates and populates a fuzzer_rule structure according to +** the content of the row. +** +** If successful, *ppRule is set to point to the new object and SQLITE_OK +** is returned. Otherwise, *ppRule is zeroed, *pzErr may be set to point +** to an error message and an SQLite error code returned. +*/ +static int fuzzerLoadOneRule( + fuzzer_vtab *p, /* Fuzzer virtual table handle */ + sqlite3_stmt *pStmt, /* Base rule on statements current row */ + fuzzer_rule **ppRule, /* OUT: New rule object */ + char **pzErr /* OUT: Error message */ +){ + sqlite3_int64 iRuleset = sqlite3_column_int64(pStmt, 0); + const char *zFrom = (const char *)sqlite3_column_text(pStmt, 1); + const char *zTo = (const char *)sqlite3_column_text(pStmt, 2); + int nCost = sqlite3_column_int(pStmt, 3); + + int rc = SQLITE_OK; /* Return code */ + int nFrom; /* Size of string zFrom, in bytes */ + int nTo; /* Size of string zTo, in bytes */ + fuzzer_rule *pRule = 0; /* New rule object to return */ + + if( zFrom==0 ) zFrom = ""; + if( zTo==0 ) zTo = ""; + nFrom = (int)strlen(zFrom); + nTo = (int)strlen(zTo); + + /* Silently ignore null transformations */ + if( strcmp(zFrom, zTo)==0 ){ + *ppRule = 0; + return SQLITE_OK; + } + + if( nCost<=0 || nCost>FUZZER_MX_COST ){ + *pzErr = sqlite3_mprintf("%s: cost must be between 1 and %d", + p->zClassName, FUZZER_MX_COST + ); + rc = SQLITE_ERROR; + }else + if( nFrom>FUZZER_MX_LENGTH || nTo>FUZZER_MX_LENGTH ){ + *pzErr = sqlite3_mprintf("%s: maximum string length is %d", + p->zClassName, FUZZER_MX_LENGTH + ); + rc = SQLITE_ERROR; + }else + if( iRuleset<0 || iRuleset>FUZZER_MX_RULEID ){ + *pzErr = sqlite3_mprintf("%s: ruleset must be between 0 and %d", + p->zClassName, FUZZER_MX_RULEID + ); + rc = SQLITE_ERROR; + }else{ + + pRule = sqlite3_malloc64( sizeof(*pRule) + nFrom + nTo ); + if( pRule==0 ){ + rc = SQLITE_NOMEM; + }else{ + memset(pRule, 0, sizeof(*pRule)); + pRule->zFrom = pRule->zTo; + pRule->zFrom += nTo + 1; + pRule->nFrom = (fuzzer_len)nFrom; + memcpy(pRule->zFrom, zFrom, nFrom+1); + memcpy(pRule->zTo, zTo, nTo+1); + pRule->nTo = (fuzzer_len)nTo; + pRule->rCost = nCost; + pRule->iRuleset = (int)iRuleset; + } + } + + *ppRule = pRule; + return rc; +} + +/* +** Load the content of the fuzzer data table into memory. +*/ +static int fuzzerLoadRules( + sqlite3 *db, /* Database handle */ + fuzzer_vtab *p, /* Virtual fuzzer table to configure */ + const char *zDb, /* Database containing rules data */ + const char *zData, /* Table containing rules data */ + char **pzErr /* OUT: Error message */ +){ + int rc = SQLITE_OK; /* Return code */ + char *zSql; /* SELECT used to read from rules table */ + fuzzer_rule *pHead = 0; + + zSql = sqlite3_mprintf("SELECT * FROM %Q.%Q", zDb, zData); + if( zSql==0 ){ + rc = SQLITE_NOMEM; + }else{ + int rc2; /* finalize() return code */ + sqlite3_stmt *pStmt = 0; + rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0); + if( rc!=SQLITE_OK ){ + *pzErr = sqlite3_mprintf("%s: %s", p->zClassName, sqlite3_errmsg(db)); + }else if( sqlite3_column_count(pStmt)!=4 ){ + *pzErr = sqlite3_mprintf("%s: %s has %d columns, expected 4", + p->zClassName, zData, sqlite3_column_count(pStmt) + ); + rc = SQLITE_ERROR; + }else{ + while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ + fuzzer_rule *pRule = 0; + rc = fuzzerLoadOneRule(p, pStmt, &pRule, pzErr); + if( pRule ){ + pRule->pNext = pHead; + pHead = pRule; + } + } + } + rc2 = sqlite3_finalize(pStmt); + if( rc==SQLITE_OK ) rc = rc2; + } + sqlite3_free(zSql); + + /* All rules are now in a singly linked list starting at pHead. This + ** block sorts them by cost and then sets fuzzer_vtab.pRule to point to + ** point to the head of the sorted list. + */ + if( rc==SQLITE_OK ){ + unsigned int i; + fuzzer_rule *pX; + fuzzer_rule *a[15]; + for(i=0; i<sizeof(a)/sizeof(a[0]); i++) a[i] = 0; + while( (pX = pHead)!=0 ){ + pHead = pX->pNext; + pX->pNext = 0; + for(i=0; a[i] && i<sizeof(a)/sizeof(a[0])-1; i++){ + pX = fuzzerMergeRules(a[i], pX); + a[i] = 0; + } + a[i] = fuzzerMergeRules(a[i], pX); + } + for(pX=a[0], i=1; i<sizeof(a)/sizeof(a[0]); i++){ + pX = fuzzerMergeRules(a[i], pX); + } + p->pRule = fuzzerMergeRules(p->pRule, pX); + }else{ + /* An error has occurred. Setting p->pRule to point to the head of the + ** allocated list ensures that the list will be cleaned up in this case. + */ + assert( p->pRule==0 ); + p->pRule = pHead; + } + + return rc; +} + +/* +** This function converts an SQL quoted string into an unquoted string +** and returns a pointer to a buffer allocated using sqlite3_malloc() +** containing the result. The caller should eventually free this buffer +** using sqlite3_free. +** +** Examples: +** +** "abc" becomes abc +** 'xyz' becomes xyz +** [pqr] becomes pqr +** `mno` becomes mno +*/ +static char *fuzzerDequote(const char *zIn){ + sqlite3_int64 nIn; /* Size of input string, in bytes */ + char *zOut; /* Output (dequoted) string */ + + nIn = strlen(zIn); + zOut = sqlite3_malloc64(nIn+1); + if( zOut ){ + char q = zIn[0]; /* Quote character (if any ) */ + + if( q!='[' && q!= '\'' && q!='"' && q!='`' ){ + memcpy(zOut, zIn, (size_t)(nIn+1)); + }else{ + int iOut = 0; /* Index of next byte to write to output */ + int iIn; /* Index of next byte to read from input */ + + if( q=='[' ) q = ']'; + for(iIn=1; iIn<nIn; iIn++){ + if( zIn[iIn]==q ) iIn++; + zOut[iOut++] = zIn[iIn]; + } + } + assert( (int)strlen(zOut)<=nIn ); + } + return zOut; +} + +/* +** xDisconnect/xDestroy method for the fuzzer module. +*/ +static int fuzzerDisconnect(sqlite3_vtab *pVtab){ + fuzzer_vtab *p = (fuzzer_vtab*)pVtab; + assert( p->nCursor==0 ); + while( p->pRule ){ + fuzzer_rule *pRule = p->pRule; + p->pRule = pRule->pNext; + sqlite3_free(pRule); + } + sqlite3_free(p); + return SQLITE_OK; +} + +/* +** xConnect/xCreate method for the fuzzer module. Arguments are: +** +** argv[0] -> module name ("fuzzer") +** argv[1] -> database name +** argv[2] -> table name +** argv[3] -> fuzzer rule table name +*/ +static int fuzzerConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + int rc = SQLITE_OK; /* Return code */ + fuzzer_vtab *pNew = 0; /* New virtual table */ + const char *zModule = argv[0]; + const char *zDb = argv[1]; + + if( argc!=4 ){ + *pzErr = sqlite3_mprintf( + "%s: wrong number of CREATE VIRTUAL TABLE arguments", zModule + ); + rc = SQLITE_ERROR; + }else{ + sqlite3_int64 nModule; /* Length of zModule, in bytes */ + + nModule = strlen(zModule); + pNew = sqlite3_malloc64( sizeof(*pNew) + nModule + 1); + if( pNew==0 ){ + rc = SQLITE_NOMEM; + }else{ + char *zTab; /* Dequoted name of fuzzer data table */ + + memset(pNew, 0, sizeof(*pNew)); + pNew->zClassName = (char*)&pNew[1]; + memcpy(pNew->zClassName, zModule, (size_t)(nModule+1)); + + zTab = fuzzerDequote(argv[3]); + if( zTab==0 ){ + rc = SQLITE_NOMEM; + }else{ + rc = fuzzerLoadRules(db, pNew, zDb, zTab, pzErr); + sqlite3_free(zTab); + } + + if( rc==SQLITE_OK ){ + rc = sqlite3_declare_vtab(db, "CREATE TABLE x(word,distance,ruleset)"); + } + if( rc!=SQLITE_OK ){ + fuzzerDisconnect((sqlite3_vtab *)pNew); + pNew = 0; + }else{ + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + } + } + } + + *ppVtab = (sqlite3_vtab *)pNew; + return rc; +} + +/* +** Open a new fuzzer cursor. +*/ +static int fuzzerOpen(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ + fuzzer_vtab *p = (fuzzer_vtab*)pVTab; + fuzzer_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->pVtab = p; + *ppCursor = &pCur->base; + p->nCursor++; + return SQLITE_OK; +} + +/* +** Free all stems in a list. +*/ +static void fuzzerClearStemList(fuzzer_stem *pStem){ + while( pStem ){ + fuzzer_stem *pNext = pStem->pNext; + sqlite3_free(pStem); + pStem = pNext; + } +} + +/* +** Free up all the memory allocated by a cursor. Set it rLimit to 0 +** to indicate that it is at EOF. +*/ +static void fuzzerClearCursor(fuzzer_cursor *pCur, int clearHash){ + int i; + fuzzerClearStemList(pCur->pStem); + fuzzerClearStemList(pCur->pDone); + for(i=0; i<FUZZER_NQUEUE; i++) fuzzerClearStemList(pCur->aQueue[i]); + pCur->rLimit = (fuzzer_cost)0; + if( clearHash && pCur->nStem ){ + pCur->mxQueue = 0; + pCur->pStem = 0; + pCur->pDone = 0; + memset(pCur->aQueue, 0, sizeof(pCur->aQueue)); + memset(pCur->apHash, 0, sizeof(pCur->apHash)); + } + pCur->nStem = 0; +} + +/* +** Close a fuzzer cursor. +*/ +static int fuzzerClose(sqlite3_vtab_cursor *cur){ + fuzzer_cursor *pCur = (fuzzer_cursor *)cur; + fuzzerClearCursor(pCur, 0); + sqlite3_free(pCur->zBuf); + pCur->pVtab->nCursor--; + sqlite3_free(pCur); + return SQLITE_OK; +} + +/* +** Compute the current output term for a fuzzer_stem. +*/ +static int fuzzerRender( + fuzzer_stem *pStem, /* The stem to be rendered */ + char **pzBuf, /* Write results into this buffer. realloc if needed */ + int *pnBuf /* Size of the buffer */ +){ + const fuzzer_rule *pRule = pStem->pRule; + int n; /* Size of output term without nul-term */ + char *z; /* Buffer to assemble output term in */ + + n = pStem->nBasis + pRule->nTo - pRule->nFrom; + if( (*pnBuf)<n+1 ){ + (*pzBuf) = sqlite3_realloc((*pzBuf), n+100); + if( (*pzBuf)==0 ) return SQLITE_NOMEM; + (*pnBuf) = n+100; + } + n = pStem->n; + z = *pzBuf; + if( n<0 ){ + memcpy(z, pStem->zBasis, pStem->nBasis+1); + }else{ + memcpy(z, pStem->zBasis, n); + memcpy(&z[n], pRule->zTo, pRule->nTo); + memcpy(&z[n+pRule->nTo], &pStem->zBasis[n+pRule->nFrom], + pStem->nBasis-n-pRule->nFrom+1); + } + + assert( z[pStem->nBasis + pRule->nTo - pRule->nFrom]==0 ); + return SQLITE_OK; +} + +/* +** Compute a hash on zBasis. +*/ +static unsigned int fuzzerHash(const char *z){ + unsigned int h = 0; + while( *z ){ h = (h<<3) ^ (h>>29) ^ *(z++); } + return h % FUZZER_HASH; +} + +/* +** Current cost of a stem +*/ +static fuzzer_cost fuzzerCost(fuzzer_stem *pStem){ + return pStem->rCostX = pStem->rBaseCost + pStem->pRule->rCost; +} + +#if 0 +/* +** Print a description of a fuzzer_stem on stderr. +*/ +static void fuzzerStemPrint( + const char *zPrefix, + fuzzer_stem *pStem, + const char *zSuffix +){ + if( pStem->n<0 ){ + fprintf(stderr, "%s[%s](%d)-->self%s", + zPrefix, + pStem->zBasis, pStem->rBaseCost, + zSuffix + ); + }else{ + char *zBuf = 0; + int nBuf = 0; + if( fuzzerRender(pStem, &zBuf, &nBuf)!=SQLITE_OK ) return; + fprintf(stderr, "%s[%s](%d)-->{%s}(%d)%s", + zPrefix, + pStem->zBasis, pStem->rBaseCost, zBuf, pStem->, + zSuffix + ); + sqlite3_free(zBuf); + } +} +#endif + +/* +** Return 1 if the string to which the cursor is point has already +** been emitted. Return 0 if not. Return -1 on a memory allocation +** failures. +*/ +static int fuzzerSeen(fuzzer_cursor *pCur, fuzzer_stem *pStem){ + unsigned int h; + fuzzer_stem *pLookup; + + if( fuzzerRender(pStem, &pCur->zBuf, &pCur->nBuf)==SQLITE_NOMEM ){ + return -1; + } + h = fuzzerHash(pCur->zBuf); + pLookup = pCur->apHash[h]; + while( pLookup && strcmp(pLookup->zBasis, pCur->zBuf)!=0 ){ + pLookup = pLookup->pHash; + } + return pLookup!=0; +} + +/* +** If argument pRule is NULL, this function returns false. +** +** Otherwise, it returns true if rule pRule should be skipped. A rule +** should be skipped if it does not belong to rule-set iRuleset, or if +** applying it to stem pStem would create a string longer than +** FUZZER_MX_OUTPUT_LENGTH bytes. +*/ +static int fuzzerSkipRule( + const fuzzer_rule *pRule, /* Determine whether or not to skip this */ + fuzzer_stem *pStem, /* Stem rule may be applied to */ + int iRuleset /* Rule-set used by the current query */ +){ + return pRule && ( + (pRule->iRuleset!=iRuleset) + || (pStem->nBasis + pRule->nTo - pRule->nFrom)>FUZZER_MX_OUTPUT_LENGTH + ); +} + +/* +** Advance a fuzzer_stem to its next value. Return 0 if there are +** no more values that can be generated by this fuzzer_stem. Return +** -1 on a memory allocation failure. +*/ +static int fuzzerAdvance(fuzzer_cursor *pCur, fuzzer_stem *pStem){ + const fuzzer_rule *pRule; + while( (pRule = pStem->pRule)!=0 ){ + assert( pRule==&pCur->nullRule || pRule->iRuleset==pCur->iRuleset ); + while( pStem->n < pStem->nBasis - pRule->nFrom ){ + pStem->n++; + if( pRule->nFrom==0 + || memcmp(&pStem->zBasis[pStem->n], pRule->zFrom, pRule->nFrom)==0 + ){ + /* Found a rewrite case. Make sure it is not a duplicate */ + int rc = fuzzerSeen(pCur, pStem); + if( rc<0 ) return -1; + if( rc==0 ){ + fuzzerCost(pStem); + return 1; + } + } + } + pStem->n = -1; + do{ + pRule = pRule->pNext; + }while( fuzzerSkipRule(pRule, pStem, pCur->iRuleset) ); + pStem->pRule = pRule; + if( pRule && fuzzerCost(pStem)>pCur->rLimit ) pStem->pRule = 0; + } + return 0; +} + +/* +** The two input stem lists are both sorted in order of increasing +** rCostX. Merge them together into a single list, sorted by rCostX, and +** return a pointer to the head of that new list. +*/ +static fuzzer_stem *fuzzerMergeStems(fuzzer_stem *pA, fuzzer_stem *pB){ + fuzzer_stem head; + fuzzer_stem *pTail; + + pTail = &head; + while( pA && pB ){ + if( pA->rCostX<=pB->rCostX ){ + pTail->pNext = pA; + pTail = pA; + pA = pA->pNext; + }else{ + pTail->pNext = pB; + pTail = pB; + pB = pB->pNext; + } + } + if( pA==0 ){ + pTail->pNext = pB; + }else{ + pTail->pNext = pA; + } + return head.pNext; +} + +/* +** Load pCur->pStem with the lowest-cost stem. Return a pointer +** to the lowest-cost stem. +*/ +static fuzzer_stem *fuzzerLowestCostStem(fuzzer_cursor *pCur){ + fuzzer_stem *pBest, *pX; + int iBest; + int i; + + if( pCur->pStem==0 ){ + iBest = -1; + pBest = 0; + for(i=0; i<=pCur->mxQueue; i++){ + pX = pCur->aQueue[i]; + if( pX==0 ) continue; + if( pBest==0 || pBest->rCostX>pX->rCostX ){ + pBest = pX; + iBest = i; + } + } + if( pBest ){ + pCur->aQueue[iBest] = pBest->pNext; + pBest->pNext = 0; + pCur->pStem = pBest; + } + } + return pCur->pStem; +} + +/* +** Insert pNew into queue of pending stems. Then find the stem +** with the lowest rCostX and move it into pCur->pStem. +** list. The insert is done such the pNew is in the correct order +** according to fuzzer_stem.zBaseCost+fuzzer_stem.pRule->rCost. +*/ +static fuzzer_stem *fuzzerInsert(fuzzer_cursor *pCur, fuzzer_stem *pNew){ + fuzzer_stem *pX; + int i; + + /* If pCur->pStem exists and is greater than pNew, then make pNew + ** the new pCur->pStem and insert the old pCur->pStem instead. + */ + if( (pX = pCur->pStem)!=0 && pX->rCostX>pNew->rCostX ){ + pNew->pNext = 0; + pCur->pStem = pNew; + pNew = pX; + } + + /* Insert the new value */ + pNew->pNext = 0; + pX = pNew; + for(i=0; i<=pCur->mxQueue; i++){ + if( pCur->aQueue[i] ){ + pX = fuzzerMergeStems(pX, pCur->aQueue[i]); + pCur->aQueue[i] = 0; + }else{ + pCur->aQueue[i] = pX; + break; + } + } + if( i>pCur->mxQueue ){ + if( i<FUZZER_NQUEUE ){ + pCur->mxQueue = i; + pCur->aQueue[i] = pX; + }else{ + assert( pCur->mxQueue==FUZZER_NQUEUE-1 ); + pX = fuzzerMergeStems(pX, pCur->aQueue[FUZZER_NQUEUE-1]); + pCur->aQueue[FUZZER_NQUEUE-1] = pX; + } + } + + return fuzzerLowestCostStem(pCur); +} + +/* +** Allocate a new fuzzer_stem. Add it to the hash table but do not +** link it into either the pCur->pStem or pCur->pDone lists. +*/ +static fuzzer_stem *fuzzerNewStem( + fuzzer_cursor *pCur, + const char *zWord, + fuzzer_cost rBaseCost +){ + fuzzer_stem *pNew; + fuzzer_rule *pRule; + unsigned int h; + + pNew = sqlite3_malloc64( sizeof(*pNew) + strlen(zWord) + 1 ); + if( pNew==0 ) return 0; + memset(pNew, 0, sizeof(*pNew)); + pNew->zBasis = (char*)&pNew[1]; + pNew->nBasis = (fuzzer_len)strlen(zWord); + memcpy(pNew->zBasis, zWord, pNew->nBasis+1); + pRule = pCur->pVtab->pRule; + while( fuzzerSkipRule(pRule, pNew, pCur->iRuleset) ){ + pRule = pRule->pNext; + } + pNew->pRule = pRule; + pNew->n = -1; + pNew->rBaseCost = pNew->rCostX = rBaseCost; + h = fuzzerHash(pNew->zBasis); + pNew->pHash = pCur->apHash[h]; + pCur->apHash[h] = pNew; + pCur->nStem++; + return pNew; +} + + +/* +** Advance a cursor to its next row of output +*/ +static int fuzzerNext(sqlite3_vtab_cursor *cur){ + fuzzer_cursor *pCur = (fuzzer_cursor*)cur; + int rc; + fuzzer_stem *pStem, *pNew; + + pCur->iRowid++; + + /* Use the element the cursor is currently point to to create + ** a new stem and insert the new stem into the priority queue. + */ + pStem = pCur->pStem; + if( pStem->rCostX>0 ){ + rc = fuzzerRender(pStem, &pCur->zBuf, &pCur->nBuf); + if( rc==SQLITE_NOMEM ) return SQLITE_NOMEM; + pNew = fuzzerNewStem(pCur, pCur->zBuf, pStem->rCostX); + if( pNew ){ + if( fuzzerAdvance(pCur, pNew)==0 ){ + pNew->pNext = pCur->pDone; + pCur->pDone = pNew; + }else{ + if( fuzzerInsert(pCur, pNew)==pNew ){ + return SQLITE_OK; + } + } + }else{ + return SQLITE_NOMEM; + } + } + + /* Adjust the priority queue so that the first element of the + ** stem list is the next lowest cost word. + */ + while( (pStem = pCur->pStem)!=0 ){ + int res = fuzzerAdvance(pCur, pStem); + if( res<0 ){ + return SQLITE_NOMEM; + }else if( res>0 ){ + pCur->pStem = 0; + pStem = fuzzerInsert(pCur, pStem); + if( (rc = fuzzerSeen(pCur, pStem))!=0 ){ + if( rc<0 ) return SQLITE_NOMEM; + continue; + } + return SQLITE_OK; /* New word found */ + } + pCur->pStem = 0; + pStem->pNext = pCur->pDone; + pCur->pDone = pStem; + if( fuzzerLowestCostStem(pCur) ){ + rc = fuzzerSeen(pCur, pCur->pStem); + if( rc<0 ) return SQLITE_NOMEM; + if( rc==0 ){ + return SQLITE_OK; + } + } + } + + /* Reach this point only if queue has been exhausted and there is + ** nothing left to be output. */ + pCur->rLimit = (fuzzer_cost)0; + return SQLITE_OK; +} + +/* +** Called to "rewind" a cursor back to the beginning so that +** it starts its output over again. Always called at least once +** prior to any fuzzerColumn, fuzzerRowid, or fuzzerEof call. +*/ +static int fuzzerFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + fuzzer_cursor *pCur = (fuzzer_cursor *)pVtabCursor; + const char *zWord = ""; + fuzzer_stem *pStem; + int idx; + + fuzzerClearCursor(pCur, 1); + pCur->rLimit = 2147483647; + idx = 0; + if( idxNum & 1 ){ + zWord = (const char*)sqlite3_value_text(argv[0]); + idx++; + } + if( idxNum & 2 ){ + pCur->rLimit = (fuzzer_cost)sqlite3_value_int(argv[idx]); + idx++; + } + if( idxNum & 4 ){ + pCur->iRuleset = (fuzzer_cost)sqlite3_value_int(argv[idx]); + idx++; + } + pCur->nullRule.pNext = pCur->pVtab->pRule; + pCur->nullRule.rCost = 0; + pCur->nullRule.nFrom = 0; + pCur->nullRule.nTo = 0; + pCur->nullRule.zFrom = ""; + pCur->iRowid = 1; + assert( pCur->pStem==0 ); + + /* If the query term is longer than FUZZER_MX_OUTPUT_LENGTH bytes, this + ** query will return zero rows. */ + if( (int)strlen(zWord)<FUZZER_MX_OUTPUT_LENGTH ){ + pCur->pStem = pStem = fuzzerNewStem(pCur, zWord, (fuzzer_cost)0); + if( pStem==0 ) return SQLITE_NOMEM; + pStem->pRule = &pCur->nullRule; + pStem->n = pStem->nBasis; + }else{ + pCur->rLimit = 0; + } + + return SQLITE_OK; +} + +/* +** Only the word and distance columns have values. All other columns +** return NULL +*/ +static int fuzzerColumn(sqlite3_vtab_cursor *cur, sqlite3_context *ctx, int i){ + fuzzer_cursor *pCur = (fuzzer_cursor*)cur; + if( i==0 ){ + /* the "word" column */ + if( fuzzerRender(pCur->pStem, &pCur->zBuf, &pCur->nBuf)==SQLITE_NOMEM ){ + return SQLITE_NOMEM; + } + sqlite3_result_text(ctx, pCur->zBuf, -1, SQLITE_TRANSIENT); + }else if( i==1 ){ + /* the "distance" column */ + sqlite3_result_int(ctx, pCur->pStem->rCostX); + }else{ + /* All other columns are NULL */ + sqlite3_result_null(ctx); + } + return SQLITE_OK; +} + +/* +** The rowid. +*/ +static int fuzzerRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + fuzzer_cursor *pCur = (fuzzer_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** When the fuzzer_cursor.rLimit value is 0 or less, that is a signal +** that the cursor has nothing more to output. +*/ +static int fuzzerEof(sqlite3_vtab_cursor *cur){ + fuzzer_cursor *pCur = (fuzzer_cursor*)cur; + return pCur->rLimit<=(fuzzer_cost)0; +} + +/* +** Search for terms of these forms: +** +** (A) word MATCH $str +** (B1) distance < $value +** (B2) distance <= $value +** (C) ruleid == $ruleid +** +** The distance< and distance<= are both treated as distance<=. +** The query plan number is a bit vector: +** +** bit 1: Term of the form (A) found +** bit 2: Term like (B1) or (B2) found +** bit 3: Term like (C) found +** +** If bit-1 is set, $str is always in filter.argv[0]. If bit-2 is set +** then $value is in filter.argv[0] if bit-1 is clear and is in +** filter.argv[1] if bit-1 is set. If bit-3 is set, then $ruleid is +** in filter.argv[0] if bit-1 and bit-2 are both zero, is in +** filter.argv[1] if exactly one of bit-1 and bit-2 are set, and is in +** filter.argv[2] if both bit-1 and bit-2 are set. +*/ +static int fuzzerBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdxInfo){ + int iPlan = 0; + int iDistTerm = -1; + int iRulesetTerm = -1; + int i; + int seenMatch = 0; + const struct sqlite3_index_constraint *pConstraint; + double rCost = 1e12; + + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->iColumn==0 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_MATCH ){ + seenMatch = 1; + } + if( pConstraint->usable==0 ) continue; + if( (iPlan & 1)==0 + && pConstraint->iColumn==0 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_MATCH + ){ + iPlan |= 1; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + rCost /= 1e6; + } + if( (iPlan & 2)==0 + && pConstraint->iColumn==1 + && (pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT + || pConstraint->op==SQLITE_INDEX_CONSTRAINT_LE) + ){ + iPlan |= 2; + iDistTerm = i; + rCost /= 10.0; + } + if( (iPlan & 4)==0 + && pConstraint->iColumn==2 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= 4; + pIdxInfo->aConstraintUsage[i].omit = 1; + iRulesetTerm = i; + rCost /= 10.0; + } + } + if( iPlan & 2 ){ + pIdxInfo->aConstraintUsage[iDistTerm].argvIndex = 1+((iPlan&1)!=0); + } + if( iPlan & 4 ){ + int idx = 1; + if( iPlan & 1 ) idx++; + if( iPlan & 2 ) idx++; + pIdxInfo->aConstraintUsage[iRulesetTerm].argvIndex = idx; + } + pIdxInfo->idxNum = iPlan; + if( pIdxInfo->nOrderBy==1 + && pIdxInfo->aOrderBy[0].iColumn==1 + && pIdxInfo->aOrderBy[0].desc==0 + ){ + pIdxInfo->orderByConsumed = 1; + } + if( seenMatch && (iPlan&1)==0 ) rCost = 1e99; + pIdxInfo->estimatedCost = rCost; + + return SQLITE_OK; +} + +/* +** A virtual table module that implements the "fuzzer". +*/ +static sqlite3_module fuzzerModule = { + 0, /* iVersion */ + fuzzerConnect, + fuzzerConnect, + fuzzerBestIndex, + fuzzerDisconnect, + fuzzerDisconnect, + fuzzerOpen, /* xOpen - open a cursor */ + fuzzerClose, /* xClose - close a cursor */ + fuzzerFilter, /* xFilter - configure scan constraints */ + fuzzerNext, /* xNext - advance a cursor */ + fuzzerEof, /* xEof - check for end of scan */ + fuzzerColumn, /* xColumn - read data */ + fuzzerRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_fuzzer_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "fuzzer", &fuzzerModule, 0); +#endif + return rc; +} diff --git a/ext/misc/ieee754.c b/ext/misc/ieee754.c new file mode 100644 index 0000000..ff5d2d3 --- /dev/null +++ b/ext/misc/ieee754.c @@ -0,0 +1,293 @@ +/* +** 2013-04-17 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements functions for the exact display +** and input of IEEE754 Binary64 floating-point numbers. +** +** ieee754(X) +** ieee754(Y,Z) +** +** In the first form, the value X should be a floating-point number. +** The function will return a string of the form 'ieee754(Y,Z)' where +** Y and Z are integers such that X==Y*pow(2,Z). +** +** In the second form, Y and Z are integers which are the mantissa and +** base-2 exponent of a new floating point number. The function returns +** a floating-point value equal to Y*pow(2,Z). +** +** Examples: +** +** ieee754(2.0) -> 'ieee754(2,0)' +** ieee754(45.25) -> 'ieee754(181,-2)' +** ieee754(2, 0) -> 2.0 +** ieee754(181, -2) -> 45.25 +** +** Two additional functions break apart the one-argument ieee754() +** result into separate integer values: +** +** ieee754_mantissa(45.25) -> 181 +** ieee754_exponent(45.25) -> -2 +** +** These functions convert binary64 numbers into blobs and back again. +** +** ieee754_from_blob(x'3ff0000000000000') -> 1.0 +** ieee754_to_blob(1.0) -> x'3ff0000000000000' +** +** In all single-argument functions, if the argument is an 8-byte blob +** then that blob is interpreted as a big-endian binary64 value. +** +** +** EXACT DECIMAL REPRESENTATION OF BINARY64 VALUES +** ----------------------------------------------- +** +** This extension in combination with the separate 'decimal' extension +** can be used to compute the exact decimal representation of binary64 +** values. To begin, first compute a table of exponent values: +** +** CREATE TABLE pow2(x INTEGER PRIMARY KEY, v TEXT); +** WITH RECURSIVE c(x,v) AS ( +** VALUES(0,'1') +** UNION ALL +** SELECT x+1, decimal_mul(v,'2') FROM c WHERE x+1<=971 +** ) INSERT INTO pow2(x,v) SELECT x, v FROM c; +** WITH RECURSIVE c(x,v) AS ( +** VALUES(-1,'0.5') +** UNION ALL +** SELECT x-1, decimal_mul(v,'0.5') FROM c WHERE x-1>=-1075 +** ) INSERT INTO pow2(x,v) SELECT x, v FROM c; +** +** Then, to compute the exact decimal representation of a floating +** point value (the value 47.49 is used in the example) do: +** +** WITH c(n) AS (VALUES(47.49)) +** ---------------^^^^^---- Replace with whatever you want +** SELECT decimal_mul(ieee754_mantissa(c.n),pow2.v) +** FROM pow2, c WHERE pow2.x=ieee754_exponent(c.n); +** +** Here is a query to show various boundry values for the binary64 +** number format: +** +** WITH c(name,bin) AS (VALUES +** ('minimum positive value', x'0000000000000001'), +** ('maximum subnormal value', x'000fffffffffffff'), +** ('mininum positive nornal value', x'0010000000000000'), +** ('maximum value', x'7fefffffffffffff')) +** SELECT c.name, decimal_mul(ieee754_mantissa(c.bin),pow2.v) +** FROM pow2, c WHERE pow2.x=ieee754_exponent(c.bin); +** +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* Mark a function parameter as unused, to suppress nuisance compiler +** warnings. */ +#ifndef UNUSED_PARAMETER +# define UNUSED_PARAMETER(X) (void)(X) +#endif + +/* +** Implementation of the ieee754() function +*/ +static void ieee754func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + if( argc==1 ){ + sqlite3_int64 m, a; + double r; + int e; + int isNeg; + char zResult[100]; + assert( sizeof(m)==sizeof(r) ); + if( sqlite3_value_type(argv[0])==SQLITE_BLOB + && sqlite3_value_bytes(argv[0])==sizeof(r) + ){ + const unsigned char *x = sqlite3_value_blob(argv[0]); + unsigned int i; + sqlite3_uint64 v = 0; + for(i=0; i<sizeof(r); i++){ + v = (v<<8) | x[i]; + } + memcpy(&r, &v, sizeof(r)); + }else{ + r = sqlite3_value_double(argv[0]); + } + if( r<0.0 ){ + isNeg = 1; + r = -r; + }else{ + isNeg = 0; + } + memcpy(&a,&r,sizeof(a)); + if( a==0 ){ + e = 0; + m = 0; + }else{ + e = a>>52; + m = a & ((((sqlite3_int64)1)<<52)-1); + if( e==0 ){ + m <<= 1; + }else{ + m |= ((sqlite3_int64)1)<<52; + } + while( e<1075 && m>0 && (m&1)==0 ){ + m >>= 1; + e++; + } + if( isNeg ) m = -m; + } + switch( *(int*)sqlite3_user_data(context) ){ + case 0: + sqlite3_snprintf(sizeof(zResult), zResult, "ieee754(%lld,%d)", + m, e-1075); + sqlite3_result_text(context, zResult, -1, SQLITE_TRANSIENT); + break; + case 1: + sqlite3_result_int64(context, m); + break; + case 2: + sqlite3_result_int(context, e-1075); + break; + } + }else{ + sqlite3_int64 m, e, a; + double r; + int isNeg = 0; + m = sqlite3_value_int64(argv[0]); + e = sqlite3_value_int64(argv[1]); + + /* Limit the range of e. Ticket 22dea1cfdb9151e4 2021-03-02 */ + if( e>10000 ){ + e = 10000; + }else if( e<-10000 ){ + e = -10000; + } + + if( m<0 ){ + isNeg = 1; + m = -m; + if( m<0 ) return; + }else if( m==0 && e>-1000 && e<1000 ){ + sqlite3_result_double(context, 0.0); + return; + } + while( (m>>32)&0xffe00000 ){ + m >>= 1; + e++; + } + while( m!=0 && ((m>>32)&0xfff00000)==0 ){ + m <<= 1; + e--; + } + e += 1075; + if( e<=0 ){ + /* Subnormal */ + if( 1-e >= 64 ){ + m = 0; + }else{ + m >>= 1-e; + } + e = 0; + }else if( e>0x7ff ){ + e = 0x7ff; + } + a = m & ((((sqlite3_int64)1)<<52)-1); + a |= e<<52; + if( isNeg ) a |= ((sqlite3_uint64)1)<<63; + memcpy(&r, &a, sizeof(r)); + sqlite3_result_double(context, r); + } +} + +/* +** Functions to convert between blobs and floats. +*/ +static void ieee754func_from_blob( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + UNUSED_PARAMETER(argc); + if( sqlite3_value_type(argv[0])==SQLITE_BLOB + && sqlite3_value_bytes(argv[0])==sizeof(double) + ){ + double r; + const unsigned char *x = sqlite3_value_blob(argv[0]); + unsigned int i; + sqlite3_uint64 v = 0; + for(i=0; i<sizeof(r); i++){ + v = (v<<8) | x[i]; + } + memcpy(&r, &v, sizeof(r)); + sqlite3_result_double(context, r); + } +} +static void ieee754func_to_blob( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + UNUSED_PARAMETER(argc); + if( sqlite3_value_type(argv[0])==SQLITE_FLOAT + || sqlite3_value_type(argv[0])==SQLITE_INTEGER + ){ + double r = sqlite3_value_double(argv[0]); + sqlite3_uint64 v; + unsigned char a[sizeof(r)]; + unsigned int i; + memcpy(&v, &r, sizeof(r)); + for(i=1; i<=sizeof(r); i++){ + a[sizeof(r)-i] = v&0xff; + v >>= 8; + } + sqlite3_result_blob(context, a, sizeof(r), SQLITE_TRANSIENT); + } +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_ieee_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + static const struct { + char *zFName; + int nArg; + int iAux; + void (*xFunc)(sqlite3_context*,int,sqlite3_value**); + } aFunc[] = { + { "ieee754", 1, 0, ieee754func }, + { "ieee754", 2, 0, ieee754func }, + { "ieee754_mantissa", 1, 1, ieee754func }, + { "ieee754_exponent", 1, 2, ieee754func }, + { "ieee754_to_blob", 1, 0, ieee754func_to_blob }, + { "ieee754_from_blob", 1, 0, ieee754func_from_blob }, + + }; + unsigned int i; + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + for(i=0; i<sizeof(aFunc)/sizeof(aFunc[0]) && rc==SQLITE_OK; i++){ + rc = sqlite3_create_function(db, aFunc[i].zFName, aFunc[i].nArg, + SQLITE_UTF8|SQLITE_INNOCUOUS, + (void*)&aFunc[i].iAux, + aFunc[i].xFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/memstat.c b/ext/misc/memstat.c new file mode 100644 index 0000000..800a86e --- /dev/null +++ b/ext/misc/memstat.c @@ -0,0 +1,428 @@ +/* +** 2018-09-27 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates an eponymous virtual table that returns information +** from sqlite3_status64() and sqlite3_db_status(). +** +** Usage example: +** +** .load ./memstat +** .mode quote +** .header on +** SELECT * FROM memstat; +*/ +#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_MEMSTATVTAB) +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* memstat_vtab is a subclass of sqlite3_vtab which will +** serve as the underlying representation of a memstat virtual table +*/ +typedef struct memstat_vtab memstat_vtab; +struct memstat_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this memstat vtab */ +}; + +/* memstat_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct memstat_cursor memstat_cursor; +struct memstat_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this cursor */ + int iRowid; /* Current row in aMemstatColumn[] */ + int iDb; /* Which schema we are looking at */ + int nDb; /* Number of schemas */ + char **azDb; /* Names of all schemas */ + sqlite3_int64 aVal[2]; /* Result values */ +}; + +/* +** The memstatConnect() method is invoked to create a new +** memstat_vtab that describes the memstat virtual table. +** +** Think of this routine as the constructor for memstat_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the memstat_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against memstat will look like. +*/ +static int memstatConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + memstat_vtab *pNew; + int rc; + +/* Column numbers */ +#define MSV_COLUMN_NAME 0 /* Name of quantity being measured */ +#define MSV_COLUMN_SCHEMA 1 /* schema name */ +#define MSV_COLUMN_VALUE 2 /* Current value */ +#define MSV_COLUMN_HIWTR 3 /* Highwater mark */ + + rc = sqlite3_declare_vtab(db,"CREATE TABLE x(name,schema,value,hiwtr)"); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + } + return rc; +} + +/* +** This method is the destructor for memstat_cursor objects. +*/ +static int memstatDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new memstat_cursor object. +*/ +static int memstatOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + memstat_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->db = ((memstat_vtab*)p)->db; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Clear all the schema names from a cursor +*/ +static void memstatClearSchema(memstat_cursor *pCur){ + int i; + if( pCur->azDb==0 ) return; + for(i=0; i<pCur->nDb; i++){ + sqlite3_free(pCur->azDb[i]); + } + sqlite3_free(pCur->azDb); + pCur->azDb = 0; + pCur->nDb = 0; +} + +/* +** Fill in the azDb[] array for the cursor. +*/ +static int memstatFindSchemas(memstat_cursor *pCur){ + sqlite3_stmt *pStmt = 0; + int rc; + if( pCur->nDb ) return SQLITE_OK; + rc = sqlite3_prepare_v2(pCur->db, "PRAGMA database_list", -1, &pStmt, 0); + if( rc ){ + sqlite3_finalize(pStmt); + return rc; + } + while( sqlite3_step(pStmt)==SQLITE_ROW ){ + char **az, *z; + az = sqlite3_realloc64(pCur->azDb, sizeof(char*)*(pCur->nDb+1)); + if( az==0 ){ + memstatClearSchema(pCur); + return SQLITE_NOMEM; + } + pCur->azDb = az; + z = sqlite3_mprintf("%s", sqlite3_column_text(pStmt, 1)); + if( z==0 ){ + memstatClearSchema(pCur); + return SQLITE_NOMEM; + } + pCur->azDb[pCur->nDb] = z; + pCur->nDb++; + } + sqlite3_finalize(pStmt); + return SQLITE_OK; +} + + +/* +** Destructor for a memstat_cursor. +*/ +static int memstatClose(sqlite3_vtab_cursor *cur){ + memstat_cursor *pCur = (memstat_cursor*)cur; + memstatClearSchema(pCur); + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Allowed values for aMemstatColumn[].eType +*/ +#define MSV_GSTAT 0 /* sqlite3_status64() information */ +#define MSV_DB 1 /* sqlite3_db_status() information */ +#define MSV_ZIPVFS 2 /* ZIPVFS file-control with 64-bit return */ + +/* +** An array of quantities that can be measured and reported by +** this virtual table +*/ +static const struct MemstatColumns { + const char *zName; /* Symbolic name */ + unsigned char eType; /* Type of interface */ + unsigned char mNull; /* Bitmask of which columns are NULL */ + /* 2: dbname, 4: current, 8: hiwtr */ + int eOp; /* Opcode */ +} aMemstatColumn[] = { + {"MEMORY_USED", MSV_GSTAT, 2, SQLITE_STATUS_MEMORY_USED }, + {"MALLOC_SIZE", MSV_GSTAT, 6, SQLITE_STATUS_MALLOC_SIZE }, + {"MALLOC_COUNT", MSV_GSTAT, 2, SQLITE_STATUS_MALLOC_COUNT }, + {"PAGECACHE_USED", MSV_GSTAT, 2, SQLITE_STATUS_PAGECACHE_USED }, + {"PAGECACHE_OVERFLOW", MSV_GSTAT, 2, SQLITE_STATUS_PAGECACHE_OVERFLOW }, + {"PAGECACHE_SIZE", MSV_GSTAT, 6, SQLITE_STATUS_PAGECACHE_SIZE }, + {"PARSER_STACK", MSV_GSTAT, 6, SQLITE_STATUS_PARSER_STACK }, + {"DB_LOOKASIDE_USED", MSV_DB, 2, SQLITE_DBSTATUS_LOOKASIDE_USED }, + {"DB_LOOKASIDE_HIT", MSV_DB, 6, SQLITE_DBSTATUS_LOOKASIDE_HIT }, + {"DB_LOOKASIDE_MISS_SIZE", MSV_DB, 6, SQLITE_DBSTATUS_LOOKASIDE_MISS_SIZE}, + {"DB_LOOKASIDE_MISS_FULL", MSV_DB, 6, SQLITE_DBSTATUS_LOOKASIDE_MISS_FULL}, + {"DB_CACHE_USED", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_USED }, +#if SQLITE_VERSION_NUMBER >= 3140000 + {"DB_CACHE_USED_SHARED", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_USED_SHARED }, +#endif + {"DB_SCHEMA_USED", MSV_DB, 10, SQLITE_DBSTATUS_SCHEMA_USED }, + {"DB_STMT_USED", MSV_DB, 10, SQLITE_DBSTATUS_STMT_USED }, + {"DB_CACHE_HIT", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_HIT }, + {"DB_CACHE_MISS", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_MISS }, + {"DB_CACHE_WRITE", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_WRITE }, +#if SQLITE_VERSION_NUMBER >= 3230000 + {"DB_CACHE_SPILL", MSV_DB, 10, SQLITE_DBSTATUS_CACHE_SPILL }, +#endif + {"DB_DEFERRED_FKS", MSV_DB, 10, SQLITE_DBSTATUS_DEFERRED_FKS }, +#ifdef SQLITE_ENABLE_ZIPVFS + {"ZIPVFS_CACHE_USED", MSV_ZIPVFS, 8, 231454 }, + {"ZIPVFS_CACHE_HIT", MSV_ZIPVFS, 8, 231455 }, + {"ZIPVFS_CACHE_MISS", MSV_ZIPVFS, 8, 231456 }, + {"ZIPVFS_CACHE_WRITE", MSV_ZIPVFS, 8, 231457 }, + {"ZIPVFS_DIRECT_READ", MSV_ZIPVFS, 8, 231458 }, + {"ZIPVFS_DIRECT_BYTES", MSV_ZIPVFS, 8, 231459 }, +#endif /* SQLITE_ENABLE_ZIPVFS */ +}; +#define MSV_NROW (sizeof(aMemstatColumn)/sizeof(aMemstatColumn[0])) + +/* +** Advance a memstat_cursor to its next row of output. +*/ +static int memstatNext(sqlite3_vtab_cursor *cur){ + memstat_cursor *pCur = (memstat_cursor*)cur; + int i; + assert( pCur->iRowid<=MSV_NROW ); + while(1){ + i = (int)pCur->iRowid - 1; + if( i<0 || (aMemstatColumn[i].mNull & 2)!=0 || (++pCur->iDb)>=pCur->nDb ){ + pCur->iRowid++; + if( pCur->iRowid>MSV_NROW ) return SQLITE_OK; /* End of the table */ + pCur->iDb = 0; + i++; + } + pCur->aVal[0] = 0; + pCur->aVal[1] = 0; + switch( aMemstatColumn[i].eType ){ + case MSV_GSTAT: { + if( sqlite3_libversion_number()>=3010000 ){ + sqlite3_status64(aMemstatColumn[i].eOp, + &pCur->aVal[0], &pCur->aVal[1],0); + }else{ + int xCur, xHiwtr; + sqlite3_status(aMemstatColumn[i].eOp, &xCur, &xHiwtr, 0); + pCur->aVal[0] = xCur; + pCur->aVal[1] = xHiwtr; + } + break; + } + case MSV_DB: { + int xCur, xHiwtr; + sqlite3_db_status(pCur->db, aMemstatColumn[i].eOp, &xCur, &xHiwtr, 0); + pCur->aVal[0] = xCur; + pCur->aVal[1] = xHiwtr; + break; + } + case MSV_ZIPVFS: { + int rc; + rc = sqlite3_file_control(pCur->db, pCur->azDb[pCur->iDb], + aMemstatColumn[i].eOp, (void*)&pCur->aVal[0]); + if( rc!=SQLITE_OK ) continue; + break; + } + } + break; + } + return SQLITE_OK; +} + + +/* +** Return values of columns for the row at which the memstat_cursor +** is currently pointing. +*/ +static int memstatColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int iCol /* Which column to return */ +){ + memstat_cursor *pCur = (memstat_cursor*)cur; + int i; + assert( pCur->iRowid>0 && pCur->iRowid<=MSV_NROW ); + i = (int)pCur->iRowid - 1; + if( (aMemstatColumn[i].mNull & (1<<iCol))!=0 ){ + return SQLITE_OK; + } + switch( iCol ){ + case MSV_COLUMN_NAME: { + sqlite3_result_text(ctx, aMemstatColumn[i].zName, -1, SQLITE_STATIC); + break; + } + case MSV_COLUMN_SCHEMA: { + sqlite3_result_text(ctx, pCur->azDb[pCur->iDb], -1, 0); + break; + } + case MSV_COLUMN_VALUE: { + sqlite3_result_int64(ctx, pCur->aVal[0]); + break; + } + case MSV_COLUMN_HIWTR: { + sqlite3_result_int64(ctx, pCur->aVal[1]); + break; + } + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int memstatRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + memstat_cursor *pCur = (memstat_cursor*)cur; + *pRowid = pCur->iRowid*1000 + pCur->iDb; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int memstatEof(sqlite3_vtab_cursor *cur){ + memstat_cursor *pCur = (memstat_cursor*)cur; + return pCur->iRowid>MSV_NROW; +} + +/* +** This method is called to "rewind" the memstat_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to memstatColumn() or memstatRowid() or +** memstatEof(). +*/ +static int memstatFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + memstat_cursor *pCur = (memstat_cursor *)pVtabCursor; + int rc = memstatFindSchemas(pCur); + if( rc ) return rc; + pCur->iRowid = 0; + pCur->iDb = 0; + return memstatNext(pVtabCursor); +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the memstat virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int memstatBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + pIdxInfo->estimatedCost = (double)500; + pIdxInfo->estimatedRows = 500; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** memstat virtual table. +*/ +static sqlite3_module memstatModule = { + 0, /* iVersion */ + 0, /* xCreate */ + memstatConnect, /* xConnect */ + memstatBestIndex, /* xBestIndex */ + memstatDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + memstatOpen, /* xOpen - open a cursor */ + memstatClose, /* xClose - close a cursor */ + memstatFilter, /* xFilter - configure scan constraints */ + memstatNext, /* xNext - advance a cursor */ + memstatEof, /* xEof - check for end of scan */ + memstatColumn, /* xColumn - read data */ + memstatRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0, /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +int sqlite3MemstatVtabInit(sqlite3 *db){ + int rc = SQLITE_OK; +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "sqlite_memstat", &memstatModule, 0); +#endif + return rc; +} + +#ifndef SQLITE_CORE +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_memstat_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3MemstatVtabInit(db); +#endif + return rc; +} +#endif /* SQLITE_CORE */ +#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_MEMSTATVTAB) */ diff --git a/ext/misc/memtrace.c b/ext/misc/memtrace.c new file mode 100644 index 0000000..19ff4c4 --- /dev/null +++ b/ext/misc/memtrace.c @@ -0,0 +1,108 @@ +/* +** 2019-01-21 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements an extension that uses the SQLITE_CONFIG_MALLOC +** mechanism to add a tracing layer on top of SQLite. If this extension +** is registered prior to sqlite3_initialize(), it will cause all memory +** allocation activities to be logged on standard output, or to some other +** FILE specified by the initializer. +** +** This file needs to be compiled into the application that uses it. +** +** This extension is used to implement the --memtrace option of the +** command-line shell. +*/ +#include <assert.h> +#include <string.h> +#include <stdio.h> + +/* The original memory allocation routines */ +static sqlite3_mem_methods memtraceBase; +static FILE *memtraceOut; + +/* Methods that trace memory allocations */ +static void *memtraceMalloc(int n){ + if( memtraceOut ){ + fprintf(memtraceOut, "MEMTRACE: allocate %d bytes\n", + memtraceBase.xRoundup(n)); + } + return memtraceBase.xMalloc(n); +} +static void memtraceFree(void *p){ + if( p==0 ) return; + if( memtraceOut ){ + fprintf(memtraceOut, "MEMTRACE: free %d bytes\n", memtraceBase.xSize(p)); + } + memtraceBase.xFree(p); +} +static void *memtraceRealloc(void *p, int n){ + if( p==0 ) return memtraceMalloc(n); + if( n==0 ){ + memtraceFree(p); + return 0; + } + if( memtraceOut ){ + fprintf(memtraceOut, "MEMTRACE: resize %d -> %d bytes\n", + memtraceBase.xSize(p), memtraceBase.xRoundup(n)); + } + return memtraceBase.xRealloc(p, n); +} +static int memtraceSize(void *p){ + return memtraceBase.xSize(p); +} +static int memtraceRoundup(int n){ + return memtraceBase.xRoundup(n); +} +static int memtraceInit(void *p){ + return memtraceBase.xInit(p); +} +static void memtraceShutdown(void *p){ + memtraceBase.xShutdown(p); +} + +/* The substitute memory allocator */ +static sqlite3_mem_methods ersaztMethods = { + memtraceMalloc, + memtraceFree, + memtraceRealloc, + memtraceSize, + memtraceRoundup, + memtraceInit, + memtraceShutdown, + 0 +}; + +/* Begin tracing memory allocations to out. */ +int sqlite3MemTraceActivate(FILE *out){ + int rc = SQLITE_OK; + if( memtraceBase.xMalloc==0 ){ + rc = sqlite3_config(SQLITE_CONFIG_GETMALLOC, &memtraceBase); + if( rc==SQLITE_OK ){ + rc = sqlite3_config(SQLITE_CONFIG_MALLOC, &ersaztMethods); + } + } + memtraceOut = out; + return rc; +} + +/* Deactivate memory tracing */ +int sqlite3MemTraceDeactivate(void){ + int rc = SQLITE_OK; + if( memtraceBase.xMalloc!=0 ){ + rc = sqlite3_config(SQLITE_CONFIG_MALLOC, &memtraceBase); + if( rc==SQLITE_OK ){ + memset(&memtraceBase, 0, sizeof(memtraceBase)); + } + } + memtraceOut = 0; + return rc; +} diff --git a/ext/misc/memvfs.c b/ext/misc/memvfs.c new file mode 100644 index 0000000..83fc946 --- /dev/null +++ b/ext/misc/memvfs.c @@ -0,0 +1,575 @@ +/* +** 2016-09-07 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This is an in-memory VFS implementation. The application supplies +** a chunk of memory to hold the database file. +** +** Because there is place to store a rollback or wal journal, the database +** must use one of journal_mode=MEMORY or journal_mode=NONE. +** +** USAGE: +** +** sqlite3_open_v2("file:/whatever?ptr=0xf05538&sz=14336&max=65536", &db, +** SQLITE_OPEN_READWRITE | SQLITE_OPEN_URI, +** "memvfs"); +** +** These are the query parameters: +** +** ptr= The address of the memory buffer that holds the database. +** +** sz= The current size the database file +** +** maxsz= The maximum size of the database. In other words, the +** amount of space allocated for the ptr= buffer. +** +** freeonclose= If true, then sqlite3_free() is called on the ptr= +** value when the connection closes. +** +** The ptr= and sz= query parameters are required. If maxsz= is omitted, +** then it defaults to the sz= value. Parameter values can be in either +** decimal or hexadecimal. The filename in the URI is ignored. +*/ +#include <sqlite3ext.h> +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> + + +/* +** Forward declaration of objects used by this utility +*/ +typedef struct sqlite3_vfs MemVfs; +typedef struct MemFile MemFile; + +/* Access to a lower-level VFS that (might) implement dynamic loading, +** access to randomness, etc. +*/ +#define ORIGVFS(p) ((sqlite3_vfs*)((p)->pAppData)) + +/* An open file */ +struct MemFile { + sqlite3_file base; /* IO methods */ + sqlite3_int64 sz; /* Size of the file */ + sqlite3_int64 szMax; /* Space allocated to aData */ + unsigned char *aData; /* content of the file */ + int bFreeOnClose; /* Invoke sqlite3_free() on aData at close */ +}; + +/* +** Methods for MemFile +*/ +static int memClose(sqlite3_file*); +static int memRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); +static int memWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); +static int memTruncate(sqlite3_file*, sqlite3_int64 size); +static int memSync(sqlite3_file*, int flags); +static int memFileSize(sqlite3_file*, sqlite3_int64 *pSize); +static int memLock(sqlite3_file*, int); +static int memUnlock(sqlite3_file*, int); +static int memCheckReservedLock(sqlite3_file*, int *pResOut); +static int memFileControl(sqlite3_file*, int op, void *pArg); +static int memSectorSize(sqlite3_file*); +static int memDeviceCharacteristics(sqlite3_file*); +static int memShmMap(sqlite3_file*, int iPg, int pgsz, int, void volatile**); +static int memShmLock(sqlite3_file*, int offset, int n, int flags); +static void memShmBarrier(sqlite3_file*); +static int memShmUnmap(sqlite3_file*, int deleteFlag); +static int memFetch(sqlite3_file*, sqlite3_int64 iOfst, int iAmt, void **pp); +static int memUnfetch(sqlite3_file*, sqlite3_int64 iOfst, void *p); + +/* +** Methods for MemVfs +*/ +static int memOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); +static int memDelete(sqlite3_vfs*, const char *zName, int syncDir); +static int memAccess(sqlite3_vfs*, const char *zName, int flags, int *); +static int memFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); +static void *memDlOpen(sqlite3_vfs*, const char *zFilename); +static void memDlError(sqlite3_vfs*, int nByte, char *zErrMsg); +static void (*memDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); +static void memDlClose(sqlite3_vfs*, void*); +static int memRandomness(sqlite3_vfs*, int nByte, char *zOut); +static int memSleep(sqlite3_vfs*, int microseconds); +static int memCurrentTime(sqlite3_vfs*, double*); +static int memGetLastError(sqlite3_vfs*, int, char *); +static int memCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); + +static sqlite3_vfs mem_vfs = { + 2, /* iVersion */ + 0, /* szOsFile (set when registered) */ + 1024, /* mxPathname */ + 0, /* pNext */ + "memvfs", /* zName */ + 0, /* pAppData (set when registered) */ + memOpen, /* xOpen */ + memDelete, /* xDelete */ + memAccess, /* xAccess */ + memFullPathname, /* xFullPathname */ + memDlOpen, /* xDlOpen */ + memDlError, /* xDlError */ + memDlSym, /* xDlSym */ + memDlClose, /* xDlClose */ + memRandomness, /* xRandomness */ + memSleep, /* xSleep */ + memCurrentTime, /* xCurrentTime */ + memGetLastError, /* xGetLastError */ + memCurrentTimeInt64 /* xCurrentTimeInt64 */ +}; + +static const sqlite3_io_methods mem_io_methods = { + 3, /* iVersion */ + memClose, /* xClose */ + memRead, /* xRead */ + memWrite, /* xWrite */ + memTruncate, /* xTruncate */ + memSync, /* xSync */ + memFileSize, /* xFileSize */ + memLock, /* xLock */ + memUnlock, /* xUnlock */ + memCheckReservedLock, /* xCheckReservedLock */ + memFileControl, /* xFileControl */ + memSectorSize, /* xSectorSize */ + memDeviceCharacteristics, /* xDeviceCharacteristics */ + memShmMap, /* xShmMap */ + memShmLock, /* xShmLock */ + memShmBarrier, /* xShmBarrier */ + memShmUnmap, /* xShmUnmap */ + memFetch, /* xFetch */ + memUnfetch /* xUnfetch */ +}; + + + +/* +** Close an mem-file. +** +** The pData pointer is owned by the application, so there is nothing +** to free. +*/ +static int memClose(sqlite3_file *pFile){ + MemFile *p = (MemFile *)pFile; + if( p->bFreeOnClose ) sqlite3_free(p->aData); + return SQLITE_OK; +} + +/* +** Read data from an mem-file. +*/ +static int memRead( + sqlite3_file *pFile, + void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + MemFile *p = (MemFile *)pFile; + memcpy(zBuf, p->aData+iOfst, iAmt); + return SQLITE_OK; +} + +/* +** Write data to an mem-file. +*/ +static int memWrite( + sqlite3_file *pFile, + const void *z, + int iAmt, + sqlite_int64 iOfst +){ + MemFile *p = (MemFile *)pFile; + if( iOfst+iAmt>p->sz ){ + if( iOfst+iAmt>p->szMax ) return SQLITE_FULL; + if( iOfst>p->sz ) memset(p->aData+p->sz, 0, iOfst-p->sz); + p->sz = iOfst+iAmt; + } + memcpy(p->aData+iOfst, z, iAmt); + return SQLITE_OK; +} + +/* +** Truncate an mem-file. +*/ +static int memTruncate(sqlite3_file *pFile, sqlite_int64 size){ + MemFile *p = (MemFile *)pFile; + if( size>p->sz ){ + if( size>p->szMax ) return SQLITE_FULL; + memset(p->aData+p->sz, 0, size-p->sz); + } + p->sz = size; + return SQLITE_OK; +} + +/* +** Sync an mem-file. +*/ +static int memSync(sqlite3_file *pFile, int flags){ + return SQLITE_OK; +} + +/* +** Return the current file-size of an mem-file. +*/ +static int memFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ + MemFile *p = (MemFile *)pFile; + *pSize = p->sz; + return SQLITE_OK; +} + +/* +** Lock an mem-file. +*/ +static int memLock(sqlite3_file *pFile, int eLock){ + return SQLITE_OK; +} + +/* +** Unlock an mem-file. +*/ +static int memUnlock(sqlite3_file *pFile, int eLock){ + return SQLITE_OK; +} + +/* +** Check if another file-handle holds a RESERVED lock on an mem-file. +*/ +static int memCheckReservedLock(sqlite3_file *pFile, int *pResOut){ + *pResOut = 0; + return SQLITE_OK; +} + +/* +** File control method. For custom operations on an mem-file. +*/ +static int memFileControl(sqlite3_file *pFile, int op, void *pArg){ + MemFile *p = (MemFile *)pFile; + int rc = SQLITE_NOTFOUND; + if( op==SQLITE_FCNTL_VFSNAME ){ + *(char**)pArg = sqlite3_mprintf("mem(%p,%lld)", p->aData, p->sz); + rc = SQLITE_OK; + } + return rc; +} + +/* +** Return the sector-size in bytes for an mem-file. +*/ +static int memSectorSize(sqlite3_file *pFile){ + return 1024; +} + +/* +** Return the device characteristic flags supported by an mem-file. +*/ +static int memDeviceCharacteristics(sqlite3_file *pFile){ + return SQLITE_IOCAP_ATOMIC | + SQLITE_IOCAP_POWERSAFE_OVERWRITE | + SQLITE_IOCAP_SAFE_APPEND | + SQLITE_IOCAP_SEQUENTIAL; +} + +/* Create a shared memory file mapping */ +static int memShmMap( + sqlite3_file *pFile, + int iPg, + int pgsz, + int bExtend, + void volatile **pp +){ + return SQLITE_IOERR_SHMMAP; +} + +/* Perform locking on a shared-memory segment */ +static int memShmLock(sqlite3_file *pFile, int offset, int n, int flags){ + return SQLITE_IOERR_SHMLOCK; +} + +/* Memory barrier operation on shared memory */ +static void memShmBarrier(sqlite3_file *pFile){ + return; +} + +/* Unmap a shared memory segment */ +static int memShmUnmap(sqlite3_file *pFile, int deleteFlag){ + return SQLITE_OK; +} + +/* Fetch a page of a memory-mapped file */ +static int memFetch( + sqlite3_file *pFile, + sqlite3_int64 iOfst, + int iAmt, + void **pp +){ + MemFile *p = (MemFile *)pFile; + *pp = (void*)(p->aData + iOfst); + return SQLITE_OK; +} + +/* Release a memory-mapped page */ +static int memUnfetch(sqlite3_file *pFile, sqlite3_int64 iOfst, void *pPage){ + return SQLITE_OK; +} + +/* +** Open an mem file handle. +*/ +static int memOpen( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_file *pFile, + int flags, + int *pOutFlags +){ + MemFile *p = (MemFile*)pFile; + memset(p, 0, sizeof(*p)); + if( (flags & SQLITE_OPEN_MAIN_DB)==0 ) return SQLITE_CANTOPEN; + p->aData = (unsigned char*)sqlite3_uri_int64(zName,"ptr",0); + if( p->aData==0 ) return SQLITE_CANTOPEN; + p->sz = sqlite3_uri_int64(zName,"sz",0); + if( p->sz<0 ) return SQLITE_CANTOPEN; + p->szMax = sqlite3_uri_int64(zName,"max",p->sz); + if( p->szMax<p->sz ) return SQLITE_CANTOPEN; + p->bFreeOnClose = sqlite3_uri_boolean(zName,"freeonclose",0); + pFile->pMethods = &mem_io_methods; + return SQLITE_OK; +} + +/* +** Delete the file located at zPath. If the dirSync argument is true, +** ensure the file-system modifications are synced to disk before +** returning. +*/ +static int memDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ + return SQLITE_IOERR_DELETE; +} + +/* +** Test for access permissions. Return true if the requested permission +** is available, or false otherwise. +*/ +static int memAccess( + sqlite3_vfs *pVfs, + const char *zPath, + int flags, + int *pResOut +){ + *pResOut = 0; + return SQLITE_OK; +} + +/* +** Populate buffer zOut with the full canonical pathname corresponding +** to the pathname in zPath. zOut is guaranteed to point to a buffer +** of at least (INST_MAX_PATHNAME+1) bytes. +*/ +static int memFullPathname( + sqlite3_vfs *pVfs, + const char *zPath, + int nOut, + char *zOut +){ + sqlite3_snprintf(nOut, zOut, "%s", zPath); + return SQLITE_OK; +} + +/* +** Open the dynamic library located at zPath and return a handle. +*/ +static void *memDlOpen(sqlite3_vfs *pVfs, const char *zPath){ + return ORIGVFS(pVfs)->xDlOpen(ORIGVFS(pVfs), zPath); +} + +/* +** Populate the buffer zErrMsg (size nByte bytes) with a human readable +** utf-8 string describing the most recent error encountered associated +** with dynamic libraries. +*/ +static void memDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ + ORIGVFS(pVfs)->xDlError(ORIGVFS(pVfs), nByte, zErrMsg); +} + +/* +** Return a pointer to the symbol zSymbol in the dynamic library pHandle. +*/ +static void (*memDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ + return ORIGVFS(pVfs)->xDlSym(ORIGVFS(pVfs), p, zSym); +} + +/* +** Close the dynamic library handle pHandle. +*/ +static void memDlClose(sqlite3_vfs *pVfs, void *pHandle){ + ORIGVFS(pVfs)->xDlClose(ORIGVFS(pVfs), pHandle); +} + +/* +** Populate the buffer pointed to by zBufOut with nByte bytes of +** random data. +*/ +static int memRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ + return ORIGVFS(pVfs)->xRandomness(ORIGVFS(pVfs), nByte, zBufOut); +} + +/* +** Sleep for nMicro microseconds. Return the number of microseconds +** actually slept. +*/ +static int memSleep(sqlite3_vfs *pVfs, int nMicro){ + return ORIGVFS(pVfs)->xSleep(ORIGVFS(pVfs), nMicro); +} + +/* +** Return the current time as a Julian Day number in *pTimeOut. +*/ +static int memCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ + return ORIGVFS(pVfs)->xCurrentTime(ORIGVFS(pVfs), pTimeOut); +} + +static int memGetLastError(sqlite3_vfs *pVfs, int a, char *b){ + return ORIGVFS(pVfs)->xGetLastError(ORIGVFS(pVfs), a, b); +} +static int memCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ + return ORIGVFS(pVfs)->xCurrentTimeInt64(ORIGVFS(pVfs), p); +} + +#ifdef MEMVFS_TEST +/* +** memvfs_from_file(FILENAME, MAXSIZE) +** +** This an SQL function used to help in testing the memvfs VFS. The +** function reads the content of a file into memory and then returns +** a URI that can be handed to ATTACH to attach the memory buffer as +** a database. Example: +** +** ATTACH memvfs_from_file('test.db',1048576) AS inmem; +** +** The optional MAXSIZE argument gives the size of the memory allocation +** used to hold the database. If omitted, it defaults to the size of the +** file on disk. +*/ +#include <stdio.h> +static void memvfsFromFileFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char *p; + sqlite3_int64 sz; + sqlite3_int64 szMax; + FILE *in; + const char *zFilename = (const char*)sqlite3_value_text(argv[0]); + char *zUri; + + if( zFilename==0 ) return; + in = fopen(zFilename, "rb"); + if( in==0 ) return; + fseek(in, 0, SEEK_END); + szMax = sz = ftell(in); + rewind(in); + if( argc>=2 ){ + szMax = sqlite3_value_int64(argv[1]); + if( szMax<sz ) szMax = sz; + } + p = sqlite3_malloc64( szMax ); + if( p==0 ){ + fclose(in); + sqlite3_result_error_nomem(context); + return; + } + fread(p, sz, 1, in); + fclose(in); + zUri = sqlite3_mprintf( + "file:/mem?vfs=memvfs&ptr=%lld&sz=%lld&max=%lld&freeonclose=1", + (sqlite3_int64)p, sz, szMax); + sqlite3_result_text(context, zUri, -1, sqlite3_free); +} +#endif /* MEMVFS_TEST */ + +#ifdef MEMVFS_TEST +/* +** memvfs_to_file(SCHEMA, FILENAME) +** +** The schema identified by SCHEMA must be a memvfs database. Write +** the content of this database into FILENAME. +*/ +static void memvfsToFileFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + MemFile *p = 0; + FILE *out; + int rc; + sqlite3 *db = sqlite3_context_db_handle(context); + sqlite3_vfs *pVfs = 0; + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + const char *zFilename = (const char*)sqlite3_value_text(argv[1]); + + if( zFilename==0 ) return; + out = fopen(zFilename, "wb"); + if( out==0 ) return; + rc = sqlite3_file_control(db, zSchema, SQLITE_FCNTL_VFS_POINTER, &pVfs); + if( rc || pVfs==0 ) return; + if( strcmp(pVfs->zName,"memvfs")!=0 ) return; + rc = sqlite3_file_control(db, zSchema, SQLITE_FCNTL_FILE_POINTER, &p); + if( rc ) return; + fwrite(p->aData, 1, (size_t)p->sz, out); + fclose(out); +} +#endif /* MEMVFS_TEST */ + +#ifdef MEMVFS_TEST +/* Called for each new database connection */ +static int memvfsRegister( + sqlite3 *db, + char **pzErrMsg, + const struct sqlite3_api_routines *pThunk +){ + sqlite3_create_function(db, "memvfs_from_file", 1, SQLITE_UTF8, 0, + memvfsFromFileFunc, 0, 0); + sqlite3_create_function(db, "memvfs_from_file", 2, SQLITE_UTF8, 0, + memvfsFromFileFunc, 0, 0); + sqlite3_create_function(db, "memvfs_to_file", 2, SQLITE_UTF8, 0, + memvfsToFileFunc, 0, 0); + return SQLITE_OK; +} +#endif /* MEMVFS_TEST */ + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +/* +** This routine is called when the extension is loaded. +** Register the new VFS. +*/ +int sqlite3_memvfs_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + mem_vfs.pAppData = sqlite3_vfs_find(0); + if( mem_vfs.pAppData==0 ) return SQLITE_ERROR; + mem_vfs.szOsFile = sizeof(MemFile); + rc = sqlite3_vfs_register(&mem_vfs, 1); +#ifdef MEMVFS_TEST + if( rc==SQLITE_OK ){ + rc = sqlite3_auto_extension((void(*)(void))memvfsRegister); + } + if( rc==SQLITE_OK ){ + rc = memvfsRegister(db, pzErrMsg, pApi); + } +#endif + if( rc==SQLITE_OK ) rc = SQLITE_OK_LOAD_PERMANENTLY; + return rc; +} diff --git a/ext/misc/mmapwarm.c b/ext/misc/mmapwarm.c new file mode 100644 index 0000000..5afa47b --- /dev/null +++ b/ext/misc/mmapwarm.c @@ -0,0 +1,107 @@ +/* +** 2017-09-18 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +*/ + +#include "sqlite3.h" + + +/* +** This function is used to touch each page of a mapping of a memory +** mapped SQLite database. Assuming that the system has sufficient free +** memory and supports sufficiently large mappings, this causes the OS +** to cache the entire database in main memory, making subsequent +** database accesses faster. +** +** If the second parameter to this function is not NULL, it is the name of +** the specific database to operate on (i.e. "main" or the name of an +** attached database). +** +** SQLITE_OK is returned if successful, or an SQLite error code otherwise. +** It is not considered an error if the file is not memory-mapped, or if +** the mapping does not span the entire file. If an error does occur, a +** transaction may be left open on the database file. +** +** It is illegal to call this function when the database handle has an +** open transaction. SQLITE_MISUSE is returned in this case. +*/ +int sqlite3_mmap_warm(sqlite3 *db, const char *zDb){ + int rc = SQLITE_OK; + char *zSql = 0; + int pgsz = 0; + int nTotal = 0; + + if( 0==sqlite3_get_autocommit(db) ) return SQLITE_MISUSE; + + /* Open a read-only transaction on the file in question */ + zSql = sqlite3_mprintf("BEGIN; SELECT * FROM %s%q%ssqlite_schema", + (zDb ? "'" : ""), (zDb ? zDb : ""), (zDb ? "'." : "") + ); + if( zSql==0 ) return SQLITE_NOMEM; + rc = sqlite3_exec(db, zSql, 0, 0, 0); + sqlite3_free(zSql); + + /* Find the SQLite page size of the file */ + if( rc==SQLITE_OK ){ + zSql = sqlite3_mprintf("PRAGMA %s%q%spage_size", + (zDb ? "'" : ""), (zDb ? zDb : ""), (zDb ? "'." : "") + ); + if( zSql==0 ){ + rc = SQLITE_NOMEM; + }else{ + sqlite3_stmt *pPgsz = 0; + rc = sqlite3_prepare_v2(db, zSql, -1, &pPgsz, 0); + sqlite3_free(zSql); + if( rc==SQLITE_OK ){ + if( sqlite3_step(pPgsz)==SQLITE_ROW ){ + pgsz = sqlite3_column_int(pPgsz, 0); + } + rc = sqlite3_finalize(pPgsz); + } + if( rc==SQLITE_OK && pgsz==0 ){ + rc = SQLITE_ERROR; + } + } + } + + /* Touch each mmap'd page of the file */ + if( rc==SQLITE_OK ){ + int rc2; + sqlite3_file *pFd = 0; + rc = sqlite3_file_control(db, zDb, SQLITE_FCNTL_FILE_POINTER, &pFd); + if( rc==SQLITE_OK && pFd->pMethods->iVersion>=3 ){ + sqlite3_int64 iPg = 1; + sqlite3_io_methods const *p = pFd->pMethods; + while( 1 ){ + unsigned char *pMap; + rc = p->xFetch(pFd, pgsz*iPg, pgsz, (void**)&pMap); + if( rc!=SQLITE_OK || pMap==0 ) break; + + nTotal += pMap[0]; + nTotal += pMap[pgsz-1]; + + rc = p->xUnfetch(pFd, pgsz*iPg, (void*)pMap); + if( rc!=SQLITE_OK ) break; + iPg++; + } + sqlite3_log(SQLITE_OK, + "sqlite3_mmap_warm_cache: Warmed up %d pages of %s", iPg==1?0:iPg, + sqlite3_db_filename(db, zDb) + ); + } + + rc2 = sqlite3_exec(db, "END", 0, 0, 0); + if( rc==SQLITE_OK ) rc = rc2; + } + + return rc; +} diff --git a/ext/misc/nextchar.c b/ext/misc/nextchar.c new file mode 100644 index 0000000..60fa3db --- /dev/null +++ b/ext/misc/nextchar.c @@ -0,0 +1,314 @@ +/* +** 2013-02-28 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains code to implement the next_char(A,T,F,W,C) SQL function. +** +** The next_char(A,T,F,W,C) function finds all valid "next" characters for +** string A given the vocabulary in T.F. If the W value exists and is a +** non-empty string, then it is an SQL expression that limits the entries +** in T.F that will be considered. If C exists and is a non-empty string, +** then it is the name of the collating sequence to use for comparison. If +** +** Only the first three arguments are required. If the C parameter is +** omitted or is NULL or is an empty string, then the default collating +** sequence of T.F is used for comparision. If the W parameter is omitted +** or is NULL or is an empty string, then no filtering of the output is +** done. +** +** The T.F column should be indexed using collation C or else this routine +** will be quite slow. +** +** For example, suppose an application has a dictionary like this: +** +** CREATE TABLE dictionary(word TEXT UNIQUE); +** +** Further suppose that for user keypad entry, it is desired to disable +** (gray out) keys that are not valid as the next character. If the +** the user has previously entered (say) 'cha' then to find all allowed +** next characters (and thereby determine when keys should not be grayed +** out) run the following query: +** +** SELECT next_char('cha','dictionary','word'); +** +** IMPLEMENTATION NOTES: +** +** The next_char function is implemented using recursive SQL that makes +** use of the table name and column name as part of a query. If either +** the table name or column name are keywords or contain special characters, +** then they should be escaped. For example: +** +** SELECT next_char('cha','[dictionary]','[word]'); +** +** This also means that the table name can be a subquery: +** +** SELECT next_char('cha','(SELECT word AS w FROM dictionary)','w'); +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <string.h> + +/* +** A structure to hold context of the next_char() computation across +** nested function calls. +*/ +typedef struct nextCharContext nextCharContext; +struct nextCharContext { + sqlite3 *db; /* Database connection */ + sqlite3_stmt *pStmt; /* Prepared statement used to query */ + const unsigned char *zPrefix; /* Prefix to scan */ + int nPrefix; /* Size of zPrefix in bytes */ + int nAlloc; /* Space allocated to aResult */ + int nUsed; /* Space used in aResult */ + unsigned int *aResult; /* Array of next characters */ + int mallocFailed; /* True if malloc fails */ + int otherError; /* True for any other failure */ +}; + +/* +** Append a result character if the character is not already in the +** result. +*/ +static void nextCharAppend(nextCharContext *p, unsigned c){ + int i; + for(i=0; i<p->nUsed; i++){ + if( p->aResult[i]==c ) return; + } + if( p->nUsed+1 > p->nAlloc ){ + unsigned int *aNew; + int n = p->nAlloc*2 + 30; + aNew = sqlite3_realloc64(p->aResult, n*sizeof(unsigned int)); + if( aNew==0 ){ + p->mallocFailed = 1; + return; + }else{ + p->aResult = aNew; + p->nAlloc = n; + } + } + p->aResult[p->nUsed++] = c; +} + +/* +** Write a character into z[] as UTF8. Return the number of bytes needed +** to hold the character +*/ +static int writeUtf8(unsigned char *z, unsigned c){ + if( c<0x00080 ){ + z[0] = (unsigned char)(c&0xff); + return 1; + } + if( c<0x00800 ){ + z[0] = 0xC0 + (unsigned char)((c>>6)&0x1F); + z[1] = 0x80 + (unsigned char)(c & 0x3F); + return 2; + } + if( c<0x10000 ){ + z[0] = 0xE0 + (unsigned char)((c>>12)&0x0F); + z[1] = 0x80 + (unsigned char)((c>>6) & 0x3F); + z[2] = 0x80 + (unsigned char)(c & 0x3F); + return 3; + } + z[0] = 0xF0 + (unsigned char)((c>>18) & 0x07); + z[1] = 0x80 + (unsigned char)((c>>12) & 0x3F); + z[2] = 0x80 + (unsigned char)((c>>6) & 0x3F); + z[3] = 0x80 + (unsigned char)(c & 0x3F); + return 4; +} + +/* +** Read a UTF8 character out of z[] and write it into *pOut. Return +** the number of bytes in z[] that were used to construct the character. +*/ +static int readUtf8(const unsigned char *z, unsigned *pOut){ + static const unsigned char validBits[] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x00, 0x01, 0x02, 0x03, 0x00, 0x01, 0x00, 0x00, + }; + unsigned c = z[0]; + if( c<0xc0 ){ + *pOut = c; + return 1; + }else{ + int n = 1; + c = validBits[c-0xc0]; + while( (z[n] & 0xc0)==0x80 ){ + c = (c<<6) + (0x3f & z[n++]); + } + if( c<0x80 || (c&0xFFFFF800)==0xD800 || (c&0xFFFFFFFE)==0xFFFE ){ + c = 0xFFFD; + } + *pOut = c; + return n; + } +} + +/* +** The nextCharContext structure has been set up. Add all "next" characters +** to the result set. +*/ +static void findNextChars(nextCharContext *p){ + unsigned cPrev = 0; + unsigned char zPrev[8]; + int n, rc; + + for(;;){ + sqlite3_bind_text(p->pStmt, 1, (char*)p->zPrefix, p->nPrefix, + SQLITE_STATIC); + n = writeUtf8(zPrev, cPrev+1); + sqlite3_bind_text(p->pStmt, 2, (char*)zPrev, n, SQLITE_STATIC); + rc = sqlite3_step(p->pStmt); + if( rc==SQLITE_DONE ){ + sqlite3_reset(p->pStmt); + return; + }else if( rc!=SQLITE_ROW ){ + p->otherError = rc; + return; + }else{ + const unsigned char *zOut = sqlite3_column_text(p->pStmt, 0); + unsigned cNext; + n = readUtf8(zOut+p->nPrefix, &cNext); + sqlite3_reset(p->pStmt); + nextCharAppend(p, cNext); + cPrev = cNext; + if( p->mallocFailed ) return; + } + } +} + + +/* +** next_character(A,T,F,W) +** +** Return a string composted of all next possible characters after +** A for elements of T.F. If W is supplied, then it is an SQL expression +** that limits the elements in T.F that are considered. +*/ +static void nextCharFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + nextCharContext c; + const unsigned char *zTable = sqlite3_value_text(argv[1]); + const unsigned char *zField = sqlite3_value_text(argv[2]); + const unsigned char *zWhere; + const unsigned char *zCollName; + char *zWhereClause = 0; + char *zColl = 0; + char *zSql; + int rc; + + memset(&c, 0, sizeof(c)); + c.db = sqlite3_context_db_handle(context); + c.zPrefix = sqlite3_value_text(argv[0]); + c.nPrefix = sqlite3_value_bytes(argv[0]); + if( zTable==0 || zField==0 || c.zPrefix==0 ) return; + if( argc>=4 + && (zWhere = sqlite3_value_text(argv[3]))!=0 + && zWhere[0]!=0 + ){ + zWhereClause = sqlite3_mprintf("AND (%s)", zWhere); + if( zWhereClause==0 ){ + sqlite3_result_error_nomem(context); + return; + } + }else{ + zWhereClause = ""; + } + if( argc>=5 + && (zCollName = sqlite3_value_text(argv[4]))!=0 + && zCollName[0]!=0 + ){ + zColl = sqlite3_mprintf("collate \"%w\"", zCollName); + if( zColl==0 ){ + sqlite3_result_error_nomem(context); + if( zWhereClause[0] ) sqlite3_free(zWhereClause); + return; + } + }else{ + zColl = ""; + } + zSql = sqlite3_mprintf( + "SELECT %s FROM %s" + " WHERE %s>=(?1 || ?2) %s" + " AND %s<=(?1 || char(1114111)) %s" /* 1114111 == 0x10ffff */ + " %s" + " ORDER BY 1 %s ASC LIMIT 1", + zField, zTable, zField, zColl, zField, zColl, zWhereClause, zColl + ); + if( zWhereClause[0] ) sqlite3_free(zWhereClause); + if( zColl[0] ) sqlite3_free(zColl); + if( zSql==0 ){ + sqlite3_result_error_nomem(context); + return; + } + + rc = sqlite3_prepare_v2(c.db, zSql, -1, &c.pStmt, 0); + sqlite3_free(zSql); + if( rc ){ + sqlite3_result_error(context, sqlite3_errmsg(c.db), -1); + return; + } + findNextChars(&c); + if( c.mallocFailed ){ + sqlite3_result_error_nomem(context); + }else{ + unsigned char *pRes; + pRes = sqlite3_malloc64( c.nUsed*4 + 1 ); + if( pRes==0 ){ + sqlite3_result_error_nomem(context); + }else{ + int i; + int n = 0; + for(i=0; i<c.nUsed; i++){ + n += writeUtf8(pRes+n, c.aResult[i]); + } + pRes[n] = 0; + sqlite3_result_text(context, (const char*)pRes, n, sqlite3_free); + } + } + sqlite3_finalize(c.pStmt); + sqlite3_free(c.aResult); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_nextchar_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "next_char", 3, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + nextCharFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "next_char", 4, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + nextCharFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "next_char", 5, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + nextCharFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/noop.c b/ext/misc/noop.c new file mode 100644 index 0000000..d3a5867 --- /dev/null +++ b/ext/misc/noop.c @@ -0,0 +1,68 @@ +/* +** 2020-01-08 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements a noop() function used for testing. +** +** Variants: +** +** noop(X) The default. Deterministic. +** noop_i(X) Deterministic and innocuous. +** noop_do(X) Deterministic and direct-only. +** noop_nd(X) Non-deterministic. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* +** Implementation of the noop() function. +** +** The function returns its argument, unchanged. +*/ +static void noopfunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + assert( argc==1 ); + sqlite3_result_value(context, argv[0]); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_noop_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "noop", 1, + SQLITE_UTF8 | SQLITE_DETERMINISTIC, + 0, noopfunc, 0, 0); + if( rc ) return rc; + rc = sqlite3_create_function(db, "noop_i", 1, + SQLITE_UTF8 | SQLITE_DETERMINISTIC | SQLITE_INNOCUOUS, + 0, noopfunc, 0, 0); + if( rc ) return rc; + rc = sqlite3_create_function(db, "noop_do", 1, + SQLITE_UTF8 | SQLITE_DETERMINISTIC | SQLITE_DIRECTONLY, + 0, noopfunc, 0, 0); + if( rc ) return rc; + rc = sqlite3_create_function(db, "noop_nd", 1, + SQLITE_UTF8, + 0, noopfunc, 0, 0); + return rc; +} diff --git a/ext/misc/normalize.c b/ext/misc/normalize.c new file mode 100644 index 0000000..08d7733 --- /dev/null +++ b/ext/misc/normalize.c @@ -0,0 +1,716 @@ +/* +** 2018-01-08 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains code to implement the sqlite3_normalize() function. +** +** char *sqlite3_normalize(const char *zSql); +** +** This function takes an SQL string as input and returns a "normalized" +** version of that string in memory obtained from sqlite3_malloc64(). The +** caller is responsible for ensuring that the returned memory is freed. +** +** If a memory allocation error occurs, this routine returns NULL. +** +** The normalization consists of the following transformations: +** +** (1) Convert every literal (string, blob literal, numeric constant, +** or "NULL" constant) into a ? +** +** (2) Remove all superfluous whitespace, including comments. Change +** all required whitespace to a single space character. +** +** (3) Lowercase all ASCII characters. +** +** (4) If an IN or NOT IN operator is followed by a list of 1 or more +** values, convert that list into "(?,?,?)". +** +** The purpose of normalization is two-fold: +** +** (1) Sanitize queries by removing potentially private or sensitive +** information contained in literals. +** +** (2) Identify structurally identical queries by comparing their +** normalized forms. +** +** Command-Line Utility +** -------------------- +** +** This file also contains code for a command-line utility that converts +** SQL queries in text files into their normalized forms. To build the +** command-line program, compile this file with -DSQLITE_NORMALIZE_CLI +** and link it against the SQLite library. +*/ +#include <sqlite3.h> +#include <string.h> + +/* +** Implementation note: +** +** Much of the tokenizer logic is copied out of the tokenize.c source file +** of SQLite. That logic could be simplified for this particular application, +** but that would impose a risk of introducing subtle errors. It is best to +** keep the code as close to the original as possible. +** +** The tokenize code is in sync with the SQLite core as of 2018-01-08. +** Any future changes to the core tokenizer might require corresponding +** adjustments to the tokenizer logic in this module. +*/ + + +/* Character classes for tokenizing +** +** In the sqlite3GetToken() function, a switch() on aiClass[c] is implemented +** using a lookup table, whereas a switch() directly on c uses a binary search. +** The lookup table is much faster. To maximize speed, and to ensure that +** a lookup table is used, all of the classes need to be small integers and +** all of them need to be used within the switch. +*/ +#define CC_X 0 /* The letter 'x', or start of BLOB literal */ +#define CC_KYWD 1 /* Alphabetics or '_'. Usable in a keyword */ +#define CC_ID 2 /* unicode characters usable in IDs */ +#define CC_DIGIT 3 /* Digits */ +#define CC_DOLLAR 4 /* '$' */ +#define CC_VARALPHA 5 /* '@', '#', ':'. Alphabetic SQL variables */ +#define CC_VARNUM 6 /* '?'. Numeric SQL variables */ +#define CC_SPACE 7 /* Space characters */ +#define CC_QUOTE 8 /* '"', '\'', or '`'. String literals, quoted ids */ +#define CC_QUOTE2 9 /* '['. [...] style quoted ids */ +#define CC_PIPE 10 /* '|'. Bitwise OR or concatenate */ +#define CC_MINUS 11 /* '-'. Minus or SQL-style comment */ +#define CC_LT 12 /* '<'. Part of < or <= or <> */ +#define CC_GT 13 /* '>'. Part of > or >= */ +#define CC_EQ 14 /* '='. Part of = or == */ +#define CC_BANG 15 /* '!'. Part of != */ +#define CC_SLASH 16 /* '/'. / or c-style comment */ +#define CC_LP 17 /* '(' */ +#define CC_RP 18 /* ')' */ +#define CC_SEMI 19 /* ';' */ +#define CC_PLUS 20 /* '+' */ +#define CC_STAR 21 /* '*' */ +#define CC_PERCENT 22 /* '%' */ +#define CC_COMMA 23 /* ',' */ +#define CC_AND 24 /* '&' */ +#define CC_TILDA 25 /* '~' */ +#define CC_DOT 26 /* '.' */ +#define CC_ILLEGAL 27 /* Illegal character */ + +static const unsigned char aiClass[] = { +/* x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 xa xb xc xd xe xf */ +/* 0x */ 27, 27, 27, 27, 27, 27, 27, 27, 27, 7, 7, 27, 7, 7, 27, 27, +/* 1x */ 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, +/* 2x */ 7, 15, 8, 5, 4, 22, 24, 8, 17, 18, 21, 20, 23, 11, 26, 16, +/* 3x */ 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 5, 19, 12, 14, 13, 6, +/* 4x */ 5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, +/* 5x */ 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 9, 27, 27, 27, 1, +/* 6x */ 8, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, +/* 7x */ 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 27, 10, 27, 25, 27, +/* 8x */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* 9x */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Ax */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Bx */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Cx */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Dx */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Ex */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, +/* Fx */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 +}; + +/* An array to map all upper-case characters into their corresponding +** lower-case character. +** +** SQLite only considers US-ASCII (or EBCDIC) characters. We do not +** handle case conversions for the UTF character set since the tables +** involved are nearly as big or bigger than SQLite itself. +*/ +static const unsigned char sqlite3UpperToLower[] = { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, + 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, + 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, + 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 97, 98, 99,100,101,102,103, + 104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121, + 122, 91, 92, 93, 94, 95, 96, 97, 98, 99,100,101,102,103,104,105,106,107, + 108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125, + 126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143, + 144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161, + 162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179, + 180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197, + 198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215, + 216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233, + 234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251, + 252,253,254,255 +}; + +/* +** The following 256 byte lookup table is used to support SQLites built-in +** equivalents to the following standard library functions: +** +** isspace() 0x01 +** isalpha() 0x02 +** isdigit() 0x04 +** isalnum() 0x06 +** isxdigit() 0x08 +** toupper() 0x20 +** SQLite identifier character 0x40 +** Quote character 0x80 +** +** Bit 0x20 is set if the mapped character requires translation to upper +** case. i.e. if the character is a lower-case ASCII character. +** If x is a lower-case ASCII character, then its upper-case equivalent +** is (x - 0x20). Therefore toupper() can be implemented as: +** +** (x & ~(map[x]&0x20)) +** +** The equivalent of tolower() is implemented using the sqlite3UpperToLower[] +** array. tolower() is used more often than toupper() by SQLite. +** +** Bit 0x40 is set if the character is non-alphanumeric and can be used in an +** SQLite identifier. Identifiers are alphanumerics, "_", "$", and any +** non-ASCII UTF character. Hence the test for whether or not a character is +** part of an identifier is 0x46. +*/ +static const unsigned char sqlite3CtypeMap[256] = { + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 00..07 ........ */ + 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x00, 0x00, /* 08..0f ........ */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 10..17 ........ */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 18..1f ........ */ + 0x01, 0x00, 0x80, 0x00, 0x40, 0x00, 0x00, 0x80, /* 20..27 !"#$%&' */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 28..2f ()*+,-./ */ + 0x0c, 0x0c, 0x0c, 0x0c, 0x0c, 0x0c, 0x0c, 0x0c, /* 30..37 01234567 */ + 0x0c, 0x0c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 38..3f 89:;<=>? */ + + 0x00, 0x0a, 0x0a, 0x0a, 0x0a, 0x0a, 0x0a, 0x02, /* 40..47 @ABCDEFG */ + 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, /* 48..4f HIJKLMNO */ + 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, /* 50..57 PQRSTUVW */ + 0x02, 0x02, 0x02, 0x80, 0x00, 0x00, 0x00, 0x40, /* 58..5f XYZ[\]^_ */ + 0x80, 0x2a, 0x2a, 0x2a, 0x2a, 0x2a, 0x2a, 0x22, /* 60..67 `abcdefg */ + 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, /* 68..6f hijklmno */ + 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, /* 70..77 pqrstuvw */ + 0x22, 0x22, 0x22, 0x00, 0x00, 0x00, 0x00, 0x00, /* 78..7f xyz{|}~. */ + + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* 80..87 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* 88..8f ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* 90..97 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* 98..9f ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* a0..a7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* a8..af ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* b0..b7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* b8..bf ........ */ + + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* c0..c7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* c8..cf ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* d0..d7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* d8..df ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* e0..e7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* e8..ef ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, /* f0..f7 ........ */ + 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40 /* f8..ff ........ */ +}; +#define sqlite3Toupper(x) ((x)&~(sqlite3CtypeMap[(unsigned char)(x)]&0x20)) +#define sqlite3Isspace(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x01) +#define sqlite3Isalnum(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x06) +#define sqlite3Isalpha(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x02) +#define sqlite3Isdigit(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x04) +#define sqlite3Isxdigit(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x08) +#define sqlite3Tolower(x) (sqlite3UpperToLower[(unsigned char)(x)]) +#define sqlite3Isquote(x) (sqlite3CtypeMap[(unsigned char)(x)]&0x80) + + +/* +** If X is a character that can be used in an identifier then +** IdChar(X) will be true. Otherwise it is false. +** +** For ASCII, any character with the high-order bit set is +** allowed in an identifier. For 7-bit characters, +** sqlite3IsIdChar[X] must be 1. +** +** For EBCDIC, the rules are more complex but have the same +** end result. +** +** Ticket #1066. the SQL standard does not allow '$' in the +** middle of identifiers. But many SQL implementations do. +** SQLite will allow '$' in identifiers for compatibility. +** But the feature is undocumented. +*/ +#define IdChar(C) ((sqlite3CtypeMap[(unsigned char)C]&0x46)!=0) + +/* +** Ignore testcase() macros +*/ +#define testcase(X) + +/* +** Token values +*/ +#define TK_SPACE 0 +#define TK_NAME 1 +#define TK_LITERAL 2 +#define TK_PUNCT 3 +#define TK_ERROR 4 + +#define TK_MINUS TK_PUNCT +#define TK_LP TK_PUNCT +#define TK_RP TK_PUNCT +#define TK_SEMI TK_PUNCT +#define TK_PLUS TK_PUNCT +#define TK_STAR TK_PUNCT +#define TK_SLASH TK_PUNCT +#define TK_REM TK_PUNCT +#define TK_EQ TK_PUNCT +#define TK_LE TK_PUNCT +#define TK_NE TK_PUNCT +#define TK_LSHIFT TK_PUNCT +#define TK_LT TK_PUNCT +#define TK_GE TK_PUNCT +#define TK_RSHIFT TK_PUNCT +#define TK_GT TK_PUNCT +#define TK_GE TK_PUNCT +#define TK_BITOR TK_PUNCT +#define TK_CONCAT TK_PUNCT +#define TK_COMMA TK_PUNCT +#define TK_BITAND TK_PUNCT +#define TK_BITNOT TK_PUNCT +#define TK_STRING TK_LITERAL +#define TK_ID TK_NAME +#define TK_ILLEGAL TK_ERROR +#define TK_DOT TK_PUNCT +#define TK_INTEGER TK_LITERAL +#define TK_FLOAT TK_LITERAL +#define TK_VARIABLE TK_LITERAL +#define TK_BLOB TK_LITERAL + +/* Disable nuisence warnings about case fall-through */ +#if !defined(deliberate_fall_through) && defined(__GCC__) && __GCC__>=7 +# define deliberate_fall_through __attribute__((fallthrough)); +#else +# define deliberate_fall_through +#endif + +/* +** Return the length (in bytes) of the token that begins at z[0]. +** Store the token type in *tokenType before returning. +*/ +static int sqlite3GetToken(const unsigned char *z, int *tokenType){ + int i, c; + switch( aiClass[*z] ){ /* Switch on the character-class of the first byte + ** of the token. See the comment on the CC_ defines + ** above. */ + case CC_SPACE: { + for(i=1; sqlite3Isspace(z[i]); i++){} + *tokenType = TK_SPACE; + return i; + } + case CC_MINUS: { + if( z[1]=='-' ){ + for(i=2; (c=z[i])!=0 && c!='\n'; i++){} + *tokenType = TK_SPACE; + return i; + } + *tokenType = TK_MINUS; + return 1; + } + case CC_LP: { + *tokenType = TK_LP; + return 1; + } + case CC_RP: { + *tokenType = TK_RP; + return 1; + } + case CC_SEMI: { + *tokenType = TK_SEMI; + return 1; + } + case CC_PLUS: { + *tokenType = TK_PLUS; + return 1; + } + case CC_STAR: { + *tokenType = TK_STAR; + return 1; + } + case CC_SLASH: { + if( z[1]!='*' || z[2]==0 ){ + *tokenType = TK_SLASH; + return 1; + } + for(i=3, c=z[2]; (c!='*' || z[i]!='/') && (c=z[i])!=0; i++){} + if( c ) i++; + *tokenType = TK_SPACE; + return i; + } + case CC_PERCENT: { + *tokenType = TK_REM; + return 1; + } + case CC_EQ: { + *tokenType = TK_EQ; + return 1 + (z[1]=='='); + } + case CC_LT: { + if( (c=z[1])=='=' ){ + *tokenType = TK_LE; + return 2; + }else if( c=='>' ){ + *tokenType = TK_NE; + return 2; + }else if( c=='<' ){ + *tokenType = TK_LSHIFT; + return 2; + }else{ + *tokenType = TK_LT; + return 1; + } + } + case CC_GT: { + if( (c=z[1])=='=' ){ + *tokenType = TK_GE; + return 2; + }else if( c=='>' ){ + *tokenType = TK_RSHIFT; + return 2; + }else{ + *tokenType = TK_GT; + return 1; + } + } + case CC_BANG: { + if( z[1]!='=' ){ + *tokenType = TK_ILLEGAL; + return 1; + }else{ + *tokenType = TK_NE; + return 2; + } + } + case CC_PIPE: { + if( z[1]!='|' ){ + *tokenType = TK_BITOR; + return 1; + }else{ + *tokenType = TK_CONCAT; + return 2; + } + } + case CC_COMMA: { + *tokenType = TK_COMMA; + return 1; + } + case CC_AND: { + *tokenType = TK_BITAND; + return 1; + } + case CC_TILDA: { + *tokenType = TK_BITNOT; + return 1; + } + case CC_QUOTE: { + int delim = z[0]; + testcase( delim=='`' ); + testcase( delim=='\'' ); + testcase( delim=='"' ); + for(i=1; (c=z[i])!=0; i++){ + if( c==delim ){ + if( z[i+1]==delim ){ + i++; + }else{ + break; + } + } + } + if( c=='\'' ){ + *tokenType = TK_STRING; + return i+1; + }else if( c!=0 ){ + *tokenType = TK_ID; + return i+1; + }else{ + *tokenType = TK_ILLEGAL; + return i; + } + } + case CC_DOT: { + if( !sqlite3Isdigit(z[1]) ){ + *tokenType = TK_DOT; + return 1; + } + /* If the next character is a digit, this is a floating point + ** number that begins with ".". Fall thru into the next case */ + /* no break */ deliberate_fall_through + } + case CC_DIGIT: { + *tokenType = TK_INTEGER; + if( z[0]=='0' && (z[1]=='x' || z[1]=='X') && sqlite3Isxdigit(z[2]) ){ + for(i=3; sqlite3Isxdigit(z[i]); i++){} + return i; + } + for(i=0; sqlite3Isdigit(z[i]); i++){} + if( z[i]=='.' ){ + i++; + while( sqlite3Isdigit(z[i]) ){ i++; } + *tokenType = TK_FLOAT; + } + if( (z[i]=='e' || z[i]=='E') && + ( sqlite3Isdigit(z[i+1]) + || ((z[i+1]=='+' || z[i+1]=='-') && sqlite3Isdigit(z[i+2])) + ) + ){ + i += 2; + while( sqlite3Isdigit(z[i]) ){ i++; } + *tokenType = TK_FLOAT; + } + while( IdChar(z[i]) ){ + *tokenType = TK_ILLEGAL; + i++; + } + return i; + } + case CC_QUOTE2: { + for(i=1, c=z[0]; c!=']' && (c=z[i])!=0; i++){} + *tokenType = c==']' ? TK_ID : TK_ILLEGAL; + return i; + } + case CC_VARNUM: { + *tokenType = TK_VARIABLE; + for(i=1; sqlite3Isdigit(z[i]); i++){} + return i; + } + case CC_DOLLAR: + case CC_VARALPHA: { + int n = 0; + testcase( z[0]=='$' ); testcase( z[0]=='@' ); + testcase( z[0]==':' ); testcase( z[0]=='#' ); + *tokenType = TK_VARIABLE; + for(i=1; (c=z[i])!=0; i++){ + if( IdChar(c) ){ + n++; + }else if( c=='(' && n>0 ){ + do{ + i++; + }while( (c=z[i])!=0 && !sqlite3Isspace(c) && c!=')' ); + if( c==')' ){ + i++; + }else{ + *tokenType = TK_ILLEGAL; + } + break; + }else if( c==':' && z[i+1]==':' ){ + i++; + }else{ + break; + } + } + if( n==0 ) *tokenType = TK_ILLEGAL; + return i; + } + case CC_KYWD: { + for(i=1; aiClass[z[i]]<=CC_KYWD; i++){} + if( IdChar(z[i]) ){ + /* This token started out using characters that can appear in keywords, + ** but z[i] is a character not allowed within keywords, so this must + ** be an identifier instead */ + i++; + break; + } + *tokenType = TK_ID; + return i; + } + case CC_X: { + testcase( z[0]=='x' ); testcase( z[0]=='X' ); + if( z[1]=='\'' ){ + *tokenType = TK_BLOB; + for(i=2; sqlite3Isxdigit(z[i]); i++){} + if( z[i]!='\'' || i%2 ){ + *tokenType = TK_ILLEGAL; + while( z[i] && z[i]!='\'' ){ i++; } + } + if( z[i] ) i++; + return i; + } + /* If it is not a BLOB literal, then it must be an ID, since no + ** SQL keywords start with the letter 'x'. Fall through */ + /* no break */ deliberate_fall_through + } + case CC_ID: { + i = 1; + break; + } + default: { + *tokenType = TK_ILLEGAL; + return 1; + } + } + while( IdChar(z[i]) ){ i++; } + *tokenType = TK_ID; + return i; +} + +char *sqlite3_normalize(const char *zSql){ + char *z; /* The output string */ + sqlite3_int64 nZ; /* Size of the output string in bytes */ + sqlite3_int64 nSql; /* Size of the input string in bytes */ + int i; /* Next character to read from zSql[] */ + int j; /* Next slot to fill in on z[] */ + int tokenType; /* Type of the next token */ + int n; /* Size of the next token */ + int k; /* Loop counter */ + + nSql = strlen(zSql); + nZ = nSql; + z = sqlite3_malloc64( nZ+2 ); + if( z==0 ) return 0; + for(i=j=0; zSql[i]; i += n){ + n = sqlite3GetToken((unsigned char*)zSql+i, &tokenType); + switch( tokenType ){ + case TK_SPACE: { + break; + } + case TK_ERROR: { + sqlite3_free(z); + return 0; + } + case TK_LITERAL: { + z[j++] = '?'; + break; + } + case TK_PUNCT: + case TK_NAME: { + if( n==4 && sqlite3_strnicmp(zSql+i,"NULL",4)==0 ){ + if( (j>=3 && strncmp(z+j-2,"is",2)==0 && !IdChar(z[j-3])) + || (j>=4 && strncmp(z+j-3,"not",3)==0 && !IdChar(z[j-4])) + ){ + /* NULL is a keyword in this case, not a literal value */ + }else{ + /* Here the NULL is a literal value */ + z[j++] = '?'; + break; + } + } + if( j>0 && IdChar(z[j-1]) && IdChar(zSql[i]) ) z[j++] = ' '; + for(k=0; k<n; k++){ + z[j++] = sqlite3Tolower(zSql[i+k]); + } + break; + } + } + } + while( j>0 && z[j-1]==' ' ){ j--; } + if( j>0 && z[j-1]!=';' ){ z[j++] = ';'; } + z[j] = 0; + + /* Make a second pass converting "in(...)" where the "..." is not a + ** SELECT statement into "in(?,?,?)" */ + for(i=0; i<j; i=n){ + char *zIn = strstr(z+i, "in("); + int nParen; + if( zIn==0 ) break; + n = (int)(zIn-z)+3; /* Index of first char past "in(" */ + if( n && IdChar(zIn[-1]) ) continue; + if( strncmp(zIn, "in(select",9)==0 && !IdChar(zIn[9]) ) continue; + if( strncmp(zIn, "in(with",7)==0 && !IdChar(zIn[7]) ) continue; + for(nParen=1, k=0; z[n+k]; k++){ + if( z[n+k]=='(' ) nParen++; + if( z[n+k]==')' ){ + nParen--; + if( nParen==0 ) break; + } + } + /* k is the number of bytes in the "..." within "in(...)" */ + if( k<5 ){ + z = sqlite3_realloc64(z, j+(5-k)+1); + if( z==0 ) return 0; + memmove(z+n+5, z+n+k, j-(n+k)); + }else if( k>5 ){ + memmove(z+n+5, z+n+k, j-(n+k)); + } + j = j-k+5; + z[j] = 0; + memcpy(z+n, "?,?,?", 5); + } + return z; +} + +/* +** For testing purposes, or to build a stand-alone SQL normalizer program, +** compile this one source file with the -DSQLITE_NORMALIZE_CLI and link +** it against any SQLite library. The resulting command-line program will +** run sqlite3_normalize() over the text of all files named on the command- +** line and show the result on standard output. +*/ +#ifdef SQLITE_NORMALIZE_CLI +#include <stdio.h> +#include <stdlib.h> + +/* +** Break zIn up into separate SQL statements and run sqlite3_normalize() +** on each one. Print the result of each run. +*/ +static void normalizeFile(char *zIn){ + int i; + if( zIn==0 ) return; + for(i=0; zIn[i]; i++){ + char cSaved; + if( zIn[i]!=';' ) continue; + cSaved = zIn[i+1]; + zIn[i+1] = 0; + if( sqlite3_complete(zIn) ){ + char *zOut = sqlite3_normalize(zIn); + if( zOut ){ + printf("%s\n", zOut); + sqlite3_free(zOut); + }else{ + fprintf(stderr, "ERROR: %s\n", zIn); + } + zIn[i+1] = cSaved; + zIn += i+1; + i = -1; + }else{ + zIn[i+1] = cSaved; + } + } +} + +/* +** The main routine for "sql_normalize". Read files named on the +** command-line and run the text of each through sqlite3_normalize(). +*/ +int main(int argc, char **argv){ + int i; + FILE *in; + char *zBuf = 0; + sqlite3_int64 sz, got; + + for(i=1; i<argc; i++){ + in = fopen(argv[i], "rb"); + if( in==0 ){ + fprintf(stderr, "cannot open \"%s\"\n", argv[i]); + continue; + } + fseek(in, 0, SEEK_END); + sz = ftell(in); + rewind(in); + zBuf = sqlite3_realloc64(zBuf, sz+1); + if( zBuf==0 ){ + fprintf(stderr, "failed to malloc for %lld bytes\n", sz); + exit(1); + } + got = fread(zBuf, 1, sz, in); + fclose(in); + if( got!=sz ){ + fprintf(stderr, "only able to read %lld of %lld bytes from \"%s\"\n", + got, sz, argv[i]); + }else{ + zBuf[got] = 0; + normalizeFile(zBuf); + } + } + sqlite3_free(zBuf); +} +#endif /* SQLITE_NORMALIZE_CLI */ diff --git a/ext/misc/percentile.c b/ext/misc/percentile.c new file mode 100644 index 0000000..d83bc5b --- /dev/null +++ b/ext/misc/percentile.c @@ -0,0 +1,220 @@ +/* +** 2013-05-28 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains code to implement the percentile(Y,P) SQL function +** as described below: +** +** (1) The percentile(Y,P) function is an aggregate function taking +** exactly two arguments. +** +** (2) If the P argument to percentile(Y,P) is not the same for every +** row in the aggregate then an error is thrown. The word "same" +** in the previous sentence means that the value differ by less +** than 0.001. +** +** (3) If the P argument to percentile(Y,P) evaluates to anything other +** than a number in the range of 0.0 to 100.0 inclusive then an +** error is thrown. +** +** (4) If any Y argument to percentile(Y,P) evaluates to a value that +** is not NULL and is not numeric then an error is thrown. +** +** (5) If any Y argument to percentile(Y,P) evaluates to plus or minus +** infinity then an error is thrown. (SQLite always interprets NaN +** values as NULL.) +** +** (6) Both Y and P in percentile(Y,P) can be arbitrary expressions, +** including CASE WHEN expressions. +** +** (7) The percentile(Y,P) aggregate is able to handle inputs of at least +** one million (1,000,000) rows. +** +** (8) If there are no non-NULL values for Y, then percentile(Y,P) +** returns NULL. +** +** (9) If there is exactly one non-NULL value for Y, the percentile(Y,P) +** returns the one Y value. +** +** (10) If there N non-NULL values of Y where N is two or more and +** the Y values are ordered from least to greatest and a graph is +** drawn from 0 to N-1 such that the height of the graph at J is +** the J-th Y value and such that straight lines are drawn between +** adjacent Y values, then the percentile(Y,P) function returns +** the height of the graph at P*(N-1)/100. +** +** (11) The percentile(Y,P) function always returns either a floating +** point number or NULL. +** +** (12) The percentile(Y,P) is implemented as a single C99 source-code +** file that compiles into a shared-library or DLL that can be loaded +** into SQLite using the sqlite3_load_extension() interface. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <stdlib.h> + +/* The following object is the session context for a single percentile() +** function. We have to remember all input Y values until the very end. +** Those values are accumulated in the Percentile.a[] array. +*/ +typedef struct Percentile Percentile; +struct Percentile { + unsigned nAlloc; /* Number of slots allocated for a[] */ + unsigned nUsed; /* Number of slots actually used in a[] */ + double rPct; /* 1.0 more than the value for P */ + double *a; /* Array of Y values */ +}; + +/* +** Return TRUE if the input floating-point number is an infinity. +*/ +static int isInfinity(double r){ + sqlite3_uint64 u; + assert( sizeof(u)==sizeof(r) ); + memcpy(&u, &r, sizeof(u)); + return ((u>>52)&0x7ff)==0x7ff; +} + +/* +** Return TRUE if two doubles differ by 0.001 or less +*/ +static int sameValue(double a, double b){ + a -= b; + return a>=-0.001 && a<=0.001; +} + +/* +** The "step" function for percentile(Y,P) is called once for each +** input row. +*/ +static void percentStep(sqlite3_context *pCtx, int argc, sqlite3_value **argv){ + Percentile *p; + double rPct; + int eType; + double y; + assert( argc==2 ); + + /* Requirement 3: P must be a number between 0 and 100 */ + eType = sqlite3_value_numeric_type(argv[1]); + rPct = sqlite3_value_double(argv[1]); + if( (eType!=SQLITE_INTEGER && eType!=SQLITE_FLOAT) + || rPct<0.0 || rPct>100.0 ){ + sqlite3_result_error(pCtx, "2nd argument to percentile() is not " + "a number between 0.0 and 100.0", -1); + return; + } + + /* Allocate the session context. */ + p = (Percentile*)sqlite3_aggregate_context(pCtx, sizeof(*p)); + if( p==0 ) return; + + /* Remember the P value. Throw an error if the P value is different + ** from any prior row, per Requirement (2). */ + if( p->rPct==0.0 ){ + p->rPct = rPct+1.0; + }else if( !sameValue(p->rPct,rPct+1.0) ){ + sqlite3_result_error(pCtx, "2nd argument to percentile() is not the " + "same for all input rows", -1); + return; + } + + /* Ignore rows for which Y is NULL */ + eType = sqlite3_value_type(argv[0]); + if( eType==SQLITE_NULL ) return; + + /* If not NULL, then Y must be numeric. Otherwise throw an error. + ** Requirement 4 */ + if( eType!=SQLITE_INTEGER && eType!=SQLITE_FLOAT ){ + sqlite3_result_error(pCtx, "1st argument to percentile() is not " + "numeric", -1); + return; + } + + /* Throw an error if the Y value is infinity or NaN */ + y = sqlite3_value_double(argv[0]); + if( isInfinity(y) ){ + sqlite3_result_error(pCtx, "Inf input to percentile()", -1); + return; + } + + /* Allocate and store the Y */ + if( p->nUsed>=p->nAlloc ){ + unsigned n = p->nAlloc*2 + 250; + double *a = sqlite3_realloc64(p->a, sizeof(double)*n); + if( a==0 ){ + sqlite3_free(p->a); + memset(p, 0, sizeof(*p)); + sqlite3_result_error_nomem(pCtx); + return; + } + p->nAlloc = n; + p->a = a; + } + p->a[p->nUsed++] = y; +} + +/* +** Compare to doubles for sorting using qsort() +*/ +static int SQLITE_CDECL doubleCmp(const void *pA, const void *pB){ + double a = *(double*)pA; + double b = *(double*)pB; + if( a==b ) return 0; + if( a<b ) return -1; + return +1; +} + +/* +** Called to compute the final output of percentile() and to clean +** up all allocated memory. +*/ +static void percentFinal(sqlite3_context *pCtx){ + Percentile *p; + unsigned i1, i2; + double v1, v2; + double ix, vx; + p = (Percentile*)sqlite3_aggregate_context(pCtx, 0); + if( p==0 ) return; + if( p->a==0 ) return; + if( p->nUsed ){ + qsort(p->a, p->nUsed, sizeof(double), doubleCmp); + ix = (p->rPct-1.0)*(p->nUsed-1)*0.01; + i1 = (unsigned)ix; + i2 = ix==(double)i1 || i1==p->nUsed-1 ? i1 : i1+1; + v1 = p->a[i1]; + v2 = p->a[i2]; + vx = v1 + (v2-v1)*(ix-i1); + sqlite3_result_double(pCtx, vx); + } + sqlite3_free(p->a); + memset(p, 0, sizeof(*p)); +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_percentile_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "percentile", 2, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + 0, percentStep, percentFinal); + return rc; +} diff --git a/ext/misc/prefixes.c b/ext/misc/prefixes.c new file mode 100644 index 0000000..3f053b7 --- /dev/null +++ b/ext/misc/prefixes.c @@ -0,0 +1,320 @@ +/* +** 2018-04-19 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a table-valued function: +** +** prefixes('abcdefg') +** +** The function has a single (non-HIDDEN) column named prefix that takes +** on all prefixes of the string in its argument, including an empty string +** and the input string itself. The order of prefixes is from longest +** to shortest. +*/ +#if !defined(SQLITE_CORE) || !defined(SQLITE_OMIT_VIRTUALTABLE) +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> + +/* prefixes_vtab is a subclass of sqlite3_vtab which is +** underlying representation of the virtual table +*/ +typedef struct prefixes_vtab prefixes_vtab; +struct prefixes_vtab { + sqlite3_vtab base; /* Base class - must be first */ + /* No additional fields are necessary */ +}; + +/* prefixes_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct prefixes_cursor prefixes_cursor; +struct prefixes_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iRowid; /* The rowid */ + char *zStr; /* Original string to be prefixed */ + int nStr; /* Length of the string in bytes */ +}; + +/* +** The prefixesConnect() method is invoked to create a new +** template virtual table. +** +** Think of this routine as the constructor for prefixes_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the prefixes_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against the virtual table will look like. +*/ +static int prefixesConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + prefixes_vtab *pNew; + int rc; + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE prefixes(prefix TEXT, original_string TEXT HIDDEN)" + ); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + } + return rc; +} + +/* +** This method is the destructor for prefixes_vtab objects. +*/ +static int prefixesDisconnect(sqlite3_vtab *pVtab){ + prefixes_vtab *p = (prefixes_vtab*)pVtab; + sqlite3_free(p); + return SQLITE_OK; +} + +/* +** Constructor for a new prefixes_cursor object. +*/ +static int prefixesOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + prefixes_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a prefixes_cursor. +*/ +static int prefixesClose(sqlite3_vtab_cursor *cur){ + prefixes_cursor *pCur = (prefixes_cursor*)cur; + sqlite3_free(pCur->zStr); + sqlite3_free(pCur); + return SQLITE_OK; +} + + +/* +** Advance a prefixes_cursor to its next row of output. +*/ +static int prefixesNext(sqlite3_vtab_cursor *cur){ + prefixes_cursor *pCur = (prefixes_cursor*)cur; + pCur->iRowid++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the prefixes_cursor +** is currently pointing. +*/ +static int prefixesColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + prefixes_cursor *pCur = (prefixes_cursor*)cur; + switch( i ){ + case 0: + sqlite3_result_text(ctx, pCur->zStr, pCur->nStr - (int)pCur->iRowid, + 0); + break; + default: + sqlite3_result_text(ctx, pCur->zStr, pCur->nStr, 0); + break; + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int prefixesRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + prefixes_cursor *pCur = (prefixes_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int prefixesEof(sqlite3_vtab_cursor *cur){ + prefixes_cursor *pCur = (prefixes_cursor*)cur; + return pCur->iRowid>pCur->nStr; +} + +/* +** This method is called to "rewind" the prefixes_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to prefixesColumn() or prefixesRowid() or +** prefixesEof(). +*/ +static int prefixesFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + prefixes_cursor *pCur = (prefixes_cursor *)pVtabCursor; + sqlite3_free(pCur->zStr); + if( argc>0 ){ + pCur->zStr = sqlite3_mprintf("%s", sqlite3_value_text(argv[0])); + pCur->nStr = pCur->zStr ? (int)strlen(pCur->zStr) : 0; + }else{ + pCur->zStr = 0; + pCur->nStr = 0; + } + pCur->iRowid = 0; + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int prefixesBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + /* Search for a usable equality constraint against column 1 + ** (original_string) and use it if at all possible */ + int i; + const struct sqlite3_index_constraint *p; + + for(i=0, p=pIdxInfo->aConstraint; i<pIdxInfo->nConstraint; i++, p++){ + if( p->iColumn!=1 ) continue; + if( p->op!=SQLITE_INDEX_CONSTRAINT_EQ ) continue; + if( !p->usable ) continue; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + pIdxInfo->estimatedCost = (double)10; + pIdxInfo->estimatedRows = 10; + return SQLITE_OK; + } + pIdxInfo->estimatedCost = (double)1000000000; + pIdxInfo->estimatedRows = 1000000000; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** virtual table. +*/ +static sqlite3_module prefixesModule = { + /* iVersion */ 0, + /* xCreate */ 0, + /* xConnect */ prefixesConnect, + /* xBestIndex */ prefixesBestIndex, + /* xDisconnect */ prefixesDisconnect, + /* xDestroy */ 0, + /* xOpen */ prefixesOpen, + /* xClose */ prefixesClose, + /* xFilter */ prefixesFilter, + /* xNext */ prefixesNext, + /* xEof */ prefixesEof, + /* xColumn */ prefixesColumn, + /* xRowid */ prefixesRowid, + /* xUpdate */ 0, + /* xBegin */ 0, + /* xSync */ 0, + /* xCommit */ 0, + /* xRollback */ 0, + /* xFindMethod */ 0, + /* xRename */ 0, + /* xSavepoint */ 0, + /* xRelease */ 0, + /* xRollbackTo */ 0, + /* xShadowName */ 0 +}; + +/* +** This is a copy of the SQLITE_SKIP_UTF8(zIn) macro in sqliteInt.h. +** +** Assuming zIn points to the first byte of a UTF-8 character, +** advance zIn to point to the first byte of the next UTF-8 character. +*/ +#define PREFIX_SKIP_UTF8(zIn) { \ + if( (*(zIn++))>=0xc0 ){ \ + while( (*zIn & 0xc0)==0x80 ){ zIn++; } \ + } \ +} + +/* +** Implementation of function prefix_length(). This function accepts two +** strings as arguments and returns the length in characters (not bytes), +** of the longest prefix shared by the two strings. For example: +** +** prefix_length('abcdxxx', 'abcyy') == 3 +** prefix_length('abcdxxx', 'bcyyy') == 0 +** prefix_length('abcdxxx', 'ab') == 2 +** prefix_length('ab', 'abcd') == 2 +** +** This function assumes the input is well-formed utf-8. If it is not, +** it is possible for this function to return -1. +*/ +static void prefixLengthFunc( + sqlite3_context *ctx, + int nVal, + sqlite3_value **apVal +){ + int nByte; /* Number of bytes to compare */ + int nRet = 0; /* Return value */ + const unsigned char *zL = sqlite3_value_text(apVal[0]); + const unsigned char *zR = sqlite3_value_text(apVal[1]); + int nL = sqlite3_value_bytes(apVal[0]); + int nR = sqlite3_value_bytes(apVal[1]); + int i; + + nByte = (nL > nR ? nL : nR); + for(i=0; i<nByte; i++){ + if( zL[i]!=zR[i] ) break; + if( (zL[i] & 0xC0)!=0x80 ) nRet++; + } + + if( (zL[i] & 0xC0)==0x80 ) nRet--; + sqlite3_result_int(ctx, nRet); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_prefixes_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_create_module(db, "prefixes", &prefixesModule, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function( + db, "prefix_length", 2, SQLITE_UTF8, 0, prefixLengthFunc, 0, 0 + ); + } + return rc; +} +#endif /* !defined(SQLITE_CORE) || !defined(SQLITE_OMIT_VIRTUALTABLE) */ diff --git a/ext/misc/qpvtab.c b/ext/misc/qpvtab.c new file mode 100644 index 0000000..fb0c155 --- /dev/null +++ b/ext/misc/qpvtab.c @@ -0,0 +1,461 @@ +/* +** 2022-01-19 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a virtual-table that returns information about +** how the query planner called the xBestIndex method. This virtual table +** is intended for testing and debugging only. +** +** The schema of the virtual table is this: +** +** CREATE TABLE qpvtab( +** vn TEXT, -- Name of an sqlite3_index_info field +** ix INTEGER, -- Array index or value +** cn TEXT, -- Column name +** op INTEGER, -- operator +** ux BOOLEAN, -- "usable" field +** rhs TEXT, -- sqlite3_vtab_rhs_value() +** +** a, b, c, d, e, -- Extra columns to attach constraints to +** +** flags INTEGER HIDDEN -- control flags +** ); +** +** The virtual table returns a description of the sqlite3_index_info object +** that was provided to the (successful) xBestIndex method. There is one +** row in the result table for each field in the sqlite3_index_info object. +** +** The values of the "a" through "e" columns are one of: +** +** 1. TEXT - the same as the column name +** 2. INTEGER - 1 for "a", 2 for "b", and so forth +** +** Option 1 is the default behavior. 2 is use if there is a usable +** constraint on "flags" with an integer right-hand side that where the +** value of the right-hand side has its 0x001 bit set. +** +** All constraints on columns "a" through "e" are marked as "omit". +** +** If there is a usable constraint on "flags" that has a RHS value that +** is an integer and that integer has its 0x02 bit set, then the +** orderByConsumed flag is set. +** +** FLAGS SUMMARY: +** +** 0x001 Columns 'a' through 'e' have INT values +** 0x002 orderByConsumed is set +** 0x004 OFFSET and LIMIT have omit set +** +** COMPILE: +** +** gcc -Wall -g -shared -fPIC -I. qpvtab.c -o qqvtab.so +** +** EXAMPLE USAGE: +** +** .load ./qpvtab +** SELECT rowid, *, flags FROM qpvtab(102) +** WHERE a=19 +** AND b BETWEEN 4.5 and 'hello' +** AND c<>x'aabbcc' +** ORDER BY d, e DESC; +*/ +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> +#include <stdlib.h> + +#if !defined(SQLITE_OMIT_VIRTUALTABLE) + +/* qpvtab_vtab is a subclass of sqlite3_vtab which is +** underlying representation of the virtual table +*/ +typedef struct qpvtab_vtab qpvtab_vtab; +struct qpvtab_vtab { + sqlite3_vtab base; /* Base class - must be first */ +}; + +/* qpvtab_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct qpvtab_cursor qpvtab_cursor; +struct qpvtab_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iRowid; /* The rowid */ + const char *zData; /* Data to return */ + int nData; /* Number of bytes of data */ + int flags; /* Flags value */ +}; + +/* +** Names of columns +*/ +static const char *azColname[] = { + "vn", + "ix", + "cn", + "op", + "ux", + "rhs", + "a", "b", "c", "d", "e", + "flags", + "" +}; + +/* +** The qpvtabConnect() method is invoked to create a new +** qpvtab virtual table. +*/ +static int qpvtabConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + qpvtab_vtab *pNew; + int rc; + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(" + " vn TEXT," + " ix INT," + " cn TEXT," + " op INT," + " ux BOOLEAN," + " rhs TEXT," + " a, b, c, d, e," + " flags INT HIDDEN)" + ); +#define QPVTAB_VN 0 +#define QPVTAB_IX 1 +#define QPVTAB_CN 2 +#define QPVTAB_OP 3 +#define QPVTAB_UX 4 +#define QPVTAB_RHS 5 +#define QPVTAB_A 6 +#define QPVTAB_B 7 +#define QPVTAB_C 8 +#define QPVTAB_D 9 +#define QPVTAB_E 10 +#define QPVTAB_FLAGS 11 +#define QPVTAB_NONE 12 + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + } + return rc; +} + +/* +** This method is the destructor for qpvtab_vtab objects. +*/ +static int qpvtabDisconnect(sqlite3_vtab *pVtab){ + qpvtab_vtab *p = (qpvtab_vtab*)pVtab; + sqlite3_free(p); + return SQLITE_OK; +} + +/* +** Constructor for a new qpvtab_cursor object. +*/ +static int qpvtabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + qpvtab_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a qpvtab_cursor. +*/ +static int qpvtabClose(sqlite3_vtab_cursor *cur){ + qpvtab_cursor *pCur = (qpvtab_cursor*)cur; + sqlite3_free(pCur); + return SQLITE_OK; +} + + +/* +** Advance a qpvtab_cursor to its next row of output. +*/ +static int qpvtabNext(sqlite3_vtab_cursor *cur){ + qpvtab_cursor *pCur = (qpvtab_cursor*)cur; + if( pCur->iRowid<pCur->nData ){ + const char *z = &pCur->zData[pCur->iRowid]; + const char *zEnd = strchr(z, '\n'); + if( zEnd ) zEnd++; + pCur->iRowid = (int)(zEnd - pCur->zData); + } + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the qpvtab_cursor +** is currently pointing. +*/ +static int qpvtabColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + qpvtab_cursor *pCur = (qpvtab_cursor*)cur; + if( i>=QPVTAB_VN && i<=QPVTAB_RHS && pCur->iRowid<pCur->nData ){ + const char *z = &pCur->zData[pCur->iRowid]; + const char *zEnd; + int j; + j = QPVTAB_VN; + while(1){ + zEnd = strchr(z, j==QPVTAB_RHS ? '\n' : ','); + if( j==i || zEnd==0 ) break; + z = zEnd+1; + j++; + } + if( zEnd==z ){ + sqlite3_result_null(ctx); + }else if( i==QPVTAB_IX || i==QPVTAB_OP || i==QPVTAB_UX ){ + sqlite3_result_int(ctx, atoi(z)); + }else{ + sqlite3_result_text64(ctx, z, zEnd-z, SQLITE_TRANSIENT, SQLITE_UTF8); + } + }else if( i>=QPVTAB_A && i<=QPVTAB_E ){ + if( pCur->flags & 0x001 ){ + sqlite3_result_int(ctx, i-QPVTAB_A+1); + }else{ + char x = 'a'+i-QPVTAB_A; + sqlite3_result_text64(ctx, &x, 1, SQLITE_TRANSIENT, SQLITE_UTF8); + } + }else if( i==QPVTAB_FLAGS ){ + sqlite3_result_int(ctx, pCur->flags); + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int qpvtabRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + qpvtab_cursor *pCur = (qpvtab_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int qpvtabEof(sqlite3_vtab_cursor *cur){ + qpvtab_cursor *pCur = (qpvtab_cursor*)cur; + return pCur->iRowid>=pCur->nData; +} + +/* +** This method is called to "rewind" the qpvtab_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to qpvtabColumn() or qpvtabRowid() or +** qpvtabEof(). +*/ +static int qpvtabFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + qpvtab_cursor *pCur = (qpvtab_cursor *)pVtabCursor; + pCur->iRowid = 0; + pCur->zData = idxStr; + pCur->nData = (int)strlen(idxStr); + pCur->flags = idxNum; + return SQLITE_OK; +} + +/* +** Append the text of a value to pStr +*/ +static void qpvtabStrAppendValue( + sqlite3_str *pStr, + sqlite3_value *pVal +){ + switch( sqlite3_value_type(pVal) ){ + case SQLITE_NULL: + sqlite3_str_appendf(pStr, "NULL"); + break; + case SQLITE_INTEGER: + sqlite3_str_appendf(pStr, "%lld", sqlite3_value_int64(pVal)); + break; + case SQLITE_FLOAT: + sqlite3_str_appendf(pStr, "%!f", sqlite3_value_double(pVal)); + break; + case SQLITE_TEXT: { + int i; + const char *a = (const char*)sqlite3_value_text(pVal); + int n = sqlite3_value_bytes(pVal); + sqlite3_str_append(pStr, "'", 1); + for(i=0; i<n; i++){ + char c = a[i]; + if( c=='\n' ) c = ' '; + sqlite3_str_append(pStr, &c, 1); + if( c=='\'' ) sqlite3_str_append(pStr, &c, 1); + } + sqlite3_str_append(pStr, "'", 1); + break; + } + case SQLITE_BLOB: { + int i; + const unsigned char *a = sqlite3_value_blob(pVal); + int n = sqlite3_value_bytes(pVal); + sqlite3_str_append(pStr, "x'", 2); + for(i=0; i<n; i++){ + sqlite3_str_appendf(pStr, "%02x", a[i]); + } + sqlite3_str_append(pStr, "'", 1); + break; + } + } +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int qpvtabBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + sqlite3_str *pStr = sqlite3_str_new(0); + int i, k = 0; + int rc; + sqlite3_str_appendf(pStr, "nConstraint,%d,,,,\n", pIdxInfo->nConstraint); + for(i=0; i<pIdxInfo->nConstraint; i++){ + sqlite3_value *pVal; + int iCol = pIdxInfo->aConstraint[i].iColumn; + int op = pIdxInfo->aConstraint[i].op; + if( iCol==QPVTAB_FLAGS && pIdxInfo->aConstraint[i].usable ){ + pVal = 0; + rc = sqlite3_vtab_rhs_value(pIdxInfo, i, &pVal); + assert( rc==SQLITE_OK || pVal==0 ); + if( pVal ){ + pIdxInfo->idxNum = sqlite3_value_int(pVal); + if( pIdxInfo->idxNum & 0x002 ) pIdxInfo->orderByConsumed = 1; + } + } + if( op==SQLITE_INDEX_CONSTRAINT_LIMIT + || op==SQLITE_INDEX_CONSTRAINT_OFFSET + ){ + iCol = QPVTAB_NONE; + } + sqlite3_str_appendf(pStr,"aConstraint,%d,%s,%d,%d,", + i, + azColname[iCol], + op, + pIdxInfo->aConstraint[i].usable); + pVal = 0; + rc = sqlite3_vtab_rhs_value(pIdxInfo, i, &pVal); + assert( rc==SQLITE_OK || pVal==0 ); + if( pVal ){ + qpvtabStrAppendValue(pStr, pVal); + } + sqlite3_str_append(pStr, "\n", 1); + } + for(i=0; i<pIdxInfo->nConstraint; i++){ + int iCol = pIdxInfo->aConstraint[i].iColumn; + int op = pIdxInfo->aConstraint[i].op; + if( op==SQLITE_INDEX_CONSTRAINT_LIMIT + || op==SQLITE_INDEX_CONSTRAINT_OFFSET + ){ + iCol = QPVTAB_NONE; + } + if( iCol>=QPVTAB_A && pIdxInfo->aConstraint[i].usable ){ + pIdxInfo->aConstraintUsage[i].argvIndex = ++k; + if( iCol<=QPVTAB_FLAGS || (pIdxInfo->idxNum & 0x004)!=0 ){ + pIdxInfo->aConstraintUsage[i].omit = 1; + } + } + } + sqlite3_str_appendf(pStr, "nOrderBy,%d,,,,\n", pIdxInfo->nOrderBy); + for(i=0; i<pIdxInfo->nOrderBy; i++){ + int iCol = pIdxInfo->aOrderBy[i].iColumn; + sqlite3_str_appendf(pStr, "aOrderBy,%d,%s,%d,,\n",i, + iCol>=0 ? azColname[iCol] : "rowid", + pIdxInfo->aOrderBy[i].desc + ); + } + sqlite3_str_appendf(pStr, "sqlite3_vtab_distinct,%d,,,,\n", + sqlite3_vtab_distinct(pIdxInfo)); + sqlite3_str_appendf(pStr, "idxFlags,%d,,,,\n", pIdxInfo->idxFlags); + sqlite3_str_appendf(pStr, "colUsed,%d,,,,\n", (int)pIdxInfo->colUsed); + pIdxInfo->estimatedCost = (double)10; + pIdxInfo->estimatedRows = 10; + sqlite3_str_appendf(pStr, "idxNum,%d,,,,\n", pIdxInfo->idxNum); + sqlite3_str_appendf(pStr, "orderByConsumed,%d,,,,\n", + pIdxInfo->orderByConsumed); + pIdxInfo->idxStr = sqlite3_str_finish(pStr); + pIdxInfo->needToFreeIdxStr = 1; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** virtual table. +*/ +static sqlite3_module qpvtabModule = { + /* iVersion */ 0, + /* xCreate */ 0, + /* xConnect */ qpvtabConnect, + /* xBestIndex */ qpvtabBestIndex, + /* xDisconnect */ qpvtabDisconnect, + /* xDestroy */ 0, + /* xOpen */ qpvtabOpen, + /* xClose */ qpvtabClose, + /* xFilter */ qpvtabFilter, + /* xNext */ qpvtabNext, + /* xEof */ qpvtabEof, + /* xColumn */ qpvtabColumn, + /* xRowid */ qpvtabRowid, + /* xUpdate */ 0, + /* xBegin */ 0, + /* xSync */ 0, + /* xCommit */ 0, + /* xRollback */ 0, + /* xFindMethod */ 0, + /* xRename */ 0, + /* xSavepoint */ 0, + /* xRelease */ 0, + /* xRollbackTo */ 0, + /* xShadowName */ 0 +}; +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_qpvtab_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "qpvtab", &qpvtabModule, 0); +#endif + return rc; +} diff --git a/ext/misc/regexp.c b/ext/misc/regexp.c new file mode 100644 index 0000000..086ef56 --- /dev/null +++ b/ext/misc/regexp.c @@ -0,0 +1,880 @@ +/* +** 2012-11-13 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** The code in this file implements a compact but reasonably +** efficient regular-expression matcher for posix extended regular +** expressions against UTF8 text. +** +** This file is an SQLite extension. It registers a single function +** named "regexp(A,B)" where A is the regular expression and B is the +** string to be matched. By registering this function, SQLite will also +** then implement the "B regexp A" operator. Note that with the function +** the regular expression comes first, but with the operator it comes +** second. +** +** The following regular expression syntax is supported: +** +** X* zero or more occurrences of X +** X+ one or more occurrences of X +** X? zero or one occurrences of X +** X{p,q} between p and q occurrences of X +** (X) match X +** X|Y X or Y +** ^X X occurring at the beginning of the string +** X$ X occurring at the end of the string +** . Match any single character +** \c Character c where c is one of \{}()[]|*+?. +** \c C-language escapes for c in afnrtv. ex: \t or \n +** \uXXXX Where XXXX is exactly 4 hex digits, unicode value XXXX +** \xXX Where XX is exactly 2 hex digits, unicode value XX +** [abc] Any single character from the set abc +** [^abc] Any single character not in the set abc +** [a-z] Any single character in the range a-z +** [^a-z] Any single character not in the range a-z +** \b Word boundary +** \w Word character. [A-Za-z0-9_] +** \W Non-word character +** \d Digit +** \D Non-digit +** \s Whitespace character +** \S Non-whitespace character +** +** A nondeterministic finite automaton (NFA) is used for matching, so the +** performance is bounded by O(N*M) where N is the size of the regular +** expression and M is the size of the input string. The matcher never +** exhibits exponential behavior. Note that the X{p,q} operator expands +** to p copies of X following by q-p copies of X? and that the size of the +** regular expression in the O(N*M) performance bound is computed after +** this expansion. +*/ +#include <string.h> +#include <stdlib.h> +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 + +/* +** The following #defines change the names of some functions implemented in +** this file to prevent name collisions with C-library functions of the +** same name. +*/ +#define re_match sqlite3re_match +#define re_compile sqlite3re_compile +#define re_free sqlite3re_free + +/* The end-of-input character */ +#define RE_EOF 0 /* End of input */ +#define RE_START 0xfffffff /* Start of input - larger than an UTF-8 */ + +/* The NFA is implemented as sequence of opcodes taken from the following +** set. Each opcode has a single integer argument. +*/ +#define RE_OP_MATCH 1 /* Match the one character in the argument */ +#define RE_OP_ANY 2 /* Match any one character. (Implements ".") */ +#define RE_OP_ANYSTAR 3 /* Special optimized version of .* */ +#define RE_OP_FORK 4 /* Continue to both next and opcode at iArg */ +#define RE_OP_GOTO 5 /* Jump to opcode at iArg */ +#define RE_OP_ACCEPT 6 /* Halt and indicate a successful match */ +#define RE_OP_CC_INC 7 /* Beginning of a [...] character class */ +#define RE_OP_CC_EXC 8 /* Beginning of a [^...] character class */ +#define RE_OP_CC_VALUE 9 /* Single value in a character class */ +#define RE_OP_CC_RANGE 10 /* Range of values in a character class */ +#define RE_OP_WORD 11 /* Perl word character [A-Za-z0-9_] */ +#define RE_OP_NOTWORD 12 /* Not a perl word character */ +#define RE_OP_DIGIT 13 /* digit: [0-9] */ +#define RE_OP_NOTDIGIT 14 /* Not a digit */ +#define RE_OP_SPACE 15 /* space: [ \t\n\r\v\f] */ +#define RE_OP_NOTSPACE 16 /* Not a digit */ +#define RE_OP_BOUNDARY 17 /* Boundary between word and non-word */ +#define RE_OP_ATSTART 18 /* Currently at the start of the string */ + +#if defined(SQLITE_DEBUG) +/* Opcode names used for symbolic debugging */ +static const char *ReOpName[] = { + "EOF", + "MATCH", + "ANY", + "ANYSTAR", + "FORK", + "GOTO", + "ACCEPT", + "CC_INC", + "CC_EXC", + "CC_VALUE", + "CC_RANGE", + "WORD", + "NOTWORD", + "DIGIT", + "NOTDIGIT", + "SPACE", + "NOTSPACE", + "BOUNDARY", + "ATSTART", +}; +#endif /* SQLITE_DEBUG */ + + +/* Each opcode is a "state" in the NFA */ +typedef unsigned short ReStateNumber; + +/* Because this is an NFA and not a DFA, multiple states can be active at +** once. An instance of the following object records all active states in +** the NFA. The implementation is optimized for the common case where the +** number of actives states is small. +*/ +typedef struct ReStateSet { + unsigned nState; /* Number of current states */ + ReStateNumber *aState; /* Current states */ +} ReStateSet; + +/* An input string read one character at a time. +*/ +typedef struct ReInput ReInput; +struct ReInput { + const unsigned char *z; /* All text */ + int i; /* Next byte to read */ + int mx; /* EOF when i>=mx */ +}; + +/* A compiled NFA (or an NFA that is in the process of being compiled) is +** an instance of the following object. +*/ +typedef struct ReCompiled ReCompiled; +struct ReCompiled { + ReInput sIn; /* Regular expression text */ + const char *zErr; /* Error message to return */ + char *aOp; /* Operators for the virtual machine */ + int *aArg; /* Arguments to each operator */ + unsigned (*xNextChar)(ReInput*); /* Next character function */ + unsigned char zInit[12]; /* Initial text to match */ + int nInit; /* Number of bytes in zInit */ + unsigned nState; /* Number of entries in aOp[] and aArg[] */ + unsigned nAlloc; /* Slots allocated for aOp[] and aArg[] */ +}; + +/* Add a state to the given state set if it is not already there */ +static void re_add_state(ReStateSet *pSet, int newState){ + unsigned i; + for(i=0; i<pSet->nState; i++) if( pSet->aState[i]==newState ) return; + pSet->aState[pSet->nState++] = (ReStateNumber)newState; +} + +/* Extract the next unicode character from *pzIn and return it. Advance +** *pzIn to the first byte past the end of the character returned. To +** be clear: this routine converts utf8 to unicode. This routine is +** optimized for the common case where the next character is a single byte. +*/ +static unsigned re_next_char(ReInput *p){ + unsigned c; + if( p->i>=p->mx ) return 0; + c = p->z[p->i++]; + if( c>=0x80 ){ + if( (c&0xe0)==0xc0 && p->i<p->mx && (p->z[p->i]&0xc0)==0x80 ){ + c = (c&0x1f)<<6 | (p->z[p->i++]&0x3f); + if( c<0x80 ) c = 0xfffd; + }else if( (c&0xf0)==0xe0 && p->i+1<p->mx && (p->z[p->i]&0xc0)==0x80 + && (p->z[p->i+1]&0xc0)==0x80 ){ + c = (c&0x0f)<<12 | ((p->z[p->i]&0x3f)<<6) | (p->z[p->i+1]&0x3f); + p->i += 2; + if( c<=0x7ff || (c>=0xd800 && c<=0xdfff) ) c = 0xfffd; + }else if( (c&0xf8)==0xf0 && p->i+2<p->mx && (p->z[p->i]&0xc0)==0x80 + && (p->z[p->i+1]&0xc0)==0x80 && (p->z[p->i+2]&0xc0)==0x80 ){ + c = (c&0x07)<<18 | ((p->z[p->i]&0x3f)<<12) | ((p->z[p->i+1]&0x3f)<<6) + | (p->z[p->i+2]&0x3f); + p->i += 3; + if( c<=0xffff || c>0x10ffff ) c = 0xfffd; + }else{ + c = 0xfffd; + } + } + return c; +} +static unsigned re_next_char_nocase(ReInput *p){ + unsigned c = re_next_char(p); + if( c>='A' && c<='Z' ) c += 'a' - 'A'; + return c; +} + +/* Return true if c is a perl "word" character: [A-Za-z0-9_] */ +static int re_word_char(int c){ + return (c>='0' && c<='9') || (c>='a' && c<='z') + || (c>='A' && c<='Z') || c=='_'; +} + +/* Return true if c is a "digit" character: [0-9] */ +static int re_digit_char(int c){ + return (c>='0' && c<='9'); +} + +/* Return true if c is a perl "space" character: [ \t\r\n\v\f] */ +static int re_space_char(int c){ + return c==' ' || c=='\t' || c=='\n' || c=='\r' || c=='\v' || c=='\f'; +} + +/* Run a compiled regular expression on the zero-terminated input +** string zIn[]. Return true on a match and false if there is no match. +*/ +static int re_match(ReCompiled *pRe, const unsigned char *zIn, int nIn){ + ReStateSet aStateSet[2], *pThis, *pNext; + ReStateNumber aSpace[100]; + ReStateNumber *pToFree; + unsigned int i = 0; + unsigned int iSwap = 0; + int c = RE_START; + int cPrev = 0; + int rc = 0; + ReInput in; + + in.z = zIn; + in.i = 0; + in.mx = nIn>=0 ? nIn : (int)strlen((char const*)zIn); + + /* Look for the initial prefix match, if there is one. */ + if( pRe->nInit ){ + unsigned char x = pRe->zInit[0]; + while( in.i+pRe->nInit<=in.mx + && (zIn[in.i]!=x || + strncmp((const char*)zIn+in.i, (const char*)pRe->zInit, pRe->nInit)!=0) + ){ + in.i++; + } + if( in.i+pRe->nInit>in.mx ) return 0; + c = RE_START-1; + } + + if( pRe->nState<=(sizeof(aSpace)/(sizeof(aSpace[0])*2)) ){ + pToFree = 0; + aStateSet[0].aState = aSpace; + }else{ + pToFree = sqlite3_malloc64( sizeof(ReStateNumber)*2*pRe->nState ); + if( pToFree==0 ) return -1; + aStateSet[0].aState = pToFree; + } + aStateSet[1].aState = &aStateSet[0].aState[pRe->nState]; + pNext = &aStateSet[1]; + pNext->nState = 0; + re_add_state(pNext, 0); + while( c!=RE_EOF && pNext->nState>0 ){ + cPrev = c; + c = pRe->xNextChar(&in); + pThis = pNext; + pNext = &aStateSet[iSwap]; + iSwap = 1 - iSwap; + pNext->nState = 0; + for(i=0; i<pThis->nState; i++){ + int x = pThis->aState[i]; + switch( pRe->aOp[x] ){ + case RE_OP_MATCH: { + if( pRe->aArg[x]==c ) re_add_state(pNext, x+1); + break; + } + case RE_OP_ATSTART: { + if( cPrev==RE_START ) re_add_state(pThis, x+1); + break; + } + case RE_OP_ANY: { + if( c!=0 ) re_add_state(pNext, x+1); + break; + } + case RE_OP_WORD: { + if( re_word_char(c) ) re_add_state(pNext, x+1); + break; + } + case RE_OP_NOTWORD: { + if( !re_word_char(c) && c!=0 ) re_add_state(pNext, x+1); + break; + } + case RE_OP_DIGIT: { + if( re_digit_char(c) ) re_add_state(pNext, x+1); + break; + } + case RE_OP_NOTDIGIT: { + if( !re_digit_char(c) && c!=0 ) re_add_state(pNext, x+1); + break; + } + case RE_OP_SPACE: { + if( re_space_char(c) ) re_add_state(pNext, x+1); + break; + } + case RE_OP_NOTSPACE: { + if( !re_space_char(c) && c!=0 ) re_add_state(pNext, x+1); + break; + } + case RE_OP_BOUNDARY: { + if( re_word_char(c)!=re_word_char(cPrev) ) re_add_state(pThis, x+1); + break; + } + case RE_OP_ANYSTAR: { + re_add_state(pNext, x); + re_add_state(pThis, x+1); + break; + } + case RE_OP_FORK: { + re_add_state(pThis, x+pRe->aArg[x]); + re_add_state(pThis, x+1); + break; + } + case RE_OP_GOTO: { + re_add_state(pThis, x+pRe->aArg[x]); + break; + } + case RE_OP_ACCEPT: { + rc = 1; + goto re_match_end; + } + case RE_OP_CC_EXC: { + if( c==0 ) break; + /* fall-through */ goto re_op_cc_inc; + } + case RE_OP_CC_INC: re_op_cc_inc: { + int j = 1; + int n = pRe->aArg[x]; + int hit = 0; + for(j=1; j>0 && j<n; j++){ + if( pRe->aOp[x+j]==RE_OP_CC_VALUE ){ + if( pRe->aArg[x+j]==c ){ + hit = 1; + j = -1; + } + }else{ + if( pRe->aArg[x+j]<=c && pRe->aArg[x+j+1]>=c ){ + hit = 1; + j = -1; + }else{ + j++; + } + } + } + if( pRe->aOp[x]==RE_OP_CC_EXC ) hit = !hit; + if( hit ) re_add_state(pNext, x+n); + break; + } + } + } + } + for(i=0; i<pNext->nState; i++){ + int x = pNext->aState[i]; + while( pRe->aOp[x]==RE_OP_GOTO ) x += pRe->aArg[x]; + if( pRe->aOp[x]==RE_OP_ACCEPT ){ rc = 1; break; } + } +re_match_end: + sqlite3_free(pToFree); + return rc; +} + +/* Resize the opcode and argument arrays for an RE under construction. +*/ +static int re_resize(ReCompiled *p, int N){ + char *aOp; + int *aArg; + aOp = sqlite3_realloc64(p->aOp, N*sizeof(p->aOp[0])); + if( aOp==0 ) return 1; + p->aOp = aOp; + aArg = sqlite3_realloc64(p->aArg, N*sizeof(p->aArg[0])); + if( aArg==0 ) return 1; + p->aArg = aArg; + p->nAlloc = N; + return 0; +} + +/* Insert a new opcode and argument into an RE under construction. The +** insertion point is just prior to existing opcode iBefore. +*/ +static int re_insert(ReCompiled *p, int iBefore, int op, int arg){ + int i; + if( p->nAlloc<=p->nState && re_resize(p, p->nAlloc*2) ) return 0; + for(i=p->nState; i>iBefore; i--){ + p->aOp[i] = p->aOp[i-1]; + p->aArg[i] = p->aArg[i-1]; + } + p->nState++; + p->aOp[iBefore] = (char)op; + p->aArg[iBefore] = arg; + return iBefore; +} + +/* Append a new opcode and argument to the end of the RE under construction. +*/ +static int re_append(ReCompiled *p, int op, int arg){ + return re_insert(p, p->nState, op, arg); +} + +/* Make a copy of N opcodes starting at iStart onto the end of the RE +** under construction. +*/ +static void re_copy(ReCompiled *p, int iStart, int N){ + if( p->nState+N>=p->nAlloc && re_resize(p, p->nAlloc*2+N) ) return; + memcpy(&p->aOp[p->nState], &p->aOp[iStart], N*sizeof(p->aOp[0])); + memcpy(&p->aArg[p->nState], &p->aArg[iStart], N*sizeof(p->aArg[0])); + p->nState += N; +} + +/* Return true if c is a hexadecimal digit character: [0-9a-fA-F] +** If c is a hex digit, also set *pV = (*pV)*16 + valueof(c). If +** c is not a hex digit *pV is unchanged. +*/ +static int re_hex(int c, int *pV){ + if( c>='0' && c<='9' ){ + c -= '0'; + }else if( c>='a' && c<='f' ){ + c -= 'a' - 10; + }else if( c>='A' && c<='F' ){ + c -= 'A' - 10; + }else{ + return 0; + } + *pV = (*pV)*16 + (c & 0xff); + return 1; +} + +/* A backslash character has been seen, read the next character and +** return its interpretation. +*/ +static unsigned re_esc_char(ReCompiled *p){ + static const char zEsc[] = "afnrtv\\()*.+?[$^{|}]"; + static const char zTrans[] = "\a\f\n\r\t\v"; + int i, v = 0; + char c; + if( p->sIn.i>=p->sIn.mx ) return 0; + c = p->sIn.z[p->sIn.i]; + if( c=='u' && p->sIn.i+4<p->sIn.mx ){ + const unsigned char *zIn = p->sIn.z + p->sIn.i; + if( re_hex(zIn[1],&v) + && re_hex(zIn[2],&v) + && re_hex(zIn[3],&v) + && re_hex(zIn[4],&v) + ){ + p->sIn.i += 5; + return v; + } + } + if( c=='x' && p->sIn.i+2<p->sIn.mx ){ + const unsigned char *zIn = p->sIn.z + p->sIn.i; + if( re_hex(zIn[1],&v) + && re_hex(zIn[2],&v) + ){ + p->sIn.i += 3; + return v; + } + } + for(i=0; zEsc[i] && zEsc[i]!=c; i++){} + if( zEsc[i] ){ + if( i<6 ) c = zTrans[i]; + p->sIn.i++; + }else{ + p->zErr = "unknown \\ escape"; + } + return c; +} + +/* Forward declaration */ +static const char *re_subcompile_string(ReCompiled*); + +/* Peek at the next byte of input */ +static unsigned char rePeek(ReCompiled *p){ + return p->sIn.i<p->sIn.mx ? p->sIn.z[p->sIn.i] : 0; +} + +/* Compile RE text into a sequence of opcodes. Continue up to the +** first unmatched ")" character, then return. If an error is found, +** return a pointer to the error message string. +*/ +static const char *re_subcompile_re(ReCompiled *p){ + const char *zErr; + int iStart, iEnd, iGoto; + iStart = p->nState; + zErr = re_subcompile_string(p); + if( zErr ) return zErr; + while( rePeek(p)=='|' ){ + iEnd = p->nState; + re_insert(p, iStart, RE_OP_FORK, iEnd + 2 - iStart); + iGoto = re_append(p, RE_OP_GOTO, 0); + p->sIn.i++; + zErr = re_subcompile_string(p); + if( zErr ) return zErr; + p->aArg[iGoto] = p->nState - iGoto; + } + return 0; +} + +/* Compile an element of regular expression text (anything that can be +** an operand to the "|" operator). Return NULL on success or a pointer +** to the error message if there is a problem. +*/ +static const char *re_subcompile_string(ReCompiled *p){ + int iPrev = -1; + int iStart; + unsigned c; + const char *zErr; + while( (c = p->xNextChar(&p->sIn))!=0 ){ + iStart = p->nState; + switch( c ){ + case '|': + case ')': { + p->sIn.i--; + return 0; + } + case '(': { + zErr = re_subcompile_re(p); + if( zErr ) return zErr; + if( rePeek(p)!=')' ) return "unmatched '('"; + p->sIn.i++; + break; + } + case '.': { + if( rePeek(p)=='*' ){ + re_append(p, RE_OP_ANYSTAR, 0); + p->sIn.i++; + }else{ + re_append(p, RE_OP_ANY, 0); + } + break; + } + case '*': { + if( iPrev<0 ) return "'*' without operand"; + re_insert(p, iPrev, RE_OP_GOTO, p->nState - iPrev + 1); + re_append(p, RE_OP_FORK, iPrev - p->nState + 1); + break; + } + case '+': { + if( iPrev<0 ) return "'+' without operand"; + re_append(p, RE_OP_FORK, iPrev - p->nState); + break; + } + case '?': { + if( iPrev<0 ) return "'?' without operand"; + re_insert(p, iPrev, RE_OP_FORK, p->nState - iPrev+1); + break; + } + case '$': { + re_append(p, RE_OP_MATCH, RE_EOF); + break; + } + case '^': { + re_append(p, RE_OP_ATSTART, 0); + break; + } + case '{': { + int m = 0, n = 0; + int sz, j; + if( iPrev<0 ) return "'{m,n}' without operand"; + while( (c=rePeek(p))>='0' && c<='9' ){ m = m*10 + c - '0'; p->sIn.i++; } + n = m; + if( c==',' ){ + p->sIn.i++; + n = 0; + while( (c=rePeek(p))>='0' && c<='9' ){ n = n*10 + c-'0'; p->sIn.i++; } + } + if( c!='}' ) return "unmatched '{'"; + if( n>0 && n<m ) return "n less than m in '{m,n}'"; + p->sIn.i++; + sz = p->nState - iPrev; + if( m==0 ){ + if( n==0 ) return "both m and n are zero in '{m,n}'"; + re_insert(p, iPrev, RE_OP_FORK, sz+1); + iPrev++; + n--; + }else{ + for(j=1; j<m; j++) re_copy(p, iPrev, sz); + } + for(j=m; j<n; j++){ + re_append(p, RE_OP_FORK, sz+1); + re_copy(p, iPrev, sz); + } + if( n==0 && m>0 ){ + re_append(p, RE_OP_FORK, -sz); + } + break; + } + case '[': { + int iFirst = p->nState; + if( rePeek(p)=='^' ){ + re_append(p, RE_OP_CC_EXC, 0); + p->sIn.i++; + }else{ + re_append(p, RE_OP_CC_INC, 0); + } + while( (c = p->xNextChar(&p->sIn))!=0 ){ + if( c=='[' && rePeek(p)==':' ){ + return "POSIX character classes not supported"; + } + if( c=='\\' ) c = re_esc_char(p); + if( rePeek(p)=='-' ){ + re_append(p, RE_OP_CC_RANGE, c); + p->sIn.i++; + c = p->xNextChar(&p->sIn); + if( c=='\\' ) c = re_esc_char(p); + re_append(p, RE_OP_CC_RANGE, c); + }else{ + re_append(p, RE_OP_CC_VALUE, c); + } + if( rePeek(p)==']' ){ p->sIn.i++; break; } + } + if( c==0 ) return "unclosed '['"; + p->aArg[iFirst] = p->nState - iFirst; + break; + } + case '\\': { + int specialOp = 0; + switch( rePeek(p) ){ + case 'b': specialOp = RE_OP_BOUNDARY; break; + case 'd': specialOp = RE_OP_DIGIT; break; + case 'D': specialOp = RE_OP_NOTDIGIT; break; + case 's': specialOp = RE_OP_SPACE; break; + case 'S': specialOp = RE_OP_NOTSPACE; break; + case 'w': specialOp = RE_OP_WORD; break; + case 'W': specialOp = RE_OP_NOTWORD; break; + } + if( specialOp ){ + p->sIn.i++; + re_append(p, specialOp, 0); + }else{ + c = re_esc_char(p); + re_append(p, RE_OP_MATCH, c); + } + break; + } + default: { + re_append(p, RE_OP_MATCH, c); + break; + } + } + iPrev = iStart; + } + return 0; +} + +/* Free and reclaim all the memory used by a previously compiled +** regular expression. Applications should invoke this routine once +** for every call to re_compile() to avoid memory leaks. +*/ +static void re_free(ReCompiled *pRe){ + if( pRe ){ + sqlite3_free(pRe->aOp); + sqlite3_free(pRe->aArg); + sqlite3_free(pRe); + } +} + +/* +** Compile a textual regular expression in zIn[] into a compiled regular +** expression suitable for us by re_match() and return a pointer to the +** compiled regular expression in *ppRe. Return NULL on success or an +** error message if something goes wrong. +*/ +static const char *re_compile(ReCompiled **ppRe, const char *zIn, int noCase){ + ReCompiled *pRe; + const char *zErr; + int i, j; + + *ppRe = 0; + pRe = sqlite3_malloc( sizeof(*pRe) ); + if( pRe==0 ){ + return "out of memory"; + } + memset(pRe, 0, sizeof(*pRe)); + pRe->xNextChar = noCase ? re_next_char_nocase : re_next_char; + if( re_resize(pRe, 30) ){ + re_free(pRe); + return "out of memory"; + } + if( zIn[0]=='^' ){ + zIn++; + }else{ + re_append(pRe, RE_OP_ANYSTAR, 0); + } + pRe->sIn.z = (unsigned char*)zIn; + pRe->sIn.i = 0; + pRe->sIn.mx = (int)strlen(zIn); + zErr = re_subcompile_re(pRe); + if( zErr ){ + re_free(pRe); + return zErr; + } + if( pRe->sIn.i>=pRe->sIn.mx ){ + re_append(pRe, RE_OP_ACCEPT, 0); + *ppRe = pRe; + }else{ + re_free(pRe); + return "unrecognized character"; + } + + /* The following is a performance optimization. If the regex begins with + ** ".*" (if the input regex lacks an initial "^") and afterwards there are + ** one or more matching characters, enter those matching characters into + ** zInit[]. The re_match() routine can then search ahead in the input + ** string looking for the initial match without having to run the whole + ** regex engine over the string. Do not worry about trying to match + ** unicode characters beyond plane 0 - those are very rare and this is + ** just an optimization. */ + if( pRe->aOp[0]==RE_OP_ANYSTAR && !noCase ){ + for(j=0, i=1; j<(int)sizeof(pRe->zInit)-2 && pRe->aOp[i]==RE_OP_MATCH; i++){ + unsigned x = pRe->aArg[i]; + if( x<=0x7f ){ + pRe->zInit[j++] = (unsigned char)x; + }else if( x<=0x7ff ){ + pRe->zInit[j++] = (unsigned char)(0xc0 | (x>>6)); + pRe->zInit[j++] = 0x80 | (x&0x3f); + }else if( x<=0xffff ){ + pRe->zInit[j++] = (unsigned char)(0xe0 | (x>>12)); + pRe->zInit[j++] = 0x80 | ((x>>6)&0x3f); + pRe->zInit[j++] = 0x80 | (x&0x3f); + }else{ + break; + } + } + if( j>0 && pRe->zInit[j-1]==0 ) j--; + pRe->nInit = j; + } + return pRe->zErr; +} + +/* +** Implementation of the regexp() SQL function. This function implements +** the build-in REGEXP operator. The first argument to the function is the +** pattern and the second argument is the string. So, the SQL statements: +** +** A REGEXP B +** +** is implemented as regexp(B,A). +*/ +static void re_sql_func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + ReCompiled *pRe; /* Compiled regular expression */ + const char *zPattern; /* The regular expression */ + const unsigned char *zStr;/* String being searched */ + const char *zErr; /* Compile error message */ + int setAux = 0; /* True to invoke sqlite3_set_auxdata() */ + + (void)argc; /* Unused */ + pRe = sqlite3_get_auxdata(context, 0); + if( pRe==0 ){ + zPattern = (const char*)sqlite3_value_text(argv[0]); + if( zPattern==0 ) return; + zErr = re_compile(&pRe, zPattern, sqlite3_user_data(context)!=0); + if( zErr ){ + re_free(pRe); + sqlite3_result_error(context, zErr, -1); + return; + } + if( pRe==0 ){ + sqlite3_result_error_nomem(context); + return; + } + setAux = 1; + } + zStr = (const unsigned char*)sqlite3_value_text(argv[1]); + if( zStr!=0 ){ + sqlite3_result_int(context, re_match(pRe, zStr, -1)); + } + if( setAux ){ + sqlite3_set_auxdata(context, 0, pRe, (void(*)(void*))re_free); + } +} + +#if defined(SQLITE_DEBUG) +/* +** This function is used for testing and debugging only. It is only available +** if the SQLITE_DEBUG compile-time option is used. +** +** Compile a regular expression and then convert the compiled expression into +** text and return that text. +*/ +static void re_bytecode_func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zPattern; + const char *zErr; + ReCompiled *pRe; + sqlite3_str *pStr; + int i; + int n; + char *z; + + zPattern = (const char*)sqlite3_value_text(argv[0]); + if( zPattern==0 ) return; + zErr = re_compile(&pRe, zPattern, sqlite3_user_data(context)!=0); + if( zErr ){ + re_free(pRe); + sqlite3_result_error(context, zErr, -1); + return; + } + if( pRe==0 ){ + sqlite3_result_error_nomem(context); + return; + } + pStr = sqlite3_str_new(0); + if( pStr==0 ) goto re_bytecode_func_err; + if( pRe->nInit>0 ){ + sqlite3_str_appendf(pStr, "INIT "); + for(i=0; i<pRe->nInit; i++){ + sqlite3_str_appendf(pStr, "%02x", pRe->zInit[i]); + } + sqlite3_str_appendf(pStr, "\n"); + } + for(i=0; (unsigned)i<pRe->nState; i++){ + sqlite3_str_appendf(pStr, "%-8s %4d\n", + ReOpName[(unsigned char)pRe->aOp[i]], pRe->aArg[i]); + } + n = sqlite3_str_length(pStr); + z = sqlite3_str_finish(pStr); + if( n==0 ){ + sqlite3_free(z); + }else{ + sqlite3_result_text(context, z, n-1, sqlite3_free); + } + +re_bytecode_func_err: + re_free(pRe); +} + +#endif /* SQLITE_DEBUG */ + + +/* +** Invoke this routine to register the regexp() function with the +** SQLite database connection. +*/ +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_regexp_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused */ + rc = sqlite3_create_function(db, "regexp", 2, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, re_sql_func, 0, 0); + if( rc==SQLITE_OK ){ + /* The regexpi(PATTERN,STRING) function is a case-insensitive version + ** of regexp(PATTERN,STRING). */ + rc = sqlite3_create_function(db, "regexpi", 2, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + (void*)db, re_sql_func, 0, 0); +#if defined(SQLITE_DEBUG) + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "regexp_bytecode", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, re_bytecode_func, 0, 0); + } +#endif /* SQLITE_DEBUG */ + } + return rc; +} diff --git a/ext/misc/remember.c b/ext/misc/remember.c new file mode 100644 index 0000000..18f6ede --- /dev/null +++ b/ext/misc/remember.c @@ -0,0 +1,72 @@ +/* +** 2016-08-09 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates how to create an SQL function that is a pass-through +** for integer values (it returns a copy of its argument) but also saves the +** value that is passed through into a C-language variable. The address of +** the C-language variable is supplied as the second argument. +** +** This allows, for example, a counter to incremented and the original +** value retrieved, atomically, using a single statement: +** +** UPDATE counterTab SET cnt=remember(cnt,$PTR)+1 WHERE id=$ID +** +** Prepare the above statement once. Then to use it, bind the address +** of the output variable to $PTR using sqlite3_bind_pointer() with a +** pointer type of "carray" and bind the id of the counter to $ID and +** run the prepared statement. +** +** This implementation of the remember() function uses a "carray" +** pointer so that it can share pointers with the carray() extension. +** +** One can imagine doing similar things with floating-point values and +** strings, but this demonstration extension will stick to using just +** integers. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> + +/* +** remember(V,PTR) +** +** Return the integer value V. Also save the value of V in a +** C-language variable whose address is PTR. +*/ +static void rememberFunc( + sqlite3_context *pCtx, + int argc, + sqlite3_value **argv +){ + sqlite3_int64 v; + sqlite3_int64 *ptr; + assert( argc==2 ); + v = sqlite3_value_int64(argv[0]); + ptr = sqlite3_value_pointer(argv[1], "carray"); + if( ptr ) *ptr = v; + sqlite3_result_int64(pCtx, v); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_remember_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_create_function(db, "remember", 2, SQLITE_UTF8, 0, + rememberFunc, 0, 0); + return rc; +} diff --git a/ext/misc/rot13.c b/ext/misc/rot13.c new file mode 100644 index 0000000..05b4a18 --- /dev/null +++ b/ext/misc/rot13.c @@ -0,0 +1,115 @@ +/* +** 2013-05-15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements a rot13() function and a rot13 +** collating sequence. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* +** Perform rot13 encoding on a single ASCII character. +*/ +static unsigned char rot13(unsigned char c){ + if( c>='a' && c<='z' ){ + c += 13; + if( c>'z' ) c -= 26; + }else if( c>='A' && c<='Z' ){ + c += 13; + if( c>'Z' ) c -= 26; + } + return c; +} + +/* +** Implementation of the rot13() function. +** +** Rotate ASCII alphabetic characters by 13 character positions. +** Non-ASCII characters are unchanged. rot13(rot13(X)) should always +** equal X. +*/ +static void rot13func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *zIn; + int nIn; + unsigned char *zOut; + unsigned char *zToFree = 0; + int i; + unsigned char zTemp[100]; + assert( argc==1 ); + if( sqlite3_value_type(argv[0])==SQLITE_NULL ) return; + zIn = (const unsigned char*)sqlite3_value_text(argv[0]); + nIn = sqlite3_value_bytes(argv[0]); + if( nIn<sizeof(zTemp)-1 ){ + zOut = zTemp; + }else{ + zOut = zToFree = (unsigned char*)sqlite3_malloc64( nIn+1 ); + if( zOut==0 ){ + sqlite3_result_error_nomem(context); + return; + } + } + for(i=0; i<nIn; i++) zOut[i] = rot13(zIn[i]); + zOut[i] = 0; + sqlite3_result_text(context, (char*)zOut, i, SQLITE_TRANSIENT); + sqlite3_free(zToFree); +} + +/* +** Implement the rot13 collating sequence so that if +** +** x=y COLLATE rot13 +** +** Then +** +** rot13(x)=rot13(y) COLLATE binary +*/ +static int rot13CollFunc( + void *notUsed, + int nKey1, const void *pKey1, + int nKey2, const void *pKey2 +){ + const char *zA = (const char*)pKey1; + const char *zB = (const char*)pKey2; + int i, x; + for(i=0; i<nKey1 && i<nKey2; i++){ + x = (int)rot13(zA[i]) - (int)rot13(zB[i]); + if( x!=0 ) return x; + } + return nKey1 - nKey2; +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_rot_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "rot13", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, rot13func, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_collation(db, "rot13", SQLITE_UTF8, 0, rot13CollFunc); + } + return rc; +} diff --git a/ext/misc/scrub.c b/ext/misc/scrub.c new file mode 100644 index 0000000..9fbf2ae --- /dev/null +++ b/ext/misc/scrub.c @@ -0,0 +1,610 @@ +/* +** 2016-05-05 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file implements a utility function (and a utility program) that +** makes a copy of an SQLite database while simultaneously zeroing out all +** deleted content. +** +** Normally (when PRAGMA secure_delete=OFF, which is the default) when SQLite +** deletes content, it does not overwrite the deleted content but rather marks +** the region of the file that held that content as being reusable. This can +** cause deleted content to recoverable from the database file. This stale +** content is removed by the VACUUM command, but VACUUM can be expensive for +** large databases. When in PRAGMA secure_delete=ON mode, the deleted content +** is zeroed, but secure_delete=ON has overhead as well. +** +** This utility attempts to make a copy of a complete SQLite database where +** all of the deleted content is zeroed out in the copy, and it attempts to +** do so while being faster than running VACUUM. +** +** Usage: +** +** int sqlite3_scrub_backup( +** const char *zSourceFile, // Source database filename +** const char *zDestFile, // Destination database filename +** char **pzErrMsg // Write error message here +** ); +** +** Simply call the API above specifying the filename of the source database +** and the name of the backup copy. The source database must already exist +** and can be in active use. (A read lock is held during the backup.) The +** destination file should not previously exist. If the pzErrMsg parameter +** is non-NULL and if an error occurs, then an error message might be written +** into memory obtained from sqlite3_malloc() and *pzErrMsg made to point to +** that error message. But if the error is an OOM, the error might not be +** reported. The routine always returns non-zero if there is an error. +** +** If compiled with -DSCRUB_STANDALONE then a main() procedure is added and +** this file becomes a standalone program that can be run as follows: +** +** ./sqlite3scrub SOURCE DEST +*/ +#include "sqlite3.h" +#include <assert.h> +#include <stdio.h> +#include <stdlib.h> +#include <stdarg.h> +#include <string.h> + +typedef struct ScrubState ScrubState; +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; + + +/* State information for a scrub-and-backup operation */ +struct ScrubState { + const char *zSrcFile; /* Name of the source file */ + const char *zDestFile; /* Name of the destination file */ + int rcErr; /* Error code */ + char *zErr; /* Error message text */ + sqlite3 *dbSrc; /* Source database connection */ + sqlite3_file *pSrc; /* Source file handle */ + sqlite3 *dbDest; /* Destination database connection */ + sqlite3_file *pDest; /* Destination file handle */ + u32 szPage; /* Page size */ + u32 szUsable; /* Usable bytes on each page */ + u32 nPage; /* Number of pages */ + u32 iLastPage; /* Page number of last page written so far*/ + u8 *page1; /* Content of page 1 */ +}; + +/* Store an error message */ +static void scrubBackupErr(ScrubState *p, const char *zFormat, ...){ + va_list ap; + sqlite3_free(p->zErr); + va_start(ap, zFormat); + p->zErr = sqlite3_vmprintf(zFormat, ap); + va_end(ap); + if( p->rcErr==0 ) p->rcErr = SQLITE_ERROR; +} + +/* Allocate memory to hold a single page of content */ +static u8 *scrubBackupAllocPage(ScrubState *p){ + u8 *pPage; + if( p->rcErr ) return 0; + pPage = sqlite3_malloc( p->szPage ); + if( pPage==0 ) p->rcErr = SQLITE_NOMEM; + return pPage; +} + +/* Read a page from the source database into memory. Use the memory +** provided by pBuf if not NULL or allocate a new page if pBuf==NULL. +*/ +static u8 *scrubBackupRead(ScrubState *p, int pgno, u8 *pBuf){ + int rc; + sqlite3_int64 iOff; + u8 *pOut = pBuf; + if( p->rcErr ) return 0; + if( pOut==0 ){ + pOut = scrubBackupAllocPage(p); + if( pOut==0 ) return 0; + } + iOff = (pgno-1)*(sqlite3_int64)p->szPage; + rc = p->pSrc->pMethods->xRead(p->pSrc, pOut, p->szPage, iOff); + if( rc!=SQLITE_OK ){ + if( pBuf==0 ) sqlite3_free(pOut); + pOut = 0; + scrubBackupErr(p, "read failed for page %d", pgno); + p->rcErr = SQLITE_IOERR; + } + return pOut; +} + +/* Write a page to the destination database */ +static void scrubBackupWrite(ScrubState *p, int pgno, const u8 *pData){ + int rc; + sqlite3_int64 iOff; + if( p->rcErr ) return; + iOff = (pgno-1)*(sqlite3_int64)p->szPage; + rc = p->pDest->pMethods->xWrite(p->pDest, pData, p->szPage, iOff); + if( rc!=SQLITE_OK ){ + scrubBackupErr(p, "write failed for page %d", pgno); + p->rcErr = SQLITE_IOERR; + } + if( (u32)pgno>p->iLastPage ) p->iLastPage = pgno; +} + +/* Prepare a statement against the "db" database. */ +static sqlite3_stmt *scrubBackupPrepare( + ScrubState *p, /* Backup context */ + sqlite3 *db, /* Database to prepare against */ + const char *zSql /* SQL statement */ +){ + sqlite3_stmt *pStmt; + if( p->rcErr ) return 0; + p->rcErr = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0); + if( p->rcErr ){ + scrubBackupErr(p, "SQL error \"%s\" on \"%s\"", + sqlite3_errmsg(db), zSql); + sqlite3_finalize(pStmt); + return 0; + } + return pStmt; +} + + +/* Open the source database file */ +static void scrubBackupOpenSrc(ScrubState *p){ + sqlite3_stmt *pStmt; + int rc; + /* Open the source database file */ + p->rcErr = sqlite3_open_v2(p->zSrcFile, &p->dbSrc, + SQLITE_OPEN_READWRITE | + SQLITE_OPEN_URI | SQLITE_OPEN_PRIVATECACHE, 0); + if( p->rcErr ){ + scrubBackupErr(p, "cannot open source database: %s", + sqlite3_errmsg(p->dbSrc)); + return; + } + p->rcErr = sqlite3_exec(p->dbSrc, "SELECT 1 FROM sqlite_schema; BEGIN;", + 0, 0, 0); + if( p->rcErr ){ + scrubBackupErr(p, + "cannot start a read transaction on the source database: %s", + sqlite3_errmsg(p->dbSrc)); + return; + } + rc = sqlite3_wal_checkpoint_v2(p->dbSrc, "main", SQLITE_CHECKPOINT_FULL, + 0, 0); + if( rc ){ + scrubBackupErr(p, "cannot checkpoint the source database"); + return; + } + pStmt = scrubBackupPrepare(p, p->dbSrc, "PRAGMA page_size"); + if( pStmt==0 ) return; + rc = sqlite3_step(pStmt); + if( rc==SQLITE_ROW ){ + p->szPage = sqlite3_column_int(pStmt, 0); + }else{ + scrubBackupErr(p, "unable to determine the page size"); + } + sqlite3_finalize(pStmt); + if( p->rcErr ) return; + pStmt = scrubBackupPrepare(p, p->dbSrc, "PRAGMA page_count"); + if( pStmt==0 ) return; + rc = sqlite3_step(pStmt); + if( rc==SQLITE_ROW ){ + p->nPage = sqlite3_column_int(pStmt, 0); + }else{ + scrubBackupErr(p, "unable to determine the size of the source database"); + } + sqlite3_finalize(pStmt); + sqlite3_file_control(p->dbSrc, "main", SQLITE_FCNTL_FILE_POINTER, &p->pSrc); + if( p->pSrc==0 || p->pSrc->pMethods==0 ){ + scrubBackupErr(p, "cannot get the source file handle"); + p->rcErr = SQLITE_ERROR; + } +} + +/* Create and open the destination file */ +static void scrubBackupOpenDest(ScrubState *p){ + sqlite3_stmt *pStmt; + int rc; + char *zSql; + if( p->rcErr ) return; + p->rcErr = sqlite3_open_v2(p->zDestFile, &p->dbDest, + SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | + SQLITE_OPEN_URI | SQLITE_OPEN_PRIVATECACHE, 0); + if( p->rcErr ){ + scrubBackupErr(p, "cannot open destination database: %s", + sqlite3_errmsg(p->dbDest)); + return; + } + zSql = sqlite3_mprintf("PRAGMA page_size(%u);", p->szPage); + if( zSql==0 ){ + p->rcErr = SQLITE_NOMEM; + return; + } + p->rcErr = sqlite3_exec(p->dbDest, zSql, 0, 0, 0); + sqlite3_free(zSql); + if( p->rcErr ){ + scrubBackupErr(p, + "cannot set the page size on the destination database: %s", + sqlite3_errmsg(p->dbDest)); + return; + } + sqlite3_exec(p->dbDest, "PRAGMA journal_mode=OFF;", 0, 0, 0); + p->rcErr = sqlite3_exec(p->dbDest, "BEGIN EXCLUSIVE;", 0, 0, 0); + if( p->rcErr ){ + scrubBackupErr(p, + "cannot start a write transaction on the destination database: %s", + sqlite3_errmsg(p->dbDest)); + return; + } + pStmt = scrubBackupPrepare(p, p->dbDest, "PRAGMA page_count;"); + if( pStmt==0 ) return; + rc = sqlite3_step(pStmt); + if( rc!=SQLITE_ROW ){ + scrubBackupErr(p, "cannot measure the size of the destination"); + }else if( sqlite3_column_int(pStmt, 0)>1 ){ + scrubBackupErr(p, "destination database is not empty - holds %d pages", + sqlite3_column_int(pStmt, 0)); + } + sqlite3_finalize(pStmt); + sqlite3_file_control(p->dbDest, "main", SQLITE_FCNTL_FILE_POINTER, &p->pDest); + if( p->pDest==0 || p->pDest->pMethods==0 ){ + scrubBackupErr(p, "cannot get the destination file handle"); + p->rcErr = SQLITE_ERROR; + } +} + +/* Read a 32-bit big-endian integer */ +static u32 scrubBackupInt32(const u8 *a){ + u32 v = a[3]; + v += ((u32)a[2])<<8; + v += ((u32)a[1])<<16; + v += ((u32)a[0])<<24; + return v; +} + +/* Read a 16-bit big-endian integer */ +static u32 scrubBackupInt16(const u8 *a){ + return (a[0]<<8) + a[1]; +} + +/* +** Read a varint. Put the value in *pVal and return the number of bytes. +*/ +static int scrubBackupVarint(const u8 *z, sqlite3_int64 *pVal){ + sqlite3_int64 v = 0; + int i; + for(i=0; i<8; i++){ + v = (v<<7) + (z[i]&0x7f); + if( (z[i]&0x80)==0 ){ *pVal = v; return i+1; } + } + v = (v<<8) + (z[i]&0xff); + *pVal = v; + return 9; +} + +/* +** Return the number of bytes in a varint. +*/ +static int scrubBackupVarintSize(const u8 *z){ + int i; + for(i=0; i<8; i++){ + if( (z[i]&0x80)==0 ){ return i+1; } + } + return 9; +} + +/* +** Copy the freelist trunk page given, and all its descendents, +** zeroing out as much as possible in the process. +*/ +static void scrubBackupFreelist(ScrubState *p, int pgno, u32 nFree){ + u8 *a, *aBuf; + u32 n, mx; + + if( p->rcErr ) return; + aBuf = scrubBackupAllocPage(p); + if( aBuf==0 ) return; + + while( pgno && nFree){ + a = scrubBackupRead(p, pgno, aBuf); + if( a==0 ) break; + n = scrubBackupInt32(&a[4]); + mx = p->szUsable/4 - 2; + if( n<mx ){ + memset(&a[n*4+8], 0, 4*(mx-n)); + } + scrubBackupWrite(p, pgno, a); + pgno = scrubBackupInt32(a); +#if 0 + /* There is really no point in copying the freelist leaf pages. + ** Simply leave them uninitialized in the destination database. The + ** OS filesystem should zero those pages for us automatically. + */ + for(i=0; i<n && nFree; i++){ + u32 iLeaf = scrubBackupInt32(&a[i*4+8]); + if( aZero==0 ){ + aZero = scrubBackupAllocPage(p); + if( aZero==0 ){ pgno = 0; break; } + memset(aZero, 0, p->szPage); + } + scrubBackupWrite(p, iLeaf, aZero); + nFree--; + } +#endif + } + sqlite3_free(aBuf); +} + +/* +** Copy an overflow chain from source to destination. Zero out any +** unused tail at the end of the overflow chain. +*/ +static void scrubBackupOverflow(ScrubState *p, int pgno, u32 nByte){ + u8 *a, *aBuf; + + aBuf = scrubBackupAllocPage(p); + if( aBuf==0 ) return; + while( nByte>0 && pgno!=0 ){ + a = scrubBackupRead(p, pgno, aBuf); + if( a==0 ) break; + if( nByte >= (p->szUsable)-4 ){ + nByte -= (p->szUsable) - 4; + }else{ + u32 x = (p->szUsable - 4) - nByte; + u32 i = p->szUsable - x; + memset(&a[i], 0, x); + nByte = 0; + } + scrubBackupWrite(p, pgno, a); + pgno = scrubBackupInt32(a); + } + sqlite3_free(aBuf); +} + + +/* +** Copy B-Tree page pgno, and all of its children, from source to destination. +** Zero out deleted content during the copy. +*/ +static void scrubBackupBtree(ScrubState *p, int pgno, int iDepth){ + u8 *a; + u32 i, n, pc; + u32 nCell; + u32 nPrefix; + u32 szHdr; + u32 iChild; + u8 *aTop; + u8 *aCell; + u32 x, y; + int ln = 0; + + + if( p->rcErr ) return; + if( iDepth>50 ){ + scrubBackupErr(p, "corrupt: b-tree too deep at page %d", pgno); + return; + } + if( pgno==1 ){ + a = p->page1; + }else{ + a = scrubBackupRead(p, pgno, 0); + if( a==0 ) return; + } + nPrefix = pgno==1 ? 100 : 0; + aTop = &a[nPrefix]; + szHdr = 8 + 4*(aTop[0]==0x02 || aTop[0]==0x05); + aCell = aTop + szHdr; + nCell = scrubBackupInt16(&aTop[3]); + + /* Zero out the gap between the cell index and the start of the + ** cell content area */ + x = scrubBackupInt16(&aTop[5]); /* First byte of cell content area */ + if( x>p->szUsable ){ ln=__LINE__; goto btree_corrupt; } + y = szHdr + nPrefix + nCell*2; + if( y>x ){ ln=__LINE__; goto btree_corrupt; } + if( y<x ) memset(a+y, 0, x-y); /* Zero the gap */ + + /* Zero out all the free blocks */ + pc = scrubBackupInt16(&aTop[1]); + if( pc>0 && pc<x ){ ln=__LINE__; goto btree_corrupt; } + while( pc ){ + if( pc>(p->szUsable)-4 ){ ln=__LINE__; goto btree_corrupt; } + n = scrubBackupInt16(&a[pc+2]); + if( pc+n>(p->szUsable) ){ ln=__LINE__; goto btree_corrupt; } + if( n>4 ) memset(&a[pc+4], 0, n-4); + x = scrubBackupInt16(&a[pc]); + if( x<pc+4 && x>0 ){ ln=__LINE__; goto btree_corrupt; } + pc = x; + } + + /* Write this one page */ + scrubBackupWrite(p, pgno, a); + + /* Walk the tree and process child pages */ + for(i=0; i<nCell; i++){ + u32 X, M, K, nLocal; + sqlite3_int64 P; + pc = scrubBackupInt16(&aCell[i*2]); + if( pc <= szHdr ){ ln=__LINE__; goto btree_corrupt; } + if( pc > p->szUsable-3 ){ ln=__LINE__; goto btree_corrupt; } + if( aTop[0]==0x05 || aTop[0]==0x02 ){ + if( pc+4 > p->szUsable ){ ln=__LINE__; goto btree_corrupt; } + iChild = scrubBackupInt32(&a[pc]); + pc += 4; + scrubBackupBtree(p, iChild, iDepth+1); + if( aTop[0]==0x05 ) continue; + } + pc += scrubBackupVarint(&a[pc], &P); + if( pc >= p->szUsable ){ ln=__LINE__; goto btree_corrupt; } + if( aTop[0]==0x0d ){ + X = p->szUsable - 35; + }else{ + X = ((p->szUsable - 12)*64/255) - 23; + } + if( P<=X ){ + /* All content is local. No overflow */ + continue; + } + M = ((p->szUsable - 12)*32/255)-23; + K = M + ((P-M)%(p->szUsable-4)); + if( aTop[0]==0x0d ){ + pc += scrubBackupVarintSize(&a[pc]); + if( pc > (p->szUsable-4) ){ ln=__LINE__; goto btree_corrupt; } + } + nLocal = K<=X ? K : M; + if( pc+nLocal > p->szUsable-4 ){ ln=__LINE__; goto btree_corrupt; } + iChild = scrubBackupInt32(&a[pc+nLocal]); + scrubBackupOverflow(p, iChild, (u32)(P-nLocal)); + } + + /* Walk the right-most tree */ + if( aTop[0]==0x05 || aTop[0]==0x02 ){ + iChild = scrubBackupInt32(&aTop[8]); + scrubBackupBtree(p, iChild, iDepth+1); + } + + /* All done */ + if( pgno>1 ) sqlite3_free(a); + return; + +btree_corrupt: + scrubBackupErr(p, "corruption on page %d of source database (errid=%d)", + pgno, ln); + if( pgno>1 ) sqlite3_free(a); +} + +/* +** Copy all ptrmap pages from source to destination. +** This routine is only called if the source database is in autovacuum +** or incremental vacuum mode. +*/ +static void scrubBackupPtrmap(ScrubState *p){ + u32 pgno = 2; + u32 J = p->szUsable/5; + u32 iLock = (1073742335/p->szPage)+1; + u8 *a, *pBuf; + if( p->rcErr ) return; + pBuf = scrubBackupAllocPage(p); + if( pBuf==0 ) return; + while( pgno<=p->nPage ){ + a = scrubBackupRead(p, pgno, pBuf); + if( a==0 ) break; + scrubBackupWrite(p, pgno, a); + pgno += J+1; + if( pgno==iLock ) pgno++; + } + sqlite3_free(pBuf); +} + +int sqlite3_scrub_backup( + const char *zSrcFile, /* Source file */ + const char *zDestFile, /* Destination file */ + char **pzErr /* Write error here if non-NULL */ +){ + ScrubState s; + u32 n, i; + sqlite3_stmt *pStmt; + + memset(&s, 0, sizeof(s)); + s.zSrcFile = zSrcFile; + s.zDestFile = zDestFile; + + /* Open both source and destination databases */ + scrubBackupOpenSrc(&s); + scrubBackupOpenDest(&s); + + /* Read in page 1 */ + s.page1 = scrubBackupRead(&s, 1, 0); + if( s.page1==0 ) goto scrub_abort; + s.szUsable = s.szPage - s.page1[20]; + + /* Copy the freelist */ + n = scrubBackupInt32(&s.page1[36]); + i = scrubBackupInt32(&s.page1[32]); + if( n ) scrubBackupFreelist(&s, i, n); + + /* Copy ptrmap pages */ + n = scrubBackupInt32(&s.page1[52]); + if( n ) scrubBackupPtrmap(&s); + + /* Copy all of the btrees */ + scrubBackupBtree(&s, 1, 0); + pStmt = scrubBackupPrepare(&s, s.dbSrc, + "SELECT rootpage FROM sqlite_schema WHERE coalesce(rootpage,0)>0"); + if( pStmt==0 ) goto scrub_abort; + while( sqlite3_step(pStmt)==SQLITE_ROW ){ + i = (u32)sqlite3_column_int(pStmt, 0); + scrubBackupBtree(&s, i, 0); + } + sqlite3_finalize(pStmt); + + /* If the last page of the input db file is a free-list leaf, then the + ** backup file on disk is still smaller than the size indicated within + ** the database header. In this case, write a page of zeroes to the + ** last page of the backup database so that SQLite does not mistakenly + ** think the db is corrupt. */ + if( s.iLastPage<s.nPage ){ + u8 *aZero = scrubBackupAllocPage(&s); + if( aZero ){ + memset(aZero, 0, s.szPage); + scrubBackupWrite(&s, s.nPage, aZero); + sqlite3_free(aZero); + } + } + +scrub_abort: + /* Close the destination database without closing the transaction. If we + ** commit, page zero will be overwritten. */ + sqlite3_close(s.dbDest); + + /* But do close out the read-transaction on the source database */ + sqlite3_exec(s.dbSrc, "COMMIT;", 0, 0, 0); + sqlite3_close(s.dbSrc); + sqlite3_free(s.page1); + if( pzErr ){ + *pzErr = s.zErr; + }else{ + sqlite3_free(s.zErr); + } + return s.rcErr; +} + +#ifdef SCRUB_STANDALONE +/* Error and warning log */ +static void errorLogCallback(void *pNotUsed, int iErr, const char *zMsg){ + const char *zType; + switch( iErr&0xff ){ + case SQLITE_WARNING: zType = "WARNING"; break; + case SQLITE_NOTICE: zType = "NOTICE"; break; + default: zType = "ERROR"; break; + } + fprintf(stderr, "%s: %s\n", zType, zMsg); +} + +/* The main() routine when this utility is run as a stand-alone program */ +int main(int argc, char **argv){ + char *zErr = 0; + int rc; + if( argc!=3 ){ + fprintf(stderr,"Usage: %s SOURCE DESTINATION\n", argv[0]); + exit(1); + } + sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, 0); + rc = sqlite3_scrub_backup(argv[1], argv[2], &zErr); + if( rc==SQLITE_NOMEM ){ + fprintf(stderr, "%s: out of memory\n", argv[0]); + exit(1); + } + if( zErr ){ + fprintf(stderr, "%s: %s\n", argv[0], zErr); + sqlite3_free(zErr); + exit(1); + } + return 0; +} +#endif diff --git a/ext/misc/series.c b/ext/misc/series.c new file mode 100644 index 0000000..3941d96 --- /dev/null +++ b/ext/misc/series.c @@ -0,0 +1,459 @@ +/* +** 2015-08-18 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates how to create a table-valued-function using +** a virtual table. This demo implements the generate_series() function +** which gives similar results to the eponymous function in PostgreSQL. +** Examples: +** +** SELECT * FROM generate_series(0,100,5); +** +** The query above returns integers from 0 through 100 counting by steps +** of 5. +** +** SELECT * FROM generate_series(0,100); +** +** Integers from 0 through 100 with a step size of 1. +** +** SELECT * FROM generate_series(20) LIMIT 10; +** +** Integers 20 through 29. +** +** HOW IT WORKS +** +** The generate_series "function" is really a virtual table with the +** following schema: +** +** CREATE TABLE generate_series( +** value, +** start HIDDEN, +** stop HIDDEN, +** step HIDDEN +** ); +** +** Function arguments in queries against this virtual table are translated +** into equality constraints against successive hidden columns. In other +** words, the following pairs of queries are equivalent to each other: +** +** SELECT * FROM generate_series(0,100,5); +** SELECT * FROM generate_series WHERE start=0 AND stop=100 AND step=5; +** +** SELECT * FROM generate_series(0,100); +** SELECT * FROM generate_series WHERE start=0 AND stop=100; +** +** SELECT * FROM generate_series(20) LIMIT 10; +** SELECT * FROM generate_series WHERE start=20 LIMIT 10; +** +** The generate_series virtual table implementation leaves the xCreate method +** set to NULL. This means that it is not possible to do a CREATE VIRTUAL +** TABLE command with "generate_series" as the USING argument. Instead, there +** is a single generate_series virtual table that is always available without +** having to be created first. +** +** The xBestIndex method looks for equality constraints against the hidden +** start, stop, and step columns, and if present, it uses those constraints +** to bound the sequence of generated values. If the equality constraints +** are missing, it uses 0 for start, 4294967295 for stop, and 1 for step. +** xBestIndex returns a small cost when both start and stop are available, +** and a very large cost if either start or stop are unavailable. This +** encourages the query planner to order joins such that the bounds of the +** series are well-defined. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + + +/* series_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct series_cursor series_cursor; +struct series_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + int isDesc; /* True to count down rather than up */ + sqlite3_int64 iRowid; /* The rowid */ + sqlite3_int64 iValue; /* Current value ("value") */ + sqlite3_int64 mnValue; /* Mimimum value ("start") */ + sqlite3_int64 mxValue; /* Maximum value ("stop") */ + sqlite3_int64 iStep; /* Increment ("step") */ +}; + +/* +** The seriesConnect() method is invoked to create a new +** series_vtab that describes the generate_series virtual table. +** +** Think of this routine as the constructor for series_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the series_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against generate_series will look like. +*/ +static int seriesConnect( + sqlite3 *db, + void *pUnused, + int argcUnused, const char *const*argvUnused, + sqlite3_vtab **ppVtab, + char **pzErrUnused +){ + sqlite3_vtab *pNew; + int rc; + +/* Column numbers */ +#define SERIES_COLUMN_VALUE 0 +#define SERIES_COLUMN_START 1 +#define SERIES_COLUMN_STOP 2 +#define SERIES_COLUMN_STEP 3 + + (void)pUnused; + (void)argcUnused; + (void)argvUnused; + (void)pzErrUnused; + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(value,start hidden,stop hidden,step hidden)"); + if( rc==SQLITE_OK ){ + pNew = *ppVtab = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + } + return rc; +} + +/* +** This method is the destructor for series_cursor objects. +*/ +static int seriesDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new series_cursor object. +*/ +static int seriesOpen(sqlite3_vtab *pUnused, sqlite3_vtab_cursor **ppCursor){ + series_cursor *pCur; + (void)pUnused; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a series_cursor. +*/ +static int seriesClose(sqlite3_vtab_cursor *cur){ + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a series_cursor to its next row of output. +*/ +static int seriesNext(sqlite3_vtab_cursor *cur){ + series_cursor *pCur = (series_cursor*)cur; + if( pCur->isDesc ){ + pCur->iValue -= pCur->iStep; + }else{ + pCur->iValue += pCur->iStep; + } + pCur->iRowid++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the series_cursor +** is currently pointing. +*/ +static int seriesColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + series_cursor *pCur = (series_cursor*)cur; + sqlite3_int64 x = 0; + switch( i ){ + case SERIES_COLUMN_START: x = pCur->mnValue; break; + case SERIES_COLUMN_STOP: x = pCur->mxValue; break; + case SERIES_COLUMN_STEP: x = pCur->iStep; break; + default: x = pCur->iValue; break; + } + sqlite3_result_int64(ctx, x); + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** first row returned is assigned rowid value 1, and each subsequent +** row a value 1 more than that of the previous. +*/ +static int seriesRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + series_cursor *pCur = (series_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int seriesEof(sqlite3_vtab_cursor *cur){ + series_cursor *pCur = (series_cursor*)cur; + if( pCur->isDesc ){ + return pCur->iValue < pCur->mnValue; + }else{ + return pCur->iValue > pCur->mxValue; + } +} + +/* True to cause run-time checking of the start=, stop=, and/or step= +** parameters. The only reason to do this is for testing the +** constraint checking logic for virtual tables in the SQLite core. +*/ +#ifndef SQLITE_SERIES_CONSTRAINT_VERIFY +# define SQLITE_SERIES_CONSTRAINT_VERIFY 0 +#endif + +/* +** This method is called to "rewind" the series_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to seriesColumn() or seriesRowid() or +** seriesEof(). +** +** The query plan selected by seriesBestIndex is passed in the idxNum +** parameter. (idxStr is not used in this implementation.) idxNum +** is a bitmask showing which constraints are available: +** +** 1: start=VALUE +** 2: stop=VALUE +** 4: step=VALUE +** +** Also, if bit 8 is set, that means that the series should be output +** in descending order rather than in ascending order. If bit 16 is +** set, then output must appear in ascending order. +** +** This routine should initialize the cursor and position it so that it +** is pointing at the first row, or pointing off the end of the table +** (so that seriesEof() will return true) if the table is empty. +*/ +static int seriesFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStrUnused, + int argc, sqlite3_value **argv +){ + series_cursor *pCur = (series_cursor *)pVtabCursor; + int i = 0; + (void)idxStrUnused; + if( idxNum & 1 ){ + pCur->mnValue = sqlite3_value_int64(argv[i++]); + }else{ + pCur->mnValue = 0; + } + if( idxNum & 2 ){ + pCur->mxValue = sqlite3_value_int64(argv[i++]); + }else{ + pCur->mxValue = 0xffffffff; + } + if( idxNum & 4 ){ + pCur->iStep = sqlite3_value_int64(argv[i++]); + if( pCur->iStep==0 ){ + pCur->iStep = 1; + }else if( pCur->iStep<0 ){ + pCur->iStep = -pCur->iStep; + if( (idxNum & 16)==0 ) idxNum |= 8; + } + }else{ + pCur->iStep = 1; + } + for(i=0; i<argc; i++){ + if( sqlite3_value_type(argv[i])==SQLITE_NULL ){ + /* If any of the constraints have a NULL value, then return no rows. + ** See ticket https://www.sqlite.org/src/info/fac496b61722daf2 */ + pCur->mnValue = 1; + pCur->mxValue = 0; + break; + } + } + if( idxNum & 8 ){ + pCur->isDesc = 1; + pCur->iValue = pCur->mxValue; + if( pCur->iStep>0 ){ + pCur->iValue -= (pCur->mxValue - pCur->mnValue)%pCur->iStep; + } + }else{ + pCur->isDesc = 0; + pCur->iValue = pCur->mnValue; + } + pCur->iRowid = 1; + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the generate_series virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +** +** In this implementation idxNum is used to represent the +** query plan. idxStr is unused. +** +** The query plan is represented by bits in idxNum: +** +** (1) start = $value -- constraint exists +** (2) stop = $value -- constraint exists +** (4) step = $value -- constraint exists +** (8) output in descending order +*/ +static int seriesBestIndex( + sqlite3_vtab *pVTab, + sqlite3_index_info *pIdxInfo +){ + int i, j; /* Loop over constraints */ + int idxNum = 0; /* The query plan bitmask */ + int bStartSeen = 0; /* EQ constraint seen on the START column */ + int unusableMask = 0; /* Mask of unusable constraints */ + int nArg = 0; /* Number of arguments that seriesFilter() expects */ + int aIdx[3]; /* Constraints on start, stop, and step */ + const struct sqlite3_index_constraint *pConstraint; + + /* This implementation assumes that the start, stop, and step columns + ** are the last three columns in the virtual table. */ + assert( SERIES_COLUMN_STOP == SERIES_COLUMN_START+1 ); + assert( SERIES_COLUMN_STEP == SERIES_COLUMN_START+2 ); + + aIdx[0] = aIdx[1] = aIdx[2] = -1; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + int iCol; /* 0 for start, 1 for stop, 2 for step */ + int iMask; /* bitmask for those column */ + if( pConstraint->iColumn<SERIES_COLUMN_START ) continue; + iCol = pConstraint->iColumn - SERIES_COLUMN_START; + assert( iCol>=0 && iCol<=2 ); + iMask = 1 << iCol; + if( iCol==0 ) bStartSeen = 1; + if( pConstraint->usable==0 ){ + unusableMask |= iMask; + continue; + }else if( pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ ){ + idxNum |= iMask; + aIdx[iCol] = i; + } + } + for(i=0; i<3; i++){ + if( (j = aIdx[i])>=0 ){ + pIdxInfo->aConstraintUsage[j].argvIndex = ++nArg; + pIdxInfo->aConstraintUsage[j].omit = !SQLITE_SERIES_CONSTRAINT_VERIFY; + } + } + /* The current generate_column() implementation requires at least one + ** argument (the START value). Legacy versions assumed START=0 if the + ** first argument was omitted. Compile with -DZERO_ARGUMENT_GENERATE_SERIES + ** to obtain the legacy behavior */ +#ifndef ZERO_ARGUMENT_GENERATE_SERIES + if( !bStartSeen ){ + sqlite3_free(pVTab->zErrMsg); + pVTab->zErrMsg = sqlite3_mprintf( + "first argument to \"generate_series()\" missing or unusable"); + return SQLITE_ERROR; + } +#endif + if( (unusableMask & ~idxNum)!=0 ){ + /* The start, stop, and step columns are inputs. Therefore if there + ** are unusable constraints on any of start, stop, or step then + ** this plan is unusable */ + return SQLITE_CONSTRAINT; + } + if( (idxNum & 3)==3 ){ + /* Both start= and stop= boundaries are available. This is the + ** the preferred case */ + pIdxInfo->estimatedCost = (double)(2 - ((idxNum&4)!=0)); + pIdxInfo->estimatedRows = 1000; + if( pIdxInfo->nOrderBy>=1 && pIdxInfo->aOrderBy[0].iColumn==0 ){ + if( pIdxInfo->aOrderBy[0].desc ){ + idxNum |= 8; + }else{ + idxNum |= 16; + } + pIdxInfo->orderByConsumed = 1; + } + }else{ + /* If either boundary is missing, we have to generate a huge span + ** of numbers. Make this case very expensive so that the query + ** planner will work hard to avoid it. */ + pIdxInfo->estimatedRows = 2147483647; + } + pIdxInfo->idxNum = idxNum; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** generate_series virtual table. +*/ +static sqlite3_module seriesModule = { + 0, /* iVersion */ + 0, /* xCreate */ + seriesConnect, /* xConnect */ + seriesBestIndex, /* xBestIndex */ + seriesDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + seriesOpen, /* xOpen - open a cursor */ + seriesClose, /* xClose - close a cursor */ + seriesFilter, /* xFilter - configure scan constraints */ + seriesNext, /* xNext - advance a cursor */ + seriesEof, /* xEof - check for end of scan */ + seriesColumn, /* xColumn - read data */ + seriesRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_series_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + if( sqlite3_libversion_number()<3008012 && pzErrMsg!=0 ){ + *pzErrMsg = sqlite3_mprintf( + "generate_series() requires SQLite 3.8.12 or later"); + return SQLITE_ERROR; + } + rc = sqlite3_create_module(db, "generate_series", &seriesModule, 0); +#endif + return rc; +} diff --git a/ext/misc/sha1.c b/ext/misc/sha1.c new file mode 100644 index 0000000..9790a1d --- /dev/null +++ b/ext/misc/sha1.c @@ -0,0 +1,393 @@ +/* +** 2017-01-27 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements functions that compute SHA1 hashes. +** Two SQL functions are implemented: +** +** sha1(X) +** sha1_query(Y) +** +** The sha1(X) function computes the SHA1 hash of the input X, or NULL if +** X is NULL. +** +** The sha1_query(Y) function evalutes all queries in the SQL statements of Y +** and returns a hash of their results. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <stdarg.h> + +/****************************************************************************** +** The Hash Engine +*/ +/* Context for the SHA1 hash */ +typedef struct SHA1Context SHA1Context; +struct SHA1Context { + unsigned int state[5]; + unsigned int count[2]; + unsigned char buffer[64]; +}; + +#define SHA_ROT(x,l,r) ((x) << (l) | (x) >> (r)) +#define rol(x,k) SHA_ROT(x,k,32-(k)) +#define ror(x,k) SHA_ROT(x,32-(k),k) + +#define blk0le(i) (block[i] = (ror(block[i],8)&0xFF00FF00) \ + |(rol(block[i],8)&0x00FF00FF)) +#define blk0be(i) block[i] +#define blk(i) (block[i&15] = rol(block[(i+13)&15]^block[(i+8)&15] \ + ^block[(i+2)&15]^block[i&15],1)) + +/* + * (R0+R1), R2, R3, R4 are the different operations (rounds) used in SHA1 + * + * Rl0() for little-endian and Rb0() for big-endian. Endianness is + * determined at run-time. + */ +#define Rl0(v,w,x,y,z,i) \ + z+=((w&(x^y))^y)+blk0le(i)+0x5A827999+rol(v,5);w=ror(w,2); +#define Rb0(v,w,x,y,z,i) \ + z+=((w&(x^y))^y)+blk0be(i)+0x5A827999+rol(v,5);w=ror(w,2); +#define R1(v,w,x,y,z,i) \ + z+=((w&(x^y))^y)+blk(i)+0x5A827999+rol(v,5);w=ror(w,2); +#define R2(v,w,x,y,z,i) \ + z+=(w^x^y)+blk(i)+0x6ED9EBA1+rol(v,5);w=ror(w,2); +#define R3(v,w,x,y,z,i) \ + z+=(((w|x)&y)|(w&x))+blk(i)+0x8F1BBCDC+rol(v,5);w=ror(w,2); +#define R4(v,w,x,y,z,i) \ + z+=(w^x^y)+blk(i)+0xCA62C1D6+rol(v,5);w=ror(w,2); + +/* + * Hash a single 512-bit block. This is the core of the algorithm. + */ +static void SHA1Transform(unsigned int state[5], const unsigned char buffer[64]){ + unsigned int qq[5]; /* a, b, c, d, e; */ + static int one = 1; + unsigned int block[16]; + memcpy(block, buffer, 64); + memcpy(qq,state,5*sizeof(unsigned int)); + +#define a qq[0] +#define b qq[1] +#define c qq[2] +#define d qq[3] +#define e qq[4] + + /* Copy p->state[] to working vars */ + /* + a = state[0]; + b = state[1]; + c = state[2]; + d = state[3]; + e = state[4]; + */ + + /* 4 rounds of 20 operations each. Loop unrolled. */ + if( 1 == *(unsigned char*)&one ){ + Rl0(a,b,c,d,e, 0); Rl0(e,a,b,c,d, 1); Rl0(d,e,a,b,c, 2); Rl0(c,d,e,a,b, 3); + Rl0(b,c,d,e,a, 4); Rl0(a,b,c,d,e, 5); Rl0(e,a,b,c,d, 6); Rl0(d,e,a,b,c, 7); + Rl0(c,d,e,a,b, 8); Rl0(b,c,d,e,a, 9); Rl0(a,b,c,d,e,10); Rl0(e,a,b,c,d,11); + Rl0(d,e,a,b,c,12); Rl0(c,d,e,a,b,13); Rl0(b,c,d,e,a,14); Rl0(a,b,c,d,e,15); + }else{ + Rb0(a,b,c,d,e, 0); Rb0(e,a,b,c,d, 1); Rb0(d,e,a,b,c, 2); Rb0(c,d,e,a,b, 3); + Rb0(b,c,d,e,a, 4); Rb0(a,b,c,d,e, 5); Rb0(e,a,b,c,d, 6); Rb0(d,e,a,b,c, 7); + Rb0(c,d,e,a,b, 8); Rb0(b,c,d,e,a, 9); Rb0(a,b,c,d,e,10); Rb0(e,a,b,c,d,11); + Rb0(d,e,a,b,c,12); Rb0(c,d,e,a,b,13); Rb0(b,c,d,e,a,14); Rb0(a,b,c,d,e,15); + } + R1(e,a,b,c,d,16); R1(d,e,a,b,c,17); R1(c,d,e,a,b,18); R1(b,c,d,e,a,19); + R2(a,b,c,d,e,20); R2(e,a,b,c,d,21); R2(d,e,a,b,c,22); R2(c,d,e,a,b,23); + R2(b,c,d,e,a,24); R2(a,b,c,d,e,25); R2(e,a,b,c,d,26); R2(d,e,a,b,c,27); + R2(c,d,e,a,b,28); R2(b,c,d,e,a,29); R2(a,b,c,d,e,30); R2(e,a,b,c,d,31); + R2(d,e,a,b,c,32); R2(c,d,e,a,b,33); R2(b,c,d,e,a,34); R2(a,b,c,d,e,35); + R2(e,a,b,c,d,36); R2(d,e,a,b,c,37); R2(c,d,e,a,b,38); R2(b,c,d,e,a,39); + R3(a,b,c,d,e,40); R3(e,a,b,c,d,41); R3(d,e,a,b,c,42); R3(c,d,e,a,b,43); + R3(b,c,d,e,a,44); R3(a,b,c,d,e,45); R3(e,a,b,c,d,46); R3(d,e,a,b,c,47); + R3(c,d,e,a,b,48); R3(b,c,d,e,a,49); R3(a,b,c,d,e,50); R3(e,a,b,c,d,51); + R3(d,e,a,b,c,52); R3(c,d,e,a,b,53); R3(b,c,d,e,a,54); R3(a,b,c,d,e,55); + R3(e,a,b,c,d,56); R3(d,e,a,b,c,57); R3(c,d,e,a,b,58); R3(b,c,d,e,a,59); + R4(a,b,c,d,e,60); R4(e,a,b,c,d,61); R4(d,e,a,b,c,62); R4(c,d,e,a,b,63); + R4(b,c,d,e,a,64); R4(a,b,c,d,e,65); R4(e,a,b,c,d,66); R4(d,e,a,b,c,67); + R4(c,d,e,a,b,68); R4(b,c,d,e,a,69); R4(a,b,c,d,e,70); R4(e,a,b,c,d,71); + R4(d,e,a,b,c,72); R4(c,d,e,a,b,73); R4(b,c,d,e,a,74); R4(a,b,c,d,e,75); + R4(e,a,b,c,d,76); R4(d,e,a,b,c,77); R4(c,d,e,a,b,78); R4(b,c,d,e,a,79); + + /* Add the working vars back into context.state[] */ + state[0] += a; + state[1] += b; + state[2] += c; + state[3] += d; + state[4] += e; + +#undef a +#undef b +#undef c +#undef d +#undef e +} + + +/* Initialize a SHA1 context */ +static void hash_init(SHA1Context *p){ + /* SHA1 initialization constants */ + p->state[0] = 0x67452301; + p->state[1] = 0xEFCDAB89; + p->state[2] = 0x98BADCFE; + p->state[3] = 0x10325476; + p->state[4] = 0xC3D2E1F0; + p->count[0] = p->count[1] = 0; +} + +/* Add new content to the SHA1 hash */ +static void hash_step( + SHA1Context *p, /* Add content to this context */ + const unsigned char *data, /* Data to be added */ + unsigned int len /* Number of bytes in data */ +){ + unsigned int i, j; + + j = p->count[0]; + if( (p->count[0] += len << 3) < j ){ + p->count[1] += (len>>29)+1; + } + j = (j >> 3) & 63; + if( (j + len) > 63 ){ + (void)memcpy(&p->buffer[j], data, (i = 64-j)); + SHA1Transform(p->state, p->buffer); + for(; i + 63 < len; i += 64){ + SHA1Transform(p->state, &data[i]); + } + j = 0; + }else{ + i = 0; + } + (void)memcpy(&p->buffer[j], &data[i], len - i); +} + +/* Compute a string using sqlite3_vsnprintf() and hash it */ +static void hash_step_vformat( + SHA1Context *p, /* Add content to this context */ + const char *zFormat, + ... +){ + va_list ap; + int n; + char zBuf[50]; + va_start(ap, zFormat); + sqlite3_vsnprintf(sizeof(zBuf),zBuf,zFormat,ap); + va_end(ap); + n = (int)strlen(zBuf); + hash_step(p, (unsigned char*)zBuf, n); +} + + +/* Add padding and compute the message digest. Render the +** message digest as lower-case hexadecimal and put it into +** zOut[]. zOut[] must be at least 41 bytes long. */ +static void hash_finish( + SHA1Context *p, /* The SHA1 context to finish and render */ + char *zOut /* Store hexadecimal hash here */ +){ + unsigned int i; + unsigned char finalcount[8]; + unsigned char digest[20]; + static const char zEncode[] = "0123456789abcdef"; + + for (i = 0; i < 8; i++){ + finalcount[i] = (unsigned char)((p->count[(i >= 4 ? 0 : 1)] + >> ((3-(i & 3)) * 8) ) & 255); /* Endian independent */ + } + hash_step(p, (const unsigned char *)"\200", 1); + while ((p->count[0] & 504) != 448){ + hash_step(p, (const unsigned char *)"\0", 1); + } + hash_step(p, finalcount, 8); /* Should cause a SHA1Transform() */ + for (i = 0; i < 20; i++){ + digest[i] = (unsigned char)((p->state[i>>2] >> ((3-(i & 3)) * 8) ) & 255); + } + for(i=0; i<20; i++){ + zOut[i*2] = zEncode[(digest[i]>>4)&0xf]; + zOut[i*2+1] = zEncode[digest[i] & 0xf]; + } + zOut[i*2]= 0; +} +/* End of the hashing logic +*****************************************************************************/ + +/* +** Implementation of the sha1(X) function. +** +** Return a lower-case hexadecimal rendering of the SHA1 hash of the +** argument X. If X is a BLOB, it is hashed as is. For all other +** types of input, X is converted into a UTF-8 string and the string +** is hash without the trailing 0x00 terminator. The hash of a NULL +** value is NULL. +*/ +static void sha1Func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + SHA1Context cx; + int eType = sqlite3_value_type(argv[0]); + int nByte = sqlite3_value_bytes(argv[0]); + char zOut[44]; + + assert( argc==1 ); + if( eType==SQLITE_NULL ) return; + hash_init(&cx); + if( eType==SQLITE_BLOB ){ + hash_step(&cx, sqlite3_value_blob(argv[0]), nByte); + }else{ + hash_step(&cx, sqlite3_value_text(argv[0]), nByte); + } + hash_finish(&cx, zOut); + sqlite3_result_text(context, zOut, 40, SQLITE_TRANSIENT); +} + +/* +** Implementation of the sha1_query(SQL) function. +** +** This function compiles and runs the SQL statement(s) given in the +** argument. The results are hashed using SHA1 and that hash is returned. +** +** The original SQL text is included as part of the hash. +** +** The hash is not just a concatenation of the outputs. Each query +** is delimited and each row and value within the query is delimited, +** with all values being marked with their datatypes. +*/ +static void sha1QueryFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zSql = (const char*)sqlite3_value_text(argv[0]); + sqlite3_stmt *pStmt = 0; + int nCol; /* Number of columns in the result set */ + int i; /* Loop counter */ + int rc; + int n; + const char *z; + SHA1Context cx; + char zOut[44]; + + assert( argc==1 ); + if( zSql==0 ) return; + hash_init(&cx); + while( zSql[0] ){ + rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, &zSql); + if( rc ){ + char *zMsg = sqlite3_mprintf("error SQL statement [%s]: %s", + zSql, sqlite3_errmsg(db)); + sqlite3_finalize(pStmt); + sqlite3_result_error(context, zMsg, -1); + sqlite3_free(zMsg); + return; + } + if( !sqlite3_stmt_readonly(pStmt) ){ + char *zMsg = sqlite3_mprintf("non-query: [%s]", sqlite3_sql(pStmt)); + sqlite3_finalize(pStmt); + sqlite3_result_error(context, zMsg, -1); + sqlite3_free(zMsg); + return; + } + nCol = sqlite3_column_count(pStmt); + z = sqlite3_sql(pStmt); + n = (int)strlen(z); + hash_step_vformat(&cx,"S%d:",n); + hash_step(&cx,(unsigned char*)z,n); + + /* Compute a hash over the result of the query */ + while( SQLITE_ROW==sqlite3_step(pStmt) ){ + hash_step(&cx,(const unsigned char*)"R",1); + for(i=0; i<nCol; i++){ + switch( sqlite3_column_type(pStmt,i) ){ + case SQLITE_NULL: { + hash_step(&cx, (const unsigned char*)"N",1); + break; + } + case SQLITE_INTEGER: { + sqlite3_uint64 u; + int j; + unsigned char x[9]; + sqlite3_int64 v = sqlite3_column_int64(pStmt,i); + memcpy(&u, &v, 8); + for(j=8; j>=1; j--){ + x[j] = u & 0xff; + u >>= 8; + } + x[0] = 'I'; + hash_step(&cx, x, 9); + break; + } + case SQLITE_FLOAT: { + sqlite3_uint64 u; + int j; + unsigned char x[9]; + double r = sqlite3_column_double(pStmt,i); + memcpy(&u, &r, 8); + for(j=8; j>=1; j--){ + x[j] = u & 0xff; + u >>= 8; + } + x[0] = 'F'; + hash_step(&cx,x,9); + break; + } + case SQLITE_TEXT: { + int n2 = sqlite3_column_bytes(pStmt, i); + const unsigned char *z2 = sqlite3_column_text(pStmt, i); + hash_step_vformat(&cx,"T%d:",n2); + hash_step(&cx, z2, n2); + break; + } + case SQLITE_BLOB: { + int n2 = sqlite3_column_bytes(pStmt, i); + const unsigned char *z2 = sqlite3_column_blob(pStmt, i); + hash_step_vformat(&cx,"B%d:",n2); + hash_step(&cx, z2, n2); + break; + } + } + } + } + sqlite3_finalize(pStmt); + } + hash_finish(&cx, zOut); + sqlite3_result_text(context, zOut, 40, SQLITE_TRANSIENT); +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_sha_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "sha1", 1, + SQLITE_UTF8 | SQLITE_INNOCUOUS | SQLITE_DETERMINISTIC, + 0, sha1Func, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "sha1_query", 1, + SQLITE_UTF8|SQLITE_DIRECTONLY, 0, + sha1QueryFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/shathree.c b/ext/misc/shathree.c new file mode 100644 index 0000000..f80dd85 --- /dev/null +++ b/ext/misc/shathree.c @@ -0,0 +1,724 @@ +/* +** 2017-03-08 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements functions that compute SHA3 hashes. +** Two SQL functions are implemented: +** +** sha3(X,SIZE) +** sha3_query(Y,SIZE) +** +** The sha3(X) function computes the SHA3 hash of the input X, or NULL if +** X is NULL. +** +** The sha3_query(Y) function evaluates all queries in the SQL statements of Y +** and returns a hash of their results. +** +** The SIZE argument is optional. If omitted, the SHA3-256 hash algorithm +** is used. If SIZE is included it must be one of the integers 224, 256, +** 384, or 512, to determine SHA3 hash variant that is computed. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <stdarg.h> + +#ifndef SQLITE_AMALGAMATION +typedef sqlite3_uint64 u64; +#endif /* SQLITE_AMALGAMATION */ + +/****************************************************************************** +** The Hash Engine +*/ +/* +** Macros to determine whether the machine is big or little endian, +** and whether or not that determination is run-time or compile-time. +** +** For best performance, an attempt is made to guess at the byte-order +** using C-preprocessor macros. If that is unsuccessful, or if +** -DSHA3_BYTEORDER=0 is set, then byte-order is determined +** at run-time. +*/ +#ifndef SHA3_BYTEORDER +# if defined(i386) || defined(__i386__) || defined(_M_IX86) || \ + defined(__x86_64) || defined(__x86_64__) || defined(_M_X64) || \ + defined(_M_AMD64) || defined(_M_ARM) || defined(__x86) || \ + defined(__arm__) +# define SHA3_BYTEORDER 1234 +# elif defined(sparc) || defined(__ppc__) +# define SHA3_BYTEORDER 4321 +# else +# define SHA3_BYTEORDER 0 +# endif +#endif + + +/* +** State structure for a SHA3 hash in progress +*/ +typedef struct SHA3Context SHA3Context; +struct SHA3Context { + union { + u64 s[25]; /* Keccak state. 5x5 lines of 64 bits each */ + unsigned char x[1600]; /* ... or 1600 bytes */ + } u; + unsigned nRate; /* Bytes of input accepted per Keccak iteration */ + unsigned nLoaded; /* Input bytes loaded into u.x[] so far this cycle */ + unsigned ixMask; /* Insert next input into u.x[nLoaded^ixMask]. */ +}; + +/* +** A single step of the Keccak mixing function for a 1600-bit state +*/ +static void KeccakF1600Step(SHA3Context *p){ + int i; + u64 b0, b1, b2, b3, b4; + u64 c0, c1, c2, c3, c4; + u64 d0, d1, d2, d3, d4; + static const u64 RC[] = { + 0x0000000000000001ULL, 0x0000000000008082ULL, + 0x800000000000808aULL, 0x8000000080008000ULL, + 0x000000000000808bULL, 0x0000000080000001ULL, + 0x8000000080008081ULL, 0x8000000000008009ULL, + 0x000000000000008aULL, 0x0000000000000088ULL, + 0x0000000080008009ULL, 0x000000008000000aULL, + 0x000000008000808bULL, 0x800000000000008bULL, + 0x8000000000008089ULL, 0x8000000000008003ULL, + 0x8000000000008002ULL, 0x8000000000000080ULL, + 0x000000000000800aULL, 0x800000008000000aULL, + 0x8000000080008081ULL, 0x8000000000008080ULL, + 0x0000000080000001ULL, 0x8000000080008008ULL + }; +# define a00 (p->u.s[0]) +# define a01 (p->u.s[1]) +# define a02 (p->u.s[2]) +# define a03 (p->u.s[3]) +# define a04 (p->u.s[4]) +# define a10 (p->u.s[5]) +# define a11 (p->u.s[6]) +# define a12 (p->u.s[7]) +# define a13 (p->u.s[8]) +# define a14 (p->u.s[9]) +# define a20 (p->u.s[10]) +# define a21 (p->u.s[11]) +# define a22 (p->u.s[12]) +# define a23 (p->u.s[13]) +# define a24 (p->u.s[14]) +# define a30 (p->u.s[15]) +# define a31 (p->u.s[16]) +# define a32 (p->u.s[17]) +# define a33 (p->u.s[18]) +# define a34 (p->u.s[19]) +# define a40 (p->u.s[20]) +# define a41 (p->u.s[21]) +# define a42 (p->u.s[22]) +# define a43 (p->u.s[23]) +# define a44 (p->u.s[24]) +# define ROL64(a,x) ((a<<x)|(a>>(64-x))) + + for(i=0; i<24; i+=4){ + c0 = a00^a10^a20^a30^a40; + c1 = a01^a11^a21^a31^a41; + c2 = a02^a12^a22^a32^a42; + c3 = a03^a13^a23^a33^a43; + c4 = a04^a14^a24^a34^a44; + d0 = c4^ROL64(c1, 1); + d1 = c0^ROL64(c2, 1); + d2 = c1^ROL64(c3, 1); + d3 = c2^ROL64(c4, 1); + d4 = c3^ROL64(c0, 1); + + b0 = (a00^d0); + b1 = ROL64((a11^d1), 44); + b2 = ROL64((a22^d2), 43); + b3 = ROL64((a33^d3), 21); + b4 = ROL64((a44^d4), 14); + a00 = b0 ^((~b1)& b2 ); + a00 ^= RC[i]; + a11 = b1 ^((~b2)& b3 ); + a22 = b2 ^((~b3)& b4 ); + a33 = b3 ^((~b4)& b0 ); + a44 = b4 ^((~b0)& b1 ); + + b2 = ROL64((a20^d0), 3); + b3 = ROL64((a31^d1), 45); + b4 = ROL64((a42^d2), 61); + b0 = ROL64((a03^d3), 28); + b1 = ROL64((a14^d4), 20); + a20 = b0 ^((~b1)& b2 ); + a31 = b1 ^((~b2)& b3 ); + a42 = b2 ^((~b3)& b4 ); + a03 = b3 ^((~b4)& b0 ); + a14 = b4 ^((~b0)& b1 ); + + b4 = ROL64((a40^d0), 18); + b0 = ROL64((a01^d1), 1); + b1 = ROL64((a12^d2), 6); + b2 = ROL64((a23^d3), 25); + b3 = ROL64((a34^d4), 8); + a40 = b0 ^((~b1)& b2 ); + a01 = b1 ^((~b2)& b3 ); + a12 = b2 ^((~b3)& b4 ); + a23 = b3 ^((~b4)& b0 ); + a34 = b4 ^((~b0)& b1 ); + + b1 = ROL64((a10^d0), 36); + b2 = ROL64((a21^d1), 10); + b3 = ROL64((a32^d2), 15); + b4 = ROL64((a43^d3), 56); + b0 = ROL64((a04^d4), 27); + a10 = b0 ^((~b1)& b2 ); + a21 = b1 ^((~b2)& b3 ); + a32 = b2 ^((~b3)& b4 ); + a43 = b3 ^((~b4)& b0 ); + a04 = b4 ^((~b0)& b1 ); + + b3 = ROL64((a30^d0), 41); + b4 = ROL64((a41^d1), 2); + b0 = ROL64((a02^d2), 62); + b1 = ROL64((a13^d3), 55); + b2 = ROL64((a24^d4), 39); + a30 = b0 ^((~b1)& b2 ); + a41 = b1 ^((~b2)& b3 ); + a02 = b2 ^((~b3)& b4 ); + a13 = b3 ^((~b4)& b0 ); + a24 = b4 ^((~b0)& b1 ); + + c0 = a00^a20^a40^a10^a30; + c1 = a11^a31^a01^a21^a41; + c2 = a22^a42^a12^a32^a02; + c3 = a33^a03^a23^a43^a13; + c4 = a44^a14^a34^a04^a24; + d0 = c4^ROL64(c1, 1); + d1 = c0^ROL64(c2, 1); + d2 = c1^ROL64(c3, 1); + d3 = c2^ROL64(c4, 1); + d4 = c3^ROL64(c0, 1); + + b0 = (a00^d0); + b1 = ROL64((a31^d1), 44); + b2 = ROL64((a12^d2), 43); + b3 = ROL64((a43^d3), 21); + b4 = ROL64((a24^d4), 14); + a00 = b0 ^((~b1)& b2 ); + a00 ^= RC[i+1]; + a31 = b1 ^((~b2)& b3 ); + a12 = b2 ^((~b3)& b4 ); + a43 = b3 ^((~b4)& b0 ); + a24 = b4 ^((~b0)& b1 ); + + b2 = ROL64((a40^d0), 3); + b3 = ROL64((a21^d1), 45); + b4 = ROL64((a02^d2), 61); + b0 = ROL64((a33^d3), 28); + b1 = ROL64((a14^d4), 20); + a40 = b0 ^((~b1)& b2 ); + a21 = b1 ^((~b2)& b3 ); + a02 = b2 ^((~b3)& b4 ); + a33 = b3 ^((~b4)& b0 ); + a14 = b4 ^((~b0)& b1 ); + + b4 = ROL64((a30^d0), 18); + b0 = ROL64((a11^d1), 1); + b1 = ROL64((a42^d2), 6); + b2 = ROL64((a23^d3), 25); + b3 = ROL64((a04^d4), 8); + a30 = b0 ^((~b1)& b2 ); + a11 = b1 ^((~b2)& b3 ); + a42 = b2 ^((~b3)& b4 ); + a23 = b3 ^((~b4)& b0 ); + a04 = b4 ^((~b0)& b1 ); + + b1 = ROL64((a20^d0), 36); + b2 = ROL64((a01^d1), 10); + b3 = ROL64((a32^d2), 15); + b4 = ROL64((a13^d3), 56); + b0 = ROL64((a44^d4), 27); + a20 = b0 ^((~b1)& b2 ); + a01 = b1 ^((~b2)& b3 ); + a32 = b2 ^((~b3)& b4 ); + a13 = b3 ^((~b4)& b0 ); + a44 = b4 ^((~b0)& b1 ); + + b3 = ROL64((a10^d0), 41); + b4 = ROL64((a41^d1), 2); + b0 = ROL64((a22^d2), 62); + b1 = ROL64((a03^d3), 55); + b2 = ROL64((a34^d4), 39); + a10 = b0 ^((~b1)& b2 ); + a41 = b1 ^((~b2)& b3 ); + a22 = b2 ^((~b3)& b4 ); + a03 = b3 ^((~b4)& b0 ); + a34 = b4 ^((~b0)& b1 ); + + c0 = a00^a40^a30^a20^a10; + c1 = a31^a21^a11^a01^a41; + c2 = a12^a02^a42^a32^a22; + c3 = a43^a33^a23^a13^a03; + c4 = a24^a14^a04^a44^a34; + d0 = c4^ROL64(c1, 1); + d1 = c0^ROL64(c2, 1); + d2 = c1^ROL64(c3, 1); + d3 = c2^ROL64(c4, 1); + d4 = c3^ROL64(c0, 1); + + b0 = (a00^d0); + b1 = ROL64((a21^d1), 44); + b2 = ROL64((a42^d2), 43); + b3 = ROL64((a13^d3), 21); + b4 = ROL64((a34^d4), 14); + a00 = b0 ^((~b1)& b2 ); + a00 ^= RC[i+2]; + a21 = b1 ^((~b2)& b3 ); + a42 = b2 ^((~b3)& b4 ); + a13 = b3 ^((~b4)& b0 ); + a34 = b4 ^((~b0)& b1 ); + + b2 = ROL64((a30^d0), 3); + b3 = ROL64((a01^d1), 45); + b4 = ROL64((a22^d2), 61); + b0 = ROL64((a43^d3), 28); + b1 = ROL64((a14^d4), 20); + a30 = b0 ^((~b1)& b2 ); + a01 = b1 ^((~b2)& b3 ); + a22 = b2 ^((~b3)& b4 ); + a43 = b3 ^((~b4)& b0 ); + a14 = b4 ^((~b0)& b1 ); + + b4 = ROL64((a10^d0), 18); + b0 = ROL64((a31^d1), 1); + b1 = ROL64((a02^d2), 6); + b2 = ROL64((a23^d3), 25); + b3 = ROL64((a44^d4), 8); + a10 = b0 ^((~b1)& b2 ); + a31 = b1 ^((~b2)& b3 ); + a02 = b2 ^((~b3)& b4 ); + a23 = b3 ^((~b4)& b0 ); + a44 = b4 ^((~b0)& b1 ); + + b1 = ROL64((a40^d0), 36); + b2 = ROL64((a11^d1), 10); + b3 = ROL64((a32^d2), 15); + b4 = ROL64((a03^d3), 56); + b0 = ROL64((a24^d4), 27); + a40 = b0 ^((~b1)& b2 ); + a11 = b1 ^((~b2)& b3 ); + a32 = b2 ^((~b3)& b4 ); + a03 = b3 ^((~b4)& b0 ); + a24 = b4 ^((~b0)& b1 ); + + b3 = ROL64((a20^d0), 41); + b4 = ROL64((a41^d1), 2); + b0 = ROL64((a12^d2), 62); + b1 = ROL64((a33^d3), 55); + b2 = ROL64((a04^d4), 39); + a20 = b0 ^((~b1)& b2 ); + a41 = b1 ^((~b2)& b3 ); + a12 = b2 ^((~b3)& b4 ); + a33 = b3 ^((~b4)& b0 ); + a04 = b4 ^((~b0)& b1 ); + + c0 = a00^a30^a10^a40^a20; + c1 = a21^a01^a31^a11^a41; + c2 = a42^a22^a02^a32^a12; + c3 = a13^a43^a23^a03^a33; + c4 = a34^a14^a44^a24^a04; + d0 = c4^ROL64(c1, 1); + d1 = c0^ROL64(c2, 1); + d2 = c1^ROL64(c3, 1); + d3 = c2^ROL64(c4, 1); + d4 = c3^ROL64(c0, 1); + + b0 = (a00^d0); + b1 = ROL64((a01^d1), 44); + b2 = ROL64((a02^d2), 43); + b3 = ROL64((a03^d3), 21); + b4 = ROL64((a04^d4), 14); + a00 = b0 ^((~b1)& b2 ); + a00 ^= RC[i+3]; + a01 = b1 ^((~b2)& b3 ); + a02 = b2 ^((~b3)& b4 ); + a03 = b3 ^((~b4)& b0 ); + a04 = b4 ^((~b0)& b1 ); + + b2 = ROL64((a10^d0), 3); + b3 = ROL64((a11^d1), 45); + b4 = ROL64((a12^d2), 61); + b0 = ROL64((a13^d3), 28); + b1 = ROL64((a14^d4), 20); + a10 = b0 ^((~b1)& b2 ); + a11 = b1 ^((~b2)& b3 ); + a12 = b2 ^((~b3)& b4 ); + a13 = b3 ^((~b4)& b0 ); + a14 = b4 ^((~b0)& b1 ); + + b4 = ROL64((a20^d0), 18); + b0 = ROL64((a21^d1), 1); + b1 = ROL64((a22^d2), 6); + b2 = ROL64((a23^d3), 25); + b3 = ROL64((a24^d4), 8); + a20 = b0 ^((~b1)& b2 ); + a21 = b1 ^((~b2)& b3 ); + a22 = b2 ^((~b3)& b4 ); + a23 = b3 ^((~b4)& b0 ); + a24 = b4 ^((~b0)& b1 ); + + b1 = ROL64((a30^d0), 36); + b2 = ROL64((a31^d1), 10); + b3 = ROL64((a32^d2), 15); + b4 = ROL64((a33^d3), 56); + b0 = ROL64((a34^d4), 27); + a30 = b0 ^((~b1)& b2 ); + a31 = b1 ^((~b2)& b3 ); + a32 = b2 ^((~b3)& b4 ); + a33 = b3 ^((~b4)& b0 ); + a34 = b4 ^((~b0)& b1 ); + + b3 = ROL64((a40^d0), 41); + b4 = ROL64((a41^d1), 2); + b0 = ROL64((a42^d2), 62); + b1 = ROL64((a43^d3), 55); + b2 = ROL64((a44^d4), 39); + a40 = b0 ^((~b1)& b2 ); + a41 = b1 ^((~b2)& b3 ); + a42 = b2 ^((~b3)& b4 ); + a43 = b3 ^((~b4)& b0 ); + a44 = b4 ^((~b0)& b1 ); + } +} + +/* +** Initialize a new hash. iSize determines the size of the hash +** in bits and should be one of 224, 256, 384, or 512. Or iSize +** can be zero to use the default hash size of 256 bits. +*/ +static void SHA3Init(SHA3Context *p, int iSize){ + memset(p, 0, sizeof(*p)); + if( iSize>=128 && iSize<=512 ){ + p->nRate = (1600 - ((iSize + 31)&~31)*2)/8; + }else{ + p->nRate = (1600 - 2*256)/8; + } +#if SHA3_BYTEORDER==1234 + /* Known to be little-endian at compile-time. No-op */ +#elif SHA3_BYTEORDER==4321 + p->ixMask = 7; /* Big-endian */ +#else + { + static unsigned int one = 1; + if( 1==*(unsigned char*)&one ){ + /* Little endian. No byte swapping. */ + p->ixMask = 0; + }else{ + /* Big endian. Byte swap. */ + p->ixMask = 7; + } + } +#endif +} + +/* +** Make consecutive calls to the SHA3Update function to add new content +** to the hash +*/ +static void SHA3Update( + SHA3Context *p, + const unsigned char *aData, + unsigned int nData +){ + unsigned int i = 0; + if( aData==0 ) return; +#if SHA3_BYTEORDER==1234 + if( (p->nLoaded % 8)==0 && ((aData - (const unsigned char*)0)&7)==0 ){ + for(; i+7<nData; i+=8){ + p->u.s[p->nLoaded/8] ^= *(u64*)&aData[i]; + p->nLoaded += 8; + if( p->nLoaded>=p->nRate ){ + KeccakF1600Step(p); + p->nLoaded = 0; + } + } + } +#endif + for(; i<nData; i++){ +#if SHA3_BYTEORDER==1234 + p->u.x[p->nLoaded] ^= aData[i]; +#elif SHA3_BYTEORDER==4321 + p->u.x[p->nLoaded^0x07] ^= aData[i]; +#else + p->u.x[p->nLoaded^p->ixMask] ^= aData[i]; +#endif + p->nLoaded++; + if( p->nLoaded==p->nRate ){ + KeccakF1600Step(p); + p->nLoaded = 0; + } + } +} + +/* +** After all content has been added, invoke SHA3Final() to compute +** the final hash. The function returns a pointer to the binary +** hash value. +*/ +static unsigned char *SHA3Final(SHA3Context *p){ + unsigned int i; + if( p->nLoaded==p->nRate-1 ){ + const unsigned char c1 = 0x86; + SHA3Update(p, &c1, 1); + }else{ + const unsigned char c2 = 0x06; + const unsigned char c3 = 0x80; + SHA3Update(p, &c2, 1); + p->nLoaded = p->nRate - 1; + SHA3Update(p, &c3, 1); + } + for(i=0; i<p->nRate; i++){ + p->u.x[i+p->nRate] = p->u.x[i^p->ixMask]; + } + return &p->u.x[p->nRate]; +} +/* End of the hashing logic +*****************************************************************************/ + +/* +** Implementation of the sha3(X,SIZE) function. +** +** Return a BLOB which is the SIZE-bit SHA3 hash of X. The default +** size is 256. If X is a BLOB, it is hashed as is. +** For all other non-NULL types of input, X is converted into a UTF-8 string +** and the string is hashed without the trailing 0x00 terminator. The hash +** of a NULL value is NULL. +*/ +static void sha3Func( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + SHA3Context cx; + int eType = sqlite3_value_type(argv[0]); + int nByte = sqlite3_value_bytes(argv[0]); + int iSize; + if( argc==1 ){ + iSize = 256; + }else{ + iSize = sqlite3_value_int(argv[1]); + if( iSize!=224 && iSize!=256 && iSize!=384 && iSize!=512 ){ + sqlite3_result_error(context, "SHA3 size should be one of: 224 256 " + "384 512", -1); + return; + } + } + if( eType==SQLITE_NULL ) return; + SHA3Init(&cx, iSize); + if( eType==SQLITE_BLOB ){ + SHA3Update(&cx, sqlite3_value_blob(argv[0]), nByte); + }else{ + SHA3Update(&cx, sqlite3_value_text(argv[0]), nByte); + } + sqlite3_result_blob(context, SHA3Final(&cx), iSize/8, SQLITE_TRANSIENT); +} + +/* Compute a string using sqlite3_vsnprintf() with a maximum length +** of 50 bytes and add it to the hash. +*/ +static void hash_step_vformat( + SHA3Context *p, /* Add content to this context */ + const char *zFormat, + ... +){ + va_list ap; + int n; + char zBuf[50]; + va_start(ap, zFormat); + sqlite3_vsnprintf(sizeof(zBuf),zBuf,zFormat,ap); + va_end(ap); + n = (int)strlen(zBuf); + SHA3Update(p, (unsigned char*)zBuf, n); +} + +/* +** Implementation of the sha3_query(SQL,SIZE) function. +** +** This function compiles and runs the SQL statement(s) given in the +** argument. The results are hashed using a SIZE-bit SHA3. The default +** size is 256. +** +** The format of the byte stream that is hashed is summarized as follows: +** +** S<n>:<sql> +** R +** N +** I<int> +** F<ieee-float> +** B<size>:<bytes> +** T<size>:<text> +** +** <sql> is the original SQL text for each statement run and <n> is +** the size of that text. The SQL text is UTF-8. A single R character +** occurs before the start of each row. N means a NULL value. +** I mean an 8-byte little-endian integer <int>. F is a floating point +** number with an 8-byte little-endian IEEE floating point value <ieee-float>. +** B means blobs of <size> bytes. T means text rendered as <size> +** bytes of UTF-8. The <n> and <size> values are expressed as an ASCII +** text integers. +** +** For each SQL statement in the X input, there is one S segment. Each +** S segment is followed by zero or more R segments, one for each row in the +** result set. After each R, there are one or more N, I, F, B, or T segments, +** one for each column in the result set. Segments are concatentated directly +** with no delimiters of any kind. +*/ +static void sha3QueryFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zSql = (const char*)sqlite3_value_text(argv[0]); + sqlite3_stmt *pStmt = 0; + int nCol; /* Number of columns in the result set */ + int i; /* Loop counter */ + int rc; + int n; + const char *z; + SHA3Context cx; + int iSize; + + if( argc==1 ){ + iSize = 256; + }else{ + iSize = sqlite3_value_int(argv[1]); + if( iSize!=224 && iSize!=256 && iSize!=384 && iSize!=512 ){ + sqlite3_result_error(context, "SHA3 size should be one of: 224 256 " + "384 512", -1); + return; + } + } + if( zSql==0 ) return; + SHA3Init(&cx, iSize); + while( zSql[0] ){ + rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, &zSql); + if( rc ){ + char *zMsg = sqlite3_mprintf("error SQL statement [%s]: %s", + zSql, sqlite3_errmsg(db)); + sqlite3_finalize(pStmt); + sqlite3_result_error(context, zMsg, -1); + sqlite3_free(zMsg); + return; + } + if( !sqlite3_stmt_readonly(pStmt) ){ + char *zMsg = sqlite3_mprintf("non-query: [%s]", sqlite3_sql(pStmt)); + sqlite3_finalize(pStmt); + sqlite3_result_error(context, zMsg, -1); + sqlite3_free(zMsg); + return; + } + nCol = sqlite3_column_count(pStmt); + z = sqlite3_sql(pStmt); + if( z ){ + n = (int)strlen(z); + hash_step_vformat(&cx,"S%d:",n); + SHA3Update(&cx,(unsigned char*)z,n); + } + + /* Compute a hash over the result of the query */ + while( SQLITE_ROW==sqlite3_step(pStmt) ){ + SHA3Update(&cx,(const unsigned char*)"R",1); + for(i=0; i<nCol; i++){ + switch( sqlite3_column_type(pStmt,i) ){ + case SQLITE_NULL: { + SHA3Update(&cx, (const unsigned char*)"N",1); + break; + } + case SQLITE_INTEGER: { + sqlite3_uint64 u; + int j; + unsigned char x[9]; + sqlite3_int64 v = sqlite3_column_int64(pStmt,i); + memcpy(&u, &v, 8); + for(j=8; j>=1; j--){ + x[j] = u & 0xff; + u >>= 8; + } + x[0] = 'I'; + SHA3Update(&cx, x, 9); + break; + } + case SQLITE_FLOAT: { + sqlite3_uint64 u; + int j; + unsigned char x[9]; + double r = sqlite3_column_double(pStmt,i); + memcpy(&u, &r, 8); + for(j=8; j>=1; j--){ + x[j] = u & 0xff; + u >>= 8; + } + x[0] = 'F'; + SHA3Update(&cx,x,9); + break; + } + case SQLITE_TEXT: { + int n2 = sqlite3_column_bytes(pStmt, i); + const unsigned char *z2 = sqlite3_column_text(pStmt, i); + hash_step_vformat(&cx,"T%d:",n2); + SHA3Update(&cx, z2, n2); + break; + } + case SQLITE_BLOB: { + int n2 = sqlite3_column_bytes(pStmt, i); + const unsigned char *z2 = sqlite3_column_blob(pStmt, i); + hash_step_vformat(&cx,"B%d:",n2); + SHA3Update(&cx, z2, n2); + break; + } + } + } + } + sqlite3_finalize(pStmt); + } + sqlite3_result_blob(context, SHA3Final(&cx), iSize/8, SQLITE_TRANSIENT); +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_shathree_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "sha3", 1, + SQLITE_UTF8 | SQLITE_INNOCUOUS | SQLITE_DETERMINISTIC, + 0, sha3Func, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "sha3", 2, + SQLITE_UTF8 | SQLITE_INNOCUOUS | SQLITE_DETERMINISTIC, + 0, sha3Func, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "sha3_query", 1, + SQLITE_UTF8 | SQLITE_DIRECTONLY, + 0, sha3QueryFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "sha3_query", 2, + SQLITE_UTF8 | SQLITE_DIRECTONLY, + 0, sha3QueryFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/showauth.c b/ext/misc/showauth.c new file mode 100644 index 0000000..87a9a68 --- /dev/null +++ b/ext/misc/showauth.c @@ -0,0 +1,103 @@ +/* +** 2014-09-21 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension adds a debug "authorizer" callback to the database +** connection. The callback merely writes the authorization request to +** standard output and returns SQLITE_OK. +** +** This extension can be used (for example) in the command-line shell to +** trace the operation of the authorizer. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdio.h> + +/* +** Display the authorization request +*/ +static int authCallback( + void *pClientData, + int op, + const char *z1, + const char *z2, + const char *z3, + const char *z4 +){ + const char *zOp; + char zOpSpace[50]; + switch( op ){ + case SQLITE_CREATE_INDEX: zOp = "CREATE_INDEX"; break; + case SQLITE_CREATE_TABLE: zOp = "CREATE_TABLE"; break; + case SQLITE_CREATE_TEMP_INDEX: zOp = "CREATE_TEMP_INDEX"; break; + case SQLITE_CREATE_TEMP_TABLE: zOp = "CREATE_TEMP_TABLE"; break; + case SQLITE_CREATE_TEMP_TRIGGER: zOp = "CREATE_TEMP_TRIGGER"; break; + case SQLITE_CREATE_TEMP_VIEW: zOp = "CREATE_TEMP_VIEW"; break; + case SQLITE_CREATE_TRIGGER: zOp = "CREATE_TRIGGER"; break; + case SQLITE_CREATE_VIEW: zOp = "CREATE_VIEW"; break; + case SQLITE_DELETE: zOp = "DELETE"; break; + case SQLITE_DROP_INDEX: zOp = "DROP_INDEX"; break; + case SQLITE_DROP_TABLE: zOp = "DROP_TABLE"; break; + case SQLITE_DROP_TEMP_INDEX: zOp = "DROP_TEMP_INDEX"; break; + case SQLITE_DROP_TEMP_TABLE: zOp = "DROP_TEMP_TABLE"; break; + case SQLITE_DROP_TEMP_TRIGGER: zOp = "DROP_TEMP_TRIGGER"; break; + case SQLITE_DROP_TEMP_VIEW: zOp = "DROP_TEMP_VIEW"; break; + case SQLITE_DROP_TRIGGER: zOp = "DROP_TRIGGER"; break; + case SQLITE_DROP_VIEW: zOp = "DROP_VIEW"; break; + case SQLITE_INSERT: zOp = "INSERT"; break; + case SQLITE_PRAGMA: zOp = "PRAGMA"; break; + case SQLITE_READ: zOp = "READ"; break; + case SQLITE_SELECT: zOp = "SELECT"; break; + case SQLITE_TRANSACTION: zOp = "TRANSACTION"; break; + case SQLITE_UPDATE: zOp = "UPDATE"; break; + case SQLITE_ATTACH: zOp = "ATTACH"; break; + case SQLITE_DETACH: zOp = "DETACH"; break; + case SQLITE_ALTER_TABLE: zOp = "ALTER_TABLE"; break; + case SQLITE_REINDEX: zOp = "REINDEX"; break; + case SQLITE_ANALYZE: zOp = "ANALYZE"; break; + case SQLITE_CREATE_VTABLE: zOp = "CREATE_VTABLE"; break; + case SQLITE_DROP_VTABLE: zOp = "DROP_VTABLE"; break; + case SQLITE_FUNCTION: zOp = "FUNCTION"; break; + case SQLITE_SAVEPOINT: zOp = "SAVEPOINT"; break; + case SQLITE_COPY: zOp = "COPY"; break; + case SQLITE_RECURSIVE: zOp = "RECURSIVE"; break; + + + default: { + sqlite3_snprintf(sizeof(zOpSpace), zOpSpace, "%d", op); + zOp = zOpSpace; + break; + } + } + if( z1==0 ) z1 = "NULL"; + if( z2==0 ) z2 = "NULL"; + if( z3==0 ) z3 = "NULL"; + if( z4==0 ) z4 = "NULL"; + printf("AUTH: %s,%s,%s,%s,%s\n", zOp, z1, z2, z3, z4); + return SQLITE_OK; +} + + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_showauth_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_set_authorizer(db, authCallback, 0); + return rc; +} diff --git a/ext/misc/spellfix.c b/ext/misc/spellfix.c new file mode 100644 index 0000000..3b78d9f --- /dev/null +++ b/ext/misc/spellfix.c @@ -0,0 +1,3071 @@ +/* +** 2012 April 10 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This module implements the spellfix1 VIRTUAL TABLE that can be used +** to search a large vocabulary for close matches. See separate +** documentation (http://www.sqlite.org/spellfix1.html) for details. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 + +#ifndef SQLITE_AMALGAMATION +# if !defined(NDEBUG) && !defined(SQLITE_DEBUG) +# define NDEBUG 1 +# endif +# if defined(NDEBUG) && defined(SQLITE_DEBUG) +# undef NDEBUG +# endif +# include <string.h> +# include <stdio.h> +# include <stdlib.h> +# include <assert.h> +# define ALWAYS(X) 1 +# define NEVER(X) 0 + typedef unsigned char u8; + typedef unsigned short u16; +#endif +#include <ctype.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Character classes for ASCII characters: +** +** 0 '' Silent letters: H W +** 1 'A' Any vowel: A E I O U (Y) +** 2 'B' A bilabeal stop or fricative: B F P V W +** 3 'C' Other fricatives or back stops: C G J K Q S X Z +** 4 'D' Alveolar stops: D T +** 5 'H' Letter H at the beginning of a word +** 6 'L' Glide: L +** 7 'R' Semivowel: R +** 8 'M' Nasals: M N +** 9 'Y' Letter Y at the beginning of a word. +** 10 '9' Digits: 0 1 2 3 4 5 6 7 8 9 +** 11 ' ' White space +** 12 '?' Other. +*/ +#define CCLASS_SILENT 0 +#define CCLASS_VOWEL 1 +#define CCLASS_B 2 +#define CCLASS_C 3 +#define CCLASS_D 4 +#define CCLASS_H 5 +#define CCLASS_L 6 +#define CCLASS_R 7 +#define CCLASS_M 8 +#define CCLASS_Y 9 +#define CCLASS_DIGIT 10 +#define CCLASS_SPACE 11 +#define CCLASS_OTHER 12 + +/* +** The following table gives the character class for non-initial ASCII +** characters. +*/ +static const unsigned char midClass[] = { + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_SPACE, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_SPACE, /* */ CCLASS_SPACE, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_SPACE, + /* ! */ CCLASS_OTHER, /* " */ CCLASS_OTHER, /* # */ CCLASS_OTHER, + /* $ */ CCLASS_OTHER, /* % */ CCLASS_OTHER, /* & */ CCLASS_OTHER, + /* ' */ CCLASS_SILENT, /* ( */ CCLASS_OTHER, /* ) */ CCLASS_OTHER, + /* * */ CCLASS_OTHER, /* + */ CCLASS_OTHER, /* , */ CCLASS_OTHER, + /* - */ CCLASS_OTHER, /* . */ CCLASS_OTHER, /* / */ CCLASS_OTHER, + /* 0 */ CCLASS_DIGIT, /* 1 */ CCLASS_DIGIT, /* 2 */ CCLASS_DIGIT, + /* 3 */ CCLASS_DIGIT, /* 4 */ CCLASS_DIGIT, /* 5 */ CCLASS_DIGIT, + /* 6 */ CCLASS_DIGIT, /* 7 */ CCLASS_DIGIT, /* 8 */ CCLASS_DIGIT, + /* 9 */ CCLASS_DIGIT, /* : */ CCLASS_OTHER, /* ; */ CCLASS_OTHER, + /* < */ CCLASS_OTHER, /* = */ CCLASS_OTHER, /* > */ CCLASS_OTHER, + /* ? */ CCLASS_OTHER, /* @ */ CCLASS_OTHER, /* A */ CCLASS_VOWEL, + /* B */ CCLASS_B, /* C */ CCLASS_C, /* D */ CCLASS_D, + /* E */ CCLASS_VOWEL, /* F */ CCLASS_B, /* G */ CCLASS_C, + /* H */ CCLASS_SILENT, /* I */ CCLASS_VOWEL, /* J */ CCLASS_C, + /* K */ CCLASS_C, /* L */ CCLASS_L, /* M */ CCLASS_M, + /* N */ CCLASS_M, /* O */ CCLASS_VOWEL, /* P */ CCLASS_B, + /* Q */ CCLASS_C, /* R */ CCLASS_R, /* S */ CCLASS_C, + /* T */ CCLASS_D, /* U */ CCLASS_VOWEL, /* V */ CCLASS_B, + /* W */ CCLASS_B, /* X */ CCLASS_C, /* Y */ CCLASS_VOWEL, + /* Z */ CCLASS_C, /* [ */ CCLASS_OTHER, /* \ */ CCLASS_OTHER, + /* ] */ CCLASS_OTHER, /* ^ */ CCLASS_OTHER, /* _ */ CCLASS_OTHER, + /* ` */ CCLASS_OTHER, /* a */ CCLASS_VOWEL, /* b */ CCLASS_B, + /* c */ CCLASS_C, /* d */ CCLASS_D, /* e */ CCLASS_VOWEL, + /* f */ CCLASS_B, /* g */ CCLASS_C, /* h */ CCLASS_SILENT, + /* i */ CCLASS_VOWEL, /* j */ CCLASS_C, /* k */ CCLASS_C, + /* l */ CCLASS_L, /* m */ CCLASS_M, /* n */ CCLASS_M, + /* o */ CCLASS_VOWEL, /* p */ CCLASS_B, /* q */ CCLASS_C, + /* r */ CCLASS_R, /* s */ CCLASS_C, /* t */ CCLASS_D, + /* u */ CCLASS_VOWEL, /* v */ CCLASS_B, /* w */ CCLASS_B, + /* x */ CCLASS_C, /* y */ CCLASS_VOWEL, /* z */ CCLASS_C, + /* { */ CCLASS_OTHER, /* | */ CCLASS_OTHER, /* } */ CCLASS_OTHER, + /* ~ */ CCLASS_OTHER, /* */ CCLASS_OTHER, +}; +/* +** This tables gives the character class for ASCII characters that form the +** initial character of a word. The only difference from midClass is with +** the letters H, W, and Y. +*/ +static const unsigned char initClass[] = { + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_SPACE, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_SPACE, /* */ CCLASS_SPACE, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, + /* */ CCLASS_OTHER, /* */ CCLASS_OTHER, /* */ CCLASS_SPACE, + /* ! */ CCLASS_OTHER, /* " */ CCLASS_OTHER, /* # */ CCLASS_OTHER, + /* $ */ CCLASS_OTHER, /* % */ CCLASS_OTHER, /* & */ CCLASS_OTHER, + /* ' */ CCLASS_OTHER, /* ( */ CCLASS_OTHER, /* ) */ CCLASS_OTHER, + /* * */ CCLASS_OTHER, /* + */ CCLASS_OTHER, /* , */ CCLASS_OTHER, + /* - */ CCLASS_OTHER, /* . */ CCLASS_OTHER, /* / */ CCLASS_OTHER, + /* 0 */ CCLASS_DIGIT, /* 1 */ CCLASS_DIGIT, /* 2 */ CCLASS_DIGIT, + /* 3 */ CCLASS_DIGIT, /* 4 */ CCLASS_DIGIT, /* 5 */ CCLASS_DIGIT, + /* 6 */ CCLASS_DIGIT, /* 7 */ CCLASS_DIGIT, /* 8 */ CCLASS_DIGIT, + /* 9 */ CCLASS_DIGIT, /* : */ CCLASS_OTHER, /* ; */ CCLASS_OTHER, + /* < */ CCLASS_OTHER, /* = */ CCLASS_OTHER, /* > */ CCLASS_OTHER, + /* ? */ CCLASS_OTHER, /* @ */ CCLASS_OTHER, /* A */ CCLASS_VOWEL, + /* B */ CCLASS_B, /* C */ CCLASS_C, /* D */ CCLASS_D, + /* E */ CCLASS_VOWEL, /* F */ CCLASS_B, /* G */ CCLASS_C, + /* H */ CCLASS_SILENT, /* I */ CCLASS_VOWEL, /* J */ CCLASS_C, + /* K */ CCLASS_C, /* L */ CCLASS_L, /* M */ CCLASS_M, + /* N */ CCLASS_M, /* O */ CCLASS_VOWEL, /* P */ CCLASS_B, + /* Q */ CCLASS_C, /* R */ CCLASS_R, /* S */ CCLASS_C, + /* T */ CCLASS_D, /* U */ CCLASS_VOWEL, /* V */ CCLASS_B, + /* W */ CCLASS_B, /* X */ CCLASS_C, /* Y */ CCLASS_Y, + /* Z */ CCLASS_C, /* [ */ CCLASS_OTHER, /* \ */ CCLASS_OTHER, + /* ] */ CCLASS_OTHER, /* ^ */ CCLASS_OTHER, /* _ */ CCLASS_OTHER, + /* ` */ CCLASS_OTHER, /* a */ CCLASS_VOWEL, /* b */ CCLASS_B, + /* c */ CCLASS_C, /* d */ CCLASS_D, /* e */ CCLASS_VOWEL, + /* f */ CCLASS_B, /* g */ CCLASS_C, /* h */ CCLASS_SILENT, + /* i */ CCLASS_VOWEL, /* j */ CCLASS_C, /* k */ CCLASS_C, + /* l */ CCLASS_L, /* m */ CCLASS_M, /* n */ CCLASS_M, + /* o */ CCLASS_VOWEL, /* p */ CCLASS_B, /* q */ CCLASS_C, + /* r */ CCLASS_R, /* s */ CCLASS_C, /* t */ CCLASS_D, + /* u */ CCLASS_VOWEL, /* v */ CCLASS_B, /* w */ CCLASS_B, + /* x */ CCLASS_C, /* y */ CCLASS_Y, /* z */ CCLASS_C, + /* { */ CCLASS_OTHER, /* | */ CCLASS_OTHER, /* } */ CCLASS_OTHER, + /* ~ */ CCLASS_OTHER, /* */ CCLASS_OTHER, +}; + +/* +** Mapping from the character class number (0-13) to a symbol for each +** character class. Note that initClass[] can be used to map the class +** symbol back into the class number. +*/ +static const unsigned char className[] = ".ABCDHLRMY9 ?"; + +/* +** Generate a "phonetic hash" from a string of ASCII characters +** in zIn[0..nIn-1]. +** +** * Map characters by character class as defined above. +** * Omit double-letters +** * Omit vowels beside R and L +** * Omit T when followed by CH +** * Omit W when followed by R +** * Omit D when followed by J or G +** * Omit K in KN or G in GN at the beginning of a word +** +** Space to hold the result is obtained from sqlite3_malloc() +** +** Return NULL if memory allocation fails. +*/ +static unsigned char *phoneticHash(const unsigned char *zIn, int nIn){ + unsigned char *zOut = sqlite3_malloc64( nIn + 1 ); + int i; + int nOut = 0; + char cPrev = 0x77; + char cPrevX = 0x77; + const unsigned char *aClass = initClass; + + if( zOut==0 ) return 0; + if( nIn>2 ){ + switch( zIn[0] ){ + case 'g': + case 'k': { + if( zIn[1]=='n' ){ zIn++; nIn--; } + break; + } + } + } + for(i=0; i<nIn; i++){ + unsigned char c = zIn[i]; + if( i+1<nIn ){ + if( c=='w' && zIn[i+1]=='r' ) continue; + if( c=='d' && (zIn[i+1]=='j' || zIn[i+1]=='g') ) continue; + if( i+2<nIn ){ + if( c=='t' && zIn[i+1]=='c' && zIn[i+2]=='h' ) continue; + } + } + c = aClass[c&0x7f]; + if( c==CCLASS_SPACE ) continue; + if( c==CCLASS_OTHER && cPrev!=CCLASS_DIGIT ) continue; + aClass = midClass; + if( c==CCLASS_VOWEL && (cPrevX==CCLASS_R || cPrevX==CCLASS_L) ){ + continue; /* No vowels beside L or R */ + } + if( (c==CCLASS_R || c==CCLASS_L) && cPrevX==CCLASS_VOWEL ){ + nOut--; /* No vowels beside L or R */ + } + cPrev = c; + if( c==CCLASS_SILENT ) continue; + cPrevX = c; + c = className[c]; + assert( nOut>=0 ); + if( nOut==0 || c!=zOut[nOut-1] ) zOut[nOut++] = c; + } + zOut[nOut] = 0; + return zOut; +} + +/* +** This is an SQL function wrapper around phoneticHash(). See +** the description of phoneticHash() for additional information. +*/ +static void phoneticHashSqlFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *zIn; + unsigned char *zOut; + + zIn = sqlite3_value_text(argv[0]); + if( zIn==0 ) return; + zOut = phoneticHash(zIn, sqlite3_value_bytes(argv[0])); + if( zOut==0 ){ + sqlite3_result_error_nomem(context); + }else{ + sqlite3_result_text(context, (char*)zOut, -1, sqlite3_free); + } +} + +/* +** Return the character class number for a character given its +** context. +*/ +static char characterClass(char cPrev, char c){ + return cPrev==0 ? initClass[c&0x7f] : midClass[c&0x7f]; +} + +/* +** Return the cost of inserting or deleting character c immediately +** following character cPrev. If cPrev==0, that means c is the first +** character of the word. +*/ +static int insertOrDeleteCost(char cPrev, char c, char cNext){ + char classC = characterClass(cPrev, c); + char classCprev; + + if( classC==CCLASS_SILENT ){ + /* Insert or delete "silent" characters such as H or W */ + return 1; + } + if( cPrev==c ){ + /* Repeated characters, or miss a repeat */ + return 10; + } + if( classC==CCLASS_VOWEL && (cPrev=='r' || cNext=='r') ){ + return 20; /* Insert a vowel before or after 'r' */ + } + classCprev = characterClass(cPrev, cPrev); + if( classC==classCprev ){ + if( classC==CCLASS_VOWEL ){ + /* Remove or add a new vowel to a vowel cluster */ + return 15; + }else{ + /* Remove or add a consonant not in the same class */ + return 50; + } + } + + /* any other character insertion or deletion */ + return 100; +} + +/* +** Divide the insertion cost by this factor when appending to the +** end of the word. +*/ +#define FINAL_INS_COST_DIV 4 + +/* +** Return the cost of substituting cTo in place of cFrom assuming +** the previous character is cPrev. If cPrev==0 then cTo is the first +** character of the word. +*/ +static int substituteCost(char cPrev, char cFrom, char cTo){ + char classFrom, classTo; + if( cFrom==cTo ){ + /* Exact match */ + return 0; + } + if( cFrom==(cTo^0x20) && ((cTo>='A' && cTo<='Z') || (cTo>='a' && cTo<='z')) ){ + /* differ only in case */ + return 0; + } + classFrom = characterClass(cPrev, cFrom); + classTo = characterClass(cPrev, cTo); + if( classFrom==classTo ){ + /* Same character class */ + return 40; + } + if( classFrom>=CCLASS_B && classFrom<=CCLASS_Y + && classTo>=CCLASS_B && classTo<=CCLASS_Y ){ + /* Convert from one consonant to another, but in a different class */ + return 75; + } + /* Any other subsitution */ + return 100; +} + +/* +** Given two strings zA and zB which are pure ASCII, return the cost +** of transforming zA into zB. If zA ends with '*' assume that it is +** a prefix of zB and give only minimal penalty for extra characters +** on the end of zB. +** +** Smaller numbers mean a closer match. +** +** Negative values indicate an error: +** -1 One of the inputs is NULL +** -2 Non-ASCII characters on input +** -3 Unable to allocate memory +** +** If pnMatch is not NULL, then *pnMatch is set to the number of bytes +** of zB that matched the pattern in zA. If zA does not end with a '*', +** then this value is always the number of bytes in zB (i.e. strlen(zB)). +** If zA does end in a '*', then it is the number of bytes in the prefix +** of zB that was deemed to match zA. +*/ +static int editdist1(const char *zA, const char *zB, int *pnMatch){ + int nA, nB; /* Number of characters in zA[] and zB[] */ + int xA, xB; /* Loop counters for zA[] and zB[] */ + char cA = 0, cB; /* Current character of zA and zB */ + char cAprev, cBprev; /* Previous character of zA and zB */ + char cAnext, cBnext; /* Next character in zA and zB */ + int d; /* North-west cost value */ + int dc = 0; /* North-west character value */ + int res; /* Final result */ + int *m; /* The cost matrix */ + char *cx; /* Corresponding character values */ + int *toFree = 0; /* Malloced space */ + int nMatch = 0; + int mStack[60+15]; /* Stack space to use if not too much is needed */ + + /* Early out if either input is NULL */ + if( zA==0 || zB==0 ) return -1; + + /* Skip any common prefix */ + while( zA[0] && zA[0]==zB[0] ){ dc = zA[0]; zA++; zB++; nMatch++; } + if( pnMatch ) *pnMatch = nMatch; + if( zA[0]==0 && zB[0]==0 ) return 0; + +#if 0 + printf("A=\"%s\" B=\"%s\" dc=%c\n", zA, zB, dc?dc:' '); +#endif + + /* Verify input strings and measure their lengths */ + for(nA=0; zA[nA]; nA++){ + if( zA[nA]&0x80 ) return -2; + } + for(nB=0; zB[nB]; nB++){ + if( zB[nB]&0x80 ) return -2; + } + + /* Special processing if either string is empty */ + if( nA==0 ){ + cBprev = (char)dc; + for(xB=res=0; (cB = zB[xB])!=0; xB++){ + res += insertOrDeleteCost(cBprev, cB, zB[xB+1])/FINAL_INS_COST_DIV; + cBprev = cB; + } + return res; + } + if( nB==0 ){ + cAprev = (char)dc; + for(xA=res=0; (cA = zA[xA])!=0; xA++){ + res += insertOrDeleteCost(cAprev, cA, zA[xA+1]); + cAprev = cA; + } + return res; + } + + /* A is a prefix of B */ + if( zA[0]=='*' && zA[1]==0 ) return 0; + + /* Allocate and initialize the Wagner matrix */ + if( nB<(sizeof(mStack)*4)/(sizeof(mStack[0])*5) ){ + m = mStack; + }else{ + m = toFree = sqlite3_malloc64( (nB+1)*5*sizeof(m[0])/4 ); + if( m==0 ) return -3; + } + cx = (char*)&m[nB+1]; + + /* Compute the Wagner edit distance */ + m[0] = 0; + cx[0] = (char)dc; + cBprev = (char)dc; + for(xB=1; xB<=nB; xB++){ + cBnext = zB[xB]; + cB = zB[xB-1]; + cx[xB] = cB; + m[xB] = m[xB-1] + insertOrDeleteCost(cBprev, cB, cBnext); + cBprev = cB; + } + cAprev = (char)dc; + for(xA=1; xA<=nA; xA++){ + int lastA = (xA==nA); + cA = zA[xA-1]; + cAnext = zA[xA]; + if( cA=='*' && lastA ) break; + d = m[0]; + dc = cx[0]; + m[0] = d + insertOrDeleteCost(cAprev, cA, cAnext); + cBprev = 0; + for(xB=1; xB<=nB; xB++){ + int totalCost, insCost, delCost, subCost, ncx; + cB = zB[xB-1]; + cBnext = zB[xB]; + + /* Cost to insert cB */ + insCost = insertOrDeleteCost(cx[xB-1], cB, cBnext); + if( lastA ) insCost /= FINAL_INS_COST_DIV; + + /* Cost to delete cA */ + delCost = insertOrDeleteCost(cx[xB], cA, cBnext); + + /* Cost to substitute cA->cB */ + subCost = substituteCost(cx[xB-1], cA, cB); + + /* Best cost */ + totalCost = insCost + m[xB-1]; + ncx = cB; + if( (delCost + m[xB])<totalCost ){ + totalCost = delCost + m[xB]; + ncx = cA; + } + if( (subCost + d)<totalCost ){ + totalCost = subCost + d; + } + +#if 0 + printf("%d,%d d=%4d u=%4d r=%4d dc=%c cA=%c cB=%c" + " ins=%4d del=%4d sub=%4d t=%4d ncx=%c\n", + xA, xB, d, m[xB], m[xB-1], dc?dc:' ', cA, cB, + insCost, delCost, subCost, totalCost, ncx?ncx:' '); +#endif + + /* Update the matrix */ + d = m[xB]; + dc = cx[xB]; + m[xB] = totalCost; + cx[xB] = (char)ncx; + cBprev = cB; + } + cAprev = cA; + } + + /* Free the wagner matrix and return the result */ + if( cA=='*' ){ + res = m[1]; + for(xB=1; xB<=nB; xB++){ + if( m[xB]<res ){ + res = m[xB]; + if( pnMatch ) *pnMatch = xB+nMatch; + } + } + }else{ + res = m[nB]; + /* In the current implementation, pnMatch is always NULL if zA does + ** not end in "*" */ + assert( pnMatch==0 ); + } + sqlite3_free(toFree); + return res; +} + +/* +** Function: editdist(A,B) +** +** Return the cost of transforming string A into string B. Both strings +** must be pure ASCII text. If A ends with '*' then it is assumed to be +** a prefix of B and extra characters on the end of B have minimal additional +** cost. +*/ +static void editdistSqlFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + int res = editdist1( + (const char*)sqlite3_value_text(argv[0]), + (const char*)sqlite3_value_text(argv[1]), + 0); + if( res<0 ){ + if( res==(-3) ){ + sqlite3_result_error_nomem(context); + }else if( res==(-2) ){ + sqlite3_result_error(context, "non-ASCII input to editdist()", -1); + }else{ + sqlite3_result_error(context, "NULL input to editdist()", -1); + } + }else{ + sqlite3_result_int(context, res); + } +} + +/* End of the fixed-cost edit distance implementation +****************************************************************************** +***************************************************************************** +** Begin: Configurable cost unicode edit distance routines +*/ +/* Forward declaration of structures */ +typedef struct EditDist3Cost EditDist3Cost; +typedef struct EditDist3Config EditDist3Config; +typedef struct EditDist3Point EditDist3Point; +typedef struct EditDist3From EditDist3From; +typedef struct EditDist3FromString EditDist3FromString; +typedef struct EditDist3To EditDist3To; +typedef struct EditDist3ToString EditDist3ToString; +typedef struct EditDist3Lang EditDist3Lang; + + +/* +** An entry in the edit cost table +*/ +struct EditDist3Cost { + EditDist3Cost *pNext; /* Next cost element */ + u8 nFrom; /* Number of bytes in aFrom */ + u8 nTo; /* Number of bytes in aTo */ + u16 iCost; /* Cost of this transformation */ + char a[4] ; /* FROM string followed by TO string */ + /* Additional TO and FROM string bytes appended as necessary */ +}; + +/* +** Edit costs for a particular language ID +*/ +struct EditDist3Lang { + int iLang; /* Language ID */ + int iInsCost; /* Default insertion cost */ + int iDelCost; /* Default deletion cost */ + int iSubCost; /* Default substitution cost */ + EditDist3Cost *pCost; /* Costs */ +}; + + +/* +** The default EditDist3Lang object, with default costs. +*/ +static const EditDist3Lang editDist3Lang = { 0, 100, 100, 150, 0 }; + +/* +** Complete configuration +*/ +struct EditDist3Config { + int nLang; /* Number of language IDs. Size of a[] */ + EditDist3Lang *a; /* One for each distinct language ID */ +}; + +/* +** Extra information about each character in the FROM string. +*/ +struct EditDist3From { + int nSubst; /* Number of substitution cost entries */ + int nDel; /* Number of deletion cost entries */ + int nByte; /* Number of bytes in this character */ + EditDist3Cost **apSubst; /* Array of substitution costs for this element */ + EditDist3Cost **apDel; /* Array of deletion cost entries */ +}; + +/* +** A precompiled FROM string. +* +** In the common case we expect the FROM string to be reused multiple times. +** In other words, the common case will be to measure the edit distance +** from a single origin string to multiple target strings. +*/ +struct EditDist3FromString { + char *z; /* The complete text of the FROM string */ + int n; /* Number of characters in the FROM string */ + int isPrefix; /* True if ends with '*' character */ + EditDist3From *a; /* Extra info about each char of the FROM string */ +}; + +/* +** Extra information about each character in the TO string. +*/ +struct EditDist3To { + int nIns; /* Number of insertion cost entries */ + int nByte; /* Number of bytes in this character */ + EditDist3Cost **apIns; /* Array of deletion cost entries */ +}; + +/* +** A precompiled FROM string +*/ +struct EditDist3ToString { + char *z; /* The complete text of the TO string */ + int n; /* Number of characters in the TO string */ + EditDist3To *a; /* Extra info about each char of the TO string */ +}; + +/* +** Clear or delete an instance of the object that records all edit-distance +** weights. +*/ +static void editDist3ConfigClear(EditDist3Config *p){ + int i; + if( p==0 ) return; + for(i=0; i<p->nLang; i++){ + EditDist3Cost *pCost, *pNext; + pCost = p->a[i].pCost; + while( pCost ){ + pNext = pCost->pNext; + sqlite3_free(pCost); + pCost = pNext; + } + } + sqlite3_free(p->a); + memset(p, 0, sizeof(*p)); +} +static void editDist3ConfigDelete(void *pIn){ + EditDist3Config *p = (EditDist3Config*)pIn; + editDist3ConfigClear(p); + sqlite3_free(p); +} + +/* Compare the FROM values of two EditDist3Cost objects, for sorting. +** Return negative, zero, or positive if the A is less than, equal to, +** or greater than B. +*/ +static int editDist3CostCompare(EditDist3Cost *pA, EditDist3Cost *pB){ + int n = pA->nFrom; + int rc; + if( n>pB->nFrom ) n = pB->nFrom; + rc = strncmp(pA->a, pB->a, n); + if( rc==0 ) rc = pA->nFrom - pB->nFrom; + return rc; +} + +/* +** Merge together two sorted lists of EditDist3Cost objects, in order +** of increasing FROM. +*/ +static EditDist3Cost *editDist3CostMerge( + EditDist3Cost *pA, + EditDist3Cost *pB +){ + EditDist3Cost *pHead = 0; + EditDist3Cost **ppTail = &pHead; + EditDist3Cost *p; + while( pA && pB ){ + if( editDist3CostCompare(pA,pB)<=0 ){ + p = pA; + pA = pA->pNext; + }else{ + p = pB; + pB = pB->pNext; + } + *ppTail = p; + ppTail = &p->pNext; + } + if( pA ){ + *ppTail = pA; + }else{ + *ppTail = pB; + } + return pHead; +} + +/* +** Sort a list of EditDist3Cost objects into order of increasing FROM +*/ +static EditDist3Cost *editDist3CostSort(EditDist3Cost *pList){ + EditDist3Cost *ap[60], *p; + int i; + int mx = 0; + ap[0] = 0; + ap[1] = 0; + while( pList ){ + p = pList; + pList = p->pNext; + p->pNext = 0; + for(i=0; ap[i]; i++){ + p = editDist3CostMerge(ap[i],p); + ap[i] = 0; + } + ap[i] = p; + if( i>mx ){ + mx = i; + ap[i+1] = 0; + } + } + p = 0; + for(i=0; i<=mx; i++){ + if( ap[i] ) p = editDist3CostMerge(p,ap[i]); + } + return p; +} + +/* +** Load all edit-distance weights from a table. +*/ +static int editDist3ConfigLoad( + EditDist3Config *p, /* The edit distance configuration to load */ + sqlite3 *db, /* Load from this database */ + const char *zTable /* Name of the table from which to load */ +){ + sqlite3_stmt *pStmt; + int rc, rc2; + char *zSql; + int iLangPrev = -9999; + EditDist3Lang *pLang = 0; + + zSql = sqlite3_mprintf("SELECT iLang, cFrom, cTo, iCost" + " FROM \"%w\" WHERE iLang>=0 ORDER BY iLang", zTable); + if( zSql==0 ) return SQLITE_NOMEM; + rc = sqlite3_prepare(db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ) return rc; + editDist3ConfigClear(p); + while( sqlite3_step(pStmt)==SQLITE_ROW ){ + int iLang = sqlite3_column_int(pStmt, 0); + const char *zFrom = (const char*)sqlite3_column_text(pStmt, 1); + int nFrom = zFrom ? sqlite3_column_bytes(pStmt, 1) : 0; + const char *zTo = (const char*)sqlite3_column_text(pStmt, 2); + int nTo = zTo ? sqlite3_column_bytes(pStmt, 2) : 0; + int iCost = sqlite3_column_int(pStmt, 3); + + assert( zFrom!=0 || nFrom==0 ); + assert( zTo!=0 || nTo==0 ); + if( nFrom>100 || nTo>100 ) continue; + if( iCost<0 ) continue; + if( iCost>=10000 ) continue; /* Costs above 10K are considered infinite */ + if( pLang==0 || iLang!=iLangPrev ){ + EditDist3Lang *pNew; + pNew = sqlite3_realloc64(p->a, (p->nLang+1)*sizeof(p->a[0])); + if( pNew==0 ){ rc = SQLITE_NOMEM; break; } + p->a = pNew; + pLang = &p->a[p->nLang]; + p->nLang++; + pLang->iLang = iLang; + pLang->iInsCost = 100; + pLang->iDelCost = 100; + pLang->iSubCost = 150; + pLang->pCost = 0; + iLangPrev = iLang; + } + if( nFrom==1 && zFrom[0]=='?' && nTo==0 ){ + pLang->iDelCost = iCost; + }else if( nFrom==0 && nTo==1 && zTo[0]=='?' ){ + pLang->iInsCost = iCost; + }else if( nFrom==1 && nTo==1 && zFrom[0]=='?' && zTo[0]=='?' ){ + pLang->iSubCost = iCost; + }else{ + EditDist3Cost *pCost; + int nExtra = nFrom + nTo - 4; + if( nExtra<0 ) nExtra = 0; + pCost = sqlite3_malloc64( sizeof(*pCost) + nExtra ); + if( pCost==0 ){ rc = SQLITE_NOMEM; break; } + pCost->nFrom = (u8)nFrom; + pCost->nTo = (u8)nTo; + pCost->iCost = (u16)iCost; + memcpy(pCost->a, zFrom, nFrom); + memcpy(pCost->a + nFrom, zTo, nTo); + pCost->pNext = pLang->pCost; + pLang->pCost = pCost; + } + } + rc2 = sqlite3_finalize(pStmt); + if( rc==SQLITE_OK ) rc = rc2; + if( rc==SQLITE_OK ){ + int iLang; + for(iLang=0; iLang<p->nLang; iLang++){ + p->a[iLang].pCost = editDist3CostSort(p->a[iLang].pCost); + } + } + return rc; +} + +/* +** Return the length (in bytes) of a utf-8 character. Or return a maximum +** of N. +*/ +static int utf8Len(unsigned char c, int N){ + int len = 1; + if( c>0x7f ){ + if( (c&0xe0)==0xc0 ){ + len = 2; + }else if( (c&0xf0)==0xe0 ){ + len = 3; + }else{ + len = 4; + } + } + if( len>N ) len = N; + return len; +} + +/* +** Return TRUE (non-zero) if the To side of the given cost matches +** the given string. +*/ +static int matchTo(EditDist3Cost *p, const char *z, int n){ + assert( n>0 ); + if( p->a[p->nFrom]!=z[0] ) return 0; + if( p->nTo>n ) return 0; + if( strncmp(p->a+p->nFrom, z, p->nTo)!=0 ) return 0; + return 1; +} + +/* +** Return TRUE (non-zero) if the From side of the given cost matches +** the given string. +*/ +static int matchFrom(EditDist3Cost *p, const char *z, int n){ + assert( p->nFrom<=n ); + if( p->nFrom ){ + if( p->a[0]!=z[0] ) return 0; + if( strncmp(p->a, z, p->nFrom)!=0 ) return 0; + } + return 1; +} + +/* +** Return TRUE (non-zero) of the next FROM character and the next TO +** character are the same. +*/ +static int matchFromTo( + EditDist3FromString *pStr, /* Left hand string */ + int n1, /* Index of comparison character on the left */ + const char *z2, /* Right-handl comparison character */ + int n2 /* Bytes remaining in z2[] */ +){ + int b1 = pStr->a[n1].nByte; + if( b1>n2 ) return 0; + assert( b1>0 ); + if( pStr->z[n1]!=z2[0] ) return 0; + if( strncmp(pStr->z+n1, z2, b1)!=0 ) return 0; + return 1; +} + +/* +** Delete an EditDist3FromString objecct +*/ +static void editDist3FromStringDelete(EditDist3FromString *p){ + int i; + if( p ){ + for(i=0; i<p->n; i++){ + sqlite3_free(p->a[i].apDel); + sqlite3_free(p->a[i].apSubst); + } + sqlite3_free(p); + } +} + +/* +** Create a EditDist3FromString object. +*/ +static EditDist3FromString *editDist3FromStringNew( + const EditDist3Lang *pLang, + const char *z, + int n +){ + EditDist3FromString *pStr; + EditDist3Cost *p; + int i; + + if( z==0 ) return 0; + if( n<0 ) n = (int)strlen(z); + pStr = sqlite3_malloc64( sizeof(*pStr) + sizeof(pStr->a[0])*n + n + 1 ); + if( pStr==0 ) return 0; + pStr->a = (EditDist3From*)&pStr[1]; + memset(pStr->a, 0, sizeof(pStr->a[0])*n); + pStr->n = n; + pStr->z = (char*)&pStr->a[n]; + memcpy(pStr->z, z, n+1); + if( n && z[n-1]=='*' ){ + pStr->isPrefix = 1; + n--; + pStr->n--; + pStr->z[n] = 0; + }else{ + pStr->isPrefix = 0; + } + + for(i=0; i<n; i++){ + EditDist3From *pFrom = &pStr->a[i]; + memset(pFrom, 0, sizeof(*pFrom)); + pFrom->nByte = utf8Len((unsigned char)z[i], n-i); + for(p=pLang->pCost; p; p=p->pNext){ + EditDist3Cost **apNew; + if( i+p->nFrom>n ) continue; + if( matchFrom(p, z+i, n-i)==0 ) continue; + if( p->nTo==0 ){ + apNew = sqlite3_realloc64(pFrom->apDel, + sizeof(*apNew)*(pFrom->nDel+1)); + if( apNew==0 ) break; + pFrom->apDel = apNew; + apNew[pFrom->nDel++] = p; + }else{ + apNew = sqlite3_realloc64(pFrom->apSubst, + sizeof(*apNew)*(pFrom->nSubst+1)); + if( apNew==0 ) break; + pFrom->apSubst = apNew; + apNew[pFrom->nSubst++] = p; + } + } + if( p ){ + editDist3FromStringDelete(pStr); + pStr = 0; + break; + } + } + return pStr; +} + +/* +** Update entry m[i] such that it is the minimum of its current value +** and m[j]+iCost. +*/ +static void updateCost( + unsigned int *m, + int i, + int j, + int iCost +){ + unsigned int b; + assert( iCost>=0 ); + assert( iCost<10000 ); + b = m[j] + iCost; + if( b<m[i] ) m[i] = b; +} + +/* +** How much stack space (int bytes) to use for Wagner matrix in +** editDist3Core(). If more space than this is required, the entire +** matrix is taken from the heap. To reduce the load on the memory +** allocator, make this value as large as practical for the +** architecture in use. +*/ +#ifndef SQLITE_SPELLFIX_STACKALLOC_SZ +# define SQLITE_SPELLFIX_STACKALLOC_SZ (1024) +#endif + +/* Compute the edit distance between two strings. +** +** If an error occurs, return a negative number which is the error code. +** +** If pnMatch is not NULL, then *pnMatch is set to the number of characters +** (not bytes) in z2 that matched the search pattern in *pFrom. If pFrom does +** not contain the pattern for a prefix-search, then this is always the number +** of characters in z2. If pFrom does contain a prefix search pattern, then +** it is the number of characters in the prefix of z2 that was deemed to +** match pFrom. +*/ +static int editDist3Core( + EditDist3FromString *pFrom, /* The FROM string */ + const char *z2, /* The TO string */ + int n2, /* Length of the TO string */ + const EditDist3Lang *pLang, /* Edit weights for a particular language ID */ + int *pnMatch /* OUT: Characters in matched prefix */ +){ + int k, n; + int i1, b1; + int i2, b2; + EditDist3FromString f = *pFrom; + EditDist3To *a2; + unsigned int *m; + unsigned int *pToFree; + int szRow; + EditDist3Cost *p; + int res; + sqlite3_uint64 nByte; + unsigned int stackSpace[SQLITE_SPELLFIX_STACKALLOC_SZ/sizeof(unsigned int)]; + + /* allocate the Wagner matrix and the aTo[] array for the TO string */ + n = (f.n+1)*(n2+1); + n = (n+1)&~1; + nByte = n*sizeof(m[0]) + sizeof(a2[0])*n2; + if( nByte<=sizeof(stackSpace) ){ + m = stackSpace; + pToFree = 0; + }else{ + m = pToFree = sqlite3_malloc64( nByte ); + if( m==0 ) return -1; /* Out of memory */ + } + a2 = (EditDist3To*)&m[n]; + memset(a2, 0, sizeof(a2[0])*n2); + + /* Fill in the a1[] matrix for all characters of the TO string */ + for(i2=0; i2<n2; i2++){ + a2[i2].nByte = utf8Len((unsigned char)z2[i2], n2-i2); + for(p=pLang->pCost; p; p=p->pNext){ + EditDist3Cost **apNew; + if( p->nFrom>0 ) break; + if( i2+p->nTo>n2 ) continue; + if( p->a[0]>z2[i2] ) break; + if( matchTo(p, z2+i2, n2-i2)==0 ) continue; + a2[i2].nIns++; + apNew = sqlite3_realloc64(a2[i2].apIns, sizeof(*apNew)*a2[i2].nIns); + if( apNew==0 ){ + res = -1; /* Out of memory */ + goto editDist3Abort; + } + a2[i2].apIns = apNew; + a2[i2].apIns[a2[i2].nIns-1] = p; + } + } + + /* Prepare to compute the minimum edit distance */ + szRow = f.n+1; + memset(m, 0x01, (n2+1)*szRow*sizeof(m[0])); + m[0] = 0; + + /* First fill in the top-row of the matrix with FROM deletion costs */ + for(i1=0; i1<f.n; i1 += b1){ + b1 = f.a[i1].nByte; + updateCost(m, i1+b1, i1, pLang->iDelCost); + for(k=0; k<f.a[i1].nDel; k++){ + p = f.a[i1].apDel[k]; + updateCost(m, i1+p->nFrom, i1, p->iCost); + } + } + + /* Fill in all subsequent rows, top-to-bottom, left-to-right */ + for(i2=0; i2<n2; i2 += b2){ + int rx; /* Starting index for current row */ + int rxp; /* Starting index for previous row */ + b2 = a2[i2].nByte; + rx = szRow*(i2+b2); + rxp = szRow*i2; + updateCost(m, rx, rxp, pLang->iInsCost); + for(k=0; k<a2[i2].nIns; k++){ + p = a2[i2].apIns[k]; + updateCost(m, szRow*(i2+p->nTo), rxp, p->iCost); + } + for(i1=0; i1<f.n; i1+=b1){ + int cx; /* Index of current cell */ + int cxp; /* Index of cell immediately to the left */ + int cxd; /* Index of cell to the left and one row above */ + int cxu; /* Index of cell immediately above */ + b1 = f.a[i1].nByte; + cxp = rx + i1; + cx = cxp + b1; + cxd = rxp + i1; + cxu = cxd + b1; + updateCost(m, cx, cxp, pLang->iDelCost); + for(k=0; k<f.a[i1].nDel; k++){ + p = f.a[i1].apDel[k]; + updateCost(m, cxp+p->nFrom, cxp, p->iCost); + } + updateCost(m, cx, cxu, pLang->iInsCost); + if( matchFromTo(&f, i1, z2+i2, n2-i2) ){ + updateCost(m, cx, cxd, 0); + } + updateCost(m, cx, cxd, pLang->iSubCost); + for(k=0; k<f.a[i1].nSubst; k++){ + p = f.a[i1].apSubst[k]; + if( matchTo(p, z2+i2, n2-i2) ){ + updateCost(m, cxd+p->nFrom+szRow*p->nTo, cxd, p->iCost); + } + } + } + } + +#if 0 /* Enable for debugging */ + printf(" ^"); + for(i1=0; i1<f.n; i1++) printf(" %c-%2x", f.z[i1], f.z[i1]&0xff); + printf("\n ^:"); + for(i1=0; i1<szRow; i1++){ + int v = m[i1]; + if( v>9999 ) printf(" ****"); + else printf(" %4d", v); + } + printf("\n"); + for(i2=0; i2<n2; i2++){ + printf("%c-%02x:", z2[i2], z2[i2]&0xff); + for(i1=0; i1<szRow; i1++){ + int v = m[(i2+1)*szRow+i1]; + if( v>9999 ) printf(" ****"); + else printf(" %4d", v); + } + printf("\n"); + } +#endif + + /* Free memory allocations and return the result */ + res = (int)m[szRow*(n2+1)-1]; + n = n2; + if( f.isPrefix ){ + for(i2=1; i2<=n2; i2++){ + int b = m[szRow*i2-1]; + if( b<=res ){ + res = b; + n = i2 - 1; + } + } + } + if( pnMatch ){ + int nExtra = 0; + for(k=0; k<n; k++){ + if( (z2[k] & 0xc0)==0x80 ) nExtra++; + } + *pnMatch = n - nExtra; + } + +editDist3Abort: + for(i2=0; i2<n2; i2++) sqlite3_free(a2[i2].apIns); + sqlite3_free(pToFree); + return res; +} + +/* +** Get an appropriate EditDist3Lang object. +*/ +static const EditDist3Lang *editDist3FindLang( + EditDist3Config *pConfig, + int iLang +){ + int i; + for(i=0; i<pConfig->nLang; i++){ + if( pConfig->a[i].iLang==iLang ) return &pConfig->a[i]; + } + return &editDist3Lang; +} + +/* +** Function: editdist3(A,B,iLang) +** editdist3(tablename) +** +** Return the cost of transforming string A into string B using edit +** weights for iLang. +** +** The second form loads edit weights into memory from a table. +*/ +static void editDist3SqlFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + EditDist3Config *pConfig = (EditDist3Config*)sqlite3_user_data(context); + sqlite3 *db = sqlite3_context_db_handle(context); + int rc; + if( argc==1 ){ + const char *zTable = (const char*)sqlite3_value_text(argv[0]); + rc = editDist3ConfigLoad(pConfig, db, zTable); + if( rc ) sqlite3_result_error_code(context, rc); + }else{ + const char *zA = (const char*)sqlite3_value_text(argv[0]); + const char *zB = (const char*)sqlite3_value_text(argv[1]); + int nA = sqlite3_value_bytes(argv[0]); + int nB = sqlite3_value_bytes(argv[1]); + int iLang = argc==3 ? sqlite3_value_int(argv[2]) : 0; + const EditDist3Lang *pLang = editDist3FindLang(pConfig, iLang); + EditDist3FromString *pFrom; + int dist; + + pFrom = editDist3FromStringNew(pLang, zA, nA); + if( pFrom==0 ){ + sqlite3_result_error_nomem(context); + return; + } + dist = editDist3Core(pFrom, zB, nB, pLang, 0); + editDist3FromStringDelete(pFrom); + if( dist==(-1) ){ + sqlite3_result_error_nomem(context); + }else{ + sqlite3_result_int(context, dist); + } + } +} + +/* +** Register the editDist3 function with SQLite +*/ +static int editDist3Install(sqlite3 *db){ + int rc; + EditDist3Config *pConfig = sqlite3_malloc64( sizeof(*pConfig) ); + if( pConfig==0 ) return SQLITE_NOMEM; + memset(pConfig, 0, sizeof(*pConfig)); + rc = sqlite3_create_function_v2(db, "editdist3", + 2, SQLITE_UTF8|SQLITE_DETERMINISTIC, pConfig, + editDist3SqlFunc, 0, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function_v2(db, "editdist3", + 3, SQLITE_UTF8|SQLITE_DETERMINISTIC, pConfig, + editDist3SqlFunc, 0, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function_v2(db, "editdist3", + 1, SQLITE_UTF8|SQLITE_DETERMINISTIC, pConfig, + editDist3SqlFunc, 0, 0, editDist3ConfigDelete); + }else{ + sqlite3_free(pConfig); + } + return rc; +} +/* End configurable cost unicode edit distance routines +****************************************************************************** +****************************************************************************** +** Begin transliterate unicode-to-ascii implementation +*/ + +#if !SQLITE_AMALGAMATION +/* +** This lookup table is used to help decode the first byte of +** a multi-byte UTF8 character. +*/ +static const unsigned char sqlite3Utf8Trans1[] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x00, 0x01, 0x02, 0x03, 0x00, 0x01, 0x00, 0x00, +}; +#endif + +/* +** Return the value of the first UTF-8 character in the string. +*/ +static int utf8Read(const unsigned char *z, int n, int *pSize){ + int c, i; + + /* All callers to this routine (in the current implementation) + ** always have n>0. */ + if( NEVER(n==0) ){ + c = i = 0; + }else{ + c = z[0]; + i = 1; + if( c>=0xc0 ){ + c = sqlite3Utf8Trans1[c-0xc0]; + while( i<n && (z[i] & 0xc0)==0x80 ){ + c = (c<<6) + (0x3f & z[i++]); + } + } + } + *pSize = i; + return c; +} + +/* +** Return the number of characters in the utf-8 string in the nIn byte +** buffer pointed to by zIn. +*/ +static int utf8Charlen(const char *zIn, int nIn){ + int i; + int nChar = 0; + for(i=0; i<nIn; nChar++){ + int sz; + utf8Read((const unsigned char *)&zIn[i], nIn-i, &sz); + i += sz; + } + return nChar; +} + +typedef struct Transliteration Transliteration; +struct Transliteration { + unsigned short int cFrom; + unsigned char cTo0, cTo1, cTo2, cTo3; +#ifdef SQLITE_SPELLFIX_5BYTE_MAPPINGS + unsigned char cTo4; +#endif +}; + +/* +** Table of translations from unicode characters into ASCII. +*/ +static const Transliteration translit[] = { + { 0x00A0, 0x20, 0x00, 0x00, 0x00 }, /* to */ + { 0x00B5, 0x75, 0x00, 0x00, 0x00 }, /* µ to u */ + { 0x00C0, 0x41, 0x00, 0x00, 0x00 }, /* À to A */ + { 0x00C1, 0x41, 0x00, 0x00, 0x00 }, /* Á to A */ + { 0x00C2, 0x41, 0x00, 0x00, 0x00 }, /*  to A */ + { 0x00C3, 0x41, 0x00, 0x00, 0x00 }, /* à to A */ + { 0x00C4, 0x41, 0x65, 0x00, 0x00 }, /* Ä to Ae */ + { 0x00C5, 0x41, 0x61, 0x00, 0x00 }, /* Å to Aa */ + { 0x00C6, 0x41, 0x45, 0x00, 0x00 }, /* Æ to AE */ + { 0x00C7, 0x43, 0x00, 0x00, 0x00 }, /* Ç to C */ + { 0x00C8, 0x45, 0x00, 0x00, 0x00 }, /* È to E */ + { 0x00C9, 0x45, 0x00, 0x00, 0x00 }, /* É to E */ + { 0x00CA, 0x45, 0x00, 0x00, 0x00 }, /* Ê to E */ + { 0x00CB, 0x45, 0x00, 0x00, 0x00 }, /* Ë to E */ + { 0x00CC, 0x49, 0x00, 0x00, 0x00 }, /* Ì to I */ + { 0x00CD, 0x49, 0x00, 0x00, 0x00 }, /* Í to I */ + { 0x00CE, 0x49, 0x00, 0x00, 0x00 }, /* Î to I */ + { 0x00CF, 0x49, 0x00, 0x00, 0x00 }, /* Ï to I */ + { 0x00D0, 0x44, 0x00, 0x00, 0x00 }, /* Ð to D */ + { 0x00D1, 0x4E, 0x00, 0x00, 0x00 }, /* Ñ to N */ + { 0x00D2, 0x4F, 0x00, 0x00, 0x00 }, /* Ò to O */ + { 0x00D3, 0x4F, 0x00, 0x00, 0x00 }, /* Ó to O */ + { 0x00D4, 0x4F, 0x00, 0x00, 0x00 }, /* Ô to O */ + { 0x00D5, 0x4F, 0x00, 0x00, 0x00 }, /* Õ to O */ + { 0x00D6, 0x4F, 0x65, 0x00, 0x00 }, /* Ö to Oe */ + { 0x00D7, 0x78, 0x00, 0x00, 0x00 }, /* × to x */ + { 0x00D8, 0x4F, 0x00, 0x00, 0x00 }, /* Ø to O */ + { 0x00D9, 0x55, 0x00, 0x00, 0x00 }, /* Ù to U */ + { 0x00DA, 0x55, 0x00, 0x00, 0x00 }, /* Ú to U */ + { 0x00DB, 0x55, 0x00, 0x00, 0x00 }, /* Û to U */ + { 0x00DC, 0x55, 0x65, 0x00, 0x00 }, /* Ü to Ue */ + { 0x00DD, 0x59, 0x00, 0x00, 0x00 }, /* Ý to Y */ + { 0x00DE, 0x54, 0x68, 0x00, 0x00 }, /* Þ to Th */ + { 0x00DF, 0x73, 0x73, 0x00, 0x00 }, /* ß to ss */ + { 0x00E0, 0x61, 0x00, 0x00, 0x00 }, /* à to a */ + { 0x00E1, 0x61, 0x00, 0x00, 0x00 }, /* á to a */ + { 0x00E2, 0x61, 0x00, 0x00, 0x00 }, /* â to a */ + { 0x00E3, 0x61, 0x00, 0x00, 0x00 }, /* ã to a */ + { 0x00E4, 0x61, 0x65, 0x00, 0x00 }, /* ä to ae */ + { 0x00E5, 0x61, 0x61, 0x00, 0x00 }, /* å to aa */ + { 0x00E6, 0x61, 0x65, 0x00, 0x00 }, /* æ to ae */ + { 0x00E7, 0x63, 0x00, 0x00, 0x00 }, /* ç to c */ + { 0x00E8, 0x65, 0x00, 0x00, 0x00 }, /* è to e */ + { 0x00E9, 0x65, 0x00, 0x00, 0x00 }, /* é to e */ + { 0x00EA, 0x65, 0x00, 0x00, 0x00 }, /* ê to e */ + { 0x00EB, 0x65, 0x00, 0x00, 0x00 }, /* ë to e */ + { 0x00EC, 0x69, 0x00, 0x00, 0x00 }, /* ì to i */ + { 0x00ED, 0x69, 0x00, 0x00, 0x00 }, /* í to i */ + { 0x00EE, 0x69, 0x00, 0x00, 0x00 }, /* î to i */ + { 0x00EF, 0x69, 0x00, 0x00, 0x00 }, /* ï to i */ + { 0x00F0, 0x64, 0x00, 0x00, 0x00 }, /* ð to d */ + { 0x00F1, 0x6E, 0x00, 0x00, 0x00 }, /* ñ to n */ + { 0x00F2, 0x6F, 0x00, 0x00, 0x00 }, /* ò to o */ + { 0x00F3, 0x6F, 0x00, 0x00, 0x00 }, /* ó to o */ + { 0x00F4, 0x6F, 0x00, 0x00, 0x00 }, /* ô to o */ + { 0x00F5, 0x6F, 0x00, 0x00, 0x00 }, /* õ to o */ + { 0x00F6, 0x6F, 0x65, 0x00, 0x00 }, /* ö to oe */ + { 0x00F7, 0x3A, 0x00, 0x00, 0x00 }, /* ÷ to : */ + { 0x00F8, 0x6F, 0x00, 0x00, 0x00 }, /* ø to o */ + { 0x00F9, 0x75, 0x00, 0x00, 0x00 }, /* ù to u */ + { 0x00FA, 0x75, 0x00, 0x00, 0x00 }, /* ú to u */ + { 0x00FB, 0x75, 0x00, 0x00, 0x00 }, /* û to u */ + { 0x00FC, 0x75, 0x65, 0x00, 0x00 }, /* ü to ue */ + { 0x00FD, 0x79, 0x00, 0x00, 0x00 }, /* ý to y */ + { 0x00FE, 0x74, 0x68, 0x00, 0x00 }, /* þ to th */ + { 0x00FF, 0x79, 0x00, 0x00, 0x00 }, /* ÿ to y */ + { 0x0100, 0x41, 0x00, 0x00, 0x00 }, /* Ā to A */ + { 0x0101, 0x61, 0x00, 0x00, 0x00 }, /* ā to a */ + { 0x0102, 0x41, 0x00, 0x00, 0x00 }, /* Ă to A */ + { 0x0103, 0x61, 0x00, 0x00, 0x00 }, /* ă to a */ + { 0x0104, 0x41, 0x00, 0x00, 0x00 }, /* Ą to A */ + { 0x0105, 0x61, 0x00, 0x00, 0x00 }, /* ą to a */ + { 0x0106, 0x43, 0x00, 0x00, 0x00 }, /* Ć to C */ + { 0x0107, 0x63, 0x00, 0x00, 0x00 }, /* ć to c */ + { 0x0108, 0x43, 0x68, 0x00, 0x00 }, /* Ĉ to Ch */ + { 0x0109, 0x63, 0x68, 0x00, 0x00 }, /* ĉ to ch */ + { 0x010A, 0x43, 0x00, 0x00, 0x00 }, /* Ċ to C */ + { 0x010B, 0x63, 0x00, 0x00, 0x00 }, /* ċ to c */ + { 0x010C, 0x43, 0x00, 0x00, 0x00 }, /* Č to C */ + { 0x010D, 0x63, 0x00, 0x00, 0x00 }, /* č to c */ + { 0x010E, 0x44, 0x00, 0x00, 0x00 }, /* Ď to D */ + { 0x010F, 0x64, 0x00, 0x00, 0x00 }, /* ď to d */ + { 0x0110, 0x44, 0x00, 0x00, 0x00 }, /* Đ to D */ + { 0x0111, 0x64, 0x00, 0x00, 0x00 }, /* đ to d */ + { 0x0112, 0x45, 0x00, 0x00, 0x00 }, /* Ē to E */ + { 0x0113, 0x65, 0x00, 0x00, 0x00 }, /* ē to e */ + { 0x0114, 0x45, 0x00, 0x00, 0x00 }, /* Ĕ to E */ + { 0x0115, 0x65, 0x00, 0x00, 0x00 }, /* ĕ to e */ + { 0x0116, 0x45, 0x00, 0x00, 0x00 }, /* Ė to E */ + { 0x0117, 0x65, 0x00, 0x00, 0x00 }, /* ė to e */ + { 0x0118, 0x45, 0x00, 0x00, 0x00 }, /* Ę to E */ + { 0x0119, 0x65, 0x00, 0x00, 0x00 }, /* ę to e */ + { 0x011A, 0x45, 0x00, 0x00, 0x00 }, /* Ě to E */ + { 0x011B, 0x65, 0x00, 0x00, 0x00 }, /* ě to e */ + { 0x011C, 0x47, 0x68, 0x00, 0x00 }, /* Ĝ to Gh */ + { 0x011D, 0x67, 0x68, 0x00, 0x00 }, /* ĝ to gh */ + { 0x011E, 0x47, 0x00, 0x00, 0x00 }, /* Ğ to G */ + { 0x011F, 0x67, 0x00, 0x00, 0x00 }, /* ğ to g */ + { 0x0120, 0x47, 0x00, 0x00, 0x00 }, /* Ġ to G */ + { 0x0121, 0x67, 0x00, 0x00, 0x00 }, /* ġ to g */ + { 0x0122, 0x47, 0x00, 0x00, 0x00 }, /* Ģ to G */ + { 0x0123, 0x67, 0x00, 0x00, 0x00 }, /* ģ to g */ + { 0x0124, 0x48, 0x68, 0x00, 0x00 }, /* Ĥ to Hh */ + { 0x0125, 0x68, 0x68, 0x00, 0x00 }, /* ĥ to hh */ + { 0x0126, 0x48, 0x00, 0x00, 0x00 }, /* Ħ to H */ + { 0x0127, 0x68, 0x00, 0x00, 0x00 }, /* ħ to h */ + { 0x0128, 0x49, 0x00, 0x00, 0x00 }, /* Ĩ to I */ + { 0x0129, 0x69, 0x00, 0x00, 0x00 }, /* ĩ to i */ + { 0x012A, 0x49, 0x00, 0x00, 0x00 }, /* Ī to I */ + { 0x012B, 0x69, 0x00, 0x00, 0x00 }, /* ī to i */ + { 0x012C, 0x49, 0x00, 0x00, 0x00 }, /* Ĭ to I */ + { 0x012D, 0x69, 0x00, 0x00, 0x00 }, /* ĭ to i */ + { 0x012E, 0x49, 0x00, 0x00, 0x00 }, /* Į to I */ + { 0x012F, 0x69, 0x00, 0x00, 0x00 }, /* į to i */ + { 0x0130, 0x49, 0x00, 0x00, 0x00 }, /* İ to I */ + { 0x0131, 0x69, 0x00, 0x00, 0x00 }, /* ı to i */ + { 0x0132, 0x49, 0x4A, 0x00, 0x00 }, /* IJ to IJ */ + { 0x0133, 0x69, 0x6A, 0x00, 0x00 }, /* ij to ij */ + { 0x0134, 0x4A, 0x68, 0x00, 0x00 }, /* Ĵ to Jh */ + { 0x0135, 0x6A, 0x68, 0x00, 0x00 }, /* ĵ to jh */ + { 0x0136, 0x4B, 0x00, 0x00, 0x00 }, /* Ķ to K */ + { 0x0137, 0x6B, 0x00, 0x00, 0x00 }, /* ķ to k */ + { 0x0138, 0x6B, 0x00, 0x00, 0x00 }, /* ĸ to k */ + { 0x0139, 0x4C, 0x00, 0x00, 0x00 }, /* Ĺ to L */ + { 0x013A, 0x6C, 0x00, 0x00, 0x00 }, /* ĺ to l */ + { 0x013B, 0x4C, 0x00, 0x00, 0x00 }, /* Ļ to L */ + { 0x013C, 0x6C, 0x00, 0x00, 0x00 }, /* ļ to l */ + { 0x013D, 0x4C, 0x00, 0x00, 0x00 }, /* Ľ to L */ + { 0x013E, 0x6C, 0x00, 0x00, 0x00 }, /* ľ to l */ + { 0x013F, 0x4C, 0x2E, 0x00, 0x00 }, /* Ŀ to L. */ + { 0x0140, 0x6C, 0x2E, 0x00, 0x00 }, /* ŀ to l. */ + { 0x0141, 0x4C, 0x00, 0x00, 0x00 }, /* Ł to L */ + { 0x0142, 0x6C, 0x00, 0x00, 0x00 }, /* ł to l */ + { 0x0143, 0x4E, 0x00, 0x00, 0x00 }, /* Ń to N */ + { 0x0144, 0x6E, 0x00, 0x00, 0x00 }, /* ń to n */ + { 0x0145, 0x4E, 0x00, 0x00, 0x00 }, /* Ņ to N */ + { 0x0146, 0x6E, 0x00, 0x00, 0x00 }, /* ņ to n */ + { 0x0147, 0x4E, 0x00, 0x00, 0x00 }, /* Ň to N */ + { 0x0148, 0x6E, 0x00, 0x00, 0x00 }, /* ň to n */ + { 0x0149, 0x27, 0x6E, 0x00, 0x00 }, /* ʼn to 'n */ + { 0x014A, 0x4E, 0x47, 0x00, 0x00 }, /* Ŋ to NG */ + { 0x014B, 0x6E, 0x67, 0x00, 0x00 }, /* ŋ to ng */ + { 0x014C, 0x4F, 0x00, 0x00, 0x00 }, /* Ō to O */ + { 0x014D, 0x6F, 0x00, 0x00, 0x00 }, /* ō to o */ + { 0x014E, 0x4F, 0x00, 0x00, 0x00 }, /* Ŏ to O */ + { 0x014F, 0x6F, 0x00, 0x00, 0x00 }, /* ŏ to o */ + { 0x0150, 0x4F, 0x00, 0x00, 0x00 }, /* Ő to O */ + { 0x0151, 0x6F, 0x00, 0x00, 0x00 }, /* ő to o */ + { 0x0152, 0x4F, 0x45, 0x00, 0x00 }, /* Œ to OE */ + { 0x0153, 0x6F, 0x65, 0x00, 0x00 }, /* œ to oe */ + { 0x0154, 0x52, 0x00, 0x00, 0x00 }, /* Ŕ to R */ + { 0x0155, 0x72, 0x00, 0x00, 0x00 }, /* ŕ to r */ + { 0x0156, 0x52, 0x00, 0x00, 0x00 }, /* Ŗ to R */ + { 0x0157, 0x72, 0x00, 0x00, 0x00 }, /* ŗ to r */ + { 0x0158, 0x52, 0x00, 0x00, 0x00 }, /* Ř to R */ + { 0x0159, 0x72, 0x00, 0x00, 0x00 }, /* ř to r */ + { 0x015A, 0x53, 0x00, 0x00, 0x00 }, /* Ś to S */ + { 0x015B, 0x73, 0x00, 0x00, 0x00 }, /* ś to s */ + { 0x015C, 0x53, 0x68, 0x00, 0x00 }, /* Ŝ to Sh */ + { 0x015D, 0x73, 0x68, 0x00, 0x00 }, /* ŝ to sh */ + { 0x015E, 0x53, 0x00, 0x00, 0x00 }, /* Ş to S */ + { 0x015F, 0x73, 0x00, 0x00, 0x00 }, /* ş to s */ + { 0x0160, 0x53, 0x00, 0x00, 0x00 }, /* Š to S */ + { 0x0161, 0x73, 0x00, 0x00, 0x00 }, /* š to s */ + { 0x0162, 0x54, 0x00, 0x00, 0x00 }, /* Ţ to T */ + { 0x0163, 0x74, 0x00, 0x00, 0x00 }, /* ţ to t */ + { 0x0164, 0x54, 0x00, 0x00, 0x00 }, /* Ť to T */ + { 0x0165, 0x74, 0x00, 0x00, 0x00 }, /* ť to t */ + { 0x0166, 0x54, 0x00, 0x00, 0x00 }, /* Ŧ to T */ + { 0x0167, 0x74, 0x00, 0x00, 0x00 }, /* ŧ to t */ + { 0x0168, 0x55, 0x00, 0x00, 0x00 }, /* Ũ to U */ + { 0x0169, 0x75, 0x00, 0x00, 0x00 }, /* ũ to u */ + { 0x016A, 0x55, 0x00, 0x00, 0x00 }, /* Ū to U */ + { 0x016B, 0x75, 0x00, 0x00, 0x00 }, /* ū to u */ + { 0x016C, 0x55, 0x00, 0x00, 0x00 }, /* Ŭ to U */ + { 0x016D, 0x75, 0x00, 0x00, 0x00 }, /* ŭ to u */ + { 0x016E, 0x55, 0x00, 0x00, 0x00 }, /* Ů to U */ + { 0x016F, 0x75, 0x00, 0x00, 0x00 }, /* ů to u */ + { 0x0170, 0x55, 0x00, 0x00, 0x00 }, /* Ű to U */ + { 0x0171, 0x75, 0x00, 0x00, 0x00 }, /* ű to u */ + { 0x0172, 0x55, 0x00, 0x00, 0x00 }, /* Ų to U */ + { 0x0173, 0x75, 0x00, 0x00, 0x00 }, /* ų to u */ + { 0x0174, 0x57, 0x00, 0x00, 0x00 }, /* Ŵ to W */ + { 0x0175, 0x77, 0x00, 0x00, 0x00 }, /* ŵ to w */ + { 0x0176, 0x59, 0x00, 0x00, 0x00 }, /* Ŷ to Y */ + { 0x0177, 0x79, 0x00, 0x00, 0x00 }, /* ŷ to y */ + { 0x0178, 0x59, 0x00, 0x00, 0x00 }, /* Ÿ to Y */ + { 0x0179, 0x5A, 0x00, 0x00, 0x00 }, /* Ź to Z */ + { 0x017A, 0x7A, 0x00, 0x00, 0x00 }, /* ź to z */ + { 0x017B, 0x5A, 0x00, 0x00, 0x00 }, /* Ż to Z */ + { 0x017C, 0x7A, 0x00, 0x00, 0x00 }, /* ż to z */ + { 0x017D, 0x5A, 0x00, 0x00, 0x00 }, /* Ž to Z */ + { 0x017E, 0x7A, 0x00, 0x00, 0x00 }, /* ž to z */ + { 0x017F, 0x73, 0x00, 0x00, 0x00 }, /* ſ to s */ + { 0x0192, 0x66, 0x00, 0x00, 0x00 }, /* ƒ to f */ + { 0x0218, 0x53, 0x00, 0x00, 0x00 }, /* Ș to S */ + { 0x0219, 0x73, 0x00, 0x00, 0x00 }, /* ș to s */ + { 0x021A, 0x54, 0x00, 0x00, 0x00 }, /* Ț to T */ + { 0x021B, 0x74, 0x00, 0x00, 0x00 }, /* ț to t */ + { 0x0386, 0x41, 0x00, 0x00, 0x00 }, /* Ά to A */ + { 0x0388, 0x45, 0x00, 0x00, 0x00 }, /* Έ to E */ + { 0x0389, 0x49, 0x00, 0x00, 0x00 }, /* Ή to I */ + { 0x038A, 0x49, 0x00, 0x00, 0x00 }, /* Ί to I */ + { 0x038C, 0x4f, 0x00, 0x00, 0x00 }, /* Ό to O */ + { 0x038E, 0x59, 0x00, 0x00, 0x00 }, /* Ύ to Y */ + { 0x038F, 0x4f, 0x00, 0x00, 0x00 }, /* Ώ to O */ + { 0x0390, 0x69, 0x00, 0x00, 0x00 }, /* ΐ to i */ + { 0x0391, 0x41, 0x00, 0x00, 0x00 }, /* Α to A */ + { 0x0392, 0x42, 0x00, 0x00, 0x00 }, /* Β to B */ + { 0x0393, 0x47, 0x00, 0x00, 0x00 }, /* Γ to G */ + { 0x0394, 0x44, 0x00, 0x00, 0x00 }, /* Δ to D */ + { 0x0395, 0x45, 0x00, 0x00, 0x00 }, /* Ε to E */ + { 0x0396, 0x5a, 0x00, 0x00, 0x00 }, /* Ζ to Z */ + { 0x0397, 0x49, 0x00, 0x00, 0x00 }, /* Η to I */ + { 0x0398, 0x54, 0x68, 0x00, 0x00 }, /* Θ to Th */ + { 0x0399, 0x49, 0x00, 0x00, 0x00 }, /* Ι to I */ + { 0x039A, 0x4b, 0x00, 0x00, 0x00 }, /* Κ to K */ + { 0x039B, 0x4c, 0x00, 0x00, 0x00 }, /* Λ to L */ + { 0x039C, 0x4d, 0x00, 0x00, 0x00 }, /* Μ to M */ + { 0x039D, 0x4e, 0x00, 0x00, 0x00 }, /* Ν to N */ + { 0x039E, 0x58, 0x00, 0x00, 0x00 }, /* Ξ to X */ + { 0x039F, 0x4f, 0x00, 0x00, 0x00 }, /* Ο to O */ + { 0x03A0, 0x50, 0x00, 0x00, 0x00 }, /* Π to P */ + { 0x03A1, 0x52, 0x00, 0x00, 0x00 }, /* Ρ to R */ + { 0x03A3, 0x53, 0x00, 0x00, 0x00 }, /* Σ to S */ + { 0x03A4, 0x54, 0x00, 0x00, 0x00 }, /* Τ to T */ + { 0x03A5, 0x59, 0x00, 0x00, 0x00 }, /* Υ to Y */ + { 0x03A6, 0x46, 0x00, 0x00, 0x00 }, /* Φ to F */ + { 0x03A7, 0x43, 0x68, 0x00, 0x00 }, /* Χ to Ch */ + { 0x03A8, 0x50, 0x73, 0x00, 0x00 }, /* Ψ to Ps */ + { 0x03A9, 0x4f, 0x00, 0x00, 0x00 }, /* Ω to O */ + { 0x03AA, 0x49, 0x00, 0x00, 0x00 }, /* Ϊ to I */ + { 0x03AB, 0x59, 0x00, 0x00, 0x00 }, /* Ϋ to Y */ + { 0x03AC, 0x61, 0x00, 0x00, 0x00 }, /* ά to a */ + { 0x03AD, 0x65, 0x00, 0x00, 0x00 }, /* έ to e */ + { 0x03AE, 0x69, 0x00, 0x00, 0x00 }, /* ή to i */ + { 0x03AF, 0x69, 0x00, 0x00, 0x00 }, /* ί to i */ + { 0x03B1, 0x61, 0x00, 0x00, 0x00 }, /* α to a */ + { 0x03B2, 0x62, 0x00, 0x00, 0x00 }, /* β to b */ + { 0x03B3, 0x67, 0x00, 0x00, 0x00 }, /* γ to g */ + { 0x03B4, 0x64, 0x00, 0x00, 0x00 }, /* δ to d */ + { 0x03B5, 0x65, 0x00, 0x00, 0x00 }, /* ε to e */ + { 0x03B6, 0x7a, 0x00, 0x00, 0x00 }, /* ζ to z */ + { 0x03B7, 0x69, 0x00, 0x00, 0x00 }, /* η to i */ + { 0x03B8, 0x74, 0x68, 0x00, 0x00 }, /* θ to th */ + { 0x03B9, 0x69, 0x00, 0x00, 0x00 }, /* ι to i */ + { 0x03BA, 0x6b, 0x00, 0x00, 0x00 }, /* κ to k */ + { 0x03BB, 0x6c, 0x00, 0x00, 0x00 }, /* λ to l */ + { 0x03BC, 0x6d, 0x00, 0x00, 0x00 }, /* μ to m */ + { 0x03BD, 0x6e, 0x00, 0x00, 0x00 }, /* ν to n */ + { 0x03BE, 0x78, 0x00, 0x00, 0x00 }, /* ξ to x */ + { 0x03BF, 0x6f, 0x00, 0x00, 0x00 }, /* ο to o */ + { 0x03C0, 0x70, 0x00, 0x00, 0x00 }, /* π to p */ + { 0x03C1, 0x72, 0x00, 0x00, 0x00 }, /* ρ to r */ + { 0x03C3, 0x73, 0x00, 0x00, 0x00 }, /* σ to s */ + { 0x03C4, 0x74, 0x00, 0x00, 0x00 }, /* τ to t */ + { 0x03C5, 0x79, 0x00, 0x00, 0x00 }, /* υ to y */ + { 0x03C6, 0x66, 0x00, 0x00, 0x00 }, /* φ to f */ + { 0x03C7, 0x63, 0x68, 0x00, 0x00 }, /* χ to ch */ + { 0x03C8, 0x70, 0x73, 0x00, 0x00 }, /* ψ to ps */ + { 0x03C9, 0x6f, 0x00, 0x00, 0x00 }, /* ω to o */ + { 0x03CA, 0x69, 0x00, 0x00, 0x00 }, /* ϊ to i */ + { 0x03CB, 0x79, 0x00, 0x00, 0x00 }, /* ϋ to y */ + { 0x03CC, 0x6f, 0x00, 0x00, 0x00 }, /* ό to o */ + { 0x03CD, 0x79, 0x00, 0x00, 0x00 }, /* ύ to y */ + { 0x03CE, 0x69, 0x00, 0x00, 0x00 }, /* ώ to i */ + { 0x0400, 0x45, 0x00, 0x00, 0x00 }, /* Ѐ to E */ + { 0x0401, 0x45, 0x00, 0x00, 0x00 }, /* Ё to E */ + { 0x0402, 0x44, 0x00, 0x00, 0x00 }, /* Ђ to D */ + { 0x0403, 0x47, 0x00, 0x00, 0x00 }, /* Ѓ to G */ + { 0x0404, 0x45, 0x00, 0x00, 0x00 }, /* Є to E */ + { 0x0405, 0x5a, 0x00, 0x00, 0x00 }, /* Ѕ to Z */ + { 0x0406, 0x49, 0x00, 0x00, 0x00 }, /* І to I */ + { 0x0407, 0x49, 0x00, 0x00, 0x00 }, /* Ї to I */ + { 0x0408, 0x4a, 0x00, 0x00, 0x00 }, /* Ј to J */ + { 0x0409, 0x49, 0x00, 0x00, 0x00 }, /* Љ to I */ + { 0x040A, 0x4e, 0x00, 0x00, 0x00 }, /* Њ to N */ + { 0x040B, 0x44, 0x00, 0x00, 0x00 }, /* Ћ to D */ + { 0x040C, 0x4b, 0x00, 0x00, 0x00 }, /* Ќ to K */ + { 0x040D, 0x49, 0x00, 0x00, 0x00 }, /* Ѝ to I */ + { 0x040E, 0x55, 0x00, 0x00, 0x00 }, /* Ў to U */ + { 0x040F, 0x44, 0x00, 0x00, 0x00 }, /* Џ to D */ + { 0x0410, 0x41, 0x00, 0x00, 0x00 }, /* А to A */ + { 0x0411, 0x42, 0x00, 0x00, 0x00 }, /* Б to B */ + { 0x0412, 0x56, 0x00, 0x00, 0x00 }, /* В to V */ + { 0x0413, 0x47, 0x00, 0x00, 0x00 }, /* Г to G */ + { 0x0414, 0x44, 0x00, 0x00, 0x00 }, /* Д to D */ + { 0x0415, 0x45, 0x00, 0x00, 0x00 }, /* Е to E */ + { 0x0416, 0x5a, 0x68, 0x00, 0x00 }, /* Ж to Zh */ + { 0x0417, 0x5a, 0x00, 0x00, 0x00 }, /* З to Z */ + { 0x0418, 0x49, 0x00, 0x00, 0x00 }, /* И to I */ + { 0x0419, 0x49, 0x00, 0x00, 0x00 }, /* Й to I */ + { 0x041A, 0x4b, 0x00, 0x00, 0x00 }, /* К to K */ + { 0x041B, 0x4c, 0x00, 0x00, 0x00 }, /* Л to L */ + { 0x041C, 0x4d, 0x00, 0x00, 0x00 }, /* М to M */ + { 0x041D, 0x4e, 0x00, 0x00, 0x00 }, /* Н to N */ + { 0x041E, 0x4f, 0x00, 0x00, 0x00 }, /* О to O */ + { 0x041F, 0x50, 0x00, 0x00, 0x00 }, /* П to P */ + { 0x0420, 0x52, 0x00, 0x00, 0x00 }, /* Р to R */ + { 0x0421, 0x53, 0x00, 0x00, 0x00 }, /* С to S */ + { 0x0422, 0x54, 0x00, 0x00, 0x00 }, /* Т to T */ + { 0x0423, 0x55, 0x00, 0x00, 0x00 }, /* У to U */ + { 0x0424, 0x46, 0x00, 0x00, 0x00 }, /* Ф to F */ + { 0x0425, 0x4b, 0x68, 0x00, 0x00 }, /* Х to Kh */ + { 0x0426, 0x54, 0x63, 0x00, 0x00 }, /* Ц to Tc */ + { 0x0427, 0x43, 0x68, 0x00, 0x00 }, /* Ч to Ch */ + { 0x0428, 0x53, 0x68, 0x00, 0x00 }, /* Ш to Sh */ + { 0x0429, 0x53, 0x68, 0x63, 0x68 }, /* Щ to Shch */ + { 0x042A, 0x61, 0x00, 0x00, 0x00 }, /* to A */ + { 0x042B, 0x59, 0x00, 0x00, 0x00 }, /* Ы to Y */ + { 0x042C, 0x59, 0x00, 0x00, 0x00 }, /* to Y */ + { 0x042D, 0x45, 0x00, 0x00, 0x00 }, /* Э to E */ + { 0x042E, 0x49, 0x75, 0x00, 0x00 }, /* Ю to Iu */ + { 0x042F, 0x49, 0x61, 0x00, 0x00 }, /* Я to Ia */ + { 0x0430, 0x61, 0x00, 0x00, 0x00 }, /* а to a */ + { 0x0431, 0x62, 0x00, 0x00, 0x00 }, /* б to b */ + { 0x0432, 0x76, 0x00, 0x00, 0x00 }, /* в to v */ + { 0x0433, 0x67, 0x00, 0x00, 0x00 }, /* г to g */ + { 0x0434, 0x64, 0x00, 0x00, 0x00 }, /* д to d */ + { 0x0435, 0x65, 0x00, 0x00, 0x00 }, /* е to e */ + { 0x0436, 0x7a, 0x68, 0x00, 0x00 }, /* ж to zh */ + { 0x0437, 0x7a, 0x00, 0x00, 0x00 }, /* з to z */ + { 0x0438, 0x69, 0x00, 0x00, 0x00 }, /* и to i */ + { 0x0439, 0x69, 0x00, 0x00, 0x00 }, /* й to i */ + { 0x043A, 0x6b, 0x00, 0x00, 0x00 }, /* к to k */ + { 0x043B, 0x6c, 0x00, 0x00, 0x00 }, /* л to l */ + { 0x043C, 0x6d, 0x00, 0x00, 0x00 }, /* м to m */ + { 0x043D, 0x6e, 0x00, 0x00, 0x00 }, /* н to n */ + { 0x043E, 0x6f, 0x00, 0x00, 0x00 }, /* о to o */ + { 0x043F, 0x70, 0x00, 0x00, 0x00 }, /* п to p */ + { 0x0440, 0x72, 0x00, 0x00, 0x00 }, /* р to r */ + { 0x0441, 0x73, 0x00, 0x00, 0x00 }, /* с to s */ + { 0x0442, 0x74, 0x00, 0x00, 0x00 }, /* т to t */ + { 0x0443, 0x75, 0x00, 0x00, 0x00 }, /* у to u */ + { 0x0444, 0x66, 0x00, 0x00, 0x00 }, /* ф to f */ + { 0x0445, 0x6b, 0x68, 0x00, 0x00 }, /* х to kh */ + { 0x0446, 0x74, 0x63, 0x00, 0x00 }, /* ц to tc */ + { 0x0447, 0x63, 0x68, 0x00, 0x00 }, /* ч to ch */ + { 0x0448, 0x73, 0x68, 0x00, 0x00 }, /* ш to sh */ + { 0x0449, 0x73, 0x68, 0x63, 0x68 }, /* щ to shch */ + { 0x044A, 0x61, 0x00, 0x00, 0x00 }, /* to a */ + { 0x044B, 0x79, 0x00, 0x00, 0x00 }, /* ы to y */ + { 0x044C, 0x79, 0x00, 0x00, 0x00 }, /* to y */ + { 0x044D, 0x65, 0x00, 0x00, 0x00 }, /* э to e */ + { 0x044E, 0x69, 0x75, 0x00, 0x00 }, /* ю to iu */ + { 0x044F, 0x69, 0x61, 0x00, 0x00 }, /* я to ia */ + { 0x0450, 0x65, 0x00, 0x00, 0x00 }, /* ѐ to e */ + { 0x0451, 0x65, 0x00, 0x00, 0x00 }, /* ё to e */ + { 0x0452, 0x64, 0x00, 0x00, 0x00 }, /* ђ to d */ + { 0x0453, 0x67, 0x00, 0x00, 0x00 }, /* ѓ to g */ + { 0x0454, 0x65, 0x00, 0x00, 0x00 }, /* є to e */ + { 0x0455, 0x7a, 0x00, 0x00, 0x00 }, /* ѕ to z */ + { 0x0456, 0x69, 0x00, 0x00, 0x00 }, /* і to i */ + { 0x0457, 0x69, 0x00, 0x00, 0x00 }, /* ї to i */ + { 0x0458, 0x6a, 0x00, 0x00, 0x00 }, /* ј to j */ + { 0x0459, 0x69, 0x00, 0x00, 0x00 }, /* љ to i */ + { 0x045A, 0x6e, 0x00, 0x00, 0x00 }, /* њ to n */ + { 0x045B, 0x64, 0x00, 0x00, 0x00 }, /* ћ to d */ + { 0x045C, 0x6b, 0x00, 0x00, 0x00 }, /* ќ to k */ + { 0x045D, 0x69, 0x00, 0x00, 0x00 }, /* ѝ to i */ + { 0x045E, 0x75, 0x00, 0x00, 0x00 }, /* ў to u */ + { 0x045F, 0x64, 0x00, 0x00, 0x00 }, /* џ to d */ + { 0x1E02, 0x42, 0x00, 0x00, 0x00 }, /* Ḃ to B */ + { 0x1E03, 0x62, 0x00, 0x00, 0x00 }, /* ḃ to b */ + { 0x1E0A, 0x44, 0x00, 0x00, 0x00 }, /* Ḋ to D */ + { 0x1E0B, 0x64, 0x00, 0x00, 0x00 }, /* ḋ to d */ + { 0x1E1E, 0x46, 0x00, 0x00, 0x00 }, /* Ḟ to F */ + { 0x1E1F, 0x66, 0x00, 0x00, 0x00 }, /* ḟ to f */ + { 0x1E40, 0x4D, 0x00, 0x00, 0x00 }, /* Ṁ to M */ + { 0x1E41, 0x6D, 0x00, 0x00, 0x00 }, /* ṁ to m */ + { 0x1E56, 0x50, 0x00, 0x00, 0x00 }, /* Ṗ to P */ + { 0x1E57, 0x70, 0x00, 0x00, 0x00 }, /* ṗ to p */ + { 0x1E60, 0x53, 0x00, 0x00, 0x00 }, /* Ṡ to S */ + { 0x1E61, 0x73, 0x00, 0x00, 0x00 }, /* ṡ to s */ + { 0x1E6A, 0x54, 0x00, 0x00, 0x00 }, /* Ṫ to T */ + { 0x1E6B, 0x74, 0x00, 0x00, 0x00 }, /* ṫ to t */ + { 0x1E80, 0x57, 0x00, 0x00, 0x00 }, /* Ẁ to W */ + { 0x1E81, 0x77, 0x00, 0x00, 0x00 }, /* ẁ to w */ + { 0x1E82, 0x57, 0x00, 0x00, 0x00 }, /* Ẃ to W */ + { 0x1E83, 0x77, 0x00, 0x00, 0x00 }, /* ẃ to w */ + { 0x1E84, 0x57, 0x00, 0x00, 0x00 }, /* Ẅ to W */ + { 0x1E85, 0x77, 0x00, 0x00, 0x00 }, /* ẅ to w */ + { 0x1EF2, 0x59, 0x00, 0x00, 0x00 }, /* Ỳ to Y */ + { 0x1EF3, 0x79, 0x00, 0x00, 0x00 }, /* ỳ to y */ + { 0xFB00, 0x66, 0x66, 0x00, 0x00 }, /* ff to ff */ + { 0xFB01, 0x66, 0x69, 0x00, 0x00 }, /* fi to fi */ + { 0xFB02, 0x66, 0x6C, 0x00, 0x00 }, /* fl to fl */ + { 0xFB05, 0x73, 0x74, 0x00, 0x00 }, /* ſt to st */ + { 0xFB06, 0x73, 0x74, 0x00, 0x00 }, /* st to st */ +}; + +static const Transliteration *spellfixFindTranslit(int c, int *pxTop){ + *pxTop = (sizeof(translit)/sizeof(translit[0])) - 1; + return translit; +} + +/* +** Convert the input string from UTF-8 into pure ASCII by converting +** all non-ASCII characters to some combination of characters in the +** ASCII subset. +** +** The returned string might contain more characters than the input. +** +** Space to hold the returned string comes from sqlite3_malloc() and +** should be freed by the caller. +*/ +static unsigned char *transliterate(const unsigned char *zIn, int nIn){ +#ifdef SQLITE_SPELLFIX_5BYTE_MAPPINGS + unsigned char *zOut = sqlite3_malloc64( nIn*5 + 1 ); +#else + unsigned char *zOut = sqlite3_malloc64( nIn*4 + 1 ); +#endif + int c, sz, nOut; + if( zOut==0 ) return 0; + nOut = 0; + while( nIn>0 ){ + c = utf8Read(zIn, nIn, &sz); + zIn += sz; + nIn -= sz; + if( c<=127 ){ + zOut[nOut++] = (unsigned char)c; + }else{ + int xTop, xBtm, x; + const Transliteration *tbl = spellfixFindTranslit(c, &xTop); + xBtm = 0; + while( xTop>=xBtm ){ + x = (xTop + xBtm)/2; + if( tbl[x].cFrom==c ){ + zOut[nOut++] = tbl[x].cTo0; + if( tbl[x].cTo1 ){ + zOut[nOut++] = tbl[x].cTo1; + if( tbl[x].cTo2 ){ + zOut[nOut++] = tbl[x].cTo2; + if( tbl[x].cTo3 ){ + zOut[nOut++] = tbl[x].cTo3; +#ifdef SQLITE_SPELLFIX_5BYTE_MAPPINGS + if( tbl[x].cTo4 ){ + zOut[nOut++] = tbl[x].cTo4; + } +#endif /* SQLITE_SPELLFIX_5BYTE_MAPPINGS */ + } + } + } + c = 0; + break; + }else if( tbl[x].cFrom>c ){ + xTop = x-1; + }else{ + xBtm = x+1; + } + } + if( c ) zOut[nOut++] = '?'; + } + } + zOut[nOut] = 0; + return zOut; +} + +/* +** Return the number of characters in the shortest prefix of the input +** string that transliterates to an ASCII string nTrans bytes or longer. +** Or, if the transliteration of the input string is less than nTrans +** bytes in size, return the number of characters in the input string. +*/ +static int translen_to_charlen(const char *zIn, int nIn, int nTrans){ + int i, c, sz, nOut; + int nChar; + + i = nOut = 0; + for(nChar=0; i<nIn && nOut<nTrans; nChar++){ + c = utf8Read((const unsigned char *)&zIn[i], nIn-i, &sz); + i += sz; + + nOut++; + if( c>=128 ){ + int xTop, xBtm, x; + const Transliteration *tbl = spellfixFindTranslit(c, &xTop); + xBtm = 0; + while( xTop>=xBtm ){ + x = (xTop + xBtm)/2; + if( tbl[x].cFrom==c ){ + if( tbl[x].cTo1 ){ + nOut++; + if( tbl[x].cTo2 ){ + nOut++; + if( tbl[x].cTo3 ){ + nOut++; + } + } + } + break; + }else if( tbl[x].cFrom>c ){ + xTop = x-1; + }else{ + xBtm = x+1; + } + } + } + } + + return nChar; +} + + +/* +** spellfix1_translit(X) +** +** Convert a string that contains non-ASCII Roman characters into +** pure ASCII. +*/ +static void transliterateSqlFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *zIn = sqlite3_value_text(argv[0]); + int nIn = sqlite3_value_bytes(argv[0]); + unsigned char *zOut = transliterate(zIn, nIn); + if( zOut==0 ){ + sqlite3_result_error_nomem(context); + }else{ + sqlite3_result_text(context, (char*)zOut, -1, sqlite3_free); + } +} + +/* +** spellfix1_scriptcode(X) +** +** Try to determine the dominant script used by the word X and return +** its ISO 15924 numeric code. +** +** The current implementation only understands the following scripts: +** +** 215 (Latin) +** 220 (Cyrillic) +** 200 (Greek) +** +** This routine will return 998 if the input X contains characters from +** two or more of the above scripts or 999 if X contains no characters +** from any of the above scripts. +*/ +static void scriptCodeSqlFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const unsigned char *zIn = sqlite3_value_text(argv[0]); + int nIn = sqlite3_value_bytes(argv[0]); + int c, sz; + int scriptMask = 0; + int res; + int seenDigit = 0; +# define SCRIPT_LATIN 0x0001 +# define SCRIPT_CYRILLIC 0x0002 +# define SCRIPT_GREEK 0x0004 +# define SCRIPT_HEBREW 0x0008 +# define SCRIPT_ARABIC 0x0010 + + while( nIn>0 ){ + c = utf8Read(zIn, nIn, &sz); + zIn += sz; + nIn -= sz; + if( c<0x02af ){ + if( c>=0x80 || midClass[c&0x7f]<CCLASS_DIGIT ){ + scriptMask |= SCRIPT_LATIN; + }else if( c>='0' && c<='9' ){ + seenDigit = 1; + } + }else if( c>=0x0400 && c<=0x04ff ){ + scriptMask |= SCRIPT_CYRILLIC; + }else if( c>=0x0386 && c<=0x03ce ){ + scriptMask |= SCRIPT_GREEK; + }else if( c>=0x0590 && c<=0x05ff ){ + scriptMask |= SCRIPT_HEBREW; + }else if( c>=0x0600 && c<=0x06ff ){ + scriptMask |= SCRIPT_ARABIC; + } + } + if( scriptMask==0 && seenDigit ) scriptMask = SCRIPT_LATIN; + switch( scriptMask ){ + case 0: res = 999; break; + case SCRIPT_LATIN: res = 215; break; + case SCRIPT_CYRILLIC: res = 220; break; + case SCRIPT_GREEK: res = 200; break; + case SCRIPT_HEBREW: res = 125; break; + case SCRIPT_ARABIC: res = 160; break; + default: res = 998; break; + } + sqlite3_result_int(context, res); +} + +/* End transliterate +****************************************************************************** +****************************************************************************** +** Begin spellfix1 virtual table. +*/ + +/* Maximum length of a phonehash used for querying the shadow table */ +#define SPELLFIX_MX_HASH 32 + +/* Maximum number of hash strings to examine per query */ +#define SPELLFIX_MX_RUN 1 + +typedef struct spellfix1_vtab spellfix1_vtab; +typedef struct spellfix1_cursor spellfix1_cursor; + +/* Fuzzy-search virtual table object */ +struct spellfix1_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database connection */ + char *zDbName; /* Name of database holding this table */ + char *zTableName; /* Name of the virtual table */ + char *zCostTable; /* Table holding edit-distance cost numbers */ + EditDist3Config *pConfig3; /* Parsed edit distance costs */ +}; + +/* Fuzzy-search cursor object */ +struct spellfix1_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + spellfix1_vtab *pVTab; /* The table to which this cursor belongs */ + char *zPattern; /* rhs of MATCH clause */ + int idxNum; /* idxNum value passed to xFilter() */ + int nRow; /* Number of rows of content */ + int nAlloc; /* Number of allocated rows */ + int iRow; /* Current row of content */ + int iLang; /* Value of the langid= constraint */ + int iTop; /* Value of the top= constraint */ + int iScope; /* Value of the scope= constraint */ + int nSearch; /* Number of vocabulary items checked */ + sqlite3_stmt *pFullScan; /* Shadow query for a full table scan */ + struct spellfix1_row { /* For each row of content */ + sqlite3_int64 iRowid; /* Rowid for this row */ + char *zWord; /* Text for this row */ + int iRank; /* Rank for this row */ + int iDistance; /* Distance from pattern for this row */ + int iScore; /* Score for sorting */ + int iMatchlen; /* Value of matchlen column (or -1) */ + char zHash[SPELLFIX_MX_HASH]; /* the phonehash used for this match */ + } *a; +}; + +/* +** Construct one or more SQL statements from the format string given +** and then evaluate those statements. The success code is written +** into *pRc. +** +** If *pRc is initially non-zero then this routine is a no-op. +*/ +static void spellfix1DbExec( + int *pRc, /* Success code */ + sqlite3 *db, /* Database in which to run SQL */ + const char *zFormat, /* Format string for SQL */ + ... /* Arguments to the format string */ +){ + va_list ap; + char *zSql; + if( *pRc ) return; + va_start(ap, zFormat); + zSql = sqlite3_vmprintf(zFormat, ap); + va_end(ap); + if( zSql==0 ){ + *pRc = SQLITE_NOMEM; + }else{ + *pRc = sqlite3_exec(db, zSql, 0, 0, 0); + sqlite3_free(zSql); + } +} + +/* +** xDisconnect/xDestroy method for the fuzzy-search module. +*/ +static int spellfix1Uninit(int isDestroy, sqlite3_vtab *pVTab){ + spellfix1_vtab *p = (spellfix1_vtab*)pVTab; + int rc = SQLITE_OK; + if( isDestroy ){ + sqlite3 *db = p->db; + spellfix1DbExec(&rc, db, "DROP TABLE IF EXISTS \"%w\".\"%w_vocab\"", + p->zDbName, p->zTableName); + } + if( rc==SQLITE_OK ){ + sqlite3_free(p->zTableName); + editDist3ConfigDelete(p->pConfig3); + sqlite3_free(p->zCostTable); + sqlite3_free(p); + } + return rc; +} +static int spellfix1Disconnect(sqlite3_vtab *pVTab){ + return spellfix1Uninit(0, pVTab); +} +static int spellfix1Destroy(sqlite3_vtab *pVTab){ + return spellfix1Uninit(1, pVTab); +} + +/* +** Make a copy of a string. Remove leading and trailing whitespace +** and dequote it. +*/ +static char *spellfix1Dequote(const char *zIn){ + char *zOut; + int i, j; + char c; + while( isspace((unsigned char)zIn[0]) ) zIn++; + zOut = sqlite3_mprintf("%s", zIn); + if( zOut==0 ) return 0; + i = (int)strlen(zOut); +#if 0 /* The parser will never leave spaces at the end */ + while( i>0 && isspace(zOut[i-1]) ){ i--; } +#endif + zOut[i] = 0; + c = zOut[0]; + if( c=='\'' || c=='"' ){ + for(i=1, j=0; ALWAYS(zOut[i]); i++){ + zOut[j++] = zOut[i]; + if( zOut[i]==c ){ + if( zOut[i+1]==c ){ + i++; + }else{ + zOut[j-1] = 0; + break; + } + } + } + } + return zOut; +} + + +/* +** xConnect/xCreate method for the spellfix1 module. Arguments are: +** +** argv[0] -> module name ("spellfix1") +** argv[1] -> database name +** argv[2] -> table name +** argv[3].. -> optional arguments (i.e. "edit_cost_table" parameter) +*/ +static int spellfix1Init( + int isCreate, + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVTab, + char **pzErr +){ + spellfix1_vtab *pNew = 0; + /* const char *zModule = argv[0]; // not used */ + const char *zDbName = argv[1]; + const char *zTableName = argv[2]; + int nDbName; + int rc = SQLITE_OK; + int i; + + nDbName = (int)strlen(zDbName); + pNew = sqlite3_malloc64( sizeof(*pNew) + nDbName + 1); + if( pNew==0 ){ + rc = SQLITE_NOMEM; + }else{ + memset(pNew, 0, sizeof(*pNew)); + pNew->zDbName = (char*)&pNew[1]; + memcpy(pNew->zDbName, zDbName, nDbName+1); + pNew->zTableName = sqlite3_mprintf("%s", zTableName); + pNew->db = db; + if( pNew->zTableName==0 ){ + rc = SQLITE_NOMEM; + }else{ + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(word,rank,distance,langid, " + "score, matchlen, phonehash HIDDEN, " + "top HIDDEN, scope HIDDEN, srchcnt HIDDEN, " + "soundslike HIDDEN, command HIDDEN)" + ); +#define SPELLFIX_COL_WORD 0 +#define SPELLFIX_COL_RANK 1 +#define SPELLFIX_COL_DISTANCE 2 +#define SPELLFIX_COL_LANGID 3 +#define SPELLFIX_COL_SCORE 4 +#define SPELLFIX_COL_MATCHLEN 5 +#define SPELLFIX_COL_PHONEHASH 6 +#define SPELLFIX_COL_TOP 7 +#define SPELLFIX_COL_SCOPE 8 +#define SPELLFIX_COL_SRCHCNT 9 +#define SPELLFIX_COL_SOUNDSLIKE 10 +#define SPELLFIX_COL_COMMAND 11 + } + if( rc==SQLITE_OK && isCreate ){ + spellfix1DbExec(&rc, db, + "CREATE TABLE IF NOT EXISTS \"%w\".\"%w_vocab\"(\n" + " id INTEGER PRIMARY KEY,\n" + " rank INT,\n" + " langid INT,\n" + " word TEXT,\n" + " k1 TEXT,\n" + " k2 TEXT\n" + ");\n", + zDbName, zTableName + ); + spellfix1DbExec(&rc, db, + "CREATE INDEX IF NOT EXISTS \"%w\".\"%w_vocab_index_langid_k2\" " + "ON \"%w_vocab\"(langid,k2);", + zDbName, zTableName, zTableName + ); + } + for(i=3; rc==SQLITE_OK && i<argc; i++){ + if( strncmp(argv[i],"edit_cost_table=",16)==0 && pNew->zCostTable==0 ){ + pNew->zCostTable = spellfix1Dequote(&argv[i][16]); + if( pNew->zCostTable==0 ) rc = SQLITE_NOMEM; + continue; + } + *pzErr = sqlite3_mprintf("bad argument to spellfix1(): \"%s\"", argv[i]); + rc = SQLITE_ERROR; + } + } + + if( rc && pNew ){ + *ppVTab = 0; + spellfix1Uninit(0, &pNew->base); + }else{ + *ppVTab = (sqlite3_vtab *)pNew; + } + return rc; +} + +/* +** The xConnect and xCreate methods +*/ +static int spellfix1Connect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVTab, + char **pzErr +){ + return spellfix1Init(0, db, pAux, argc, argv, ppVTab, pzErr); +} +static int spellfix1Create( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVTab, + char **pzErr +){ + return spellfix1Init(1, db, pAux, argc, argv, ppVTab, pzErr); +} + +/* +** Clear all of the content from a cursor. +*/ +static void spellfix1ResetCursor(spellfix1_cursor *pCur){ + int i; + for(i=0; i<pCur->nRow; i++){ + sqlite3_free(pCur->a[i].zWord); + } + pCur->nRow = 0; + pCur->iRow = 0; + pCur->nSearch = 0; + if( pCur->pFullScan ){ + sqlite3_finalize(pCur->pFullScan); + pCur->pFullScan = 0; + } +} + +/* +** Resize the cursor to hold up to N rows of content +*/ +static void spellfix1ResizeCursor(spellfix1_cursor *pCur, int N){ + struct spellfix1_row *aNew; + assert( N>=pCur->nRow ); + aNew = sqlite3_realloc64(pCur->a, sizeof(pCur->a[0])*N); + if( aNew==0 && N>0 ){ + spellfix1ResetCursor(pCur); + sqlite3_free(pCur->a); + pCur->nAlloc = 0; + pCur->a = 0; + }else{ + pCur->nAlloc = N; + pCur->a = aNew; + } +} + + +/* +** Close a fuzzy-search cursor. +*/ +static int spellfix1Close(sqlite3_vtab_cursor *cur){ + spellfix1_cursor *pCur = (spellfix1_cursor *)cur; + spellfix1ResetCursor(pCur); + spellfix1ResizeCursor(pCur, 0); + sqlite3_free(pCur->zPattern); + sqlite3_free(pCur); + return SQLITE_OK; +} + +#define SPELLFIX_IDXNUM_MATCH 0x01 /* word MATCH $str */ +#define SPELLFIX_IDXNUM_LANGID 0x02 /* langid == $langid */ +#define SPELLFIX_IDXNUM_TOP 0x04 /* top = $top */ +#define SPELLFIX_IDXNUM_SCOPE 0x08 /* scope = $scope */ +#define SPELLFIX_IDXNUM_DISTLT 0x10 /* distance < $distance */ +#define SPELLFIX_IDXNUM_DISTLE 0x20 /* distance <= $distance */ +#define SPELLFIX_IDXNUM_ROWID 0x40 /* rowid = $rowid */ +#define SPELLFIX_IDXNUM_DIST (0x10|0x20) /* DISTLT and DISTLE */ + +/* +** +** The plan number is a bitmask of the SPELLFIX_IDXNUM_* values defined +** above. +** +** filter.argv[*] values contains $str, $langid, $top, $scope and $rowid +** if specified and in that order. +*/ +static int spellfix1BestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdxInfo){ + int iPlan = 0; + int iLangTerm = -1; + int iTopTerm = -1; + int iScopeTerm = -1; + int iDistTerm = -1; + int iRowidTerm = -1; + int i; + const struct sqlite3_index_constraint *pConstraint; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + + /* Terms of the form: word MATCH $str */ + if( (iPlan & SPELLFIX_IDXNUM_MATCH)==0 + && pConstraint->iColumn==SPELLFIX_COL_WORD + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_MATCH + ){ + iPlan |= SPELLFIX_IDXNUM_MATCH; + pIdxInfo->aConstraintUsage[i].argvIndex = 1; + pIdxInfo->aConstraintUsage[i].omit = 1; + } + + /* Terms of the form: langid = $langid */ + if( (iPlan & SPELLFIX_IDXNUM_LANGID)==0 + && pConstraint->iColumn==SPELLFIX_COL_LANGID + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= SPELLFIX_IDXNUM_LANGID; + iLangTerm = i; + } + + /* Terms of the form: top = $top */ + if( (iPlan & SPELLFIX_IDXNUM_TOP)==0 + && pConstraint->iColumn==SPELLFIX_COL_TOP + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= SPELLFIX_IDXNUM_TOP; + iTopTerm = i; + } + + /* Terms of the form: scope = $scope */ + if( (iPlan & SPELLFIX_IDXNUM_SCOPE)==0 + && pConstraint->iColumn==SPELLFIX_COL_SCOPE + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= SPELLFIX_IDXNUM_SCOPE; + iScopeTerm = i; + } + + /* Terms of the form: distance < $dist or distance <= $dist */ + if( (iPlan & SPELLFIX_IDXNUM_DIST)==0 + && pConstraint->iColumn==SPELLFIX_COL_DISTANCE + && (pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT + || pConstraint->op==SQLITE_INDEX_CONSTRAINT_LE) + ){ + if( pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT ){ + iPlan |= SPELLFIX_IDXNUM_DISTLT; + }else{ + iPlan |= SPELLFIX_IDXNUM_DISTLE; + } + iDistTerm = i; + } + + /* Terms of the form: distance < $dist or distance <= $dist */ + if( (iPlan & SPELLFIX_IDXNUM_ROWID)==0 + && pConstraint->iColumn<0 + && pConstraint->op==SQLITE_INDEX_CONSTRAINT_EQ + ){ + iPlan |= SPELLFIX_IDXNUM_ROWID; + iRowidTerm = i; + } + } + if( iPlan&SPELLFIX_IDXNUM_MATCH ){ + int idx = 2; + pIdxInfo->idxNum = iPlan; + if( pIdxInfo->nOrderBy==1 + && pIdxInfo->aOrderBy[0].iColumn==SPELLFIX_COL_SCORE + && pIdxInfo->aOrderBy[0].desc==0 + ){ + pIdxInfo->orderByConsumed = 1; /* Default order by iScore */ + } + if( iPlan&SPELLFIX_IDXNUM_LANGID ){ + pIdxInfo->aConstraintUsage[iLangTerm].argvIndex = idx++; + pIdxInfo->aConstraintUsage[iLangTerm].omit = 1; + } + if( iPlan&SPELLFIX_IDXNUM_TOP ){ + pIdxInfo->aConstraintUsage[iTopTerm].argvIndex = idx++; + pIdxInfo->aConstraintUsage[iTopTerm].omit = 1; + } + if( iPlan&SPELLFIX_IDXNUM_SCOPE ){ + pIdxInfo->aConstraintUsage[iScopeTerm].argvIndex = idx++; + pIdxInfo->aConstraintUsage[iScopeTerm].omit = 1; + } + if( iPlan&SPELLFIX_IDXNUM_DIST ){ + pIdxInfo->aConstraintUsage[iDistTerm].argvIndex = idx++; + pIdxInfo->aConstraintUsage[iDistTerm].omit = 1; + } + pIdxInfo->estimatedCost = 1e5; + }else if( (iPlan & SPELLFIX_IDXNUM_ROWID) ){ + pIdxInfo->idxNum = SPELLFIX_IDXNUM_ROWID; + pIdxInfo->aConstraintUsage[iRowidTerm].argvIndex = 1; + pIdxInfo->aConstraintUsage[iRowidTerm].omit = 1; + pIdxInfo->estimatedCost = 5; + }else{ + pIdxInfo->idxNum = 0; + pIdxInfo->estimatedCost = 1e50; + } + return SQLITE_OK; +} + +/* +** Open a new fuzzy-search cursor. +*/ +static int spellfix1Open(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor){ + spellfix1_vtab *p = (spellfix1_vtab*)pVTab; + spellfix1_cursor *pCur; + pCur = sqlite3_malloc64( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->pVTab = p; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Adjust a distance measurement by the words rank in order to show +** preference to common words. +*/ +static int spellfix1Score(int iDistance, int iRank){ + int iLog2; + for(iLog2=0; iRank>0; iLog2++, iRank>>=1){} + return iDistance + 32 - iLog2; +} + +/* +** Compare two spellfix1_row objects for sorting purposes in qsort() such +** that they sort in order of increasing distance. +*/ +static int SQLITE_CDECL spellfix1RowCompare(const void *A, const void *B){ + const struct spellfix1_row *a = (const struct spellfix1_row*)A; + const struct spellfix1_row *b = (const struct spellfix1_row*)B; + return a->iScore - b->iScore; +} + +/* +** A structure used to pass information from spellfix1FilterForMatch() +** into spellfix1RunQuery(). +*/ +typedef struct MatchQuery { + spellfix1_cursor *pCur; /* The cursor being queried */ + sqlite3_stmt *pStmt; /* shadow table query statment */ + char zHash[SPELLFIX_MX_HASH]; /* The current phonehash for zPattern */ + const char *zPattern; /* Transliterated input string */ + int nPattern; /* Length of zPattern */ + EditDist3FromString *pMatchStr3; /* Original unicode string */ + EditDist3Config *pConfig3; /* Edit-distance cost coefficients */ + const EditDist3Lang *pLang; /* The selected language coefficients */ + int iLang; /* The language id */ + int iScope; /* Default scope */ + int iMaxDist; /* Maximum allowed edit distance, or -1 */ + int rc; /* Error code */ + int nRun; /* Number of prior runs for the same zPattern */ + char azPrior[SPELLFIX_MX_RUN][SPELLFIX_MX_HASH]; /* Prior hashes */ +} MatchQuery; + +/* +** Run a query looking for the best matches against zPattern using +** zHash as the character class seed hash. +*/ +static void spellfix1RunQuery(MatchQuery *p, const char *zQuery, int nQuery){ + const char *zK1; + const char *zWord; + int iDist; + int iRank; + int iScore; + int iWorst = 0; + int idx; + int idxWorst = -1; + int i; + int iScope = p->iScope; + spellfix1_cursor *pCur = p->pCur; + sqlite3_stmt *pStmt = p->pStmt; + char zHash1[SPELLFIX_MX_HASH]; + char zHash2[SPELLFIX_MX_HASH]; + char *zClass; + int nClass; + int rc; + + if( pCur->a==0 || p->rc ) return; /* Prior memory allocation failure */ + zClass = (char*)phoneticHash((unsigned char*)zQuery, nQuery); + if( zClass==0 ){ + p->rc = SQLITE_NOMEM; + return; + } + nClass = (int)strlen(zClass); + if( nClass>SPELLFIX_MX_HASH-2 ){ + nClass = SPELLFIX_MX_HASH-2; + zClass[nClass] = 0; + } + if( nClass<=iScope ){ + if( nClass>2 ){ + iScope = nClass-1; + }else{ + iScope = nClass; + } + } + memcpy(zHash1, zClass, iScope); + sqlite3_free(zClass); + zHash1[iScope] = 0; + memcpy(zHash2, zHash1, iScope); + zHash2[iScope] = 'Z'; + zHash2[iScope+1] = 0; +#if SPELLFIX_MX_RUN>1 + for(i=0; i<p->nRun; i++){ + if( strcmp(p->azPrior[i], zHash1)==0 ) return; + } +#endif + assert( p->nRun<SPELLFIX_MX_RUN ); + memcpy(p->azPrior[p->nRun++], zHash1, iScope+1); + if( sqlite3_bind_text(pStmt, 1, zHash1, -1, SQLITE_STATIC)==SQLITE_NOMEM + || sqlite3_bind_text(pStmt, 2, zHash2, -1, SQLITE_STATIC)==SQLITE_NOMEM + ){ + p->rc = SQLITE_NOMEM; + return; + } +#if SPELLFIX_MX_RUN>1 + for(i=0; i<pCur->nRow; i++){ + if( pCur->a[i].iScore>iWorst ){ + iWorst = pCur->a[i].iScore; + idxWorst = i; + } + } +#endif + while( sqlite3_step(pStmt)==SQLITE_ROW ){ + int iMatchlen = -1; + iRank = sqlite3_column_int(pStmt, 2); + if( p->pMatchStr3 ){ + int nWord = sqlite3_column_bytes(pStmt, 1); + zWord = (const char*)sqlite3_column_text(pStmt, 1); + iDist = editDist3Core(p->pMatchStr3, zWord, nWord, p->pLang, &iMatchlen); + }else{ + zK1 = (const char*)sqlite3_column_text(pStmt, 3); + if( zK1==0 ) continue; + iDist = editdist1(p->zPattern, zK1, 0); + } + if( iDist<0 ){ + p->rc = SQLITE_NOMEM; + break; + } + pCur->nSearch++; + + /* If there is a "distance < $dist" or "distance <= $dist" constraint, + ** check if this row meets it. If not, jump back up to the top of the + ** loop to process the next row. Otherwise, if the row does match the + ** distance constraint, check if the pCur->a[] array is already full. + ** If it is and no explicit "top = ?" constraint was present in the + ** query, grow the array to ensure there is room for the new entry. */ + assert( (p->iMaxDist>=0)==((pCur->idxNum & SPELLFIX_IDXNUM_DIST) ? 1 : 0) ); + if( p->iMaxDist>=0 ){ + if( iDist>p->iMaxDist ) continue; + if( pCur->nRow>=pCur->nAlloc && (pCur->idxNum & SPELLFIX_IDXNUM_TOP)==0 ){ + spellfix1ResizeCursor(pCur, pCur->nAlloc*2 + 10); + if( pCur->a==0 ) break; + } + } + + iScore = spellfix1Score(iDist,iRank); + if( pCur->nRow<pCur->nAlloc ){ + idx = pCur->nRow; + }else if( iScore<iWorst ){ + idx = idxWorst; + sqlite3_free(pCur->a[idx].zWord); + }else{ + continue; + } + + pCur->a[idx].zWord = sqlite3_mprintf("%s", sqlite3_column_text(pStmt, 1)); + if( pCur->a[idx].zWord==0 ){ + p->rc = SQLITE_NOMEM; + break; + } + pCur->a[idx].iRowid = sqlite3_column_int64(pStmt, 0); + pCur->a[idx].iRank = iRank; + pCur->a[idx].iDistance = iDist; + pCur->a[idx].iScore = iScore; + pCur->a[idx].iMatchlen = iMatchlen; + memcpy(pCur->a[idx].zHash, zHash1, iScope+1); + if( pCur->nRow<pCur->nAlloc ) pCur->nRow++; + if( pCur->nRow==pCur->nAlloc ){ + iWorst = pCur->a[0].iScore; + idxWorst = 0; + for(i=1; i<pCur->nRow; i++){ + iScore = pCur->a[i].iScore; + if( iWorst<iScore ){ + iWorst = iScore; + idxWorst = i; + } + } + } + } + rc = sqlite3_reset(pStmt); + if( rc ) p->rc = rc; +} + +/* +** This version of the xFilter method work if the MATCH term is present +** and we are doing a scan. +*/ +static int spellfix1FilterForMatch( + spellfix1_cursor *pCur, + int argc, + sqlite3_value **argv +){ + int idxNum = pCur->idxNum; + const unsigned char *zMatchThis; /* RHS of the MATCH operator */ + EditDist3FromString *pMatchStr3 = 0; /* zMatchThis as an editdist string */ + char *zPattern; /* Transliteration of zMatchThis */ + int nPattern; /* Length of zPattern */ + int iLimit = 20; /* Max number of rows of output */ + int iScope = 3; /* Use this many characters of zClass */ + int iLang = 0; /* Language code */ + char *zSql; /* SQL of shadow table query */ + sqlite3_stmt *pStmt = 0; /* Shadow table query */ + int rc; /* Result code */ + int idx = 1; /* Next available filter parameter */ + spellfix1_vtab *p = pCur->pVTab; /* The virtual table that owns pCur */ + MatchQuery x; /* For passing info to RunQuery() */ + + /* Load the cost table if we have not already done so */ + if( p->zCostTable!=0 && p->pConfig3==0 ){ + p->pConfig3 = sqlite3_malloc64( sizeof(p->pConfig3[0]) ); + if( p->pConfig3==0 ) return SQLITE_NOMEM; + memset(p->pConfig3, 0, sizeof(p->pConfig3[0])); + rc = editDist3ConfigLoad(p->pConfig3, p->db, p->zCostTable); + if( rc ) return rc; + } + memset(&x, 0, sizeof(x)); + x.iScope = 3; /* Default scope if none specified by "WHERE scope=N" */ + x.iMaxDist = -1; /* Maximum allowed edit distance */ + + if( idxNum&2 ){ + iLang = sqlite3_value_int(argv[idx++]); + } + if( idxNum&4 ){ + iLimit = sqlite3_value_int(argv[idx++]); + if( iLimit<1 ) iLimit = 1; + } + if( idxNum&8 ){ + x.iScope = sqlite3_value_int(argv[idx++]); + if( x.iScope<1 ) x.iScope = 1; + if( x.iScope>SPELLFIX_MX_HASH-2 ) x.iScope = SPELLFIX_MX_HASH-2; + } + if( idxNum&(16|32) ){ + x.iMaxDist = sqlite3_value_int(argv[idx++]); + if( idxNum&16 ) x.iMaxDist--; + if( x.iMaxDist<0 ) x.iMaxDist = 0; + } + spellfix1ResetCursor(pCur); + spellfix1ResizeCursor(pCur, iLimit); + zMatchThis = sqlite3_value_text(argv[0]); + if( zMatchThis==0 ) return SQLITE_OK; + if( p->pConfig3 ){ + x.pLang = editDist3FindLang(p->pConfig3, iLang); + pMatchStr3 = editDist3FromStringNew(x.pLang, (const char*)zMatchThis, -1); + if( pMatchStr3==0 ){ + x.rc = SQLITE_NOMEM; + goto filter_exit; + } + }else{ + x.pLang = 0; + } + zPattern = (char*)transliterate(zMatchThis, sqlite3_value_bytes(argv[0])); + sqlite3_free(pCur->zPattern); + pCur->zPattern = zPattern; + if( zPattern==0 ){ + x.rc = SQLITE_NOMEM; + goto filter_exit; + } + nPattern = (int)strlen(zPattern); + if( zPattern[nPattern-1]=='*' ) nPattern--; + zSql = sqlite3_mprintf( + "SELECT id, word, rank, coalesce(k1,word)" + " FROM \"%w\".\"%w_vocab\"" + " WHERE langid=%d AND k2>=?1 AND k2<?2", + p->zDbName, p->zTableName, iLang + ); + if( zSql==0 ){ + x.rc = SQLITE_NOMEM; + pStmt = 0; + goto filter_exit; + } + rc = sqlite3_prepare_v2(p->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + pCur->iLang = iLang; + x.pCur = pCur; + x.pStmt = pStmt; + x.zPattern = zPattern; + x.nPattern = nPattern; + x.pMatchStr3 = pMatchStr3; + x.iLang = iLang; + x.rc = rc; + x.pConfig3 = p->pConfig3; + if( x.rc==SQLITE_OK ){ + spellfix1RunQuery(&x, zPattern, nPattern); + } + + if( pCur->a ){ + qsort(pCur->a, pCur->nRow, sizeof(pCur->a[0]), spellfix1RowCompare); + pCur->iTop = iLimit; + pCur->iScope = iScope; + }else{ + x.rc = SQLITE_NOMEM; + } + +filter_exit: + sqlite3_finalize(pStmt); + editDist3FromStringDelete(pMatchStr3); + return x.rc; +} + +/* +** This version of xFilter handles a full-table scan case +*/ +static int spellfix1FilterForFullScan( + spellfix1_cursor *pCur, + int argc, + sqlite3_value **argv +){ + int rc = SQLITE_OK; + int idxNum = pCur->idxNum; + char *zSql; + spellfix1_vtab *pVTab = pCur->pVTab; + spellfix1ResetCursor(pCur); + assert( idxNum==0 || idxNum==64 ); + zSql = sqlite3_mprintf( + "SELECT word, rank, NULL, langid, id FROM \"%w\".\"%w_vocab\"%s", + pVTab->zDbName, pVTab->zTableName, + ((idxNum & 64) ? " WHERE rowid=?" : "") + ); + if( zSql==0 ) return SQLITE_NOMEM; + rc = sqlite3_prepare_v2(pVTab->db, zSql, -1, &pCur->pFullScan, 0); + sqlite3_free(zSql); + if( rc==SQLITE_OK && (idxNum & 64) ){ + assert( argc==1 ); + rc = sqlite3_bind_value(pCur->pFullScan, 1, argv[0]); + } + pCur->nRow = pCur->iRow = 0; + if( rc==SQLITE_OK ){ + rc = sqlite3_step(pCur->pFullScan); + if( rc==SQLITE_ROW ){ pCur->iRow = -1; rc = SQLITE_OK; } + if( rc==SQLITE_DONE ){ rc = SQLITE_OK; } + }else{ + pCur->iRow = 0; + } + return rc; +} + + +/* +** Called to "rewind" a cursor back to the beginning so that +** it starts its output over again. Always called at least once +** prior to any spellfix1Column, spellfix1Rowid, or spellfix1Eof call. +*/ +static int spellfix1Filter( + sqlite3_vtab_cursor *cur, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + spellfix1_cursor *pCur = (spellfix1_cursor *)cur; + int rc; + pCur->idxNum = idxNum; + if( idxNum & 1 ){ + rc = spellfix1FilterForMatch(pCur, argc, argv); + }else{ + rc = spellfix1FilterForFullScan(pCur, argc, argv); + } + return rc; +} + + +/* +** Advance a cursor to its next row of output +*/ +static int spellfix1Next(sqlite3_vtab_cursor *cur){ + spellfix1_cursor *pCur = (spellfix1_cursor *)cur; + int rc = SQLITE_OK; + if( pCur->iRow < pCur->nRow ){ + if( pCur->pFullScan ){ + rc = sqlite3_step(pCur->pFullScan); + if( rc!=SQLITE_ROW ) pCur->iRow = pCur->nRow; + if( rc==SQLITE_ROW || rc==SQLITE_DONE ) rc = SQLITE_OK; + }else{ + pCur->iRow++; + } + } + return rc; +} + +/* +** Return TRUE if we are at the end-of-file +*/ +static int spellfix1Eof(sqlite3_vtab_cursor *cur){ + spellfix1_cursor *pCur = (spellfix1_cursor *)cur; + return pCur->iRow>=pCur->nRow; +} + +/* +** Return columns from the current row. +*/ +static int spellfix1Column( + sqlite3_vtab_cursor *cur, + sqlite3_context *ctx, + int i +){ + spellfix1_cursor *pCur = (spellfix1_cursor*)cur; + if( pCur->pFullScan ){ + if( i<=SPELLFIX_COL_LANGID ){ + sqlite3_result_value(ctx, sqlite3_column_value(pCur->pFullScan, i)); + }else{ + sqlite3_result_null(ctx); + } + return SQLITE_OK; + } + switch( i ){ + case SPELLFIX_COL_WORD: { + sqlite3_result_text(ctx, pCur->a[pCur->iRow].zWord, -1, SQLITE_STATIC); + break; + } + case SPELLFIX_COL_RANK: { + sqlite3_result_int(ctx, pCur->a[pCur->iRow].iRank); + break; + } + case SPELLFIX_COL_DISTANCE: { + sqlite3_result_int(ctx, pCur->a[pCur->iRow].iDistance); + break; + } + case SPELLFIX_COL_LANGID: { + sqlite3_result_int(ctx, pCur->iLang); + break; + } + case SPELLFIX_COL_SCORE: { + sqlite3_result_int(ctx, pCur->a[pCur->iRow].iScore); + break; + } + case SPELLFIX_COL_MATCHLEN: { + int iMatchlen = pCur->a[pCur->iRow].iMatchlen; + if( iMatchlen<0 ){ + int nPattern = (int)strlen(pCur->zPattern); + char *zWord = pCur->a[pCur->iRow].zWord; + int nWord = (int)strlen(zWord); + + if( nPattern>0 && pCur->zPattern[nPattern-1]=='*' ){ + char *zTranslit; + int res; + zTranslit = (char *)transliterate((unsigned char *)zWord, nWord); + if( !zTranslit ) return SQLITE_NOMEM; + res = editdist1(pCur->zPattern, zTranslit, &iMatchlen); + sqlite3_free(zTranslit); + if( res<0 ) return SQLITE_NOMEM; + iMatchlen = translen_to_charlen(zWord, nWord, iMatchlen); + }else{ + iMatchlen = utf8Charlen(zWord, nWord); + } + } + + sqlite3_result_int(ctx, iMatchlen); + break; + } + case SPELLFIX_COL_PHONEHASH: { + sqlite3_result_text(ctx, pCur->a[pCur->iRow].zHash, -1, SQLITE_STATIC); + break; + } + case SPELLFIX_COL_TOP: { + sqlite3_result_int(ctx, pCur->iTop); + break; + } + case SPELLFIX_COL_SCOPE: { + sqlite3_result_int(ctx, pCur->iScope); + break; + } + case SPELLFIX_COL_SRCHCNT: { + sqlite3_result_int(ctx, pCur->nSearch); + break; + } + default: { + sqlite3_result_null(ctx); + break; + } + } + return SQLITE_OK; +} + +/* +** The rowid. +*/ +static int spellfix1Rowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + spellfix1_cursor *pCur = (spellfix1_cursor*)cur; + if( pCur->pFullScan ){ + *pRowid = sqlite3_column_int64(pCur->pFullScan, 4); + }else{ + *pRowid = pCur->a[pCur->iRow].iRowid; + } + return SQLITE_OK; +} + +/* +** This function is called by the xUpdate() method. It returns a string +** containing the conflict mode that xUpdate() should use for the current +** operation. One of: "ROLLBACK", "IGNORE", "ABORT" or "REPLACE". +*/ +static const char *spellfix1GetConflict(sqlite3 *db){ + static const char *azConflict[] = { + /* Note: Instead of "FAIL" - "ABORT". */ + "ROLLBACK", "IGNORE", "ABORT", "ABORT", "REPLACE" + }; + int eConflict = sqlite3_vtab_on_conflict(db); + + assert( eConflict==SQLITE_ROLLBACK || eConflict==SQLITE_IGNORE + || eConflict==SQLITE_FAIL || eConflict==SQLITE_ABORT + || eConflict==SQLITE_REPLACE + ); + assert( SQLITE_ROLLBACK==1 ); + assert( SQLITE_IGNORE==2 ); + assert( SQLITE_FAIL==3 ); + assert( SQLITE_ABORT==4 ); + assert( SQLITE_REPLACE==5 ); + + return azConflict[eConflict-1]; +} + +/* +** The xUpdate() method. +*/ +static int spellfix1Update( + sqlite3_vtab *pVTab, + int argc, + sqlite3_value **argv, + sqlite_int64 *pRowid +){ + int rc = SQLITE_OK; + sqlite3_int64 rowid, newRowid; + spellfix1_vtab *p = (spellfix1_vtab*)pVTab; + sqlite3 *db = p->db; + + if( argc==1 ){ + /* A delete operation on the rowid given by argv[0] */ + rowid = *pRowid = sqlite3_value_int64(argv[0]); + spellfix1DbExec(&rc, db, "DELETE FROM \"%w\".\"%w_vocab\" " + " WHERE id=%lld", + p->zDbName, p->zTableName, rowid); + }else{ + const unsigned char *zWord = sqlite3_value_text(argv[SPELLFIX_COL_WORD+2]); + int nWord = sqlite3_value_bytes(argv[SPELLFIX_COL_WORD+2]); + int iLang = sqlite3_value_int(argv[SPELLFIX_COL_LANGID+2]); + int iRank = sqlite3_value_int(argv[SPELLFIX_COL_RANK+2]); + const unsigned char *zSoundslike = + sqlite3_value_text(argv[SPELLFIX_COL_SOUNDSLIKE+2]); + int nSoundslike = sqlite3_value_bytes(argv[SPELLFIX_COL_SOUNDSLIKE+2]); + char *zK1, *zK2; + int i; + char c; + const char *zConflict = spellfix1GetConflict(db); + + if( zWord==0 ){ + /* Inserts of the form: INSERT INTO table(command) VALUES('xyzzy'); + ** cause zWord to be NULL, so we look at the "command" column to see + ** what special actions to take */ + const char *zCmd = + (const char*)sqlite3_value_text(argv[SPELLFIX_COL_COMMAND+2]); + if( zCmd==0 ){ + pVTab->zErrMsg = sqlite3_mprintf("NOT NULL constraint failed: %s.word", + p->zTableName); + return SQLITE_CONSTRAINT_NOTNULL; + } + if( strcmp(zCmd,"reset")==0 ){ + /* Reset the edit cost table (if there is one). */ + editDist3ConfigDelete(p->pConfig3); + p->pConfig3 = 0; + return SQLITE_OK; + } + if( strncmp(zCmd,"edit_cost_table=",16)==0 ){ + editDist3ConfigDelete(p->pConfig3); + p->pConfig3 = 0; + sqlite3_free(p->zCostTable); + p->zCostTable = spellfix1Dequote(zCmd+16); + if( p->zCostTable==0 ) return SQLITE_NOMEM; + if( p->zCostTable[0]==0 || sqlite3_stricmp(p->zCostTable,"null")==0 ){ + sqlite3_free(p->zCostTable); + p->zCostTable = 0; + } + return SQLITE_OK; + } + pVTab->zErrMsg = sqlite3_mprintf("unknown value for %s.command: \"%w\"", + p->zTableName, zCmd); + return SQLITE_ERROR; + } + if( iRank<1 ) iRank = 1; + if( zSoundslike ){ + zK1 = (char*)transliterate(zSoundslike, nSoundslike); + }else{ + zK1 = (char*)transliterate(zWord, nWord); + } + if( zK1==0 ) return SQLITE_NOMEM; + for(i=0; (c = zK1[i])!=0; i++){ + if( c>='A' && c<='Z' ) zK1[i] += 'a' - 'A'; + } + zK2 = (char*)phoneticHash((const unsigned char*)zK1, i); + if( zK2==0 ){ + sqlite3_free(zK1); + return SQLITE_NOMEM; + } + if( sqlite3_value_type(argv[0])==SQLITE_NULL ){ + if( sqlite3_value_type(argv[1])==SQLITE_NULL ){ + spellfix1DbExec(&rc, db, + "INSERT INTO \"%w\".\"%w_vocab\"(rank,langid,word,k1,k2) " + "VALUES(%d,%d,%Q,nullif(%Q,%Q),%Q)", + p->zDbName, p->zTableName, + iRank, iLang, zWord, zK1, zWord, zK2 + ); + }else{ + newRowid = sqlite3_value_int64(argv[1]); + spellfix1DbExec(&rc, db, + "INSERT OR %s INTO \"%w\".\"%w_vocab\"(id,rank,langid,word,k1,k2) " + "VALUES(%lld,%d,%d,%Q,nullif(%Q,%Q),%Q)", + zConflict, p->zDbName, p->zTableName, + newRowid, iRank, iLang, zWord, zK1, zWord, zK2 + ); + } + *pRowid = sqlite3_last_insert_rowid(db); + }else{ + rowid = sqlite3_value_int64(argv[0]); + newRowid = *pRowid = sqlite3_value_int64(argv[1]); + spellfix1DbExec(&rc, db, + "UPDATE OR %s \"%w\".\"%w_vocab\" SET id=%lld, rank=%d, langid=%d," + " word=%Q, k1=nullif(%Q,%Q), k2=%Q WHERE id=%lld", + zConflict, p->zDbName, p->zTableName, newRowid, iRank, iLang, + zWord, zK1, zWord, zK2, rowid + ); + } + sqlite3_free(zK1); + sqlite3_free(zK2); + } + return rc; +} + +/* +** Rename the spellfix1 table. +*/ +static int spellfix1Rename(sqlite3_vtab *pVTab, const char *zNew){ + spellfix1_vtab *p = (spellfix1_vtab*)pVTab; + sqlite3 *db = p->db; + int rc = SQLITE_OK; + char *zNewName = sqlite3_mprintf("%s", zNew); + if( zNewName==0 ){ + return SQLITE_NOMEM; + } + spellfix1DbExec(&rc, db, + "ALTER TABLE \"%w\".\"%w_vocab\" RENAME TO \"%w_vocab\"", + p->zDbName, p->zTableName, zNewName + ); + if( rc==SQLITE_OK ){ + sqlite3_free(p->zTableName); + p->zTableName = zNewName; + }else{ + sqlite3_free(zNewName); + } + return rc; +} + + +/* +** A virtual table module that provides fuzzy search. +*/ +static sqlite3_module spellfix1Module = { + 0, /* iVersion */ + spellfix1Create, /* xCreate - handle CREATE VIRTUAL TABLE */ + spellfix1Connect, /* xConnect - reconnected to an existing table */ + spellfix1BestIndex, /* xBestIndex - figure out how to do a query */ + spellfix1Disconnect, /* xDisconnect - close a connection */ + spellfix1Destroy, /* xDestroy - handle DROP TABLE */ + spellfix1Open, /* xOpen - open a cursor */ + spellfix1Close, /* xClose - close a cursor */ + spellfix1Filter, /* xFilter - configure scan constraints */ + spellfix1Next, /* xNext - advance a cursor */ + spellfix1Eof, /* xEof - check for end of scan */ + spellfix1Column, /* xColumn - read data */ + spellfix1Rowid, /* xRowid - read data */ + spellfix1Update, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + spellfix1Rename, /* xRename */ +}; + +/* +** Register the various functions and the virtual table. +*/ +static int spellfix1Register(sqlite3 *db){ + int rc = SQLITE_OK; + int i; + rc = sqlite3_create_function(db, "spellfix1_translit", 1, + SQLITE_UTF8|SQLITE_DETERMINISTIC, 0, + transliterateSqlFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "spellfix1_editdist", 2, + SQLITE_UTF8|SQLITE_DETERMINISTIC, 0, + editdistSqlFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "spellfix1_phonehash", 1, + SQLITE_UTF8|SQLITE_DETERMINISTIC, 0, + phoneticHashSqlFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "spellfix1_scriptcode", 1, + SQLITE_UTF8|SQLITE_DETERMINISTIC, 0, + scriptCodeSqlFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_module(db, "spellfix1", &spellfix1Module, 0); + } + if( rc==SQLITE_OK ){ + rc = editDist3Install(db); + } + + /* Verify sanity of the translit[] table */ + for(i=0; i<sizeof(translit)/sizeof(translit[0])-1; i++){ + assert( translit[i].cFrom<translit[i+1].cFrom ); + } + + return rc; +} + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +/* +** Extension load function. +*/ +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_spellfix_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + return spellfix1Register(db); +#endif + return SQLITE_OK; +} diff --git a/ext/misc/sqlar.c b/ext/misc/sqlar.c new file mode 100644 index 0000000..47cb68f --- /dev/null +++ b/ext/misc/sqlar.c @@ -0,0 +1,124 @@ +/* +** 2017-12-17 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** Utility functions sqlar_compress() and sqlar_uncompress(). Useful +** for working with sqlar archives and used by the shell tool's built-in +** sqlar support. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <zlib.h> +#include <assert.h> + +/* +** Implementation of the "sqlar_compress(X)" SQL function. +** +** If the type of X is SQLITE_BLOB, and compressing that blob using +** zlib utility function compress() yields a smaller blob, return the +** compressed blob. Otherwise, return a copy of X. +** +** SQLar uses the "zlib format" for compressed content. The zlib format +** contains a two-byte identification header and a four-byte checksum at +** the end. This is different from ZIP which uses the raw deflate format. +** +** Future enhancements to SQLar might add support for new compression formats. +** If so, those new formats will be identified by alternative headers in the +** compressed data. +*/ +static void sqlarCompressFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + assert( argc==1 ); + if( sqlite3_value_type(argv[0])==SQLITE_BLOB ){ + const Bytef *pData = sqlite3_value_blob(argv[0]); + uLong nData = sqlite3_value_bytes(argv[0]); + uLongf nOut = compressBound(nData); + Bytef *pOut; + + pOut = (Bytef*)sqlite3_malloc(nOut); + if( pOut==0 ){ + sqlite3_result_error_nomem(context); + return; + }else{ + if( Z_OK!=compress(pOut, &nOut, pData, nData) ){ + sqlite3_result_error(context, "error in compress()", -1); + }else if( nOut<nData ){ + sqlite3_result_blob(context, pOut, nOut, SQLITE_TRANSIENT); + }else{ + sqlite3_result_value(context, argv[0]); + } + sqlite3_free(pOut); + } + }else{ + sqlite3_result_value(context, argv[0]); + } +} + +/* +** Implementation of the "sqlar_uncompress(X,SZ)" SQL function +** +** Parameter SZ is interpreted as an integer. If it is less than or +** equal to zero, then this function returns a copy of X. Or, if +** SZ is equal to the size of X when interpreted as a blob, also +** return a copy of X. Otherwise, decompress blob X using zlib +** utility function uncompress() and return the results (another +** blob). +*/ +static void sqlarUncompressFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + uLong nData; + uLongf sz; + + assert( argc==2 ); + sz = sqlite3_value_int(argv[1]); + + if( sz<=0 || sz==(nData = sqlite3_value_bytes(argv[0])) ){ + sqlite3_result_value(context, argv[0]); + }else{ + const Bytef *pData= sqlite3_value_blob(argv[0]); + Bytef *pOut = sqlite3_malloc(sz); + if( Z_OK!=uncompress(pOut, &sz, pData, nData) ){ + sqlite3_result_error(context, "error in uncompress()", -1); + }else{ + sqlite3_result_blob(context, pOut, sz, SQLITE_TRANSIENT); + } + sqlite3_free(pOut); + } +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_sqlar_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "sqlar_compress", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + sqlarCompressFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "sqlar_uncompress", 2, + SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + sqlarUncompressFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/stmt.c b/ext/misc/stmt.c new file mode 100644 index 0000000..1687f41 --- /dev/null +++ b/ext/misc/stmt.c @@ -0,0 +1,337 @@ +/* +** 2017-05-31 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file demonstrates an eponymous virtual table that returns information +** about all prepared statements for the database connection. +** +** Usage example: +** +** .load ./stmt +** .mode line +** .header on +** SELECT * FROM stmt; +*/ +#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_STMTVTAB) +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + + +#define STMT_NUM_INTEGER_COLUMN 10 +typedef struct StmtRow StmtRow; +struct StmtRow { + sqlite3_int64 iRowid; /* Rowid value */ + char *zSql; /* column "sql" */ + int aCol[STMT_NUM_INTEGER_COLUMN+1]; /* all other column values */ + StmtRow *pNext; /* Next row to return */ +}; + +/* stmt_vtab is a subclass of sqlite3_vtab which will +** serve as the underlying representation of a stmt virtual table +*/ +typedef struct stmt_vtab stmt_vtab; +struct stmt_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this stmt vtab */ +}; + +/* stmt_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct stmt_cursor stmt_cursor; +struct stmt_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3 *db; /* Database connection for this cursor */ + StmtRow *pRow; /* Current row */ +}; + +/* +** The stmtConnect() method is invoked to create a new +** stmt_vtab that describes the stmt virtual table. +** +** Think of this routine as the constructor for stmt_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the stmt_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against stmt will look like. +*/ +static int stmtConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + stmt_vtab *pNew; + int rc; + +/* Column numbers */ +#define STMT_COLUMN_SQL 0 /* SQL for the statement */ +#define STMT_COLUMN_NCOL 1 /* Number of result columns */ +#define STMT_COLUMN_RO 2 /* True if read-only */ +#define STMT_COLUMN_BUSY 3 /* True if currently busy */ +#define STMT_COLUMN_NSCAN 4 /* SQLITE_STMTSTATUS_FULLSCAN_STEP */ +#define STMT_COLUMN_NSORT 5 /* SQLITE_STMTSTATUS_SORT */ +#define STMT_COLUMN_NAIDX 6 /* SQLITE_STMTSTATUS_AUTOINDEX */ +#define STMT_COLUMN_NSTEP 7 /* SQLITE_STMTSTATUS_VM_STEP */ +#define STMT_COLUMN_REPREP 8 /* SQLITE_STMTSTATUS_REPREPARE */ +#define STMT_COLUMN_RUN 9 /* SQLITE_STMTSTATUS_RUN */ +#define STMT_COLUMN_MEM 10 /* SQLITE_STMTSTATUS_MEMUSED */ + + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(sql,ncol,ro,busy,nscan,nsort,naidx,nstep," + "reprep,run,mem)"); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc64( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->db = db; + } + return rc; +} + +/* +** This method is the destructor for stmt_cursor objects. +*/ +static int stmtDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new stmt_cursor object. +*/ +static int stmtOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + stmt_cursor *pCur; + pCur = sqlite3_malloc64( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->db = ((stmt_vtab*)p)->db; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +static void stmtCsrReset(stmt_cursor *pCur){ + StmtRow *pRow = 0; + StmtRow *pNext = 0; + for(pRow=pCur->pRow; pRow; pRow=pNext){ + pNext = pRow->pNext; + sqlite3_free(pRow); + } + pCur->pRow = 0; +} + +/* +** Destructor for a stmt_cursor. +*/ +static int stmtClose(sqlite3_vtab_cursor *cur){ + stmtCsrReset((stmt_cursor*)cur); + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a stmt_cursor to its next row of output. +*/ +static int stmtNext(sqlite3_vtab_cursor *cur){ + stmt_cursor *pCur = (stmt_cursor*)cur; + StmtRow *pNext = pCur->pRow->pNext; + sqlite3_free(pCur->pRow); + pCur->pRow = pNext; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the stmt_cursor +** is currently pointing. +*/ +static int stmtColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + stmt_cursor *pCur = (stmt_cursor*)cur; + StmtRow *pRow = pCur->pRow; + if( i==STMT_COLUMN_SQL ){ + sqlite3_result_text(ctx, pRow->zSql, -1, SQLITE_TRANSIENT); + }else{ + sqlite3_result_int(ctx, pRow->aCol[i]); + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int stmtRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + stmt_cursor *pCur = (stmt_cursor*)cur; + *pRowid = pCur->pRow->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int stmtEof(sqlite3_vtab_cursor *cur){ + stmt_cursor *pCur = (stmt_cursor*)cur; + return pCur->pRow==0; +} + +/* +** This method is called to "rewind" the stmt_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to stmtColumn() or stmtRowid() or +** stmtEof(). +*/ +static int stmtFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + stmt_cursor *pCur = (stmt_cursor *)pVtabCursor; + sqlite3_stmt *p = 0; + sqlite3_int64 iRowid = 1; + StmtRow **ppRow = 0; + + stmtCsrReset(pCur); + ppRow = &pCur->pRow; + for(p=sqlite3_next_stmt(pCur->db, 0); p; p=sqlite3_next_stmt(pCur->db, p)){ + const char *zSql = sqlite3_sql(p); + sqlite3_int64 nSql = zSql ? strlen(zSql)+1 : 0; + StmtRow *pNew = (StmtRow*)sqlite3_malloc64(sizeof(StmtRow) + nSql); + + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(StmtRow)); + if( zSql ){ + pNew->zSql = (char*)&pNew[1]; + memcpy(pNew->zSql, zSql, nSql); + } + pNew->aCol[STMT_COLUMN_NCOL] = sqlite3_column_count(p); + pNew->aCol[STMT_COLUMN_RO] = sqlite3_stmt_readonly(p); + pNew->aCol[STMT_COLUMN_BUSY] = sqlite3_stmt_busy(p); + pNew->aCol[STMT_COLUMN_NSCAN] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_FULLSCAN_STEP, 0 + ); + pNew->aCol[STMT_COLUMN_NSORT] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_SORT, 0 + ); + pNew->aCol[STMT_COLUMN_NAIDX] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_AUTOINDEX, 0 + ); + pNew->aCol[STMT_COLUMN_NSTEP] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_VM_STEP, 0 + ); + pNew->aCol[STMT_COLUMN_REPREP] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_REPREPARE, 0 + ); + pNew->aCol[STMT_COLUMN_RUN] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_RUN, 0 + ); + pNew->aCol[STMT_COLUMN_MEM] = sqlite3_stmt_status( + p, SQLITE_STMTSTATUS_MEMUSED, 0 + ); + pNew->iRowid = iRowid++; + *ppRow = pNew; + ppRow = &pNew->pNext; + } + + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the stmt virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int stmtBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + pIdxInfo->estimatedCost = (double)500; + pIdxInfo->estimatedRows = 500; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** stmt virtual table. +*/ +static sqlite3_module stmtModule = { + 0, /* iVersion */ + 0, /* xCreate */ + stmtConnect, /* xConnect */ + stmtBestIndex, /* xBestIndex */ + stmtDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + stmtOpen, /* xOpen - open a cursor */ + stmtClose, /* xClose - close a cursor */ + stmtFilter, /* xFilter - configure scan constraints */ + stmtNext, /* xNext - advance a cursor */ + stmtEof, /* xEof - check for end of scan */ + stmtColumn, /* xColumn - read data */ + stmtRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0, /* xShadowName */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +int sqlite3StmtVtabInit(sqlite3 *db){ + int rc = SQLITE_OK; +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "sqlite_stmt", &stmtModule, 0); +#endif + return rc; +} + +#ifndef SQLITE_CORE +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_stmt_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3StmtVtabInit(db); +#endif + return rc; +} +#endif /* SQLITE_CORE */ +#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_STMTVTAB) */ diff --git a/ext/misc/templatevtab.c b/ext/misc/templatevtab.c new file mode 100644 index 0000000..d7efa2b --- /dev/null +++ b/ext/misc/templatevtab.c @@ -0,0 +1,268 @@ +/* +** 2018-04-19 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a template virtual-table. +** Developers can make a copy of this file as a baseline for writing +** new virtual tables and/or table-valued functions. +** +** Steps for writing a new virtual table implementation: +** +** (1) Make a copy of this file. Perhaps call it "mynewvtab.c" +** +** (2) Replace this header comment with something appropriate for +** the new virtual table +** +** (3) Change every occurrence of "templatevtab" to some other string +** appropriate for the new virtual table. Ideally, the new string +** should be the basename of the source file: "mynewvtab". Also +** globally change "TEMPLATEVTAB" to "MYNEWVTAB". +** +** (4) Run a test compilation to make sure the unmodified virtual +** table works. +** +** (5) Begin making incremental changes, testing as you go, to evolve +** the new virtual table to do what you want it to do. +** +** This template is minimal, in the sense that it uses only the required +** methods on the sqlite3_module object. As a result, templatevtab is +** a read-only and eponymous-only table. Those limitation can be removed +** by adding new methods. +** +** This template implements an eponymous-only virtual table with a rowid and +** two columns named "a" and "b". The table as 10 rows with fixed integer +** values. Usage example: +** +** SELECT rowid, a, b FROM templatevtab; +*/ +#if !defined(SQLITEINT_H) +#include "sqlite3ext.h" +#endif +SQLITE_EXTENSION_INIT1 +#include <string.h> +#include <assert.h> + +/* templatevtab_vtab is a subclass of sqlite3_vtab which is +** underlying representation of the virtual table +*/ +typedef struct templatevtab_vtab templatevtab_vtab; +struct templatevtab_vtab { + sqlite3_vtab base; /* Base class - must be first */ + /* Add new fields here, as necessary */ +}; + +/* templatevtab_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct templatevtab_cursor templatevtab_cursor; +struct templatevtab_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + /* Insert new fields here. For this templatevtab we only keep track + ** of the rowid */ + sqlite3_int64 iRowid; /* The rowid */ +}; + +/* +** The templatevtabConnect() method is invoked to create a new +** template virtual table. +** +** Think of this routine as the constructor for templatevtab_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the templatevtab_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against the virtual table will look like. +*/ +static int templatevtabConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + templatevtab_vtab *pNew; + int rc; + + rc = sqlite3_declare_vtab(db, + "CREATE TABLE x(a,b)" + ); + /* For convenience, define symbolic names for the index to each column. */ +#define TEMPLATEVTAB_A 0 +#define TEMPLATEVTAB_B 1 + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + } + return rc; +} + +/* +** This method is the destructor for templatevtab_vtab objects. +*/ +static int templatevtabDisconnect(sqlite3_vtab *pVtab){ + templatevtab_vtab *p = (templatevtab_vtab*)pVtab; + sqlite3_free(p); + return SQLITE_OK; +} + +/* +** Constructor for a new templatevtab_cursor object. +*/ +static int templatevtabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + templatevtab_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a templatevtab_cursor. +*/ +static int templatevtabClose(sqlite3_vtab_cursor *cur){ + templatevtab_cursor *pCur = (templatevtab_cursor*)cur; + sqlite3_free(pCur); + return SQLITE_OK; +} + + +/* +** Advance a templatevtab_cursor to its next row of output. +*/ +static int templatevtabNext(sqlite3_vtab_cursor *cur){ + templatevtab_cursor *pCur = (templatevtab_cursor*)cur; + pCur->iRowid++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the templatevtab_cursor +** is currently pointing. +*/ +static int templatevtabColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + templatevtab_cursor *pCur = (templatevtab_cursor*)cur; + switch( i ){ + case TEMPLATEVTAB_A: + sqlite3_result_int(ctx, 1000 + pCur->iRowid); + break; + default: + assert( i==TEMPLATEVTAB_B ); + sqlite3_result_int(ctx, 2000 + pCur->iRowid); + break; + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int templatevtabRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + templatevtab_cursor *pCur = (templatevtab_cursor*)cur; + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int templatevtabEof(sqlite3_vtab_cursor *cur){ + templatevtab_cursor *pCur = (templatevtab_cursor*)cur; + return pCur->iRowid>=10; +} + +/* +** This method is called to "rewind" the templatevtab_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to templatevtabColumn() or templatevtabRowid() or +** templatevtabEof(). +*/ +static int templatevtabFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + templatevtab_cursor *pCur = (templatevtab_cursor *)pVtabCursor; + pCur->iRowid = 1; + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int templatevtabBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + pIdxInfo->estimatedCost = (double)10; + pIdxInfo->estimatedRows = 10; + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** virtual table. +*/ +static sqlite3_module templatevtabModule = { + /* iVersion */ 0, + /* xCreate */ 0, + /* xConnect */ templatevtabConnect, + /* xBestIndex */ templatevtabBestIndex, + /* xDisconnect */ templatevtabDisconnect, + /* xDestroy */ 0, + /* xOpen */ templatevtabOpen, + /* xClose */ templatevtabClose, + /* xFilter */ templatevtabFilter, + /* xNext */ templatevtabNext, + /* xEof */ templatevtabEof, + /* xColumn */ templatevtabColumn, + /* xRowid */ templatevtabRowid, + /* xUpdate */ 0, + /* xBegin */ 0, + /* xSync */ 0, + /* xCommit */ 0, + /* xRollback */ 0, + /* xFindMethod */ 0, + /* xRename */ 0, + /* xSavepoint */ 0, + /* xRelease */ 0, + /* xRollbackTo */ 0, + /* xShadowName */ 0 +}; + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_templatevtab_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_create_module(db, "templatevtab", &templatevtabModule, 0); + return rc; +} diff --git a/ext/misc/totype.c b/ext/misc/totype.c new file mode 100644 index 0000000..50d7f05 --- /dev/null +++ b/ext/misc/totype.c @@ -0,0 +1,514 @@ +/* +** 2013-10-14 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements functions tointeger(X) and toreal(X). +** +** If X is an integer, real, or string value that can be +** losslessly represented as an integer, then tointeger(X) +** returns the corresponding integer value. +** If X is an 8-byte BLOB then that blob is interpreted as +** a signed two-compliment little-endian encoding of an integer +** and tointeger(X) returns the corresponding integer value. +** Otherwise tointeger(X) return NULL. +** +** If X is an integer, real, or string value that can be +** convert into a real number, preserving at least 15 digits +** of precision, then toreal(X) returns the corresponding real value. +** If X is an 8-byte BLOB then that blob is interpreted as +** a 64-bit IEEE754 big-endian floating point value +** and toreal(X) returns the corresponding real value. +** Otherwise toreal(X) return NULL. +** +** Note that tointeger(X) of an 8-byte BLOB assumes a little-endian +** encoding whereas toreal(X) of an 8-byte BLOB assumes a big-endian +** encoding. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* +** Determine if this is running on a big-endian or little-endian +** processor +*/ +#if defined(i386) || defined(__i386__) || defined(_M_IX86)\ + || defined(__x86_64) || defined(__x86_64__) +# define TOTYPE_BIGENDIAN 0 +# define TOTYPE_LITTLEENDIAN 1 +#else + const int totype_one = 1; +# define TOTYPE_BIGENDIAN (*(char *)(&totype_one)==0) +# define TOTYPE_LITTLEENDIAN (*(char *)(&totype_one)==1) +#endif + +/* +** Constants for the largest and smallest possible 64-bit signed integers. +** These macros are designed to work correctly on both 32-bit and 64-bit +** compilers. +*/ +#ifndef LARGEST_INT64 +# define LARGEST_INT64 (0xffffffff|(((sqlite3_int64)0x7fffffff)<<32)) +#endif + +#ifndef SMALLEST_INT64 +# define SMALLEST_INT64 (((sqlite3_int64)-1) - LARGEST_INT64) +#endif + +/* +** Return TRUE if character c is a whitespace character +*/ +static int totypeIsspace(unsigned char c){ + return c==' ' || c=='\t' || c=='\n' || c=='\v' || c=='\f' || c=='\r'; +} + +/* +** Return TRUE if character c is a digit +*/ +static int totypeIsdigit(unsigned char c){ + return c>='0' && c<='9'; +} + +/* +** Compare the 19-character string zNum against the text representation +** value 2^63: 9223372036854775808. Return negative, zero, or positive +** if zNum is less than, equal to, or greater than the string. +** Note that zNum must contain exactly 19 characters. +** +** Unlike memcmp() this routine is guaranteed to return the difference +** in the values of the last digit if the only difference is in the +** last digit. So, for example, +** +** totypeCompare2pow63("9223372036854775800") +** +** will return -8. +*/ +static int totypeCompare2pow63(const char *zNum){ + int c = 0; + int i; + /* 012345678901234567 */ + const char *pow63 = "922337203685477580"; + for(i=0; c==0 && i<18; i++){ + c = (zNum[i]-pow63[i])*10; + } + if( c==0 ){ + c = zNum[18] - '8'; + } + return c; +} + +/* +** Convert zNum to a 64-bit signed integer. +** +** If the zNum value is representable as a 64-bit twos-complement +** integer, then write that value into *pNum and return 0. +** +** If zNum is exactly 9223372036854665808, return 2. This special +** case is broken out because while 9223372036854665808 cannot be a +** signed 64-bit integer, its negative -9223372036854665808 can be. +** +** If zNum is too big for a 64-bit integer and is not +** 9223372036854665808 or if zNum contains any non-numeric text, +** then return 1. +** +** The string is not necessarily zero-terminated. +*/ +static int totypeAtoi64(const char *zNum, sqlite3_int64 *pNum, int length){ + sqlite3_uint64 u = 0; + int neg = 0; /* assume positive */ + int i; + int c = 0; + int nonNum = 0; + const char *zStart; + const char *zEnd = zNum + length; + + while( zNum<zEnd && totypeIsspace(*zNum) ) zNum++; + if( zNum<zEnd ){ + if( *zNum=='-' ){ + neg = 1; + zNum++; + }else if( *zNum=='+' ){ + zNum++; + } + } + zStart = zNum; + while( zNum<zEnd && zNum[0]=='0' ){ zNum++; } /* Skip leading zeros. */ + for(i=0; &zNum[i]<zEnd && (c=zNum[i])>='0' && c<='9'; i++){ + u = u*10 + c - '0'; + } + if( u>LARGEST_INT64 ){ + *pNum = SMALLEST_INT64; + }else if( neg ){ + *pNum = -(sqlite3_int64)u; + }else{ + *pNum = (sqlite3_int64)u; + } + if( (c!=0 && &zNum[i]<zEnd) || (i==0 && zStart==zNum) || i>19 || nonNum ){ + /* zNum is empty or contains non-numeric text or is longer + ** than 19 digits (thus guaranteeing that it is too large) */ + return 1; + }else if( i<19 ){ + /* Less than 19 digits, so we know that it fits in 64 bits */ + assert( u<=LARGEST_INT64 ); + return 0; + }else{ + /* zNum is a 19-digit numbers. Compare it against 9223372036854775808. */ + c = totypeCompare2pow63(zNum); + if( c<0 ){ + /* zNum is less than 9223372036854775808 so it fits */ + assert( u<=LARGEST_INT64 ); + return 0; + }else if( c>0 ){ + /* zNum is greater than 9223372036854775808 so it overflows */ + return 1; + }else{ + /* zNum is exactly 9223372036854775808. Fits if negative. The + ** special case 2 overflow if positive */ + assert( u-1==LARGEST_INT64 ); + assert( (*pNum)==SMALLEST_INT64 ); + return neg ? 0 : 2; + } + } +} + +/* +** The string z[] is an text representation of a real number. +** Convert this string to a double and write it into *pResult. +** +** The string is not necessarily zero-terminated. +** +** Return TRUE if the result is a valid real number (or integer) and FALSE +** if the string is empty or contains extraneous text. Valid numbers +** are in one of these formats: +** +** [+-]digits[E[+-]digits] +** [+-]digits.[digits][E[+-]digits] +** [+-].digits[E[+-]digits] +** +** Leading and trailing whitespace is ignored for the purpose of determining +** validity. +** +** If some prefix of the input string is a valid number, this routine +** returns FALSE but it still converts the prefix and writes the result +** into *pResult. +*/ +static int totypeAtoF(const char *z, double *pResult, int length){ + const char *zEnd = z + length; + /* sign * significand * (10 ^ (esign * exponent)) */ + int sign = 1; /* sign of significand */ + sqlite3_int64 s = 0; /* significand */ + int d = 0; /* adjust exponent for shifting decimal point */ + int esign = 1; /* sign of exponent */ + int e = 0; /* exponent */ + int eValid = 1; /* True exponent is either not used or is well-formed */ + double result; + int nDigits = 0; + int nonNum = 0; + + *pResult = 0.0; /* Default return value, in case of an error */ + + /* skip leading spaces */ + while( z<zEnd && totypeIsspace(*z) ) z++; + if( z>=zEnd ) return 0; + + /* get sign of significand */ + if( *z=='-' ){ + sign = -1; + z++; + }else if( *z=='+' ){ + z++; + } + + /* skip leading zeroes */ + while( z<zEnd && z[0]=='0' ) z++, nDigits++; + + /* copy max significant digits to significand */ + while( z<zEnd && totypeIsdigit(*z) && s<((LARGEST_INT64-9)/10) ){ + s = s*10 + (*z - '0'); + z++, nDigits++; + } + + /* skip non-significant significand digits + ** (increase exponent by d to shift decimal left) */ + while( z<zEnd && totypeIsdigit(*z) ) z++, nDigits++, d++; + if( z>=zEnd ) goto totype_atof_calc; + + /* if decimal point is present */ + if( *z=='.' ){ + z++; + /* copy digits from after decimal to significand + ** (decrease exponent by d to shift decimal right) */ + while( z<zEnd && totypeIsdigit(*z) && s<((LARGEST_INT64-9)/10) ){ + s = s*10 + (*z - '0'); + z++, nDigits++, d--; + } + /* skip non-significant digits */ + while( z<zEnd && totypeIsdigit(*z) ) z++, nDigits++; + } + if( z>=zEnd ) goto totype_atof_calc; + + /* if exponent is present */ + if( *z=='e' || *z=='E' ){ + z++; + eValid = 0; + if( z>=zEnd ) goto totype_atof_calc; + /* get sign of exponent */ + if( *z=='-' ){ + esign = -1; + z++; + }else if( *z=='+' ){ + z++; + } + /* copy digits to exponent */ + while( z<zEnd && totypeIsdigit(*z) ){ + e = e<10000 ? (e*10 + (*z - '0')) : 10000; + z++; + eValid = 1; + } + } + + /* skip trailing spaces */ + if( nDigits && eValid ){ + while( z<zEnd && totypeIsspace(*z) ) z++; + } + +totype_atof_calc: + /* adjust exponent by d, and update sign */ + e = (e*esign) + d; + if( e<0 ) { + esign = -1; + e *= -1; + } else { + esign = 1; + } + + /* if 0 significand */ + if( !s ) { + /* In the IEEE 754 standard, zero is signed. + ** Add the sign if we've seen at least one digit */ + result = (sign<0 && nDigits) ? -(double)0 : (double)0; + } else { + /* attempt to reduce exponent */ + if( esign>0 ){ + while( s<(LARGEST_INT64/10) && e>0 ) e--,s*=10; + }else{ + while( !(s%10) && e>0 ) e--,s/=10; + } + + /* adjust the sign of significand */ + s = sign<0 ? -s : s; + + /* if exponent, scale significand as appropriate + ** and store in result. */ + if( e ){ + double scale = 1.0; + /* attempt to handle extremely small/large numbers better */ + if( e>307 && e<342 ){ + while( e%308 ) { scale *= 1.0e+1; e -= 1; } + if( esign<0 ){ + result = s / scale; + result /= 1.0e+308; + }else{ + result = s * scale; + result *= 1.0e+308; + } + }else if( e>=342 ){ + if( esign<0 ){ + result = 0.0*s; + }else{ + result = 1e308*1e308*s; /* Infinity */ + } + }else{ + /* 1.0e+22 is the largest power of 10 than can be + ** represented exactly. */ + while( e%22 ) { scale *= 1.0e+1; e -= 1; } + while( e>0 ) { scale *= 1.0e+22; e -= 22; } + if( esign<0 ){ + result = s / scale; + }else{ + result = s * scale; + } + } + } else { + result = (double)s; + } + } + + /* store the result */ + *pResult = result; + + /* return true if number and no extra non-whitespace chracters after */ + return z>=zEnd && nDigits>0 && eValid && nonNum==0; +} + +/* +** tointeger(X): If X is any value (integer, double, blob, or string) that +** can be losslessly converted into an integer, then make the conversion and +** return the result. Otherwise, return NULL. +*/ +static void tointegerFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + assert( argc==1 ); + (void)argc; + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_FLOAT: { + double rVal = sqlite3_value_double(argv[0]); + sqlite3_int64 iVal = (sqlite3_int64)rVal; + if( rVal==(double)iVal ){ + sqlite3_result_int64(context, iVal); + } + break; + } + case SQLITE_INTEGER: { + sqlite3_result_int64(context, sqlite3_value_int64(argv[0])); + break; + } + case SQLITE_BLOB: { + const unsigned char *zBlob = sqlite3_value_blob(argv[0]); + if( zBlob ){ + int nBlob = sqlite3_value_bytes(argv[0]); + if( nBlob==sizeof(sqlite3_int64) ){ + sqlite3_int64 iVal; + if( TOTYPE_BIGENDIAN ){ + int i; + unsigned char zBlobRev[sizeof(sqlite3_int64)]; + for(i=0; i<sizeof(sqlite3_int64); i++){ + zBlobRev[i] = zBlob[sizeof(sqlite3_int64)-1-i]; + } + memcpy(&iVal, zBlobRev, sizeof(sqlite3_int64)); + }else{ + memcpy(&iVal, zBlob, sizeof(sqlite3_int64)); + } + sqlite3_result_int64(context, iVal); + } + } + break; + } + case SQLITE_TEXT: { + const unsigned char *zStr = sqlite3_value_text(argv[0]); + if( zStr ){ + int nStr = sqlite3_value_bytes(argv[0]); + if( nStr && !totypeIsspace(zStr[0]) ){ + sqlite3_int64 iVal; + if( !totypeAtoi64((const char*)zStr, &iVal, nStr) ){ + sqlite3_result_int64(context, iVal); + } + } + } + break; + } + default: { + assert( sqlite3_value_type(argv[0])==SQLITE_NULL ); + break; + } + } +} + +/* +** toreal(X): If X is any value (integer, double, blob, or string) that can +** be losslessly converted into a real number, then do so and return that +** real number. Otherwise return NULL. +*/ +#if defined(_MSC_VER) +#pragma warning(disable: 4748) +#pragma optimize("", off) +#endif +static void torealFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + assert( argc==1 ); + (void)argc; + switch( sqlite3_value_type(argv[0]) ){ + case SQLITE_FLOAT: { + sqlite3_result_double(context, sqlite3_value_double(argv[0])); + break; + } + case SQLITE_INTEGER: { + sqlite3_int64 iVal = sqlite3_value_int64(argv[0]); + double rVal = (double)iVal; + if( iVal==(sqlite3_int64)rVal ){ + sqlite3_result_double(context, rVal); + } + break; + } + case SQLITE_BLOB: { + const unsigned char *zBlob = sqlite3_value_blob(argv[0]); + if( zBlob ){ + int nBlob = sqlite3_value_bytes(argv[0]); + if( nBlob==sizeof(double) ){ + double rVal; + if( TOTYPE_LITTLEENDIAN ){ + int i; + unsigned char zBlobRev[sizeof(double)]; + for(i=0; i<sizeof(double); i++){ + zBlobRev[i] = zBlob[sizeof(double)-1-i]; + } + memcpy(&rVal, zBlobRev, sizeof(double)); + }else{ + memcpy(&rVal, zBlob, sizeof(double)); + } + sqlite3_result_double(context, rVal); + } + } + break; + } + case SQLITE_TEXT: { + const unsigned char *zStr = sqlite3_value_text(argv[0]); + if( zStr ){ + int nStr = sqlite3_value_bytes(argv[0]); + if( nStr && !totypeIsspace(zStr[0]) && !totypeIsspace(zStr[nStr-1]) ){ + double rVal; + if( totypeAtoF((const char*)zStr, &rVal, nStr) ){ + sqlite3_result_double(context, rVal); + return; + } + } + } + break; + } + default: { + assert( sqlite3_value_type(argv[0])==SQLITE_NULL ); + break; + } + } +} +#if defined(_MSC_VER) +#pragma optimize("", on) +#pragma warning(default: 4748) +#endif + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_totype_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "tointeger", 1, + SQLITE_UTF8 | SQLITE_DETERMINISTIC | SQLITE_INNOCUOUS, 0, + tointegerFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "toreal", 1, + SQLITE_UTF8 | SQLITE_DETERMINISTIC | SQLITE_INNOCUOUS, 0, + torealFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/uint.c b/ext/misc/uint.c new file mode 100644 index 0000000..286314f --- /dev/null +++ b/ext/misc/uint.c @@ -0,0 +1,92 @@ +/* +** 2020-04-14 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements the UINT collating sequence. +** +** UINT works like BINARY for text, except that embedded strings +** of digits compare in numeric order. +** +** * Leading zeros are handled properly, in the sense that +** they do not mess of the maginitude comparison of embedded +** strings of digits. "x00123y" is equal to "x123y". +** +** * Only unsigned integers are recognized. Plus and minus +** signs are ignored. Decimal points and exponential notation +** are ignored. +** +** * Embedded integers can be of arbitrary length. Comparison +** is *not* limited integers that can be expressed as a +** 64-bit machine integer. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <ctype.h> + +/* +** Compare text in lexicographic order, except strings of digits +** compare in numeric order. +*/ +static int uintCollFunc( + void *notUsed, + int nKey1, const void *pKey1, + int nKey2, const void *pKey2 +){ + const unsigned char *zA = (const unsigned char*)pKey1; + const unsigned char *zB = (const unsigned char*)pKey2; + int i=0, j=0, x; + (void)notUsed; + while( i<nKey1 && j<nKey2 ){ + x = zA[i] - zB[j]; + if( isdigit(zA[i]) ){ + int k; + if( !isdigit(zB[j]) ) return x; + while( i<nKey1 && zA[i]=='0' ){ i++; } + while( j<nKey2 && zB[j]=='0' ){ j++; } + k = 0; + while( i+k<nKey1 && isdigit(zA[i+k]) + && j+k<nKey2 && isdigit(zB[j+k]) ){ + k++; + } + if( i+k<nKey1 && isdigit(zA[i+k]) ){ + return +1; + }else if( j+k<nKey2 && isdigit(zB[j+k]) ){ + return -1; + }else{ + x = memcmp(zA+i, zB+j, k); + if( x ) return x; + i += k; + j += k; + } + }else if( x ){ + return x; + }else{ + i++; + j++; + } + } + return (nKey1 - i) - (nKey2 - j); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_uint_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + return sqlite3_create_collation(db, "uint", SQLITE_UTF8, 0, uintCollFunc); +} diff --git a/ext/misc/unionvtab.c b/ext/misc/unionvtab.c new file mode 100644 index 0000000..6ac7ca6 --- /dev/null +++ b/ext/misc/unionvtab.c @@ -0,0 +1,1382 @@ +/* +** 2017 July 15 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file contains the implementation of the "unionvtab" and "swarmvtab" +** virtual tables. These modules provide read-only access to multiple tables, +** possibly in multiple database files, via a single database object. +** The source tables must have the following characteristics: +** +** * They must all be rowid tables (not VIRTUAL or WITHOUT ROWID +** tables or views). +** +** * Each table must have the same set of columns, declared in +** the same order and with the same declared types. +** +** * The tables must not feature a user-defined column named "_rowid_". +** +** * Each table must contain a distinct range of rowid values. +** +** The difference between the two virtual table modules is that for +** "unionvtab", all source tables must be located in the main database or +** in databases ATTACHed to the main database by the user. For "swarmvtab", +** the tables may be located in any database file on disk. The "swarmvtab" +** implementation takes care of opening and closing database files +** automatically. +** +** UNIONVTAB +** +** A "unionvtab" virtual table is created as follows: +** +** CREATE VIRTUAL TABLE <name> USING unionvtab(<sql-statement>); +** +** The implementation evalutes <sql statement> whenever a unionvtab virtual +** table is created or opened. It should return one row for each source +** database table. The four columns required of each row are: +** +** 1. The name of the database containing the table ("main" or "temp" or +** the name of an attached database). Or NULL to indicate that all +** databases should be searched for the table in the usual fashion. +** +** 2. The name of the database table. +** +** 3. The smallest rowid in the range of rowids that may be stored in the +** database table (an integer). +** +** 4. The largest rowid in the range of rowids that may be stored in the +** database table (an integer). +** +** SWARMVTAB +** +** LEGACY SYNTAX: +** +** A "swarmvtab" virtual table is created similarly to a unionvtab table: +** +** CREATE VIRTUAL TABLE <name> +** USING swarmvtab(<sql-statement>, <callback>); +** +** The difference is that for a swarmvtab table, the first column returned +** by the <sql statement> must return a path or URI that can be used to open +** the database file containing the source table. The <callback> option +** is optional. If included, it is the name of an application-defined +** SQL function that is invoked with the URI of the file, if the file +** does not already exist on disk when required by swarmvtab. +** +** NEW SYNTAX: +** +** Using the new syntax, a swarmvtab table is created with: +** +** CREATE VIRTUAL TABLE <name> USING swarmvtab( +** <sql-statement> [, <options>] +** ); +** +** where valid <options> are: +** +** missing=<udf-function-name> +** openclose=<udf-function-name> +** maxopen=<integer> +** <sql-parameter>=<text-value> +** +** The <sql-statement> must return the same 4 columns as for a swarmvtab +** table in legacy mode. However, it may also return a 5th column - the +** "context" column. The text value returned in this column is not used +** at all by the swarmvtab implementation, except that it is passed as +** an additional argument to the two UDF functions that may be invoked +** (see below). +** +** The "missing" option, if present, specifies the name of an SQL UDF +** function to be invoked if a database file is not already present on +** disk when required by swarmvtab. If the <sql-statement> did not provide +** a context column, it is invoked as: +** +** SELECT <missing-udf>(<database filename/uri>); +** +** Or, if there was a context column: +** +** SELECT <missing-udf>(<database filename/uri>, <context>); +** +** The "openclose" option may also specify a UDF function. This function +** is invoked right before swarmvtab opens a database, and right after +** it closes one. The first argument - or first two arguments, if +** <sql-statement> supplied the context column - is the same as for +** the "missing" UDF. Following this, the UDF is passed integer value +** 0 before a db is opened, and 1 right after it is closed. If both +** a missing and openclose UDF is supplied, the application should expect +** the following sequence of calls (for a single database): +** +** SELECT <openclose-udf>(<db filename>, <context>, 0); +** if( db not already on disk ){ +** SELECT <missing-udf>(<db filename>, <context>); +** } +** ... swarmvtab uses database ... +** SELECT <openclose-udf>(<db filename>, <context>, 1); +** +** The "maxopen" option is used to configure the maximum number of +** database files swarmvtab will hold open simultaneously (default 9). +** +** If an option name begins with a ":" character, then it is assumed +** to be an SQL parameter. In this case, the specified text value is +** bound to the same variable of the <sql-statement> before it is +** executed. It is an error of the named SQL parameter does not exist. +** For example: +** +** CREATE VIRTUAL TABLE swarm USING swarmvtab( +** 'SELECT :path || localfile, tbl, min, max FROM swarmdir', +** :path='/home/user/databases/' +** missing='missing_func' +** ); +*/ + +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <stdlib.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* +** Largest and smallest possible 64-bit signed integers. These macros +** copied from sqliteInt.h. +*/ +#ifndef LARGEST_INT64 +# define LARGEST_INT64 (0xffffffff|(((sqlite3_int64)0x7fffffff)<<32)) +#endif +#ifndef SMALLEST_INT64 +# define SMALLEST_INT64 (((sqlite3_int64)-1) - LARGEST_INT64) +#endif + +/* +** The following is also copied from sqliteInt.h. To facilitate coverage +** testing. +*/ +#ifndef ALWAYS +# if defined(SQLITE_COVERAGE_TEST) || defined(SQLITE_MUTATION_TEST) +# define ALWAYS(X) (1) +# define NEVER(X) (0) +# elif !defined(NDEBUG) +# define ALWAYS(X) ((X)?1:(assert(0),0)) +# define NEVER(X) ((X)?(assert(0),1):0) +# else +# define ALWAYS(X) (X) +# define NEVER(X) (X) +# endif +#endif + +/* +** The swarmvtab module attempts to keep the number of open database files +** at or below this limit. This may not be possible if there are too many +** simultaneous queries. +*/ +#define SWARMVTAB_MAX_OPEN 9 + +typedef struct UnionCsr UnionCsr; +typedef struct UnionTab UnionTab; +typedef struct UnionSrc UnionSrc; + +/* +** Each source table (row returned by the initialization query) is +** represented by an instance of the following structure stored in the +** UnionTab.aSrc[] array. +*/ +struct UnionSrc { + char *zDb; /* Database containing source table */ + char *zTab; /* Source table name */ + sqlite3_int64 iMin; /* Minimum rowid */ + sqlite3_int64 iMax; /* Maximum rowid */ + + /* Fields used by swarmvtab only */ + char *zFile; /* Database file containing table zTab */ + char *zContext; /* Context string, if any */ + int nUser; /* Current number of users */ + sqlite3 *db; /* Database handle */ + UnionSrc *pNextClosable; /* Next in list of closable sources */ +}; + +/* +** Virtual table type for union vtab. +*/ +struct UnionTab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3 *db; /* Database handle */ + int bSwarm; /* 1 for "swarmvtab", 0 for "unionvtab" */ + int iPK; /* INTEGER PRIMARY KEY column, or -1 */ + int nSrc; /* Number of elements in the aSrc[] array */ + UnionSrc *aSrc; /* Array of source tables, sorted by rowid */ + + /* Used by swarmvtab only */ + int bHasContext; /* Has context strings */ + char *zSourceStr; /* Expected unionSourceToStr() value */ + sqlite3_stmt *pNotFound; /* UDF to invoke if file not found on open */ + sqlite3_stmt *pOpenClose; /* UDF to invoke on open and close */ + + UnionSrc *pClosable; /* First in list of closable sources */ + int nOpen; /* Current number of open sources */ + int nMaxOpen; /* Maximum number of open sources */ +}; + +/* +** Virtual table cursor type for union vtab. +*/ +struct UnionCsr { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_stmt *pStmt; /* SQL statement to run */ + + /* Used by swarmvtab only */ + sqlite3_int64 iMaxRowid; /* Last rowid to visit */ + int iTab; /* Index of table read by pStmt */ +}; + +/* +** Given UnionTab table pTab and UnionSrc object pSrc, return the database +** handle that should be used to access the table identified by pSrc. This +** is the main db handle for "unionvtab" tables, or the source-specific +** handle for "swarmvtab". +*/ +#define unionGetDb(pTab, pSrc) ((pTab)->bSwarm ? (pSrc)->db : (pTab)->db) + +/* +** If *pRc is other than SQLITE_OK when this function is called, it +** always returns NULL. Otherwise, it attempts to allocate and return +** a pointer to nByte bytes of zeroed memory. If the memory allocation +** is attempted but fails, NULL is returned and *pRc is set to +** SQLITE_NOMEM. +*/ +static void *unionMalloc(int *pRc, sqlite3_int64 nByte){ + void *pRet; + assert( nByte>0 ); + if( *pRc==SQLITE_OK ){ + pRet = sqlite3_malloc64(nByte); + if( pRet ){ + memset(pRet, 0, (size_t)nByte); + }else{ + *pRc = SQLITE_NOMEM; + } + }else{ + pRet = 0; + } + return pRet; +} + +/* +** If *pRc is other than SQLITE_OK when this function is called, it +** always returns NULL. Otherwise, it attempts to allocate and return +** a copy of the nul-terminated string passed as the second argument. +** If the allocation is attempted but fails, NULL is returned and *pRc is +** set to SQLITE_NOMEM. +*/ +static char *unionStrdup(int *pRc, const char *zIn){ + char *zRet = 0; + if( zIn ){ + sqlite3_int64 nByte = strlen(zIn) + 1; + zRet = unionMalloc(pRc, nByte); + if( zRet ){ + memcpy(zRet, zIn, (size_t)nByte); + } + } + return zRet; +} + +/* +** If the first character of the string passed as the only argument to this +** function is one of the 4 that may be used as an open quote character +** in SQL, this function assumes that the input is a well-formed quoted SQL +** string. In this case the string is dequoted in place. +** +** If the first character of the input is not an open quote, then this +** function is a no-op. +*/ +static void unionDequote(char *z){ + if( z ){ + char q = z[0]; + + /* Set stack variable q to the close-quote character */ + if( q=='[' || q=='\'' || q=='"' || q=='`' ){ + int iIn = 1; + int iOut = 0; + if( q=='[' ) q = ']'; + while( ALWAYS(z[iIn]) ){ + if( z[iIn]==q ){ + if( z[iIn+1]!=q ){ + /* Character iIn was the close quote. */ + iIn++; + break; + }else{ + /* Character iIn and iIn+1 form an escaped quote character. Skip + ** the input cursor past both and copy a single quote character + ** to the output buffer. */ + iIn += 2; + z[iOut++] = q; + } + }else{ + z[iOut++] = z[iIn++]; + } + } + z[iOut] = '\0'; + } + } +} + +/* +** This function is a no-op if *pRc is set to other than SQLITE_OK when it +** is called. NULL is returned in this case. +** +** Otherwise, the SQL statement passed as the third argument is prepared +** against the database handle passed as the second. If the statement is +** successfully prepared, a pointer to the new statement handle is +** returned. It is the responsibility of the caller to eventually free the +** statement by calling sqlite3_finalize(). Alternatively, if statement +** compilation fails, NULL is returned, *pRc is set to an SQLite error +** code and *pzErr may be set to an error message buffer allocated by +** sqlite3_malloc(). +*/ +static sqlite3_stmt *unionPrepare( + int *pRc, /* IN/OUT: Error code */ + sqlite3 *db, /* Database handle */ + const char *zSql, /* SQL statement to prepare */ + char **pzErr /* OUT: Error message */ +){ + sqlite3_stmt *pRet = 0; + assert( pzErr ); + if( *pRc==SQLITE_OK ){ + int rc = sqlite3_prepare_v2(db, zSql, -1, &pRet, 0); + if( rc!=SQLITE_OK ){ + *pzErr = sqlite3_mprintf("sql error: %s", sqlite3_errmsg(db)); + *pRc = rc; + } + } + return pRet; +} + +/* +** Like unionPrepare(), except prepare the results of vprintf(zFmt, ...) +** instead of a constant SQL string. +*/ +static sqlite3_stmt *unionPreparePrintf( + int *pRc, /* IN/OUT: Error code */ + char **pzErr, /* OUT: Error message */ + sqlite3 *db, /* Database handle */ + const char *zFmt, /* printf() format string */ + ... /* Trailing printf args */ +){ + sqlite3_stmt *pRet = 0; + char *zSql; + va_list ap; + va_start(ap, zFmt); + + zSql = sqlite3_vmprintf(zFmt, ap); + if( *pRc==SQLITE_OK ){ + if( zSql==0 ){ + *pRc = SQLITE_NOMEM; + }else{ + pRet = unionPrepare(pRc, db, zSql, pzErr); + } + } + sqlite3_free(zSql); + + va_end(ap); + return pRet; +} + + +/* +** Call sqlite3_reset() on SQL statement pStmt. If *pRc is set to +** SQLITE_OK when this function is called, then it is set to the +** value returned by sqlite3_reset() before this function exits. +** In this case, *pzErr may be set to point to an error message +** buffer allocated by sqlite3_malloc(). +*/ +#if 0 +static void unionReset(int *pRc, sqlite3_stmt *pStmt, char **pzErr){ + int rc = sqlite3_reset(pStmt); + if( *pRc==SQLITE_OK ){ + *pRc = rc; + if( rc ){ + *pzErr = sqlite3_mprintf("%s", sqlite3_errmsg(sqlite3_db_handle(pStmt))); + } + } +} +#endif + +/* +** Call sqlite3_finalize() on SQL statement pStmt. If *pRc is set to +** SQLITE_OK when this function is called, then it is set to the +** value returned by sqlite3_finalize() before this function exits. +*/ +static void unionFinalize(int *pRc, sqlite3_stmt *pStmt, char **pzErr){ + sqlite3 *db = sqlite3_db_handle(pStmt); + int rc = sqlite3_finalize(pStmt); + if( *pRc==SQLITE_OK ){ + *pRc = rc; + if( rc ){ + *pzErr = sqlite3_mprintf("%s", sqlite3_errmsg(db)); + } + } +} + +/* +** If an "openclose" UDF was supplied when this virtual table was created, +** invoke it now. The first argument passed is the name of the database +** file for source pSrc. The second is integer value bClose. +** +** If successful, return SQLITE_OK. Otherwise an SQLite error code. In this +** case if argument pzErr is not NULL, also set (*pzErr) to an English +** language error message. The caller is responsible for eventually freeing +** any error message using sqlite3_free(). +*/ +static int unionInvokeOpenClose( + UnionTab *pTab, + UnionSrc *pSrc, + int bClose, + char **pzErr +){ + int rc = SQLITE_OK; + if( pTab->pOpenClose ){ + sqlite3_bind_text(pTab->pOpenClose, 1, pSrc->zFile, -1, SQLITE_STATIC); + if( pTab->bHasContext ){ + sqlite3_bind_text(pTab->pOpenClose, 2, pSrc->zContext, -1, SQLITE_STATIC); + } + sqlite3_bind_int(pTab->pOpenClose, 2+pTab->bHasContext, bClose); + sqlite3_step(pTab->pOpenClose); + if( SQLITE_OK!=(rc = sqlite3_reset(pTab->pOpenClose)) ){ + if( pzErr ){ + *pzErr = sqlite3_mprintf("%s", sqlite3_errmsg(pTab->db)); + } + } + } + return rc; +} + +/* +** This function is a no-op for unionvtab. For swarmvtab, it attempts to +** close open database files until at most nMax are open. An SQLite error +** code is returned if an error occurs, or SQLITE_OK otherwise. +*/ +static void unionCloseSources(UnionTab *pTab, int nMax){ + while( pTab->pClosable && pTab->nOpen>nMax ){ + UnionSrc *p; + UnionSrc **pp; + for(pp=&pTab->pClosable; (*pp)->pNextClosable; pp=&(*pp)->pNextClosable); + p = *pp; + assert( p->db ); + sqlite3_close(p->db); + p->db = 0; + *pp = 0; + pTab->nOpen--; + unionInvokeOpenClose(pTab, p, 1, 0); + } +} + +/* +** xDisconnect method. +*/ +static int unionDisconnect(sqlite3_vtab *pVtab){ + if( pVtab ){ + UnionTab *pTab = (UnionTab*)pVtab; + int i; + for(i=0; i<pTab->nSrc; i++){ + UnionSrc *pSrc = &pTab->aSrc[i]; + int bHaveSrcDb = (pSrc->db!=0); + sqlite3_close(pSrc->db); + if( bHaveSrcDb ){ + unionInvokeOpenClose(pTab, pSrc, 1, 0); + } + sqlite3_free(pSrc->zDb); + sqlite3_free(pSrc->zTab); + sqlite3_free(pSrc->zFile); + sqlite3_free(pSrc->zContext); + } + sqlite3_finalize(pTab->pNotFound); + sqlite3_finalize(pTab->pOpenClose); + sqlite3_free(pTab->zSourceStr); + sqlite3_free(pTab->aSrc); + sqlite3_free(pTab); + } + return SQLITE_OK; +} + +/* +** Check that the table identified by pSrc is a rowid table. If not, +** return SQLITE_ERROR and set (*pzErr) to point to an English language +** error message. If the table is a rowid table and no error occurs, +** return SQLITE_OK and leave (*pzErr) unmodified. +*/ +static int unionIsIntkeyTable( + sqlite3 *db, /* Database handle */ + UnionSrc *pSrc, /* Source table to test */ + char **pzErr /* OUT: Error message */ +){ + int bPk = 0; + const char *zType = 0; + int rc; + + sqlite3_table_column_metadata( + db, pSrc->zDb, pSrc->zTab, "_rowid_", &zType, 0, 0, &bPk, 0 + ); + rc = sqlite3_errcode(db); + if( rc==SQLITE_ERROR + || (rc==SQLITE_OK && (!bPk || sqlite3_stricmp("integer", zType))) + ){ + rc = SQLITE_ERROR; + *pzErr = sqlite3_mprintf("no such rowid table: %s%s%s", + (pSrc->zDb ? pSrc->zDb : ""), + (pSrc->zDb ? "." : ""), + pSrc->zTab + ); + } + return rc; +} + +/* +** This function is a no-op if *pRc is other than SQLITE_OK when it is +** called. In this case it returns NULL. +** +** Otherwise, this function checks that the source table passed as the +** second argument (a) exists, (b) is not a view and (c) has a column +** named "_rowid_" of type "integer" that is the primary key. +** If this is not the case, *pRc is set to SQLITE_ERROR and NULL is +** returned. +** +** Finally, if the source table passes the checks above, a nul-terminated +** string describing the column names and types belonging to the source +** table is returned. Tables with the same set of column names and types +** cause this function to return identical strings. Is is the responsibility +** of the caller to free the returned string using sqlite3_free() when +** it is no longer required. +*/ +static char *unionSourceToStr( + int *pRc, /* IN/OUT: Error code */ + UnionTab *pTab, /* Virtual table object */ + UnionSrc *pSrc, /* Source table to test */ + char **pzErr /* OUT: Error message */ +){ + char *zRet = 0; + if( *pRc==SQLITE_OK ){ + sqlite3 *db = unionGetDb(pTab, pSrc); + int rc = unionIsIntkeyTable(db, pSrc, pzErr); + sqlite3_stmt *pStmt = unionPrepare(&rc, db, + "SELECT group_concat(quote(name) || '.' || quote(type)) " + "FROM pragma_table_info(?, ?)", pzErr + ); + if( rc==SQLITE_OK ){ + sqlite3_bind_text(pStmt, 1, pSrc->zTab, -1, SQLITE_STATIC); + sqlite3_bind_text(pStmt, 2, pSrc->zDb, -1, SQLITE_STATIC); + if( SQLITE_ROW==sqlite3_step(pStmt) ){ + const char *z = (const char*)sqlite3_column_text(pStmt, 0); + zRet = unionStrdup(&rc, z); + } + unionFinalize(&rc, pStmt, pzErr); + } + *pRc = rc; + } + + return zRet; +} + +/* +** Check that all configured source tables exist and have the same column +** names and datatypes. If this is not the case, or if some other error +** occurs, return an SQLite error code. In this case *pzErr may be set +** to point to an error message buffer allocated by sqlite3_mprintf(). +** Or, if no problems regarding the source tables are detected and no +** other error occurs, SQLITE_OK is returned. +*/ +static int unionSourceCheck(UnionTab *pTab, char **pzErr){ + int rc = SQLITE_OK; + char *z0 = 0; + int i; + + assert( *pzErr==0 ); + z0 = unionSourceToStr(&rc, pTab, &pTab->aSrc[0], pzErr); + for(i=1; i<pTab->nSrc; i++){ + char *z = unionSourceToStr(&rc, pTab, &pTab->aSrc[i], pzErr); + if( rc==SQLITE_OK && sqlite3_stricmp(z, z0) ){ + *pzErr = sqlite3_mprintf("source table schema mismatch"); + rc = SQLITE_ERROR; + } + sqlite3_free(z); + } + sqlite3_free(z0); + + return rc; +} + +/* +** Try to open the swarmvtab database. If initially unable, invoke the +** not-found callback UDF and then try again. +*/ +static int unionOpenDatabaseInner(UnionTab *pTab, UnionSrc *pSrc, char **pzErr){ + static const int openFlags = SQLITE_OPEN_READONLY | SQLITE_OPEN_URI; + int rc; + + rc = unionInvokeOpenClose(pTab, pSrc, 0, pzErr); + if( rc!=SQLITE_OK ) return rc; + + rc = sqlite3_open_v2(pSrc->zFile, &pSrc->db, openFlags, 0); + if( rc==SQLITE_OK ) return rc; + if( pTab->pNotFound ){ + sqlite3_close(pSrc->db); + pSrc->db = 0; + sqlite3_bind_text(pTab->pNotFound, 1, pSrc->zFile, -1, SQLITE_STATIC); + if( pTab->bHasContext ){ + sqlite3_bind_text(pTab->pNotFound, 2, pSrc->zContext, -1, SQLITE_STATIC); + } + sqlite3_step(pTab->pNotFound); + if( SQLITE_OK!=(rc = sqlite3_reset(pTab->pNotFound)) ){ + *pzErr = sqlite3_mprintf("%s", sqlite3_errmsg(pTab->db)); + return rc; + } + rc = sqlite3_open_v2(pSrc->zFile, &pSrc->db, openFlags, 0); + } + if( rc!=SQLITE_OK ){ + *pzErr = sqlite3_mprintf("%s", sqlite3_errmsg(pSrc->db)); + } + return rc; +} + +/* +** This function may only be called for swarmvtab tables. The results of +** calling it on a unionvtab table are undefined. +** +** For a swarmvtab table, this function ensures that source database iSrc +** is open. If the database is opened successfully and the schema is as +** expected, or if it is already open when this function is called, SQLITE_OK +** is returned. +** +** Alternatively If an error occurs while opening the databases, or if the +** database schema is unsuitable, an SQLite error code is returned and (*pzErr) +** may be set to point to an English language error message. In this case it is +** the responsibility of the caller to eventually free the error message buffer +** using sqlite3_free(). +*/ +static int unionOpenDatabase(UnionTab *pTab, int iSrc, char **pzErr){ + int rc = SQLITE_OK; + UnionSrc *pSrc = &pTab->aSrc[iSrc]; + + assert( pTab->bSwarm && iSrc<pTab->nSrc ); + if( pSrc->db==0 ){ + unionCloseSources(pTab, pTab->nMaxOpen-1); + rc = unionOpenDatabaseInner(pTab, pSrc, pzErr); + if( rc==SQLITE_OK ){ + char *z = unionSourceToStr(&rc, pTab, pSrc, pzErr); + if( rc==SQLITE_OK ){ + if( pTab->zSourceStr==0 ){ + pTab->zSourceStr = z; + }else{ + if( sqlite3_stricmp(z, pTab->zSourceStr) ){ + *pzErr = sqlite3_mprintf("source table schema mismatch"); + rc = SQLITE_ERROR; + } + sqlite3_free(z); + } + } + } + + if( rc==SQLITE_OK ){ + pSrc->pNextClosable = pTab->pClosable; + pTab->pClosable = pSrc; + pTab->nOpen++; + }else{ + sqlite3_close(pSrc->db); + pSrc->db = 0; + unionInvokeOpenClose(pTab, pSrc, 1, 0); + } + } + + return rc; +} + + +/* +** This function is a no-op for unionvtab tables. For swarmvtab, increment +** the reference count for source table iTab. If the reference count was +** zero before it was incremented, also remove the source from the closable +** list. +*/ +static void unionIncrRefcount(UnionTab *pTab, int iTab){ + if( pTab->bSwarm ){ + UnionSrc *pSrc = &pTab->aSrc[iTab]; + assert( pSrc->nUser>=0 && pSrc->db ); + if( pSrc->nUser==0 ){ + UnionSrc **pp; + for(pp=&pTab->pClosable; *pp!=pSrc; pp=&(*pp)->pNextClosable); + *pp = pSrc->pNextClosable; + pSrc->pNextClosable = 0; + } + pSrc->nUser++; + } +} + +/* +** Finalize the SQL statement pCsr->pStmt and return the result. +** +** If this is a swarmvtab table (not unionvtab) and pCsr->pStmt was not +** NULL when this function was called, also decrement the reference +** count on the associated source table. If this means the source tables +** refcount is now zero, add it to the closable list. +*/ +static int unionFinalizeCsrStmt(UnionCsr *pCsr){ + int rc = SQLITE_OK; + if( pCsr->pStmt ){ + UnionTab *pTab = (UnionTab*)pCsr->base.pVtab; + UnionSrc *pSrc = &pTab->aSrc[pCsr->iTab]; + rc = sqlite3_finalize(pCsr->pStmt); + pCsr->pStmt = 0; + if( pTab->bSwarm ){ + pSrc->nUser--; + assert( pSrc->nUser>=0 ); + if( pSrc->nUser==0 ){ + pSrc->pNextClosable = pTab->pClosable; + pTab->pClosable = pSrc; + } + unionCloseSources(pTab, pTab->nMaxOpen); + } + } + return rc; +} + +/* +** Return true if the argument is a space, tab, CR or LF character. +*/ +static int union_isspace(char c){ + return (c==' ' || c=='\n' || c=='\r' || c=='\t'); +} + +/* +** Return true if the argument is an alphanumeric character in the +** ASCII range. +*/ +static int union_isidchar(char c){ + return ((c>='a' && c<='z') || (c>='A' && c<'Z') || (c>='0' && c<='9')); +} + +/* +** This function is called to handle all arguments following the first +** (the SQL statement) passed to a swarmvtab (not unionvtab) CREATE +** VIRTUAL TABLE statement. It may bind parameters to the SQL statement +** or configure members of the UnionTab object passed as the second +** argument. +** +** Refer to header comments at the top of this file for a description +** of the arguments parsed. +** +** This function is a no-op if *pRc is other than SQLITE_OK when it is +** called. Otherwise, if an error occurs, *pRc is set to an SQLite error +** code. In this case *pzErr may be set to point to a buffer containing +** an English language error message. It is the responsibility of the +** caller to eventually free the buffer using sqlite3_free(). +*/ +static void unionConfigureVtab( + int *pRc, /* IN/OUT: Error code */ + UnionTab *pTab, /* Table to configure */ + sqlite3_stmt *pStmt, /* SQL statement to find sources */ + int nArg, /* Number of entries in azArg[] array */ + const char * const *azArg, /* Array of arguments to consider */ + char **pzErr /* OUT: Error message */ +){ + int rc = *pRc; + int i; + if( rc==SQLITE_OK ){ + pTab->bHasContext = (sqlite3_column_count(pStmt)>4); + } + for(i=0; rc==SQLITE_OK && i<nArg; i++){ + char *zArg = unionStrdup(&rc, azArg[i]); + if( zArg ){ + int nOpt = 0; /* Size of option name in bytes */ + char *zOpt; /* Pointer to option name */ + char *zVal; /* Pointer to value */ + + unionDequote(zArg); + zOpt = zArg; + while( union_isspace(*zOpt) ) zOpt++; + zVal = zOpt; + if( *zVal==':' ) zVal++; + while( union_isidchar(*zVal) ) zVal++; + nOpt = (int)(zVal-zOpt); + + while( union_isspace(*zVal) ) zVal++; + if( *zVal=='=' ){ + zOpt[nOpt] = '\0'; + zVal++; + while( union_isspace(*zVal) ) zVal++; + zVal = unionStrdup(&rc, zVal); + if( zVal ){ + unionDequote(zVal); + if( zOpt[0]==':' ){ + /* A value to bind to the SQL statement */ + int iParam = sqlite3_bind_parameter_index(pStmt, zOpt); + if( iParam==0 ){ + *pzErr = sqlite3_mprintf( + "swarmvtab: no such SQL parameter: %s", zOpt + ); + rc = SQLITE_ERROR; + }else{ + rc = sqlite3_bind_text(pStmt, iParam, zVal, -1, SQLITE_TRANSIENT); + } + }else if( nOpt==7 && 0==sqlite3_strnicmp(zOpt, "maxopen", 7) ){ + pTab->nMaxOpen = atoi(zVal); + if( pTab->nMaxOpen<=0 ){ + *pzErr = sqlite3_mprintf("swarmvtab: illegal maxopen value"); + rc = SQLITE_ERROR; + } + }else if( nOpt==7 && 0==sqlite3_strnicmp(zOpt, "missing", 7) ){ + if( pTab->pNotFound ){ + *pzErr = sqlite3_mprintf( + "swarmvtab: duplicate \"missing\" option"); + rc = SQLITE_ERROR; + }else{ + pTab->pNotFound = unionPreparePrintf(&rc, pzErr, pTab->db, + "SELECT \"%w\"(?%s)", zVal, pTab->bHasContext ? ",?" : "" + ); + } + }else if( nOpt==9 && 0==sqlite3_strnicmp(zOpt, "openclose", 9) ){ + if( pTab->pOpenClose ){ + *pzErr = sqlite3_mprintf( + "swarmvtab: duplicate \"openclose\" option"); + rc = SQLITE_ERROR; + }else{ + pTab->pOpenClose = unionPreparePrintf(&rc, pzErr, pTab->db, + "SELECT \"%w\"(?,?%s)", zVal, pTab->bHasContext ? ",?" : "" + ); + } + }else{ + *pzErr = sqlite3_mprintf("swarmvtab: unrecognized option: %s",zOpt); + rc = SQLITE_ERROR; + } + sqlite3_free(zVal); + } + }else{ + if( i==0 && nArg==1 ){ + pTab->pNotFound = unionPreparePrintf(&rc, pzErr, pTab->db, + "SELECT \"%w\"(?)", zArg + ); + }else{ + *pzErr = sqlite3_mprintf( "swarmvtab: parse error: %s", azArg[i]); + rc = SQLITE_ERROR; + } + } + sqlite3_free(zArg); + } + } + *pRc = rc; +} + +/* +** xConnect/xCreate method. +** +** The argv[] array contains the following: +** +** argv[0] -> module name ("unionvtab" or "swarmvtab") +** argv[1] -> database name +** argv[2] -> table name +** argv[3] -> SQL statement +** argv[4] -> not-found callback UDF name +*/ +static int unionConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + UnionTab *pTab = 0; + int rc = SQLITE_OK; + int bSwarm = (pAux==0 ? 0 : 1); + const char *zVtab = (bSwarm ? "swarmvtab" : "unionvtab"); + + if( sqlite3_stricmp("temp", argv[1]) ){ + /* unionvtab tables may only be created in the temp schema */ + *pzErr = sqlite3_mprintf("%s tables must be created in TEMP schema", zVtab); + rc = SQLITE_ERROR; + }else if( argc<4 || (argc>4 && bSwarm==0) ){ + *pzErr = sqlite3_mprintf("wrong number of arguments for %s", zVtab); + rc = SQLITE_ERROR; + }else{ + int nAlloc = 0; /* Allocated size of pTab->aSrc[] */ + sqlite3_stmt *pStmt = 0; /* Argument statement */ + char *zArg = unionStrdup(&rc, argv[3]); /* Copy of argument to CVT */ + + /* Prepare the SQL statement. Instead of executing it directly, sort + ** the results by the "minimum rowid" field. This makes it easier to + ** check that there are no rowid range overlaps between source tables + ** and that the UnionTab.aSrc[] array is always sorted by rowid. */ + unionDequote(zArg); + pStmt = unionPreparePrintf(&rc, pzErr, db, + "SELECT * FROM (%z) ORDER BY 3", zArg + ); + + /* Allocate the UnionTab structure */ + pTab = unionMalloc(&rc, sizeof(UnionTab)); + if( pTab ){ + assert( rc==SQLITE_OK ); + pTab->db = db; + pTab->bSwarm = bSwarm; + pTab->nMaxOpen = SWARMVTAB_MAX_OPEN; + } + + /* Parse other CVT arguments, if any */ + if( bSwarm ){ + unionConfigureVtab(&rc, pTab, pStmt, argc-4, &argv[4], pzErr); + } + + /* Iterate through the rows returned by the SQL statement specified + ** as an argument to the CREATE VIRTUAL TABLE statement. */ + while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ + const char *zDb = (const char*)sqlite3_column_text(pStmt, 0); + const char *zTab = (const char*)sqlite3_column_text(pStmt, 1); + sqlite3_int64 iMin = sqlite3_column_int64(pStmt, 2); + sqlite3_int64 iMax = sqlite3_column_int64(pStmt, 3); + UnionSrc *pSrc; + + /* Grow the pTab->aSrc[] array if required. */ + if( nAlloc<=pTab->nSrc ){ + int nNew = nAlloc ? nAlloc*2 : 8; + UnionSrc *aNew = (UnionSrc*)sqlite3_realloc64( + pTab->aSrc, nNew*sizeof(UnionSrc) + ); + if( aNew==0 ){ + rc = SQLITE_NOMEM; + break; + }else{ + memset(&aNew[pTab->nSrc], 0, (nNew-pTab->nSrc)*sizeof(UnionSrc)); + pTab->aSrc = aNew; + nAlloc = nNew; + } + } + + /* Check for problems with the specified range of rowids */ + if( iMax<iMin || (pTab->nSrc>0 && iMin<=pTab->aSrc[pTab->nSrc-1].iMax) ){ + *pzErr = sqlite3_mprintf("rowid range mismatch error"); + rc = SQLITE_ERROR; + } + + if( rc==SQLITE_OK ){ + pSrc = &pTab->aSrc[pTab->nSrc++]; + pSrc->zTab = unionStrdup(&rc, zTab); + pSrc->iMin = iMin; + pSrc->iMax = iMax; + if( bSwarm ){ + pSrc->zFile = unionStrdup(&rc, zDb); + }else{ + pSrc->zDb = unionStrdup(&rc, zDb); + } + if( pTab->bHasContext ){ + const char *zContext = (const char*)sqlite3_column_text(pStmt, 4); + pSrc->zContext = unionStrdup(&rc, zContext); + } + } + } + unionFinalize(&rc, pStmt, pzErr); + pStmt = 0; + + /* It is an error if the SELECT statement returned zero rows. If only + ** because there is no way to determine the schema of the virtual + ** table in this case. */ + if( rc==SQLITE_OK && pTab->nSrc==0 ){ + *pzErr = sqlite3_mprintf("no source tables configured"); + rc = SQLITE_ERROR; + } + + /* For unionvtab, verify that all source tables exist and have + ** compatible schemas. For swarmvtab, attach the first database and + ** check that the first table is a rowid table only. */ + if( rc==SQLITE_OK ){ + if( bSwarm ){ + rc = unionOpenDatabase(pTab, 0, pzErr); + }else{ + rc = unionSourceCheck(pTab, pzErr); + } + } + + /* Compose a CREATE TABLE statement and pass it to declare_vtab() */ + if( rc==SQLITE_OK ){ + UnionSrc *pSrc = &pTab->aSrc[0]; + sqlite3 *tdb = unionGetDb(pTab, pSrc); + pStmt = unionPreparePrintf(&rc, pzErr, tdb, "SELECT " + "'CREATE TABLE xyz('" + " || group_concat(quote(name) || ' ' || type, ', ')" + " || ')'," + "max((cid+1) * (type='INTEGER' COLLATE nocase AND pk=1))-1 " + "FROM pragma_table_info(%Q, ?)", + pSrc->zTab, pSrc->zDb + ); + } + if( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pStmt) ){ + const char *zDecl = (const char*)sqlite3_column_text(pStmt, 0); + rc = sqlite3_declare_vtab(db, zDecl); + pTab->iPK = sqlite3_column_int(pStmt, 1); + } + + unionFinalize(&rc, pStmt, pzErr); + } + + if( rc!=SQLITE_OK ){ + unionDisconnect((sqlite3_vtab*)pTab); + pTab = 0; + } + + *ppVtab = (sqlite3_vtab*)pTab; + return rc; +} + +/* +** xOpen +*/ +static int unionOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + UnionCsr *pCsr; + int rc = SQLITE_OK; + (void)p; /* Suppress harmless warning */ + pCsr = (UnionCsr*)unionMalloc(&rc, sizeof(UnionCsr)); + *ppCursor = &pCsr->base; + return rc; +} + +/* +** xClose +*/ +static int unionClose(sqlite3_vtab_cursor *cur){ + UnionCsr *pCsr = (UnionCsr*)cur; + unionFinalizeCsrStmt(pCsr); + sqlite3_free(pCsr); + return SQLITE_OK; +} + +/* +** This function does the work of the xNext() method. Except that, if it +** returns SQLITE_ROW, it should be called again within the same xNext() +** method call. See unionNext() for details. +*/ +static int doUnionNext(UnionCsr *pCsr){ + int rc = SQLITE_OK; + assert( pCsr->pStmt ); + if( sqlite3_step(pCsr->pStmt)!=SQLITE_ROW ){ + UnionTab *pTab = (UnionTab*)pCsr->base.pVtab; + rc = unionFinalizeCsrStmt(pCsr); + if( rc==SQLITE_OK && pTab->bSwarm ){ + pCsr->iTab++; + if( pCsr->iTab<pTab->nSrc ){ + UnionSrc *pSrc = &pTab->aSrc[pCsr->iTab]; + if( pCsr->iMaxRowid>=pSrc->iMin ){ + /* It is necessary to scan the next table. */ + rc = unionOpenDatabase(pTab, pCsr->iTab, &pTab->base.zErrMsg); + pCsr->pStmt = unionPreparePrintf(&rc, &pTab->base.zErrMsg, pSrc->db, + "SELECT rowid, * FROM %Q %s %lld", + pSrc->zTab, + (pSrc->iMax>pCsr->iMaxRowid ? "WHERE _rowid_ <=" : "-- "), + pCsr->iMaxRowid + ); + if( rc==SQLITE_OK ){ + assert( pCsr->pStmt ); + unionIncrRefcount(pTab, pCsr->iTab); + rc = SQLITE_ROW; + } + } + } + } + } + + return rc; +} + +/* +** xNext +*/ +static int unionNext(sqlite3_vtab_cursor *cur){ + int rc; + do { + rc = doUnionNext((UnionCsr*)cur); + }while( rc==SQLITE_ROW ); + return rc; +} + +/* +** xColumn +*/ +static int unionColumn( + sqlite3_vtab_cursor *cur, + sqlite3_context *ctx, + int i +){ + UnionCsr *pCsr = (UnionCsr*)cur; + sqlite3_result_value(ctx, sqlite3_column_value(pCsr->pStmt, i+1)); + return SQLITE_OK; +} + +/* +** xRowid +*/ +static int unionRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + UnionCsr *pCsr = (UnionCsr*)cur; + *pRowid = sqlite3_column_int64(pCsr->pStmt, 0); + return SQLITE_OK; +} + +/* +** xEof +*/ +static int unionEof(sqlite3_vtab_cursor *cur){ + UnionCsr *pCsr = (UnionCsr*)cur; + return pCsr->pStmt==0; +} + +/* +** xFilter +*/ +static int unionFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + UnionTab *pTab = (UnionTab*)(pVtabCursor->pVtab); + UnionCsr *pCsr = (UnionCsr*)pVtabCursor; + int rc = SQLITE_OK; + int i; + char *zSql = 0; + int bZero = 0; + + sqlite3_int64 iMin = SMALLEST_INT64; + sqlite3_int64 iMax = LARGEST_INT64; + + assert( idxNum==0 + || idxNum==SQLITE_INDEX_CONSTRAINT_EQ + || idxNum==SQLITE_INDEX_CONSTRAINT_LE + || idxNum==SQLITE_INDEX_CONSTRAINT_GE + || idxNum==SQLITE_INDEX_CONSTRAINT_LT + || idxNum==SQLITE_INDEX_CONSTRAINT_GT + || idxNum==(SQLITE_INDEX_CONSTRAINT_GE|SQLITE_INDEX_CONSTRAINT_LE) + ); + + (void)idxStr; /* Suppress harmless warning */ + + if( idxNum==SQLITE_INDEX_CONSTRAINT_EQ ){ + assert( argc==1 ); + iMin = iMax = sqlite3_value_int64(argv[0]); + }else{ + + if( idxNum & (SQLITE_INDEX_CONSTRAINT_LE|SQLITE_INDEX_CONSTRAINT_LT) ){ + assert( argc>=1 ); + iMax = sqlite3_value_int64(argv[0]); + if( idxNum & SQLITE_INDEX_CONSTRAINT_LT ){ + if( iMax==SMALLEST_INT64 ){ + bZero = 1; + }else{ + iMax--; + } + } + } + + if( idxNum & (SQLITE_INDEX_CONSTRAINT_GE|SQLITE_INDEX_CONSTRAINT_GT) ){ + assert( argc>=1 ); + iMin = sqlite3_value_int64(argv[argc-1]); + if( idxNum & SQLITE_INDEX_CONSTRAINT_GT ){ + if( iMin==LARGEST_INT64 ){ + bZero = 1; + }else{ + iMin++; + } + } + } + } + + unionFinalizeCsrStmt(pCsr); + if( bZero ){ + return SQLITE_OK; + } + + for(i=0; i<pTab->nSrc; i++){ + UnionSrc *pSrc = &pTab->aSrc[i]; + if( iMin>pSrc->iMax || iMax<pSrc->iMin ){ + continue; + } + + zSql = sqlite3_mprintf("%z%sSELECT rowid, * FROM %s%q%s%Q" + , zSql + , (zSql ? " UNION ALL " : "") + , (pSrc->zDb ? "'" : "") + , (pSrc->zDb ? pSrc->zDb : "") + , (pSrc->zDb ? "'." : "") + , pSrc->zTab + ); + if( zSql==0 ){ + rc = SQLITE_NOMEM; + break; + } + + if( iMin==iMax ){ + zSql = sqlite3_mprintf("%z WHERE rowid=%lld", zSql, iMin); + }else{ + const char *zWhere = "WHERE"; + if( iMin!=SMALLEST_INT64 && iMin>pSrc->iMin ){ + zSql = sqlite3_mprintf("%z WHERE rowid>=%lld", zSql, iMin); + zWhere = "AND"; + } + if( iMax!=LARGEST_INT64 && iMax<pSrc->iMax ){ + zSql = sqlite3_mprintf("%z %s rowid<=%lld", zSql, zWhere, iMax); + } + } + + if( pTab->bSwarm ){ + pCsr->iTab = i; + pCsr->iMaxRowid = iMax; + rc = unionOpenDatabase(pTab, i, &pTab->base.zErrMsg); + break; + } + } + + if( zSql==0 ){ + return rc; + }else{ + sqlite3 *db = unionGetDb(pTab, &pTab->aSrc[pCsr->iTab]); + pCsr->pStmt = unionPrepare(&rc, db, zSql, &pTab->base.zErrMsg); + if( pCsr->pStmt ){ + unionIncrRefcount(pTab, pCsr->iTab); + } + sqlite3_free(zSql); + } + if( rc!=SQLITE_OK ) return rc; + return unionNext(pVtabCursor); +} + +/* +** xBestIndex. +** +** This implementation searches for constraints on the rowid field. EQ, +** LE, LT, GE and GT are handled. +** +** If there is an EQ comparison, then idxNum is set to INDEX_CONSTRAINT_EQ. +** In this case the only argument passed to xFilter is the rhs of the == +** operator. +** +** Otherwise, if an LE or LT constraint is found, then the INDEX_CONSTRAINT_LE +** or INDEX_CONSTRAINT_LT (but not both) bit is set in idxNum. The first +** argument to xFilter is the rhs of the <= or < operator. Similarly, if +** an GE or GT constraint is found, then the INDEX_CONSTRAINT_GE or +** INDEX_CONSTRAINT_GT bit is set in idxNum. The rhs of the >= or > operator +** is passed as either the first or second argument to xFilter, depending +** on whether or not there is also a LT|LE constraint. +*/ +static int unionBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + UnionTab *pTab = (UnionTab*)tab; + int iEq = -1; + int iLt = -1; + int iGt = -1; + int i; + + for(i=0; i<pIdxInfo->nConstraint; i++){ + struct sqlite3_index_constraint *p = &pIdxInfo->aConstraint[i]; + if( p->usable && (p->iColumn<0 || p->iColumn==pTab->iPK) ){ + switch( p->op ){ + case SQLITE_INDEX_CONSTRAINT_EQ: + iEq = i; + break; + case SQLITE_INDEX_CONSTRAINT_LE: + case SQLITE_INDEX_CONSTRAINT_LT: + iLt = i; + break; + case SQLITE_INDEX_CONSTRAINT_GE: + case SQLITE_INDEX_CONSTRAINT_GT: + iGt = i; + break; + } + } + } + + if( iEq>=0 ){ + pIdxInfo->estimatedRows = 1; + pIdxInfo->idxFlags = SQLITE_INDEX_SCAN_UNIQUE; + pIdxInfo->estimatedCost = 3.0; + pIdxInfo->idxNum = SQLITE_INDEX_CONSTRAINT_EQ; + pIdxInfo->aConstraintUsage[iEq].argvIndex = 1; + pIdxInfo->aConstraintUsage[iEq].omit = 1; + }else{ + int iCons = 1; + int idxNum = 0; + sqlite3_int64 nRow = 1000000; + if( iLt>=0 ){ + nRow = nRow / 2; + pIdxInfo->aConstraintUsage[iLt].argvIndex = iCons++; + pIdxInfo->aConstraintUsage[iLt].omit = 1; + idxNum |= pIdxInfo->aConstraint[iLt].op; + } + if( iGt>=0 ){ + nRow = nRow / 2; + pIdxInfo->aConstraintUsage[iGt].argvIndex = iCons++; + pIdxInfo->aConstraintUsage[iGt].omit = 1; + idxNum |= pIdxInfo->aConstraint[iGt].op; + } + pIdxInfo->estimatedRows = nRow; + pIdxInfo->estimatedCost = 3.0 * (double)nRow; + pIdxInfo->idxNum = idxNum; + } + + return SQLITE_OK; +} + +/* +** Register the unionvtab virtual table module with database handle db. +*/ +static int createUnionVtab(sqlite3 *db){ + static sqlite3_module unionModule = { + 0, /* iVersion */ + unionConnect, + unionConnect, + unionBestIndex, /* xBestIndex - query planner */ + unionDisconnect, + unionDisconnect, + unionOpen, /* xOpen - open a cursor */ + unionClose, /* xClose - close a cursor */ + unionFilter, /* xFilter - configure scan constraints */ + unionNext, /* xNext - advance a cursor */ + unionEof, /* xEof - check for end of scan */ + unionColumn, /* xColumn - read data */ + unionRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0 /* xShadowName */ + }; + int rc; + + rc = sqlite3_create_module(db, "unionvtab", &unionModule, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_module(db, "swarmvtab", &unionModule, (void*)db); + } + return rc; +} + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_unionvtab_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Suppress harmless warning */ +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = createUnionVtab(db); +#endif + return rc; +} diff --git a/ext/misc/urifuncs.c b/ext/misc/urifuncs.c new file mode 100644 index 0000000..9697b72 --- /dev/null +++ b/ext/misc/urifuncs.c @@ -0,0 +1,209 @@ +/* +** 2020-01-11 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements various SQL functions used to access +** the following SQLite C-language APIs: +** +** sqlite3_uri_parameter() +** sqlite3_uri_boolean() +** sqlite3_uri_int64() +** sqlite3_uri_key() +** sqlite3_filename_database() +** sqlite3_filename_journal() +** sqlite3_filename_wal() +** sqlite3_db_filename() +** +** These SQL functions are for testing and demonstration purposes only. +** +** +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* +** SQL function: sqlite3_db_filename(SCHEMA) +** +** Return the filename corresponding to SCHEMA. +*/ +static void func_db_filename( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zFile = sqlite3_db_filename(db, zSchema); + sqlite3_result_text(context, zFile, -1, SQLITE_TRANSIENT); +} + +/* +** SQL function: sqlite3_uri_parameter(SCHEMA,NAME) +** +** Return the value of the NAME query parameter to the database for SCHEMA +*/ +static void func_uri_parameter( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zName = (const char*)sqlite3_value_text(argv[1]); + const char *zFile = sqlite3_db_filename(db, zSchema); + const char *zRes = sqlite3_uri_parameter(zFile, zName); + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); +} + +/* +** SQL function: sqlite3_uri_boolean(SCHEMA,NAME,DEFAULT) +** +** Return the boolean value of the NAME query parameter to +** the database for SCHEMA +*/ +static void func_uri_boolean( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zName = (const char*)sqlite3_value_text(argv[1]); + const char *zFile = sqlite3_db_filename(db, zSchema); + int iDflt = sqlite3_value_int(argv[2]); + int iRes = sqlite3_uri_boolean(zFile, zName, iDflt); + sqlite3_result_int(context, iRes); +} + +/* +** SQL function: sqlite3_uri_key(SCHEMA,N) +** +** Return the name of the Nth query parameter +*/ +static void func_uri_key( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + int N = sqlite3_value_int(argv[1]); + const char *zFile = sqlite3_db_filename(db, zSchema); + const char *zRes = sqlite3_uri_key(zFile, N); + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); +} + +/* +** SQL function: sqlite3_uri_int64(SCHEMA,NAME,DEFAULT) +** +** Return the int64 value of the NAME query parameter to +** the database for SCHEMA +*/ +static void func_uri_int64( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zName = (const char*)sqlite3_value_text(argv[1]); + const char *zFile = sqlite3_db_filename(db, zSchema); + sqlite3_int64 iDflt = sqlite3_value_int64(argv[2]); + sqlite3_int64 iRes = sqlite3_uri_int64(zFile, zName, iDflt); + sqlite3_result_int64(context, iRes); +} + +/* +** SQL function: sqlite3_filename_database(SCHEMA) +** +** Return the database filename for SCHEMA +*/ +static void func_filename_database( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zFile = sqlite3_db_filename(db, zSchema); + const char *zRes = zFile ? sqlite3_filename_database(zFile) : 0; + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); +} + +/* +** SQL function: sqlite3_filename_journal(SCHEMA) +** +** Return the rollback journal filename for SCHEMA +*/ +static void func_filename_journal( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zFile = sqlite3_db_filename(db, zSchema); + const char *zRes = zFile ? sqlite3_filename_journal(zFile) : 0; + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); +} + +/* +** SQL function: sqlite3_filename_wal(SCHEMA) +** +** Return the WAL filename for SCHEMA +*/ +static void func_filename_wal( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + const char *zSchema = (const char*)sqlite3_value_text(argv[0]); + sqlite3 *db = sqlite3_context_db_handle(context); + const char *zFile = sqlite3_db_filename(db, zSchema); + const char *zRes = zFile ? sqlite3_filename_wal(zFile) : 0; + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_urifuncs_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + static const struct { + const char *zFuncName; + int nArg; + void (*xFunc)(sqlite3_context*,int,sqlite3_value**); + } aFunc[] = { + { "sqlite3_db_filename", 1, func_db_filename }, + { "sqlite3_uri_parameter", 2, func_uri_parameter }, + { "sqlite3_uri_boolean", 3, func_uri_boolean }, + { "sqlite3_uri_int64", 3, func_uri_int64 }, + { "sqlite3_uri_key", 2, func_uri_key }, + { "sqlite3_filename_database", 1, func_filename_database }, + { "sqlite3_filename_journal", 1, func_filename_journal }, + { "sqlite3_filename_wal", 1, func_filename_wal }, + }; + int rc = SQLITE_OK; + int i; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + for(i=0; rc==SQLITE_OK && i<sizeof(aFunc)/sizeof(aFunc[0]); i++){ + rc = sqlite3_create_function(db, aFunc[i].zFuncName, aFunc[i].nArg, + SQLITE_UTF8, 0, + aFunc[i].xFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/uuid.c b/ext/misc/uuid.c new file mode 100644 index 0000000..5b5b808 --- /dev/null +++ b/ext/misc/uuid.c @@ -0,0 +1,233 @@ +/* +** 2019-10-23 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This SQLite extension implements functions that handling RFC-4122 UUIDs +** Three SQL functions are implemented: +** +** uuid() - generate a version 4 UUID as a string +** uuid_str(X) - convert a UUID X into a well-formed UUID string +** uuid_blob(X) - convert a UUID X into a 16-byte blob +** +** The output from uuid() and uuid_str(X) are always well-formed RFC-4122 +** UUID strings in this format: +** +** xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx +** +** All of the 'x', 'M', and 'N' values are lower-case hexadecimal digits. +** The M digit indicates the "version". For uuid()-generated UUIDs, the +** version is always "4" (a random UUID). The upper three bits of N digit +** are the "variant". This library only supports variant 1 (indicated +** by values of N between '8' and 'b') as those are overwhelming the most +** common. Other variants are for legacy compatibility only. +** +** The output of uuid_blob(X) is always a 16-byte blob. The UUID input +** string is converted in network byte order (big-endian) in accordance +** with RFC-4122 specifications for variant-1 UUIDs. Note that network +** byte order is *always* used, even if the input self-identifies as a +** variant-2 UUID. +** +** The input X to the uuid_str() and uuid_blob() functions can be either +** a string or a BLOB. If it is a BLOB it must be exactly 16 bytes in +** length or else a NULL is returned. If the input is a string it must +** consist of 32 hexadecimal digits, upper or lower case, optionally +** surrounded by {...} and with optional "-" characters interposed in the +** middle. The flexibility of input is inspired by the PostgreSQL +** implementation of UUID functions that accept in all of the following +** formats: +** +** A0EEBC99-9C0B-4EF8-BB6D-6BB9BD380A11 +** {a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11} +** a0eebc999c0b4ef8bb6d6bb9bd380a11 +** a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11 +** {a0eebc99-9c0b4ef8-bb6d6bb9-bd380a11} +** +** If any of the above inputs are passed into uuid_str(), the output will +** always be in the canonical RFC-4122 format: +** +** a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 +** +** If the X input string has too few or too many digits or contains +** stray characters other than {, }, or -, then NULL is returned. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> +#include <ctype.h> + +#if !defined(SQLITE_ASCII) && !defined(SQLITE_EBCDIC) +# define SQLITE_ASCII 1 +#endif + +/* +** Translate a single byte of Hex into an integer. +** This routine only works if h really is a valid hexadecimal +** character: 0..9a..fA..F +*/ +static unsigned char sqlite3UuidHexToInt(int h){ + assert( (h>='0' && h<='9') || (h>='a' && h<='f') || (h>='A' && h<='F') ); +#ifdef SQLITE_ASCII + h += 9*(1&(h>>6)); +#endif +#ifdef SQLITE_EBCDIC + h += 9*(1&~(h>>4)); +#endif + return (unsigned char)(h & 0xf); +} + +/* +** Convert a 16-byte BLOB into a well-formed RFC-4122 UUID. The output +** buffer zStr should be at least 37 bytes in length. The output will +** be zero-terminated. +*/ +static void sqlite3UuidBlobToStr( + const unsigned char *aBlob, /* Input blob */ + unsigned char *zStr /* Write the answer here */ +){ + static const char zDigits[] = "0123456789abcdef"; + int i, k; + unsigned char x; + k = 0; + for(i=0, k=0x550; i<16; i++, k=k>>1){ + if( k&1 ){ + zStr[0] = '-'; + zStr++; + } + x = aBlob[i]; + zStr[0] = zDigits[x>>4]; + zStr[1] = zDigits[x&0xf]; + zStr += 2; + } + *zStr = 0; +} + +/* +** Attempt to parse a zero-terminated input string zStr into a binary +** UUID. Return 0 on success, or non-zero if the input string is not +** parsable. +*/ +static int sqlite3UuidStrToBlob( + const unsigned char *zStr, /* Input string */ + unsigned char *aBlob /* Write results here */ +){ + int i; + if( zStr[0]=='{' ) zStr++; + for(i=0; i<16; i++){ + if( zStr[0]=='-' ) zStr++; + if( isxdigit(zStr[0]) && isxdigit(zStr[1]) ){ + aBlob[i] = (sqlite3UuidHexToInt(zStr[0])<<4) + + sqlite3UuidHexToInt(zStr[1]); + zStr += 2; + }else{ + return 1; + } + } + if( zStr[0]=='}' ) zStr++; + return zStr[0]!=0; +} + +/* +** Render sqlite3_value pIn as a 16-byte UUID blob. Return a pointer +** to the blob, or NULL if the input is not well-formed. +*/ +static const unsigned char *sqlite3UuidInputToBlob( + sqlite3_value *pIn, /* Input text */ + unsigned char *pBuf /* output buffer */ +){ + switch( sqlite3_value_type(pIn) ){ + case SQLITE_TEXT: { + const unsigned char *z = sqlite3_value_text(pIn); + if( sqlite3UuidStrToBlob(z, pBuf) ) return 0; + return pBuf; + } + case SQLITE_BLOB: { + int n = sqlite3_value_bytes(pIn); + return n==16 ? sqlite3_value_blob(pIn) : 0; + } + default: { + return 0; + } + } +} + +/* Implementation of uuid() */ +static void sqlite3UuidFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char aBlob[16]; + unsigned char zStr[37]; + (void)argc; + (void)argv; + sqlite3_randomness(16, aBlob); + aBlob[6] = (aBlob[6]&0x0f) + 0x40; + aBlob[8] = (aBlob[8]&0x3f) + 0x80; + sqlite3UuidBlobToStr(aBlob, zStr); + sqlite3_result_text(context, (char*)zStr, 36, SQLITE_TRANSIENT); +} + +/* Implementation of uuid_str() */ +static void sqlite3UuidStrFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char aBlob[16]; + unsigned char zStr[37]; + const unsigned char *pBlob; + (void)argc; + pBlob = sqlite3UuidInputToBlob(argv[0], aBlob); + if( pBlob==0 ) return; + sqlite3UuidBlobToStr(pBlob, zStr); + sqlite3_result_text(context, (char*)zStr, 36, SQLITE_TRANSIENT); +} + +/* Implementation of uuid_blob() */ +static void sqlite3UuidBlobFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + unsigned char aBlob[16]; + const unsigned char *pBlob; + (void)argc; + pBlob = sqlite3UuidInputToBlob(argv[0], aBlob); + if( pBlob==0 ) return; + sqlite3_result_blob(context, pBlob, 16, SQLITE_TRANSIENT); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_uuid_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "uuid", 0, SQLITE_UTF8|SQLITE_INNOCUOUS, 0, + sqlite3UuidFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "uuid_str", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, sqlite3UuidStrFunc, 0, 0); + } + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "uuid_blob", 1, + SQLITE_UTF8|SQLITE_INNOCUOUS|SQLITE_DETERMINISTIC, + 0, sqlite3UuidBlobFunc, 0, 0); + } + return rc; +} diff --git a/ext/misc/vfslog.c b/ext/misc/vfslog.c new file mode 100644 index 0000000..cb5bc55 --- /dev/null +++ b/ext/misc/vfslog.c @@ -0,0 +1,760 @@ +/* +** 2013-10-09 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains the implementation of an SQLite vfs wrapper for +** unix that generates per-database log files of all disk activity. +*/ + +/* +** This module contains code for a wrapper VFS that causes a log of +** most VFS calls to be written into a file on disk. +** +** Each database connection creates a separate log file in the same +** directory as the original database and named after the original +** database. A unique suffix is added to avoid name collisions. +** Separate log files are used so that concurrent processes do not +** try to write log operations to the same file at the same instant, +** resulting in overwritten or comingled log text. +** +** Each individual log file records operations by a single database +** connection on both the original database and its associated rollback +** journal. +** +** The log files are in the comma-separated-value (CSV) format. The +** log files can be imported into an SQLite database using the ".import" +** command of the SQLite command-line shell for analysis. +** +** One technique for using this module is to append the text of this +** module to the end of a standard "sqlite3.c" amalgamation file then +** add the following compile-time options: +** +** -DSQLITE_EXTRA_INIT=sqlite3_register_vfslog +** -DSQLITE_USE_FCNTL_TRACE +** +** The first compile-time option causes the sqlite3_register_vfslog() +** function, defined below, to be invoked when SQLite is initialized. +** That causes this custom VFS to become the default VFS for all +** subsequent connections. The SQLITE_USE_FCNTL_TRACE option causes +** the SQLite core to issue extra sqlite3_file_control() operations +** with SQLITE_FCNTL_TRACE to give some indication of what is going +** on in the core. +*/ + +#include "sqlite3.h" +#include <string.h> +#include <assert.h> +#include <stdio.h> +#if SQLITE_OS_UNIX +# include <unistd.h> +#endif + +/* +** Forward declaration of objects used by this utility +*/ +typedef struct VLogLog VLogLog; +typedef struct VLogVfs VLogVfs; +typedef struct VLogFile VLogFile; + +/* There is a pair (an array of size 2) of the following objects for +** each database file being logged. The first contains the filename +** and is used to log I/O with the main database. The second has +** a NULL filename and is used to log I/O for the journal. Both +** out pointers are the same. +*/ +struct VLogLog { + VLogLog *pNext; /* Next in a list of all active logs */ + VLogLog **ppPrev; /* Pointer to this in the list */ + int nRef; /* Number of references to this object */ + int nFilename; /* Length of zFilename in bytes */ + char *zFilename; /* Name of database file. NULL for journal */ + FILE *out; /* Write information here */ +}; + +struct VLogVfs { + sqlite3_vfs base; /* VFS methods */ + sqlite3_vfs *pVfs; /* Parent VFS */ +}; + +struct VLogFile { + sqlite3_file base; /* IO methods */ + sqlite3_file *pReal; /* Underlying file handle */ + VLogLog *pLog; /* The log file for this file */ +}; + +#define REALVFS(p) (((VLogVfs*)(p))->pVfs) + +/* +** Methods for VLogFile +*/ +static int vlogClose(sqlite3_file*); +static int vlogRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); +static int vlogWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); +static int vlogTruncate(sqlite3_file*, sqlite3_int64 size); +static int vlogSync(sqlite3_file*, int flags); +static int vlogFileSize(sqlite3_file*, sqlite3_int64 *pSize); +static int vlogLock(sqlite3_file*, int); +static int vlogUnlock(sqlite3_file*, int); +static int vlogCheckReservedLock(sqlite3_file*, int *pResOut); +static int vlogFileControl(sqlite3_file*, int op, void *pArg); +static int vlogSectorSize(sqlite3_file*); +static int vlogDeviceCharacteristics(sqlite3_file*); + +/* +** Methods for VLogVfs +*/ +static int vlogOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); +static int vlogDelete(sqlite3_vfs*, const char *zName, int syncDir); +static int vlogAccess(sqlite3_vfs*, const char *zName, int flags, int *); +static int vlogFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); +static void *vlogDlOpen(sqlite3_vfs*, const char *zFilename); +static void vlogDlError(sqlite3_vfs*, int nByte, char *zErrMsg); +static void (*vlogDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); +static void vlogDlClose(sqlite3_vfs*, void*); +static int vlogRandomness(sqlite3_vfs*, int nByte, char *zOut); +static int vlogSleep(sqlite3_vfs*, int microseconds); +static int vlogCurrentTime(sqlite3_vfs*, double*); +static int vlogGetLastError(sqlite3_vfs*, int, char *); +static int vlogCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); + +static VLogVfs vlog_vfs = { + { + 1, /* iVersion */ + 0, /* szOsFile (set by register_vlog()) */ + 1024, /* mxPathname */ + 0, /* pNext */ + "vfslog", /* zName */ + 0, /* pAppData */ + vlogOpen, /* xOpen */ + vlogDelete, /* xDelete */ + vlogAccess, /* xAccess */ + vlogFullPathname, /* xFullPathname */ + vlogDlOpen, /* xDlOpen */ + vlogDlError, /* xDlError */ + vlogDlSym, /* xDlSym */ + vlogDlClose, /* xDlClose */ + vlogRandomness, /* xRandomness */ + vlogSleep, /* xSleep */ + vlogCurrentTime, /* xCurrentTime */ + vlogGetLastError, /* xGetLastError */ + vlogCurrentTimeInt64 /* xCurrentTimeInt64 */ + }, + 0 +}; + +static sqlite3_io_methods vlog_io_methods = { + 1, /* iVersion */ + vlogClose, /* xClose */ + vlogRead, /* xRead */ + vlogWrite, /* xWrite */ + vlogTruncate, /* xTruncate */ + vlogSync, /* xSync */ + vlogFileSize, /* xFileSize */ + vlogLock, /* xLock */ + vlogUnlock, /* xUnlock */ + vlogCheckReservedLock, /* xCheckReservedLock */ + vlogFileControl, /* xFileControl */ + vlogSectorSize, /* xSectorSize */ + vlogDeviceCharacteristics, /* xDeviceCharacteristics */ + 0, /* xShmMap */ + 0, /* xShmLock */ + 0, /* xShmBarrier */ + 0 /* xShmUnmap */ +}; + +#if SQLITE_OS_UNIX && !defined(NO_GETTOD) +#include <sys/time.h> +static sqlite3_uint64 vlog_time(){ + struct timeval sTime; + gettimeofday(&sTime, 0); + return sTime.tv_usec + (sqlite3_uint64)sTime.tv_sec * 1000000; +} +#elif SQLITE_OS_WIN +#include <windows.h> +#include <time.h> +static sqlite3_uint64 vlog_time(){ + FILETIME ft; + sqlite3_uint64 u64time = 0; + + GetSystemTimeAsFileTime(&ft); + + u64time |= ft.dwHighDateTime; + u64time <<= 32; + u64time |= ft.dwLowDateTime; + + /* ft is 100-nanosecond intervals, we want microseconds */ + return u64time /(sqlite3_uint64)10; +} +#else +static sqlite3_uint64 vlog_time(){ + return 0; +} +#endif + + +/* +** Write a message to the log file +*/ +static void vlogLogPrint( + VLogLog *pLog, /* The log file to write into */ + sqlite3_int64 tStart, /* Start time of system call */ + sqlite3_int64 tElapse, /* Elapse time of system call */ + const char *zOp, /* Type of system call */ + sqlite3_int64 iArg1, /* First argument */ + sqlite3_int64 iArg2, /* Second argument */ + const char *zArg3, /* Third argument */ + int iRes /* Result */ +){ + char z1[40], z2[40], z3[2000]; + if( pLog==0 ) return; + if( iArg1>=0 ){ + sqlite3_snprintf(sizeof(z1), z1, "%lld", iArg1); + }else{ + z1[0] = 0; + } + if( iArg2>=0 ){ + sqlite3_snprintf(sizeof(z2), z2, "%lld", iArg2); + }else{ + z2[0] = 0; + } + if( zArg3 ){ + sqlite3_snprintf(sizeof(z3), z3, "\"%.*w\"", sizeof(z3)-4, zArg3); + }else{ + z3[0] = 0; + } + fprintf(pLog->out,"%lld,%lld,%s,%d,%s,%s,%s,%d\n", + tStart, tElapse, zOp, pLog->zFilename==0, z1, z2, z3, iRes); +} + +/* +** List of all active log connections. Protected by the master mutex. +*/ +static VLogLog *allLogs = 0; + +/* +** Close a VLogLog object +*/ +static void vlogLogClose(VLogLog *p){ + if( p ){ + sqlite3_mutex *pMutex; + p->nRef--; + if( p->nRef>0 || p->zFilename==0 ) return; + pMutex = sqlite3_mutex_alloc(SQLITE_MUTEX_STATIC_MASTER); + sqlite3_mutex_enter(pMutex); + *p->ppPrev = p->pNext; + if( p->pNext ) p->pNext->ppPrev = p->ppPrev; + sqlite3_mutex_leave(pMutex); + fclose(p->out); + sqlite3_free(p); + } +} + +/* +** Open a VLogLog object on the given file +*/ +static VLogLog *vlogLogOpen(const char *zFilename){ + int nName = (int)strlen(zFilename); + int isJournal = 0; + sqlite3_mutex *pMutex; + VLogLog *pLog, *pTemp; + sqlite3_int64 tNow = 0; + if( nName>4 && strcmp(zFilename+nName-4,"-wal")==0 ){ + return 0; /* Do not log wal files */ + }else + if( nName>8 && strcmp(zFilename+nName-8,"-journal")==0 ){ + nName -= 8; + isJournal = 1; + }else if( nName>12 + && sqlite3_strglob("-mj??????9??", zFilename+nName-12)==0 ){ + return 0; /* Do not log master journal files */ + } + pTemp = sqlite3_malloc64( sizeof(*pLog)*2 + nName + 60 ); + if( pTemp==0 ) return 0; + pMutex = sqlite3_mutex_alloc(SQLITE_MUTEX_STATIC_MASTER); + sqlite3_mutex_enter(pMutex); + for(pLog=allLogs; pLog; pLog=pLog->pNext){ + if( pLog->nFilename==nName && !memcmp(pLog->zFilename, zFilename, nName) ){ + break; + } + } + if( pLog==0 ){ + pLog = pTemp; + pTemp = 0; + memset(pLog, 0, sizeof(*pLog)*2); + pLog->zFilename = (char*)&pLog[2]; + tNow = vlog_time(); + sqlite3_snprintf(nName+60, pLog->zFilename, "%.*s-debuglog-%lld", + nName, zFilename, tNow); + pLog->out = fopen(pLog->zFilename, "a"); + if( pLog->out==0 ){ + sqlite3_mutex_leave(pMutex); + sqlite3_free(pLog); + return 0; + } + pLog->nFilename = nName; + pLog[1].out = pLog[0].out; + pLog->ppPrev = &allLogs; + if( allLogs ) allLogs->ppPrev = &pLog->pNext; + pLog->pNext = allLogs; + allLogs = pLog; + } + sqlite3_mutex_leave(pMutex); + if( pTemp ){ + sqlite3_free(pTemp); + }else{ +#if SQLITE_OS_UNIX + char zHost[200]; + zHost[0] = 0; + gethostname(zHost, sizeof(zHost)-1); + zHost[sizeof(zHost)-1] = 0; + vlogLogPrint(pLog, tNow, 0, "IDENT", getpid(), -1, zHost, 0); +#endif + } + if( pLog && isJournal ) pLog++; + pLog->nRef++; + return pLog; +} + + +/* +** Close an vlog-file. +*/ +static int vlogClose(sqlite3_file *pFile){ + sqlite3_uint64 tStart, tElapse; + int rc = SQLITE_OK; + VLogFile *p = (VLogFile *)pFile; + + tStart = vlog_time(); + if( p->pReal->pMethods ){ + rc = p->pReal->pMethods->xClose(p->pReal); + } + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "CLOSE", -1, -1, 0, rc); + vlogLogClose(p->pLog); + return rc; +} + +/* +** Compute signature for a block of content. +** +** For blocks of 16 or fewer bytes, the signature is just a hex dump of +** the entire block. +** +** For blocks of more than 16 bytes, the signature is a hex dump of the +** first 8 bytes followed by a 64-bit has of the entire block. +*/ +static void vlogSignature(unsigned char *p, int n, char *zCksum){ + unsigned int s0 = 0, s1 = 0; + unsigned int *pI; + int i; + if( n<=16 ){ + for(i=0; i<n; i++) sqlite3_snprintf(3, zCksum+i*2, "%02x", p[i]); + }else{ + pI = (unsigned int*)p; + for(i=0; i<n-7; i+=8){ + s0 += pI[0] + s1; + s1 += pI[1] + s0; + pI += 2; + } + for(i=0; i<8; i++) sqlite3_snprintf(3, zCksum+i*2, "%02x", p[i]); + sqlite3_snprintf(18, zCksum+i*2, "-%08x%08x", s0, s1); + } +} + +/* +** Convert a big-endian 32-bit integer into a native integer +*/ +static int bigToNative(const unsigned char *x){ + return (x[0]<<24) + (x[1]<<16) + (x[2]<<8) + x[3]; +} + +/* +** Read data from an vlog-file. +*/ +static int vlogRead( + sqlite3_file *pFile, + void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + char zSig[40]; + + tStart = vlog_time(); + rc = p->pReal->pMethods->xRead(p->pReal, zBuf, iAmt, iOfst); + tElapse = vlog_time() - tStart; + if( rc==SQLITE_OK ){ + vlogSignature(zBuf, iAmt, zSig); + }else{ + zSig[0] = 0; + } + vlogLogPrint(p->pLog, tStart, tElapse, "READ", iAmt, iOfst, zSig, rc); + if( rc==SQLITE_OK + && p->pLog + && p->pLog->zFilename + && iOfst<=24 + && iOfst+iAmt>=28 + ){ + unsigned char *x = ((unsigned char*)zBuf)+(24-iOfst); + unsigned iCtr, nFree = -1; + char *zFree = 0; + char zStr[12]; + iCtr = bigToNative(x); + if( iOfst+iAmt>=40 ){ + zFree = zStr; + sqlite3_snprintf(sizeof(zStr), zStr, "%d", bigToNative(x+8)); + nFree = bigToNative(x+12); + } + vlogLogPrint(p->pLog, tStart, 0, "CHNGCTR-READ", iCtr, nFree, zFree, 0); + } + return rc; +} + +/* +** Write data to an vlog-file. +*/ +static int vlogWrite( + sqlite3_file *pFile, + const void *z, + int iAmt, + sqlite_int64 iOfst +){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + char zSig[40]; + + tStart = vlog_time(); + vlogSignature((unsigned char*)z, iAmt, zSig); + rc = p->pReal->pMethods->xWrite(p->pReal, z, iAmt, iOfst); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "WRITE", iAmt, iOfst, zSig, rc); + if( rc==SQLITE_OK + && p->pLog + && p->pLog->zFilename + && iOfst<=24 + && iOfst+iAmt>=28 + ){ + unsigned char *x = ((unsigned char*)z)+(24-iOfst); + unsigned iCtr, nFree = -1; + char *zFree = 0; + char zStr[12]; + iCtr = bigToNative(x); + if( iOfst+iAmt>=40 ){ + zFree = zStr; + sqlite3_snprintf(sizeof(zStr), zStr, "%d", bigToNative(x+8)); + nFree = bigToNative(x+12); + } + vlogLogPrint(p->pLog, tStart, 0, "CHNGCTR-WRITE", iCtr, nFree, zFree, 0); + } + return rc; +} + +/* +** Truncate an vlog-file. +*/ +static int vlogTruncate(sqlite3_file *pFile, sqlite_int64 size){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xTruncate(p->pReal, size); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "TRUNCATE", size, -1, 0, rc); + return rc; +} + +/* +** Sync an vlog-file. +*/ +static int vlogSync(sqlite3_file *pFile, int flags){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xSync(p->pReal, flags); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "SYNC", flags, -1, 0, rc); + return rc; +} + +/* +** Return the current file-size of an vlog-file. +*/ +static int vlogFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xFileSize(p->pReal, pSize); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "FILESIZE", *pSize, -1, 0, rc); + return rc; +} + +/* +** Lock an vlog-file. +*/ +static int vlogLock(sqlite3_file *pFile, int eLock){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xLock(p->pReal, eLock); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "LOCK", eLock, -1, 0, rc); + return rc; +} + +/* +** Unlock an vlog-file. +*/ +static int vlogUnlock(sqlite3_file *pFile, int eLock){ + int rc; + sqlite3_uint64 tStart; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + vlogLogPrint(p->pLog, tStart, 0, "UNLOCK", eLock, -1, 0, 0); + rc = p->pReal->pMethods->xUnlock(p->pReal, eLock); + return rc; +} + +/* +** Check if another file-handle holds a RESERVED lock on an vlog-file. +*/ +static int vlogCheckReservedLock(sqlite3_file *pFile, int *pResOut){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xCheckReservedLock(p->pReal, pResOut); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "CHECKRESERVEDLOCK", + *pResOut, -1, "", rc); + return rc; +} + +/* +** File control method. For custom operations on an vlog-file. +*/ +static int vlogFileControl(sqlite3_file *pFile, int op, void *pArg){ + VLogFile *p = (VLogFile *)pFile; + sqlite3_uint64 tStart, tElapse; + int rc; + tStart = vlog_time(); + rc = p->pReal->pMethods->xFileControl(p->pReal, op, pArg); + if( op==SQLITE_FCNTL_VFSNAME && rc==SQLITE_OK ){ + *(char**)pArg = sqlite3_mprintf("vlog/%z", *(char**)pArg); + } + tElapse = vlog_time() - tStart; + if( op==SQLITE_FCNTL_TRACE ){ + vlogLogPrint(p->pLog, tStart, tElapse, "TRACE", op, -1, pArg, rc); + }else if( op==SQLITE_FCNTL_PRAGMA ){ + const char **azArg = (const char **)pArg; + vlogLogPrint(p->pLog, tStart, tElapse, "FILECONTROL", op, -1, azArg[1], rc); + }else if( op==SQLITE_FCNTL_SIZE_HINT ){ + sqlite3_int64 sz = *(sqlite3_int64*)pArg; + vlogLogPrint(p->pLog, tStart, tElapse, "FILECONTROL", op, sz, 0, rc); + }else{ + vlogLogPrint(p->pLog, tStart, tElapse, "FILECONTROL", op, -1, 0, rc); + } + return rc; +} + +/* +** Return the sector-size in bytes for an vlog-file. +*/ +static int vlogSectorSize(sqlite3_file *pFile){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xSectorSize(p->pReal); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "SECTORSIZE", -1, -1, 0, rc); + return rc; +} + +/* +** Return the device characteristic flags supported by an vlog-file. +*/ +static int vlogDeviceCharacteristics(sqlite3_file *pFile){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogFile *p = (VLogFile *)pFile; + tStart = vlog_time(); + rc = p->pReal->pMethods->xDeviceCharacteristics(p->pReal); + tElapse = vlog_time() - tStart; + vlogLogPrint(p->pLog, tStart, tElapse, "DEVCHAR", -1, -1, 0, rc); + return rc; +} + + +/* +** Open an vlog file handle. +*/ +static int vlogOpen( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_file *pFile, + int flags, + int *pOutFlags +){ + int rc; + sqlite3_uint64 tStart, tElapse; + sqlite3_int64 iArg2; + VLogFile *p = (VLogFile*)pFile; + + p->pReal = (sqlite3_file*)&p[1]; + if( (flags & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_MAIN_JOURNAL))!=0 ){ + p->pLog = vlogLogOpen(zName); + }else{ + p->pLog = 0; + } + tStart = vlog_time(); + rc = REALVFS(pVfs)->xOpen(REALVFS(pVfs), zName, p->pReal, flags, pOutFlags); + tElapse = vlog_time() - tStart; + iArg2 = pOutFlags ? *pOutFlags : -1; + vlogLogPrint(p->pLog, tStart, tElapse, "OPEN", flags, iArg2, 0, rc); + if( rc==SQLITE_OK ){ + pFile->pMethods = &vlog_io_methods; + }else{ + if( p->pLog ) vlogLogClose(p->pLog); + p->pLog = 0; + } + return rc; +} + +/* +** Delete the file located at zPath. If the dirSync argument is true, +** ensure the file-system modifications are synced to disk before +** returning. +*/ +static int vlogDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogLog *pLog; + tStart = vlog_time(); + rc = REALVFS(pVfs)->xDelete(REALVFS(pVfs), zPath, dirSync); + tElapse = vlog_time() - tStart; + pLog = vlogLogOpen(zPath); + vlogLogPrint(pLog, tStart, tElapse, "DELETE", dirSync, -1, 0, rc); + vlogLogClose(pLog); + return rc; +} + +/* +** Test for access permissions. Return true if the requested permission +** is available, or false otherwise. +*/ +static int vlogAccess( + sqlite3_vfs *pVfs, + const char *zPath, + int flags, + int *pResOut +){ + int rc; + sqlite3_uint64 tStart, tElapse; + VLogLog *pLog; + tStart = vlog_time(); + rc = REALVFS(pVfs)->xAccess(REALVFS(pVfs), zPath, flags, pResOut); + tElapse = vlog_time() - tStart; + pLog = vlogLogOpen(zPath); + vlogLogPrint(pLog, tStart, tElapse, "ACCESS", flags, *pResOut, 0, rc); + vlogLogClose(pLog); + return rc; +} + +/* +** Populate buffer zOut with the full canonical pathname corresponding +** to the pathname in zPath. zOut is guaranteed to point to a buffer +** of at least (INST_MAX_PATHNAME+1) bytes. +*/ +static int vlogFullPathname( + sqlite3_vfs *pVfs, + const char *zPath, + int nOut, + char *zOut +){ + return REALVFS(pVfs)->xFullPathname(REALVFS(pVfs), zPath, nOut, zOut); +} + +/* +** Open the dynamic library located at zPath and return a handle. +*/ +static void *vlogDlOpen(sqlite3_vfs *pVfs, const char *zPath){ + return REALVFS(pVfs)->xDlOpen(REALVFS(pVfs), zPath); +} + +/* +** Populate the buffer zErrMsg (size nByte bytes) with a human readable +** utf-8 string describing the most recent error encountered associated +** with dynamic libraries. +*/ +static void vlogDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ + REALVFS(pVfs)->xDlError(REALVFS(pVfs), nByte, zErrMsg); +} + +/* +** Return a pointer to the symbol zSymbol in the dynamic library pHandle. +*/ +static void (*vlogDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ + return REALVFS(pVfs)->xDlSym(REALVFS(pVfs), p, zSym); +} + +/* +** Close the dynamic library handle pHandle. +*/ +static void vlogDlClose(sqlite3_vfs *pVfs, void *pHandle){ + REALVFS(pVfs)->xDlClose(REALVFS(pVfs), pHandle); +} + +/* +** Populate the buffer pointed to by zBufOut with nByte bytes of +** random data. +*/ +static int vlogRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ + return REALVFS(pVfs)->xRandomness(REALVFS(pVfs), nByte, zBufOut); +} + +/* +** Sleep for nMicro microseconds. Return the number of microseconds +** actually slept. +*/ +static int vlogSleep(sqlite3_vfs *pVfs, int nMicro){ + return REALVFS(pVfs)->xSleep(REALVFS(pVfs), nMicro); +} + +/* +** Return the current time as a Julian Day number in *pTimeOut. +*/ +static int vlogCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ + return REALVFS(pVfs)->xCurrentTime(REALVFS(pVfs), pTimeOut); +} + +static int vlogGetLastError(sqlite3_vfs *pVfs, int a, char *b){ + return REALVFS(pVfs)->xGetLastError(REALVFS(pVfs), a, b); +} +static int vlogCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ + return REALVFS(pVfs)->xCurrentTimeInt64(REALVFS(pVfs), p); +} + +/* +** Register debugvfs as the default VFS for this process. +*/ +int sqlite3_register_vfslog(const char *zArg){ + vlog_vfs.pVfs = sqlite3_vfs_find(0); + if( vlog_vfs.pVfs==0 ) return SQLITE_ERROR; + vlog_vfs.base.szOsFile = sizeof(VLogFile) + vlog_vfs.pVfs->szOsFile; + return sqlite3_vfs_register(&vlog_vfs.base, 1); +} diff --git a/ext/misc/vfsstat.c b/ext/misc/vfsstat.c new file mode 100644 index 0000000..83a7a3d --- /dev/null +++ b/ext/misc/vfsstat.c @@ -0,0 +1,820 @@ +/* +** 2016-05-27 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file contains the implementation of an SQLite vfs shim that +** tracks I/O. Access to the accumulated status counts is provided using +** an eponymous virtual table. +*/ +#include <sqlite3ext.h> +SQLITE_EXTENSION_INIT1 + +/* +** This module contains code for a wrapper VFS that cause stats for +** most VFS calls to be recorded. +** +** To use this module, first compile it as a loadable extension. See +** https://www.sqlite.org/loadext.html#build for compilations instructions. +** +** After compliing, load this extension, then open database connections to be +** measured. Query usages status using the vfsstat virtual table: +** +** SELECT * FROM vfsstat; +** +** Reset counters using UPDATE statements against vfsstat: +** +** UPDATE vfsstat SET count=0; +** +** EXAMPLE SCRIPT: +** +** .load ./vfsstat +** .open test.db +** DROP TABLE IF EXISTS t1; +** CREATE TABLE t1(x,y); +** INSERT INTO t1 VALUES(123, randomblob(5000)); +** CREATE INDEX t1x ON t1(x); +** DROP TABLE t1; +** VACUUM; +** SELECT * FROM vfsstat WHERE count>0; +** +** LIMITATIONS: +** +** This module increments counters without using mutex protection. So if +** two or more threads try to use this module at the same time, race conditions +** may occur which mess up the counts. This is harmless, other than giving +** incorrect statistics. +*/ +#include <string.h> +#include <stdlib.h> +#include <assert.h> + +/* +** File types +*/ +#define VFSSTAT_MAIN 0 /* Main database file */ +#define VFSSTAT_JOURNAL 1 /* Rollback journal */ +#define VFSSTAT_WAL 2 /* Write-ahead log file */ +#define VFSSTAT_MASTERJRNL 3 /* Master journal */ +#define VFSSTAT_SUBJRNL 4 /* Subjournal */ +#define VFSSTAT_TEMPDB 5 /* TEMP database */ +#define VFSSTAT_TEMPJRNL 6 /* Journal for TEMP database */ +#define VFSSTAT_TRANSIENT 7 /* Transient database */ +#define VFSSTAT_ANY 8 /* Unspecified file type */ +#define VFSSTAT_nFile 9 /* This many file types */ + +/* Names of the file types. These are allowed values for the +** first column of the vfsstat virtual table. +*/ +static const char *azFile[] = { + "database", "journal", "wal", "master-journal", "sub-journal", + "temp-database", "temp-journal", "transient-db", "*" +}; + +/* +** Stat types +*/ +#define VFSSTAT_BYTESIN 0 /* Bytes read in */ +#define VFSSTAT_BYTESOUT 1 /* Bytes written out */ +#define VFSSTAT_READ 2 /* Read requests */ +#define VFSSTAT_WRITE 3 /* Write requests */ +#define VFSSTAT_SYNC 4 /* Syncs */ +#define VFSSTAT_OPEN 5 /* File opens */ +#define VFSSTAT_LOCK 6 /* Lock requests */ +#define VFSSTAT_ACCESS 0 /* xAccess calls. filetype==ANY only */ +#define VFSSTAT_DELETE 1 /* xDelete calls. filetype==ANY only */ +#define VFSSTAT_FULLPATH 2 /* xFullPathname calls. ANY only */ +#define VFSSTAT_RANDOM 3 /* xRandomness calls. ANY only */ +#define VFSSTAT_SLEEP 4 /* xSleep calls. ANY only */ +#define VFSSTAT_CURTIME 5 /* xCurrentTime calls. ANY only */ +#define VFSSTAT_nStat 7 /* This many stat types */ + + +/* Names for the second column of the vfsstat virtual table for all +** cases except when the first column is "*" or VFSSTAT_ANY. */ +static const char *azStat[] = { + "bytes-in", "bytes-out", "read", "write", "sync", "open", "lock", +}; +static const char *azStatAny[] = { + "access", "delete", "fullpathname", "randomness", "sleep", "currenttimestamp", + "not-used" +}; + +/* Total number of counters */ +#define VFSSTAT_MXCNT (VFSSTAT_nStat*VFSSTAT_nFile) + +/* +** Performance stats are collected in an instance of the following +** global array. +*/ +static sqlite3_uint64 aVfsCnt[VFSSTAT_MXCNT]; + +/* +** Access to a specific counter +*/ +#define STATCNT(filetype,stat) (aVfsCnt[(filetype)*VFSSTAT_nStat+(stat)]) + +/* +** Forward declaration of objects used by this utility +*/ +typedef struct VStatVfs VStatVfs; +typedef struct VStatFile VStatFile; + +/* An instance of the VFS */ +struct VStatVfs { + sqlite3_vfs base; /* VFS methods */ + sqlite3_vfs *pVfs; /* Parent VFS */ +}; + +/* An open file */ +struct VStatFile { + sqlite3_file base; /* IO methods */ + sqlite3_file *pReal; /* Underlying file handle */ + unsigned char eFiletype; /* What type of file is this */ +}; + +#define REALVFS(p) (((VStatVfs*)(p))->pVfs) + +/* +** Methods for VStatFile +*/ +static int vstatClose(sqlite3_file*); +static int vstatRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); +static int vstatWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); +static int vstatTruncate(sqlite3_file*, sqlite3_int64 size); +static int vstatSync(sqlite3_file*, int flags); +static int vstatFileSize(sqlite3_file*, sqlite3_int64 *pSize); +static int vstatLock(sqlite3_file*, int); +static int vstatUnlock(sqlite3_file*, int); +static int vstatCheckReservedLock(sqlite3_file*, int *pResOut); +static int vstatFileControl(sqlite3_file*, int op, void *pArg); +static int vstatSectorSize(sqlite3_file*); +static int vstatDeviceCharacteristics(sqlite3_file*); +static int vstatShmMap(sqlite3_file*, int iPg, int pgsz, int, void volatile**); +static int vstatShmLock(sqlite3_file*, int offset, int n, int flags); +static void vstatShmBarrier(sqlite3_file*); +static int vstatShmUnmap(sqlite3_file*, int deleteFlag); +static int vstatFetch(sqlite3_file*, sqlite3_int64 iOfst, int iAmt, void **pp); +static int vstatUnfetch(sqlite3_file*, sqlite3_int64 iOfst, void *p); + +/* +** Methods for VStatVfs +*/ +static int vstatOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); +static int vstatDelete(sqlite3_vfs*, const char *zName, int syncDir); +static int vstatAccess(sqlite3_vfs*, const char *zName, int flags, int *); +static int vstatFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); +static void *vstatDlOpen(sqlite3_vfs*, const char *zFilename); +static void vstatDlError(sqlite3_vfs*, int nByte, char *zErrMsg); +static void (*vstatDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); +static void vstatDlClose(sqlite3_vfs*, void*); +static int vstatRandomness(sqlite3_vfs*, int nByte, char *zOut); +static int vstatSleep(sqlite3_vfs*, int microseconds); +static int vstatCurrentTime(sqlite3_vfs*, double*); +static int vstatGetLastError(sqlite3_vfs*, int, char *); +static int vstatCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); + +static VStatVfs vstat_vfs = { + { + 2, /* iVersion */ + 0, /* szOsFile (set by register_vstat()) */ + 1024, /* mxPathname */ + 0, /* pNext */ + "vfslog", /* zName */ + 0, /* pAppData */ + vstatOpen, /* xOpen */ + vstatDelete, /* xDelete */ + vstatAccess, /* xAccess */ + vstatFullPathname, /* xFullPathname */ + vstatDlOpen, /* xDlOpen */ + vstatDlError, /* xDlError */ + vstatDlSym, /* xDlSym */ + vstatDlClose, /* xDlClose */ + vstatRandomness, /* xRandomness */ + vstatSleep, /* xSleep */ + vstatCurrentTime, /* xCurrentTime */ + vstatGetLastError, /* xGetLastError */ + vstatCurrentTimeInt64 /* xCurrentTimeInt64 */ + }, + 0 +}; + +static const sqlite3_io_methods vstat_io_methods = { + 3, /* iVersion */ + vstatClose, /* xClose */ + vstatRead, /* xRead */ + vstatWrite, /* xWrite */ + vstatTruncate, /* xTruncate */ + vstatSync, /* xSync */ + vstatFileSize, /* xFileSize */ + vstatLock, /* xLock */ + vstatUnlock, /* xUnlock */ + vstatCheckReservedLock, /* xCheckReservedLock */ + vstatFileControl, /* xFileControl */ + vstatSectorSize, /* xSectorSize */ + vstatDeviceCharacteristics, /* xDeviceCharacteristics */ + vstatShmMap, /* xShmMap */ + vstatShmLock, /* xShmLock */ + vstatShmBarrier, /* xShmBarrier */ + vstatShmUnmap, /* xShmUnmap */ + vstatFetch, /* xFetch */ + vstatUnfetch /* xUnfetch */ +}; + + + +/* +** Close an vstat-file. +*/ +static int vstatClose(sqlite3_file *pFile){ + VStatFile *p = (VStatFile *)pFile; + int rc = SQLITE_OK; + + if( p->pReal->pMethods ){ + rc = p->pReal->pMethods->xClose(p->pReal); + } + return rc; +} + + +/* +** Read data from an vstat-file. +*/ +static int vstatRead( + sqlite3_file *pFile, + void *zBuf, + int iAmt, + sqlite_int64 iOfst +){ + int rc; + VStatFile *p = (VStatFile *)pFile; + + rc = p->pReal->pMethods->xRead(p->pReal, zBuf, iAmt, iOfst); + STATCNT(p->eFiletype,VFSSTAT_READ)++; + if( rc==SQLITE_OK ){ + STATCNT(p->eFiletype,VFSSTAT_BYTESIN) += iAmt; + } + return rc; +} + +/* +** Write data to an vstat-file. +*/ +static int vstatWrite( + sqlite3_file *pFile, + const void *z, + int iAmt, + sqlite_int64 iOfst +){ + int rc; + VStatFile *p = (VStatFile *)pFile; + + rc = p->pReal->pMethods->xWrite(p->pReal, z, iAmt, iOfst); + STATCNT(p->eFiletype,VFSSTAT_WRITE)++; + if( rc==SQLITE_OK ){ + STATCNT(p->eFiletype,VFSSTAT_BYTESOUT) += iAmt; + } + return rc; +} + +/* +** Truncate an vstat-file. +*/ +static int vstatTruncate(sqlite3_file *pFile, sqlite_int64 size){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xTruncate(p->pReal, size); + return rc; +} + +/* +** Sync an vstat-file. +*/ +static int vstatSync(sqlite3_file *pFile, int flags){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xSync(p->pReal, flags); + STATCNT(p->eFiletype,VFSSTAT_SYNC)++; + return rc; +} + +/* +** Return the current file-size of an vstat-file. +*/ +static int vstatFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xFileSize(p->pReal, pSize); + return rc; +} + +/* +** Lock an vstat-file. +*/ +static int vstatLock(sqlite3_file *pFile, int eLock){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xLock(p->pReal, eLock); + STATCNT(p->eFiletype,VFSSTAT_LOCK)++; + return rc; +} + +/* +** Unlock an vstat-file. +*/ +static int vstatUnlock(sqlite3_file *pFile, int eLock){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xUnlock(p->pReal, eLock); + STATCNT(p->eFiletype,VFSSTAT_LOCK)++; + return rc; +} + +/* +** Check if another file-handle holds a RESERVED lock on an vstat-file. +*/ +static int vstatCheckReservedLock(sqlite3_file *pFile, int *pResOut){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xCheckReservedLock(p->pReal, pResOut); + STATCNT(p->eFiletype,VFSSTAT_LOCK)++; + return rc; +} + +/* +** File control method. For custom operations on an vstat-file. +*/ +static int vstatFileControl(sqlite3_file *pFile, int op, void *pArg){ + VStatFile *p = (VStatFile *)pFile; + int rc; + rc = p->pReal->pMethods->xFileControl(p->pReal, op, pArg); + if( op==SQLITE_FCNTL_VFSNAME && rc==SQLITE_OK ){ + *(char**)pArg = sqlite3_mprintf("vstat/%z", *(char**)pArg); + } + return rc; +} + +/* +** Return the sector-size in bytes for an vstat-file. +*/ +static int vstatSectorSize(sqlite3_file *pFile){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xSectorSize(p->pReal); + return rc; +} + +/* +** Return the device characteristic flags supported by an vstat-file. +*/ +static int vstatDeviceCharacteristics(sqlite3_file *pFile){ + int rc; + VStatFile *p = (VStatFile *)pFile; + rc = p->pReal->pMethods->xDeviceCharacteristics(p->pReal); + return rc; +} + +/* Create a shared memory file mapping */ +static int vstatShmMap( + sqlite3_file *pFile, + int iPg, + int pgsz, + int bExtend, + void volatile **pp +){ + VStatFile *p = (VStatFile *)pFile; + return p->pReal->pMethods->xShmMap(p->pReal, iPg, pgsz, bExtend, pp); +} + +/* Perform locking on a shared-memory segment */ +static int vstatShmLock(sqlite3_file *pFile, int offset, int n, int flags){ + VStatFile *p = (VStatFile *)pFile; + return p->pReal->pMethods->xShmLock(p->pReal, offset, n, flags); +} + +/* Memory barrier operation on shared memory */ +static void vstatShmBarrier(sqlite3_file *pFile){ + VStatFile *p = (VStatFile *)pFile; + p->pReal->pMethods->xShmBarrier(p->pReal); +} + +/* Unmap a shared memory segment */ +static int vstatShmUnmap(sqlite3_file *pFile, int deleteFlag){ + VStatFile *p = (VStatFile *)pFile; + return p->pReal->pMethods->xShmUnmap(p->pReal, deleteFlag); +} + +/* Fetch a page of a memory-mapped file */ +static int vstatFetch( + sqlite3_file *pFile, + sqlite3_int64 iOfst, + int iAmt, + void **pp +){ + VStatFile *p = (VStatFile *)pFile; + return p->pReal->pMethods->xFetch(p->pReal, iOfst, iAmt, pp); +} + +/* Release a memory-mapped page */ +static int vstatUnfetch(sqlite3_file *pFile, sqlite3_int64 iOfst, void *pPage){ + VStatFile *p = (VStatFile *)pFile; + return p->pReal->pMethods->xUnfetch(p->pReal, iOfst, pPage); +} + +/* +** Open an vstat file handle. +*/ +static int vstatOpen( + sqlite3_vfs *pVfs, + const char *zName, + sqlite3_file *pFile, + int flags, + int *pOutFlags +){ + int rc; + VStatFile *p = (VStatFile*)pFile; + + p->pReal = (sqlite3_file*)&p[1]; + rc = REALVFS(pVfs)->xOpen(REALVFS(pVfs), zName, p->pReal, flags, pOutFlags); + if( flags & SQLITE_OPEN_MAIN_DB ){ + p->eFiletype = VFSSTAT_MAIN; + }else if( flags & SQLITE_OPEN_MAIN_JOURNAL ){ + p->eFiletype = VFSSTAT_JOURNAL; + }else if( flags & SQLITE_OPEN_WAL ){ + p->eFiletype = VFSSTAT_WAL; + }else if( flags & SQLITE_OPEN_MASTER_JOURNAL ){ + p->eFiletype = VFSSTAT_MASTERJRNL; + }else if( flags & SQLITE_OPEN_SUBJOURNAL ){ + p->eFiletype = VFSSTAT_SUBJRNL; + }else if( flags & SQLITE_OPEN_TEMP_DB ){ + p->eFiletype = VFSSTAT_TEMPDB; + }else if( flags & SQLITE_OPEN_TEMP_JOURNAL ){ + p->eFiletype = VFSSTAT_TEMPJRNL; + }else{ + p->eFiletype = VFSSTAT_TRANSIENT; + } + STATCNT(p->eFiletype,VFSSTAT_OPEN)++; + pFile->pMethods = rc ? 0 : &vstat_io_methods; + return rc; +} + +/* +** Delete the file located at zPath. If the dirSync argument is true, +** ensure the file-system modifications are synced to disk before +** returning. +*/ +static int vstatDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ + int rc; + rc = REALVFS(pVfs)->xDelete(REALVFS(pVfs), zPath, dirSync); + STATCNT(VFSSTAT_ANY,VFSSTAT_DELETE)++; + return rc; +} + +/* +** Test for access permissions. Return true if the requested permission +** is available, or false otherwise. +*/ +static int vstatAccess( + sqlite3_vfs *pVfs, + const char *zPath, + int flags, + int *pResOut +){ + int rc; + rc = REALVFS(pVfs)->xAccess(REALVFS(pVfs), zPath, flags, pResOut); + STATCNT(VFSSTAT_ANY,VFSSTAT_ACCESS)++; + return rc; +} + +/* +** Populate buffer zOut with the full canonical pathname corresponding +** to the pathname in zPath. zOut is guaranteed to point to a buffer +** of at least (INST_MAX_PATHNAME+1) bytes. +*/ +static int vstatFullPathname( + sqlite3_vfs *pVfs, + const char *zPath, + int nOut, + char *zOut +){ + STATCNT(VFSSTAT_ANY,VFSSTAT_FULLPATH)++; + return REALVFS(pVfs)->xFullPathname(REALVFS(pVfs), zPath, nOut, zOut); +} + +/* +** Open the dynamic library located at zPath and return a handle. +*/ +static void *vstatDlOpen(sqlite3_vfs *pVfs, const char *zPath){ + return REALVFS(pVfs)->xDlOpen(REALVFS(pVfs), zPath); +} + +/* +** Populate the buffer zErrMsg (size nByte bytes) with a human readable +** utf-8 string describing the most recent error encountered associated +** with dynamic libraries. +*/ +static void vstatDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ + REALVFS(pVfs)->xDlError(REALVFS(pVfs), nByte, zErrMsg); +} + +/* +** Return a pointer to the symbol zSymbol in the dynamic library pHandle. +*/ +static void (*vstatDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ + return REALVFS(pVfs)->xDlSym(REALVFS(pVfs), p, zSym); +} + +/* +** Close the dynamic library handle pHandle. +*/ +static void vstatDlClose(sqlite3_vfs *pVfs, void *pHandle){ + REALVFS(pVfs)->xDlClose(REALVFS(pVfs), pHandle); +} + +/* +** Populate the buffer pointed to by zBufOut with nByte bytes of +** random data. +*/ +static int vstatRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ + STATCNT(VFSSTAT_ANY,VFSSTAT_RANDOM)++; + return REALVFS(pVfs)->xRandomness(REALVFS(pVfs), nByte, zBufOut); +} + +/* +** Sleep for nMicro microseconds. Return the number of microseconds +** actually slept. +*/ +static int vstatSleep(sqlite3_vfs *pVfs, int nMicro){ + STATCNT(VFSSTAT_ANY,VFSSTAT_SLEEP)++; + return REALVFS(pVfs)->xSleep(REALVFS(pVfs), nMicro); +} + +/* +** Return the current time as a Julian Day number in *pTimeOut. +*/ +static int vstatCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ + STATCNT(VFSSTAT_ANY,VFSSTAT_CURTIME)++; + return REALVFS(pVfs)->xCurrentTime(REALVFS(pVfs), pTimeOut); +} + +static int vstatGetLastError(sqlite3_vfs *pVfs, int a, char *b){ + return REALVFS(pVfs)->xGetLastError(REALVFS(pVfs), a, b); +} +static int vstatCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ + STATCNT(VFSSTAT_ANY,VFSSTAT_CURTIME)++; + return REALVFS(pVfs)->xCurrentTimeInt64(REALVFS(pVfs), p); +} + +/* +** A virtual table for accessing the stats collected by this VFS shim +*/ +static int vstattabConnect(sqlite3*, void*, int, const char*const*, + sqlite3_vtab**,char**); +static int vstattabBestIndex(sqlite3_vtab*,sqlite3_index_info*); +static int vstattabDisconnect(sqlite3_vtab*); +static int vstattabOpen(sqlite3_vtab*, sqlite3_vtab_cursor**); +static int vstattabClose(sqlite3_vtab_cursor*); +static int vstattabFilter(sqlite3_vtab_cursor*, int idxNum, const char *idxStr, + int argc, sqlite3_value **argv); +static int vstattabNext(sqlite3_vtab_cursor*); +static int vstattabEof(sqlite3_vtab_cursor*); +static int vstattabColumn(sqlite3_vtab_cursor*,sqlite3_context*,int); +static int vstattabRowid(sqlite3_vtab_cursor*,sqlite3_int64*); +static int vstattabUpdate(sqlite3_vtab*,int,sqlite3_value**,sqlite3_int64*); + +/* A cursor for the vfsstat virtual table */ +typedef struct VfsStatCursor { + sqlite3_vtab_cursor base; /* Base class. Must be first */ + int i; /* Pointing to this aVfsCnt[] value */ +} VfsStatCursor; + + +static int vstattabConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + sqlite3_vtab *pNew; + int rc; + +/* Column numbers */ +#define VSTAT_COLUMN_FILE 0 +#define VSTAT_COLUMN_STAT 1 +#define VSTAT_COLUMN_COUNT 2 + + rc = sqlite3_declare_vtab(db,"CREATE TABLE x(file,stat,count)"); + if( rc==SQLITE_OK ){ + pNew = *ppVtab = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + } + return rc; +} + +/* +** This method is the destructor for vstat table object. +*/ +static int vstattabDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new vstat table cursor object. +*/ +static int vstattabOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + VfsStatCursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + + +/* +** Destructor for a VfsStatCursor. +*/ +static int vstattabClose(sqlite3_vtab_cursor *cur){ + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a VfsStatCursor to its next row of output. +*/ +static int vstattabNext(sqlite3_vtab_cursor *cur){ + ((VfsStatCursor*)cur)->i++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the VfsStatCursor +** is currently pointing. +*/ +static int vstattabColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + VfsStatCursor *pCur = (VfsStatCursor*)cur; + switch( i ){ + case VSTAT_COLUMN_FILE: { + sqlite3_result_text(ctx, azFile[pCur->i/VFSSTAT_nStat], -1, SQLITE_STATIC); + break; + } + case VSTAT_COLUMN_STAT: { + const char **az; + az = (pCur->i/VFSSTAT_nStat)==VFSSTAT_ANY ? azStatAny : azStat; + sqlite3_result_text(ctx, az[pCur->i%VFSSTAT_nStat], -1, SQLITE_STATIC); + break; + } + case VSTAT_COLUMN_COUNT: { + sqlite3_result_int64(ctx, aVfsCnt[pCur->i]); + break; + } + } + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. +*/ +static int vstattabRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + VfsStatCursor *pCur = (VfsStatCursor*)cur; + *pRowid = pCur->i; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int vstattabEof(sqlite3_vtab_cursor *cur){ + VfsStatCursor *pCur = (VfsStatCursor*)cur; + return pCur->i >= VFSSTAT_MXCNT; +} + +/* +** Only a full table scan is supported. So xFilter simply rewinds to +** the beginning. +*/ +static int vstattabFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + VfsStatCursor *pCur = (VfsStatCursor*)pVtabCursor; + pCur->i = 0; + return SQLITE_OK; +} + +/* +** Only a forwards full table scan is supported. xBestIndex is a no-op. +*/ +static int vstattabBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + return SQLITE_OK; +} + +/* +** Any VSTAT_COLUMN_COUNT can be changed to a positive integer. +** No deletions or insertions are allowed. No changes to other +** columns are allowed. +*/ +static int vstattabUpdate( + sqlite3_vtab *tab, + int argc, sqlite3_value **argv, + sqlite3_int64 *pRowid +){ + sqlite3_int64 iRowid, x; + if( argc==1 ) return SQLITE_ERROR; + if( sqlite3_value_type(argv[0])!=SQLITE_INTEGER ) return SQLITE_ERROR; + iRowid = sqlite3_value_int64(argv[0]); + if( iRowid!=sqlite3_value_int64(argv[1]) ) return SQLITE_ERROR; + if( iRowid<0 || iRowid>=VFSSTAT_MXCNT ) return SQLITE_ERROR; + if( sqlite3_value_type(argv[VSTAT_COLUMN_COUNT+2])!=SQLITE_INTEGER ){ + return SQLITE_ERROR; + } + x = sqlite3_value_int64(argv[VSTAT_COLUMN_COUNT+2]); + if( x<0 ) return SQLITE_ERROR; + aVfsCnt[iRowid] = x; + return SQLITE_OK; +} + +static sqlite3_module VfsStatModule = { + 0, /* iVersion */ + 0, /* xCreate */ + vstattabConnect, /* xConnect */ + vstattabBestIndex, /* xBestIndex */ + vstattabDisconnect, /* xDisconnect */ + 0, /* xDestroy */ + vstattabOpen, /* xOpen - open a cursor */ + vstattabClose, /* xClose - close a cursor */ + vstattabFilter, /* xFilter - configure scan constraints */ + vstattabNext, /* xNext - advance a cursor */ + vstattabEof, /* xEof - check for end of scan */ + vstattabColumn, /* xColumn - read data */ + vstattabRowid, /* xRowid - read data */ + vstattabUpdate, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; + +/* +** This routine is an sqlite3_auto_extension() callback, invoked to register +** the vfsstat virtual table for all new database connections. +*/ +static int vstatRegister( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pThunk +){ + return sqlite3_create_module(db, "vfsstat", &VfsStatModule, 0); +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +/* +** This routine is called when the extension is loaded. +** +** Register the new VFS. Make arrangement to register the virtual table +** for each new database connection. +*/ +int sqlite3_vfsstat_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + vstat_vfs.pVfs = sqlite3_vfs_find(0); + if( vstat_vfs.pVfs==0 ) return SQLITE_ERROR; + vstat_vfs.base.szOsFile = sizeof(VStatFile) + vstat_vfs.pVfs->szOsFile; + rc = sqlite3_vfs_register(&vstat_vfs.base, 1); + if( rc==SQLITE_OK ){ + rc = vstatRegister(db, pzErrMsg, pApi); + if( rc==SQLITE_OK ){ + rc = sqlite3_auto_extension((void(*)(void))vstatRegister); + } + } + if( rc==SQLITE_OK ) rc = SQLITE_OK_LOAD_PERMANENTLY; + return rc; +} diff --git a/ext/misc/vtablog.c b/ext/misc/vtablog.c new file mode 100644 index 0000000..424b345 --- /dev/null +++ b/ext/misc/vtablog.c @@ -0,0 +1,510 @@ +/* +** 2017-08-10 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a virtual table that prints diagnostic information +** on stdout when its key interfaces are called. This is intended for +** interactive analysis and debugging of virtual table interfaces. +** +** Usage example: +** +** .load ./vtablog +** CREATE VIRTUAL TABLE temp.log USING vtablog( +** schema='CREATE TABLE x(a,b,c)', +** rows=25 +** ); +** SELECT * FROM log; +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> +#include <string.h> +#include <ctype.h> + + +/* vtablog_vtab is a subclass of sqlite3_vtab which will +** serve as the underlying representation of a vtablog virtual table +*/ +typedef struct vtablog_vtab vtablog_vtab; +struct vtablog_vtab { + sqlite3_vtab base; /* Base class - must be first */ + int nRow; /* Number of rows in the table */ + int iInst; /* Instance number for this vtablog table */ + int nCursor; /* Number of cursors created */ +}; + +/* vtablog_cursor is a subclass of sqlite3_vtab_cursor which will +** serve as the underlying representation of a cursor that scans +** over rows of the result +*/ +typedef struct vtablog_cursor vtablog_cursor; +struct vtablog_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + int iCursor; /* Cursor number */ + sqlite3_int64 iRowid; /* The rowid */ +}; + +/* Skip leading whitespace. Return a pointer to the first non-whitespace +** character, or to the zero terminator if the string has only whitespace */ +static const char *vtablog_skip_whitespace(const char *z){ + while( isspace((unsigned char)z[0]) ) z++; + return z; +} + +/* Remove trailing whitespace from the end of string z[] */ +static void vtablog_trim_whitespace(char *z){ + size_t n = strlen(z); + while( n>0 && isspace((unsigned char)z[n]) ) n--; + z[n] = 0; +} + +/* Dequote the string */ +static void vtablog_dequote(char *z){ + int j; + char cQuote = z[0]; + size_t i, n; + + if( cQuote!='\'' && cQuote!='"' ) return; + n = strlen(z); + if( n<2 || z[n-1]!=z[0] ) return; + for(i=1, j=0; i<n-1; i++){ + if( z[i]==cQuote && z[i+1]==cQuote ) i++; + z[j++] = z[i]; + } + z[j] = 0; +} + +/* Check to see if the string is of the form: "TAG = VALUE" with optional +** whitespace before and around tokens. If it is, return a pointer to the +** first character of VALUE. If it is not, return NULL. +*/ +static const char *vtablog_parameter(const char *zTag, int nTag, const char *z){ + z = vtablog_skip_whitespace(z); + if( strncmp(zTag, z, nTag)!=0 ) return 0; + z = vtablog_skip_whitespace(z+nTag); + if( z[0]!='=' ) return 0; + return vtablog_skip_whitespace(z+1); +} + +/* Decode a parameter that requires a dequoted string. +** +** Return non-zero on an error. +*/ +static int vtablog_string_parameter( + char **pzErr, /* Leave the error message here, if there is one */ + const char *zParam, /* Parameter we are checking for */ + const char *zArg, /* Raw text of the virtual table argment */ + char **pzVal /* Write the dequoted string value here */ +){ + const char *zValue; + zValue = vtablog_parameter(zParam,(int)strlen(zParam),zArg); + if( zValue==0 ) return 0; + if( *pzVal ){ + *pzErr = sqlite3_mprintf("more than one '%s' parameter", zParam); + return 1; + } + *pzVal = sqlite3_mprintf("%s", zValue); + if( *pzVal==0 ){ + *pzErr = sqlite3_mprintf("out of memory"); + return 1; + } + vtablog_trim_whitespace(*pzVal); + vtablog_dequote(*pzVal); + return 0; +} + +#if 0 /* not used - yet */ +/* Return 0 if the argument is false and 1 if it is true. Return -1 if +** we cannot really tell. +*/ +static int vtablog_boolean(const char *z){ + if( sqlite3_stricmp("yes",z)==0 + || sqlite3_stricmp("on",z)==0 + || sqlite3_stricmp("true",z)==0 + || (z[0]=='1' && z[1]==0) + ){ + return 1; + } + if( sqlite3_stricmp("no",z)==0 + || sqlite3_stricmp("off",z)==0 + || sqlite3_stricmp("false",z)==0 + || (z[0]=='0' && z[1]==0) + ){ + return 0; + } + return -1; +} +#endif + +/* +** The vtablogConnect() method is invoked to create a new +** vtablog_vtab that describes the vtablog virtual table. +** +** Think of this routine as the constructor for vtablog_vtab objects. +** +** All this routine needs to do is: +** +** (1) Allocate the vtablog_vtab object and initialize all fields. +** +** (2) Tell SQLite (via the sqlite3_declare_vtab() interface) what the +** result set of queries against vtablog will look like. +*/ +static int vtablogConnectCreate( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr, + int isCreate +){ + static int nInst = 0; + vtablog_vtab *pNew; + int i; + int rc; + int iInst = ++nInst; + char *zSchema = 0; + char *zNRow = 0; + + printf("vtablog%s(tab=%d):\n", isCreate ? "Create" : "Connect", iInst); + printf(" argc=%d\n", argc); + for(i=0; i<argc; i++){ + printf(" argv[%d] = ", i); + if( argv[i] ){ + printf("[%s]\n", argv[i]); + }else{ + printf("NULL\n"); + } + } + + for(i=3; i<argc; i++){ + const char *z = argv[i]; + if( vtablog_string_parameter(pzErr, "schema", z, &zSchema) ){ + return SQLITE_ERROR; + } + if( vtablog_string_parameter(pzErr, "rows", z, &zNRow) ){ + return SQLITE_ERROR; + } + } + + if( zSchema==0 ){ + *pzErr = sqlite3_mprintf("no schema defined"); + return SQLITE_ERROR; + } + rc = sqlite3_declare_vtab(db, zSchema); + if( rc==SQLITE_OK ){ + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + pNew->nRow = 10; + if( zNRow ) pNew->nRow = atoi(zNRow); + pNew->iInst = iInst; + } + return rc; +} +static int vtablogCreate( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + return vtablogConnectCreate(db,pAux,argc,argv,ppVtab,pzErr,1); +} +static int vtablogConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + return vtablogConnectCreate(db,pAux,argc,argv,ppVtab,pzErr,0); +} + + +/* +** This method is the destructor for vtablog_cursor objects. +*/ +static int vtablogDisconnect(sqlite3_vtab *pVtab){ + vtablog_vtab *pTab = (vtablog_vtab*)pVtab; + printf("vtablogDisconnect(%d)\n", pTab->iInst); + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** This method is the destructor for vtablog_cursor objects. +*/ +static int vtablogDestroy(sqlite3_vtab *pVtab){ + vtablog_vtab *pTab = (vtablog_vtab*)pVtab; + printf("vtablogDestroy(%d)\n", pTab->iInst); + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new vtablog_cursor object. +*/ +static int vtablogOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + vtablog_vtab *pTab = (vtablog_vtab*)p; + vtablog_cursor *pCur; + printf("vtablogOpen(tab=%d, cursor=%d)\n", pTab->iInst, ++pTab->nCursor); + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + pCur->iCursor = pTab->nCursor; + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Destructor for a vtablog_cursor. +*/ +static int vtablogClose(sqlite3_vtab_cursor *cur){ + vtablog_cursor *pCur = (vtablog_cursor*)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + printf("vtablogClose(tab=%d, cursor=%d)\n", pTab->iInst, pCur->iCursor); + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a vtablog_cursor to its next row of output. +*/ +static int vtablogNext(sqlite3_vtab_cursor *cur){ + vtablog_cursor *pCur = (vtablog_cursor*)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + printf("vtablogNext(tab=%d, cursor=%d) rowid %d -> %d\n", + pTab->iInst, pCur->iCursor, (int)pCur->iRowid, (int)pCur->iRowid+1); + pCur->iRowid++; + return SQLITE_OK; +} + +/* +** Return values of columns for the row at which the vtablog_cursor +** is currently pointing. +*/ +static int vtablogColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + vtablog_cursor *pCur = (vtablog_cursor*)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + char zVal[50]; + + if( i<26 ){ + sqlite3_snprintf(sizeof(zVal),zVal,"%c%d", + "abcdefghijklmnopqrstuvwyz"[i], pCur->iRowid); + }else{ + sqlite3_snprintf(sizeof(zVal),zVal,"{%d}%d", i, pCur->iRowid); + } + printf("vtablogColumn(tab=%d, cursor=%d, i=%d): [%s]\n", + pTab->iInst, pCur->iCursor, i, zVal); + sqlite3_result_text(ctx, zVal, -1, SQLITE_TRANSIENT); + return SQLITE_OK; +} + +/* +** Return the rowid for the current row. In this implementation, the +** rowid is the same as the output value. +*/ +static int vtablogRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + vtablog_cursor *pCur = (vtablog_cursor*)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + printf("vtablogRowid(tab=%d, cursor=%d): %d\n", + pTab->iInst, pCur->iCursor, (int)pCur->iRowid); + *pRowid = pCur->iRowid; + return SQLITE_OK; +} + +/* +** Return TRUE if the cursor has been moved off of the last +** row of output. +*/ +static int vtablogEof(sqlite3_vtab_cursor *cur){ + vtablog_cursor *pCur = (vtablog_cursor*)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + int rc = pCur->iRowid >= pTab->nRow; + printf("vtablogEof(tab=%d, cursor=%d): %d\n", + pTab->iInst, pCur->iCursor, rc); + return rc; +} + +/* +** Output an sqlite3_value object's value as an SQL literal. +*/ +static void vtablogQuote(sqlite3_value *p){ + char z[50]; + switch( sqlite3_value_type(p) ){ + case SQLITE_NULL: { + printf("NULL"); + break; + } + case SQLITE_INTEGER: { + sqlite3_snprintf(50,z,"%lld", sqlite3_value_int64(p)); + printf("%s", z); + break; + } + case SQLITE_FLOAT: { + sqlite3_snprintf(50,z,"%!.20g", sqlite3_value_double(p)); + printf("%s", z); + break; + } + case SQLITE_BLOB: { + int n = sqlite3_value_bytes(p); + const unsigned char *z = (const unsigned char*)sqlite3_value_blob(p); + int i; + printf("x'"); + for(i=0; i<n; i++) printf("%02x", z[i]); + printf("'"); + break; + } + case SQLITE_TEXT: { + const char *z = (const char*)sqlite3_value_text(p); + int i; + char c; + for(i=0; (c = z[i])!=0 && c!='\''; i++){} + if( c==0 ){ + printf("'%s'",z); + }else{ + printf("'"); + while( *z ){ + for(i=0; (c = z[i])!=0 && c!='\''; i++){} + if( c=='\'' ) i++; + if( i ){ + printf("%.*s", i, z); + z += i; + } + if( c=='\'' ){ + printf("'"); + continue; + } + if( c==0 ){ + break; + } + z++; + } + printf("'"); + } + break; + } + } +} + + +/* +** This method is called to "rewind" the vtablog_cursor object back +** to the first row of output. This method is always called at least +** once prior to any call to vtablogColumn() or vtablogRowid() or +** vtablogEof(). +*/ +static int vtablogFilter( + sqlite3_vtab_cursor *cur, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + vtablog_cursor *pCur = (vtablog_cursor *)cur; + vtablog_vtab *pTab = (vtablog_vtab*)cur->pVtab; + printf("vtablogFilter(tab=%d, cursor=%d):\n", pTab->iInst, pCur->iCursor); + pCur->iRowid = 0; + return SQLITE_OK; +} + +/* +** SQLite will invoke this method one or more times while planning a query +** that uses the vtablog virtual table. This routine needs to create +** a query plan for each invocation and compute an estimated cost for that +** plan. +*/ +static int vtablogBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + vtablog_vtab *pTab = (vtablog_vtab*)tab; + printf("vtablogBestIndex(tab=%d):\n", pTab->iInst); + pIdxInfo->estimatedCost = (double)500; + pIdxInfo->estimatedRows = 500; + return SQLITE_OK; +} + +/* +** SQLite invokes this method to INSERT, UPDATE, or DELETE content from +** the table. +** +** This implementation does not actually make any changes to the table +** content. It merely logs the fact that the method was invoked +*/ +static int vtablogUpdate( + sqlite3_vtab *tab, + int argc, + sqlite3_value **argv, + sqlite_int64 *pRowid +){ + vtablog_vtab *pTab = (vtablog_vtab*)tab; + int i; + printf("vtablogUpdate(tab=%d):\n", pTab->iInst); + printf(" argc=%d\n", argc); + for(i=0; i<argc; i++){ + printf(" argv[%d]=", i); + vtablogQuote(argv[i]); + printf("\n"); + } + return SQLITE_OK; +} + +/* +** This following structure defines all the methods for the +** vtablog virtual table. +*/ +static sqlite3_module vtablogModule = { + 0, /* iVersion */ + vtablogCreate, /* xCreate */ + vtablogConnect, /* xConnect */ + vtablogBestIndex, /* xBestIndex */ + vtablogDisconnect, /* xDisconnect */ + vtablogDestroy, /* xDestroy */ + vtablogOpen, /* xOpen - open a cursor */ + vtablogClose, /* xClose - close a cursor */ + vtablogFilter, /* xFilter - configure scan constraints */ + vtablogNext, /* xNext - advance a cursor */ + vtablogEof, /* xEof - check for end of scan */ + vtablogColumn, /* xColumn - read data */ + vtablogRowid, /* xRowid - read data */ + vtablogUpdate, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollbackTo */ + 0, /* xShadowName */ +}; + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_vtablog_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc; + SQLITE_EXTENSION_INIT2(pApi); + rc = sqlite3_create_module(db, "vtablog", &vtablogModule, 0); + return rc; +} diff --git a/ext/misc/vtshim.c b/ext/misc/vtshim.c new file mode 100644 index 0000000..0709a26 --- /dev/null +++ b/ext/misc/vtshim.c @@ -0,0 +1,553 @@ +/* +** 2013-06-12 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** A shim that sits between the SQLite virtual table interface and +** runtimes with garbage collector based memory management. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +/* Forward references */ +typedef struct vtshim_aux vtshim_aux; +typedef struct vtshim_vtab vtshim_vtab; +typedef struct vtshim_cursor vtshim_cursor; + + +/* The vtshim_aux argument is the auxiliary parameter that is passed +** into sqlite3_create_module_v2(). +*/ +struct vtshim_aux { + void *pChildAux; /* pAux for child virtual tables */ + void (*xChildDestroy)(void*); /* Destructor for pChildAux */ + sqlite3_module *pMod; /* Methods for child virtual tables */ + sqlite3 *db; /* The database to which we are attached */ + char *zName; /* Name of the module */ + int bDisposed; /* True if disposed */ + vtshim_vtab *pAllVtab; /* List of all vtshim_vtab objects */ + sqlite3_module sSelf; /* Methods used by this shim */ +}; + +/* A vtshim virtual table object */ +struct vtshim_vtab { + sqlite3_vtab base; /* Base class - must be first */ + sqlite3_vtab *pChild; /* Child virtual table */ + vtshim_aux *pAux; /* Pointer to vtshim_aux object */ + vtshim_cursor *pAllCur; /* List of all cursors */ + vtshim_vtab **ppPrev; /* Previous on list */ + vtshim_vtab *pNext; /* Next on list */ +}; + +/* A vtshim cursor object */ +struct vtshim_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_vtab_cursor *pChild; /* Cursor generated by the managed subclass */ + vtshim_cursor **ppPrev; /* Previous on list of all cursors */ + vtshim_cursor *pNext; /* Next on list of all cursors */ +}; + +/* Macro used to copy the child vtable error message to outer vtable */ +#define VTSHIM_COPY_ERRMSG() \ + do { \ + sqlite3_free(pVtab->base.zErrMsg); \ + pVtab->base.zErrMsg = sqlite3_mprintf("%s", pVtab->pChild->zErrMsg); \ + } while (0) + +/* Methods for the vtshim module */ +static int vtshimCreate( + sqlite3 *db, + void *ppAux, + int argc, + const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + vtshim_aux *pAux = (vtshim_aux*)ppAux; + vtshim_vtab *pNew; + int rc; + + assert( db==pAux->db ); + if( pAux->bDisposed ){ + if( pzErr ){ + *pzErr = sqlite3_mprintf("virtual table was disposed: \"%s\"", + pAux->zName); + } + return SQLITE_ERROR; + } + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + rc = pAux->pMod->xCreate(db, pAux->pChildAux, argc, argv, + &pNew->pChild, pzErr); + if( rc ){ + sqlite3_free(pNew); + *ppVtab = 0; + return rc; + } + pNew->pAux = pAux; + pNew->ppPrev = &pAux->pAllVtab; + pNew->pNext = pAux->pAllVtab; + if( pAux->pAllVtab ) pAux->pAllVtab->ppPrev = &pNew->pNext; + pAux->pAllVtab = pNew; + return rc; +} + +static int vtshimConnect( + sqlite3 *db, + void *ppAux, + int argc, + const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + vtshim_aux *pAux = (vtshim_aux*)ppAux; + vtshim_vtab *pNew; + int rc; + + assert( db==pAux->db ); + if( pAux->bDisposed ){ + if( pzErr ){ + *pzErr = sqlite3_mprintf("virtual table was disposed: \"%s\"", + pAux->zName); + } + return SQLITE_ERROR; + } + pNew = sqlite3_malloc( sizeof(*pNew) ); + *ppVtab = (sqlite3_vtab*)pNew; + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, sizeof(*pNew)); + rc = pAux->pMod->xConnect(db, pAux->pChildAux, argc, argv, + &pNew->pChild, pzErr); + if( rc ){ + sqlite3_free(pNew); + *ppVtab = 0; + return rc; + } + pNew->pAux = pAux; + pNew->ppPrev = &pAux->pAllVtab; + pNew->pNext = pAux->pAllVtab; + if( pAux->pAllVtab ) pAux->pAllVtab->ppPrev = &pNew->pNext; + pAux->pAllVtab = pNew; + return rc; +} + +static int vtshimBestIndex( + sqlite3_vtab *pBase, + sqlite3_index_info *pIdxInfo +){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xBestIndex(pVtab->pChild, pIdxInfo); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimDisconnect(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc = SQLITE_OK; + if( !pAux->bDisposed ){ + rc = pAux->pMod->xDisconnect(pVtab->pChild); + } + if( pVtab->pNext ) pVtab->pNext->ppPrev = pVtab->ppPrev; + *pVtab->ppPrev = pVtab->pNext; + sqlite3_free(pVtab); + return rc; +} + +static int vtshimDestroy(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc = SQLITE_OK; + if( !pAux->bDisposed ){ + rc = pAux->pMod->xDestroy(pVtab->pChild); + } + if( pVtab->pNext ) pVtab->pNext->ppPrev = pVtab->ppPrev; + *pVtab->ppPrev = pVtab->pNext; + sqlite3_free(pVtab); + return rc; +} + +static int vtshimOpen(sqlite3_vtab *pBase, sqlite3_vtab_cursor **ppCursor){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + vtshim_cursor *pCur; + int rc; + *ppCursor = 0; + if( pAux->bDisposed ) return SQLITE_ERROR; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + rc = pAux->pMod->xOpen(pVtab->pChild, &pCur->pChild); + if( rc ){ + sqlite3_free(pCur); + VTSHIM_COPY_ERRMSG(); + return rc; + } + pCur->pChild->pVtab = pVtab->pChild; + *ppCursor = &pCur->base; + pCur->ppPrev = &pVtab->pAllCur; + if( pVtab->pAllCur ) pVtab->pAllCur->ppPrev = &pCur->pNext; + pCur->pNext = pVtab->pAllCur; + pVtab->pAllCur = pCur; + return SQLITE_OK; +} + +static int vtshimClose(sqlite3_vtab_cursor *pX){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc = SQLITE_OK; + if( !pAux->bDisposed ){ + rc = pAux->pMod->xClose(pCur->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + } + if( pCur->pNext ) pCur->pNext->ppPrev = pCur->ppPrev; + *pCur->ppPrev = pCur->pNext; + sqlite3_free(pCur); + return rc; +} + +static int vtshimFilter( + sqlite3_vtab_cursor *pX, + int idxNum, + const char *idxStr, + int argc, + sqlite3_value **argv +){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xFilter(pCur->pChild, idxNum, idxStr, argc, argv); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimNext(sqlite3_vtab_cursor *pX){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xNext(pCur->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimEof(sqlite3_vtab_cursor *pX){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return 1; + rc = pAux->pMod->xEof(pCur->pChild); + VTSHIM_COPY_ERRMSG(); + return rc; +} + +static int vtshimColumn(sqlite3_vtab_cursor *pX, sqlite3_context *ctx, int i){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xColumn(pCur->pChild, ctx, i); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimRowid(sqlite3_vtab_cursor *pX, sqlite3_int64 *pRowid){ + vtshim_cursor *pCur = (vtshim_cursor*)pX; + vtshim_vtab *pVtab = (vtshim_vtab*)pCur->base.pVtab; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xRowid(pCur->pChild, pRowid); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimUpdate( + sqlite3_vtab *pBase, + int argc, + sqlite3_value **argv, + sqlite3_int64 *pRowid +){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xUpdate(pVtab->pChild, argc, argv, pRowid); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimBegin(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xBegin(pVtab->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimSync(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xSync(pVtab->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimCommit(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xCommit(pVtab->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimRollback(sqlite3_vtab *pBase){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xRollback(pVtab->pChild); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimFindFunction( + sqlite3_vtab *pBase, + int nArg, + const char *zName, + void (**pxFunc)(sqlite3_context*,int,sqlite3_value**), + void **ppArg +){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return 0; + rc = pAux->pMod->xFindFunction(pVtab->pChild, nArg, zName, pxFunc, ppArg); + VTSHIM_COPY_ERRMSG(); + return rc; +} + +static int vtshimRename(sqlite3_vtab *pBase, const char *zNewName){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xRename(pVtab->pChild, zNewName); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimSavepoint(sqlite3_vtab *pBase, int n){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xSavepoint(pVtab->pChild, n); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimRelease(sqlite3_vtab *pBase, int n){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xRelease(pVtab->pChild, n); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +static int vtshimRollbackTo(sqlite3_vtab *pBase, int n){ + vtshim_vtab *pVtab = (vtshim_vtab*)pBase; + vtshim_aux *pAux = pVtab->pAux; + int rc; + if( pAux->bDisposed ) return SQLITE_ERROR; + rc = pAux->pMod->xRollbackTo(pVtab->pChild, n); + if( rc!=SQLITE_OK ){ + VTSHIM_COPY_ERRMSG(); + } + return rc; +} + +/* The destructor function for a disposible module */ +static void vtshimAuxDestructor(void *pXAux){ + vtshim_aux *pAux = (vtshim_aux*)pXAux; + assert( pAux->pAllVtab==0 ); + if( !pAux->bDisposed && pAux->xChildDestroy ){ + pAux->xChildDestroy(pAux->pChildAux); + pAux->xChildDestroy = 0; + } + sqlite3_free(pAux->zName); + sqlite3_free(pAux->pMod); + sqlite3_free(pAux); +} + +static int vtshimCopyModule( + const sqlite3_module *pMod, /* Source module to be copied */ + sqlite3_module **ppMod /* Destination for copied module */ +){ + sqlite3_module *p; + if( !pMod || !ppMod ) return SQLITE_ERROR; + p = sqlite3_malloc( sizeof(*p) ); + if( p==0 ) return SQLITE_NOMEM; + memcpy(p, pMod, sizeof(*p)); + *ppMod = p; + return SQLITE_OK; +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +void *sqlite3_create_disposable_module( + sqlite3 *db, /* SQLite connection to register module with */ + const char *zName, /* Name of the module */ + const sqlite3_module *p, /* Methods for the module */ + void *pClientData, /* Client data for xCreate/xConnect */ + void(*xDestroy)(void*) /* Module destructor function */ +){ + vtshim_aux *pAux; + sqlite3_module *pMod; + int rc; + pAux = sqlite3_malloc( sizeof(*pAux) ); + if( pAux==0 ){ + if( xDestroy ) xDestroy(pClientData); + return 0; + } + rc = vtshimCopyModule(p, &pMod); + if( rc!=SQLITE_OK ){ + sqlite3_free(pAux); + return 0; + } + pAux->pChildAux = pClientData; + pAux->xChildDestroy = xDestroy; + pAux->pMod = pMod; + pAux->db = db; + pAux->zName = sqlite3_mprintf("%s", zName); + pAux->bDisposed = 0; + pAux->pAllVtab = 0; + pAux->sSelf.iVersion = p->iVersion<=2 ? p->iVersion : 2; + pAux->sSelf.xCreate = p->xCreate ? vtshimCreate : 0; + pAux->sSelf.xConnect = p->xConnect ? vtshimConnect : 0; + pAux->sSelf.xBestIndex = p->xBestIndex ? vtshimBestIndex : 0; + pAux->sSelf.xDisconnect = p->xDisconnect ? vtshimDisconnect : 0; + pAux->sSelf.xDestroy = p->xDestroy ? vtshimDestroy : 0; + pAux->sSelf.xOpen = p->xOpen ? vtshimOpen : 0; + pAux->sSelf.xClose = p->xClose ? vtshimClose : 0; + pAux->sSelf.xFilter = p->xFilter ? vtshimFilter : 0; + pAux->sSelf.xNext = p->xNext ? vtshimNext : 0; + pAux->sSelf.xEof = p->xEof ? vtshimEof : 0; + pAux->sSelf.xColumn = p->xColumn ? vtshimColumn : 0; + pAux->sSelf.xRowid = p->xRowid ? vtshimRowid : 0; + pAux->sSelf.xUpdate = p->xUpdate ? vtshimUpdate : 0; + pAux->sSelf.xBegin = p->xBegin ? vtshimBegin : 0; + pAux->sSelf.xSync = p->xSync ? vtshimSync : 0; + pAux->sSelf.xCommit = p->xCommit ? vtshimCommit : 0; + pAux->sSelf.xRollback = p->xRollback ? vtshimRollback : 0; + pAux->sSelf.xFindFunction = p->xFindFunction ? vtshimFindFunction : 0; + pAux->sSelf.xRename = p->xRename ? vtshimRename : 0; + if( p->iVersion>=2 ){ + pAux->sSelf.xSavepoint = p->xSavepoint ? vtshimSavepoint : 0; + pAux->sSelf.xRelease = p->xRelease ? vtshimRelease : 0; + pAux->sSelf.xRollbackTo = p->xRollbackTo ? vtshimRollbackTo : 0; + }else{ + pAux->sSelf.xSavepoint = 0; + pAux->sSelf.xRelease = 0; + pAux->sSelf.xRollbackTo = 0; + } + rc = sqlite3_create_module_v2(db, zName, &pAux->sSelf, + pAux, vtshimAuxDestructor); + return rc==SQLITE_OK ? (void*)pAux : 0; +} + +#ifdef _WIN32 +__declspec(dllexport) +#endif +void sqlite3_dispose_module(void *pX){ + vtshim_aux *pAux = (vtshim_aux*)pX; + if( !pAux->bDisposed ){ + vtshim_vtab *pVtab; + vtshim_cursor *pCur; + for(pVtab=pAux->pAllVtab; pVtab; pVtab=pVtab->pNext){ + for(pCur=pVtab->pAllCur; pCur; pCur=pCur->pNext){ + pAux->pMod->xClose(pCur->pChild); + } + pAux->pMod->xDisconnect(pVtab->pChild); + } + pAux->bDisposed = 1; + if( pAux->xChildDestroy ){ + pAux->xChildDestroy(pAux->pChildAux); + pAux->xChildDestroy = 0; + } + } +} + + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_vtshim_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + SQLITE_EXTENSION_INIT2(pApi); + return SQLITE_OK; +} diff --git a/ext/misc/wholenumber.c b/ext/misc/wholenumber.c new file mode 100644 index 0000000..03d6e69 --- /dev/null +++ b/ext/misc/wholenumber.c @@ -0,0 +1,275 @@ +/* +** 2011 April 02 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +** +** This file implements a virtual table that returns the whole numbers +** between 1 and 4294967295, inclusive. +** +** Example: +** +** CREATE VIRTUAL TABLE nums USING wholenumber; +** SELECT value FROM nums WHERE value<10; +** +** Results in: +** +** 1 2 3 4 5 6 7 8 9 +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + + +/* A wholenumber cursor object */ +typedef struct wholenumber_cursor wholenumber_cursor; +struct wholenumber_cursor { + sqlite3_vtab_cursor base; /* Base class - must be first */ + sqlite3_int64 iValue; /* Current value */ + sqlite3_int64 mxValue; /* Maximum value */ +}; + +/* Methods for the wholenumber module */ +static int wholenumberConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + sqlite3_vtab *pNew; + pNew = *ppVtab = sqlite3_malloc( sizeof(*pNew) ); + if( pNew==0 ) return SQLITE_NOMEM; + sqlite3_declare_vtab(db, "CREATE TABLE x(value)"); + sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS); + memset(pNew, 0, sizeof(*pNew)); + return SQLITE_OK; +} +/* Note that for this virtual table, the xCreate and xConnect +** methods are identical. */ + +static int wholenumberDisconnect(sqlite3_vtab *pVtab){ + sqlite3_free(pVtab); + return SQLITE_OK; +} +/* The xDisconnect and xDestroy methods are also the same */ + + +/* +** Open a new wholenumber cursor. +*/ +static int wholenumberOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCursor){ + wholenumber_cursor *pCur; + pCur = sqlite3_malloc( sizeof(*pCur) ); + if( pCur==0 ) return SQLITE_NOMEM; + memset(pCur, 0, sizeof(*pCur)); + *ppCursor = &pCur->base; + return SQLITE_OK; +} + +/* +** Close a wholenumber cursor. +*/ +static int wholenumberClose(sqlite3_vtab_cursor *cur){ + sqlite3_free(cur); + return SQLITE_OK; +} + + +/* +** Advance a cursor to its next row of output +*/ +static int wholenumberNext(sqlite3_vtab_cursor *cur){ + wholenumber_cursor *pCur = (wholenumber_cursor*)cur; + pCur->iValue++; + return SQLITE_OK; +} + +/* +** Return the value associated with a wholenumber. +*/ +static int wholenumberColumn( + sqlite3_vtab_cursor *cur, + sqlite3_context *ctx, + int i +){ + wholenumber_cursor *pCur = (wholenumber_cursor*)cur; + sqlite3_result_int64(ctx, pCur->iValue); + return SQLITE_OK; +} + +/* +** The rowid. +*/ +static int wholenumberRowid(sqlite3_vtab_cursor *cur, sqlite_int64 *pRowid){ + wholenumber_cursor *pCur = (wholenumber_cursor*)cur; + *pRowid = pCur->iValue; + return SQLITE_OK; +} + +/* +** When the wholenumber_cursor.rLimit value is 0 or less, that is a signal +** that the cursor has nothing more to output. +*/ +static int wholenumberEof(sqlite3_vtab_cursor *cur){ + wholenumber_cursor *pCur = (wholenumber_cursor*)cur; + return pCur->iValue>pCur->mxValue || pCur->iValue==0; +} + +/* +** Called to "rewind" a cursor back to the beginning so that +** it starts its output over again. Always called at least once +** prior to any wholenumberColumn, wholenumberRowid, or wholenumberEof call. +** +** idxNum Constraints +** ------ --------------------- +** 0 (none) +** 1 value > $argv0 +** 2 value >= $argv0 +** 4 value < $argv0 +** 8 value <= $argv0 +** +** 5 value > $argv0 AND value < $argv1 +** 6 value >= $argv0 AND value < $argv1 +** 9 value > $argv0 AND value <= $argv1 +** 10 value >= $argv0 AND value <= $argv1 +*/ +static int wholenumberFilter( + sqlite3_vtab_cursor *pVtabCursor, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + wholenumber_cursor *pCur = (wholenumber_cursor *)pVtabCursor; + sqlite3_int64 v; + int i = 0; + pCur->iValue = 1; + pCur->mxValue = 0xffffffff; /* 4294967295 */ + if( idxNum & 3 ){ + v = sqlite3_value_int64(argv[0]) + (idxNum&1); + if( v>pCur->iValue && v<=pCur->mxValue ) pCur->iValue = v; + i++; + } + if( idxNum & 12 ){ + v = sqlite3_value_int64(argv[i]) - ((idxNum>>2)&1); + if( v>=pCur->iValue && v<pCur->mxValue ) pCur->mxValue = v; + } + return SQLITE_OK; +} + +/* +** Search for terms of these forms: +** +** (1) value > $value +** (2) value >= $value +** (4) value < $value +** (8) value <= $value +** +** idxNum is an ORed combination of 1 or 2 with 4 or 8. +*/ +static int wholenumberBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; + int idxNum = 0; + int argvIdx = 1; + int ltIdx = -1; + int gtIdx = -1; + const struct sqlite3_index_constraint *pConstraint; + pConstraint = pIdxInfo->aConstraint; + for(i=0; i<pIdxInfo->nConstraint; i++, pConstraint++){ + if( pConstraint->usable==0 ) continue; + if( (idxNum & 3)==0 && pConstraint->op==SQLITE_INDEX_CONSTRAINT_GT ){ + idxNum |= 1; + ltIdx = i; + } + if( (idxNum & 3)==0 && pConstraint->op==SQLITE_INDEX_CONSTRAINT_GE ){ + idxNum |= 2; + ltIdx = i; + } + if( (idxNum & 12)==0 && pConstraint->op==SQLITE_INDEX_CONSTRAINT_LT ){ + idxNum |= 4; + gtIdx = i; + } + if( (idxNum & 12)==0 && pConstraint->op==SQLITE_INDEX_CONSTRAINT_LE ){ + idxNum |= 8; + gtIdx = i; + } + } + pIdxInfo->idxNum = idxNum; + if( ltIdx>=0 ){ + pIdxInfo->aConstraintUsage[ltIdx].argvIndex = argvIdx++; + pIdxInfo->aConstraintUsage[ltIdx].omit = 1; + } + if( gtIdx>=0 ){ + pIdxInfo->aConstraintUsage[gtIdx].argvIndex = argvIdx; + pIdxInfo->aConstraintUsage[gtIdx].omit = 1; + } + if( pIdxInfo->nOrderBy==1 + && pIdxInfo->aOrderBy[0].desc==0 + ){ + pIdxInfo->orderByConsumed = 1; + } + if( (idxNum & 12)==0 ){ + pIdxInfo->estimatedCost = 1e99; + }else if( (idxNum & 3)==0 ){ + pIdxInfo->estimatedCost = (double)5; + }else{ + pIdxInfo->estimatedCost = (double)1; + } + return SQLITE_OK; +} + +/* +** A virtual table module that provides read-only access to a +** Tcl global variable namespace. +*/ +static sqlite3_module wholenumberModule = { + 0, /* iVersion */ + wholenumberConnect, + wholenumberConnect, + wholenumberBestIndex, + wholenumberDisconnect, + wholenumberDisconnect, + wholenumberOpen, /* xOpen - open a cursor */ + wholenumberClose, /* xClose - close a cursor */ + wholenumberFilter, /* xFilter - configure scan constraints */ + wholenumberNext, /* xNext - advance a cursor */ + wholenumberEof, /* xEof - check for end of scan */ + wholenumberColumn, /* xColumn - read data */ + wholenumberRowid, /* xRowid - read data */ + 0, /* xUpdate */ + 0, /* xBegin */ + 0, /* xSync */ + 0, /* xCommit */ + 0, /* xRollback */ + 0, /* xFindMethod */ + 0, /* xRename */ +}; + +#endif /* SQLITE_OMIT_VIRTUALTABLE */ + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_wholenumber_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); +#ifndef SQLITE_OMIT_VIRTUALTABLE + rc = sqlite3_create_module(db, "wholenumber", &wholenumberModule, 0); +#endif + return rc; +} diff --git a/ext/misc/zipfile.c b/ext/misc/zipfile.c new file mode 100644 index 0000000..f818fbc --- /dev/null +++ b/ext/misc/zipfile.c @@ -0,0 +1,2212 @@ +/* +** 2017-12-26 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** This file implements a virtual table for reading and writing ZIP archive +** files. +** +** Usage example: +** +** SELECT name, sz, datetime(mtime,'unixepoch') FROM zipfile($filename); +** +** Current limitations: +** +** * No support for encryption +** * No support for ZIP archives spanning multiple files +** * No support for zip64 extensions +** * Only the "inflate/deflate" (zlib) compression method is supported +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <stdio.h> +#include <string.h> +#include <assert.h> + +#include <zlib.h> + +#ifndef SQLITE_OMIT_VIRTUALTABLE + +#ifndef SQLITE_AMALGAMATION + +#ifndef UINT32_TYPE +# ifdef HAVE_UINT32_T +# define UINT32_TYPE uint32_t +# else +# define UINT32_TYPE unsigned int +# endif +#endif +#ifndef UINT16_TYPE +# ifdef HAVE_UINT16_T +# define UINT16_TYPE uint16_t +# else +# define UINT16_TYPE unsigned short int +# endif +#endif +typedef sqlite3_int64 i64; +typedef unsigned char u8; +typedef UINT32_TYPE u32; /* 4-byte unsigned integer */ +typedef UINT16_TYPE u16; /* 2-byte unsigned integer */ +#define MIN(a,b) ((a)<(b) ? (a) : (b)) + +#if defined(SQLITE_COVERAGE_TEST) || defined(SQLITE_MUTATION_TEST) +# define SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS 1 +#endif +#if defined(SQLITE_OMIT_AUXILIARY_SAFETY_CHECKS) +# define ALWAYS(X) (1) +# define NEVER(X) (0) +#elif !defined(NDEBUG) +# define ALWAYS(X) ((X)?1:(assert(0),0)) +# define NEVER(X) ((X)?(assert(0),1):0) +#else +# define ALWAYS(X) (X) +# define NEVER(X) (X) +#endif + +#endif /* SQLITE_AMALGAMATION */ + +/* +** Definitions for mode bitmasks S_IFDIR, S_IFREG and S_IFLNK. +** +** In some ways it would be better to obtain these values from system +** header files. But, the dependency is undesirable and (a) these +** have been stable for decades, (b) the values are part of POSIX and +** are also made explicit in [man stat], and (c) are part of the +** file format for zip archives. +*/ +#ifndef S_IFDIR +# define S_IFDIR 0040000 +#endif +#ifndef S_IFREG +# define S_IFREG 0100000 +#endif +#ifndef S_IFLNK +# define S_IFLNK 0120000 +#endif + +static const char ZIPFILE_SCHEMA[] = + "CREATE TABLE y(" + "name PRIMARY KEY," /* 0: Name of file in zip archive */ + "mode," /* 1: POSIX mode for file */ + "mtime," /* 2: Last modification time (secs since 1970)*/ + "sz," /* 3: Size of object */ + "rawdata," /* 4: Raw data */ + "data," /* 5: Uncompressed data */ + "method," /* 6: Compression method (integer) */ + "z HIDDEN" /* 7: Name of zip file */ + ") WITHOUT ROWID;"; + +#define ZIPFILE_F_COLUMN_IDX 7 /* Index of column "file" in the above */ +#define ZIPFILE_BUFFER_SIZE (64*1024) + + +/* +** Magic numbers used to read and write zip files. +** +** ZIPFILE_NEWENTRY_MADEBY: +** Use this value for the "version-made-by" field in new zip file +** entries. The upper byte indicates "unix", and the lower byte +** indicates that the zip file matches pkzip specification 3.0. +** This is what info-zip seems to do. +** +** ZIPFILE_NEWENTRY_REQUIRED: +** Value for "version-required-to-extract" field of new entries. +** Version 2.0 is required to support folders and deflate compression. +** +** ZIPFILE_NEWENTRY_FLAGS: +** Value for "general-purpose-bit-flags" field of new entries. Bit +** 11 means "utf-8 filename and comment". +** +** ZIPFILE_SIGNATURE_CDS: +** First 4 bytes of a valid CDS record. +** +** ZIPFILE_SIGNATURE_LFH: +** First 4 bytes of a valid LFH record. +** +** ZIPFILE_SIGNATURE_EOCD +** First 4 bytes of a valid EOCD record. +*/ +#define ZIPFILE_EXTRA_TIMESTAMP 0x5455 +#define ZIPFILE_NEWENTRY_MADEBY ((3<<8) + 30) +#define ZIPFILE_NEWENTRY_REQUIRED 20 +#define ZIPFILE_NEWENTRY_FLAGS 0x800 +#define ZIPFILE_SIGNATURE_CDS 0x02014b50 +#define ZIPFILE_SIGNATURE_LFH 0x04034b50 +#define ZIPFILE_SIGNATURE_EOCD 0x06054b50 + +/* +** The sizes of the fixed-size part of each of the three main data +** structures in a zip archive. +*/ +#define ZIPFILE_LFH_FIXED_SZ 30 +#define ZIPFILE_EOCD_FIXED_SZ 22 +#define ZIPFILE_CDS_FIXED_SZ 46 + +/* +*** 4.3.16 End of central directory record: +*** +*** end of central dir signature 4 bytes (0x06054b50) +*** number of this disk 2 bytes +*** number of the disk with the +*** start of the central directory 2 bytes +*** total number of entries in the +*** central directory on this disk 2 bytes +*** total number of entries in +*** the central directory 2 bytes +*** size of the central directory 4 bytes +*** offset of start of central +*** directory with respect to +*** the starting disk number 4 bytes +*** .ZIP file comment length 2 bytes +*** .ZIP file comment (variable size) +*/ +typedef struct ZipfileEOCD ZipfileEOCD; +struct ZipfileEOCD { + u16 iDisk; + u16 iFirstDisk; + u16 nEntry; + u16 nEntryTotal; + u32 nSize; + u32 iOffset; +}; + +/* +*** 4.3.12 Central directory structure: +*** +*** ... +*** +*** central file header signature 4 bytes (0x02014b50) +*** version made by 2 bytes +*** version needed to extract 2 bytes +*** general purpose bit flag 2 bytes +*** compression method 2 bytes +*** last mod file time 2 bytes +*** last mod file date 2 bytes +*** crc-32 4 bytes +*** compressed size 4 bytes +*** uncompressed size 4 bytes +*** file name length 2 bytes +*** extra field length 2 bytes +*** file comment length 2 bytes +*** disk number start 2 bytes +*** internal file attributes 2 bytes +*** external file attributes 4 bytes +*** relative offset of local header 4 bytes +*/ +typedef struct ZipfileCDS ZipfileCDS; +struct ZipfileCDS { + u16 iVersionMadeBy; + u16 iVersionExtract; + u16 flags; + u16 iCompression; + u16 mTime; + u16 mDate; + u32 crc32; + u32 szCompressed; + u32 szUncompressed; + u16 nFile; + u16 nExtra; + u16 nComment; + u16 iDiskStart; + u16 iInternalAttr; + u32 iExternalAttr; + u32 iOffset; + char *zFile; /* Filename (sqlite3_malloc()) */ +}; + +/* +*** 4.3.7 Local file header: +*** +*** local file header signature 4 bytes (0x04034b50) +*** version needed to extract 2 bytes +*** general purpose bit flag 2 bytes +*** compression method 2 bytes +*** last mod file time 2 bytes +*** last mod file date 2 bytes +*** crc-32 4 bytes +*** compressed size 4 bytes +*** uncompressed size 4 bytes +*** file name length 2 bytes +*** extra field length 2 bytes +*** +*/ +typedef struct ZipfileLFH ZipfileLFH; +struct ZipfileLFH { + u16 iVersionExtract; + u16 flags; + u16 iCompression; + u16 mTime; + u16 mDate; + u32 crc32; + u32 szCompressed; + u32 szUncompressed; + u16 nFile; + u16 nExtra; +}; + +typedef struct ZipfileEntry ZipfileEntry; +struct ZipfileEntry { + ZipfileCDS cds; /* Parsed CDS record */ + u32 mUnixTime; /* Modification time, in UNIX format */ + u8 *aExtra; /* cds.nExtra+cds.nComment bytes of extra data */ + i64 iDataOff; /* Offset to data in file (if aData==0) */ + u8 *aData; /* cds.szCompressed bytes of compressed data */ + ZipfileEntry *pNext; /* Next element in in-memory CDS */ +}; + +/* +** Cursor type for zipfile tables. +*/ +typedef struct ZipfileCsr ZipfileCsr; +struct ZipfileCsr { + sqlite3_vtab_cursor base; /* Base class - must be first */ + i64 iId; /* Cursor ID */ + u8 bEof; /* True when at EOF */ + u8 bNoop; /* If next xNext() call is no-op */ + + /* Used outside of write transactions */ + FILE *pFile; /* Zip file */ + i64 iNextOff; /* Offset of next record in central directory */ + ZipfileEOCD eocd; /* Parse of central directory record */ + + ZipfileEntry *pFreeEntry; /* Free this list when cursor is closed or reset */ + ZipfileEntry *pCurrent; /* Current entry */ + ZipfileCsr *pCsrNext; /* Next cursor on same virtual table */ +}; + +typedef struct ZipfileTab ZipfileTab; +struct ZipfileTab { + sqlite3_vtab base; /* Base class - must be first */ + char *zFile; /* Zip file this table accesses (may be NULL) */ + sqlite3 *db; /* Host database connection */ + u8 *aBuffer; /* Temporary buffer used for various tasks */ + + ZipfileCsr *pCsrList; /* List of cursors */ + i64 iNextCsrid; + + /* The following are used by write transactions only */ + ZipfileEntry *pFirstEntry; /* Linked list of all files (if pWriteFd!=0) */ + ZipfileEntry *pLastEntry; /* Last element in pFirstEntry list */ + FILE *pWriteFd; /* File handle open on zip archive */ + i64 szCurrent; /* Current size of zip archive */ + i64 szOrig; /* Size of archive at start of transaction */ +}; + +/* +** Set the error message contained in context ctx to the results of +** vprintf(zFmt, ...). +*/ +static void zipfileCtxErrorMsg(sqlite3_context *ctx, const char *zFmt, ...){ + char *zMsg = 0; + va_list ap; + va_start(ap, zFmt); + zMsg = sqlite3_vmprintf(zFmt, ap); + sqlite3_result_error(ctx, zMsg, -1); + sqlite3_free(zMsg); + va_end(ap); +} + +/* +** If string zIn is quoted, dequote it in place. Otherwise, if the string +** is not quoted, do nothing. +*/ +static void zipfileDequote(char *zIn){ + char q = zIn[0]; + if( q=='"' || q=='\'' || q=='`' || q=='[' ){ + int iIn = 1; + int iOut = 0; + if( q=='[' ) q = ']'; + while( ALWAYS(zIn[iIn]) ){ + char c = zIn[iIn++]; + if( c==q && zIn[iIn++]!=q ) break; + zIn[iOut++] = c; + } + zIn[iOut] = '\0'; + } +} + +/* +** Construct a new ZipfileTab virtual table object. +** +** argv[0] -> module name ("zipfile") +** argv[1] -> database name +** argv[2] -> table name +** argv[...] -> "column name" and other module argument fields. +*/ +static int zipfileConnect( + sqlite3 *db, + void *pAux, + int argc, const char *const*argv, + sqlite3_vtab **ppVtab, + char **pzErr +){ + int nByte = sizeof(ZipfileTab) + ZIPFILE_BUFFER_SIZE; + int nFile = 0; + const char *zFile = 0; + ZipfileTab *pNew = 0; + int rc; + + /* If the table name is not "zipfile", require that the argument be + ** specified. This stops zipfile tables from being created as: + ** + ** CREATE VIRTUAL TABLE zzz USING zipfile(); + ** + ** It does not prevent: + ** + ** CREATE VIRTUAL TABLE zipfile USING zipfile(); + */ + assert( 0==sqlite3_stricmp(argv[0], "zipfile") ); + if( (0!=sqlite3_stricmp(argv[2], "zipfile") && argc<4) || argc>4 ){ + *pzErr = sqlite3_mprintf("zipfile constructor requires one argument"); + return SQLITE_ERROR; + } + + if( argc>3 ){ + zFile = argv[3]; + nFile = (int)strlen(zFile)+1; + } + + rc = sqlite3_declare_vtab(db, ZIPFILE_SCHEMA); + if( rc==SQLITE_OK ){ + pNew = (ZipfileTab*)sqlite3_malloc64((sqlite3_int64)nByte+nFile); + if( pNew==0 ) return SQLITE_NOMEM; + memset(pNew, 0, nByte+nFile); + pNew->db = db; + pNew->aBuffer = (u8*)&pNew[1]; + if( zFile ){ + pNew->zFile = (char*)&pNew->aBuffer[ZIPFILE_BUFFER_SIZE]; + memcpy(pNew->zFile, zFile, nFile); + zipfileDequote(pNew->zFile); + } + } + sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY); + *ppVtab = (sqlite3_vtab*)pNew; + return rc; +} + +/* +** Free the ZipfileEntry structure indicated by the only argument. +*/ +static void zipfileEntryFree(ZipfileEntry *p){ + if( p ){ + sqlite3_free(p->cds.zFile); + sqlite3_free(p); + } +} + +/* +** Release resources that should be freed at the end of a write +** transaction. +*/ +static void zipfileCleanupTransaction(ZipfileTab *pTab){ + ZipfileEntry *pEntry; + ZipfileEntry *pNext; + + if( pTab->pWriteFd ){ + fclose(pTab->pWriteFd); + pTab->pWriteFd = 0; + } + for(pEntry=pTab->pFirstEntry; pEntry; pEntry=pNext){ + pNext = pEntry->pNext; + zipfileEntryFree(pEntry); + } + pTab->pFirstEntry = 0; + pTab->pLastEntry = 0; + pTab->szCurrent = 0; + pTab->szOrig = 0; +} + +/* +** This method is the destructor for zipfile vtab objects. +*/ +static int zipfileDisconnect(sqlite3_vtab *pVtab){ + zipfileCleanupTransaction((ZipfileTab*)pVtab); + sqlite3_free(pVtab); + return SQLITE_OK; +} + +/* +** Constructor for a new ZipfileCsr object. +*/ +static int zipfileOpen(sqlite3_vtab *p, sqlite3_vtab_cursor **ppCsr){ + ZipfileTab *pTab = (ZipfileTab*)p; + ZipfileCsr *pCsr; + pCsr = sqlite3_malloc(sizeof(*pCsr)); + *ppCsr = (sqlite3_vtab_cursor*)pCsr; + if( pCsr==0 ){ + return SQLITE_NOMEM; + } + memset(pCsr, 0, sizeof(*pCsr)); + pCsr->iId = ++pTab->iNextCsrid; + pCsr->pCsrNext = pTab->pCsrList; + pTab->pCsrList = pCsr; + return SQLITE_OK; +} + +/* +** Reset a cursor back to the state it was in when first returned +** by zipfileOpen(). +*/ +static void zipfileResetCursor(ZipfileCsr *pCsr){ + ZipfileEntry *p; + ZipfileEntry *pNext; + + pCsr->bEof = 0; + if( pCsr->pFile ){ + fclose(pCsr->pFile); + pCsr->pFile = 0; + zipfileEntryFree(pCsr->pCurrent); + pCsr->pCurrent = 0; + } + + for(p=pCsr->pFreeEntry; p; p=pNext){ + pNext = p->pNext; + zipfileEntryFree(p); + } +} + +/* +** Destructor for an ZipfileCsr. +*/ +static int zipfileClose(sqlite3_vtab_cursor *cur){ + ZipfileCsr *pCsr = (ZipfileCsr*)cur; + ZipfileTab *pTab = (ZipfileTab*)(pCsr->base.pVtab); + ZipfileCsr **pp; + zipfileResetCursor(pCsr); + + /* Remove this cursor from the ZipfileTab.pCsrList list. */ + for(pp=&pTab->pCsrList; *pp!=pCsr; pp=&((*pp)->pCsrNext)); + *pp = pCsr->pCsrNext; + + sqlite3_free(pCsr); + return SQLITE_OK; +} + +/* +** Set the error message for the virtual table associated with cursor +** pCsr to the results of vprintf(zFmt, ...). +*/ +static void zipfileTableErr(ZipfileTab *pTab, const char *zFmt, ...){ + va_list ap; + va_start(ap, zFmt); + sqlite3_free(pTab->base.zErrMsg); + pTab->base.zErrMsg = sqlite3_vmprintf(zFmt, ap); + va_end(ap); +} +static void zipfileCursorErr(ZipfileCsr *pCsr, const char *zFmt, ...){ + va_list ap; + va_start(ap, zFmt); + sqlite3_free(pCsr->base.pVtab->zErrMsg); + pCsr->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap); + va_end(ap); +} + +/* +** Read nRead bytes of data from offset iOff of file pFile into buffer +** aRead[]. Return SQLITE_OK if successful, or an SQLite error code +** otherwise. +** +** If an error does occur, output variable (*pzErrmsg) may be set to point +** to an English language error message. It is the responsibility of the +** caller to eventually free this buffer using +** sqlite3_free(). +*/ +static int zipfileReadData( + FILE *pFile, /* Read from this file */ + u8 *aRead, /* Read into this buffer */ + int nRead, /* Number of bytes to read */ + i64 iOff, /* Offset to read from */ + char **pzErrmsg /* OUT: Error message (from sqlite3_malloc) */ +){ + size_t n; + fseek(pFile, (long)iOff, SEEK_SET); + n = fread(aRead, 1, nRead, pFile); + if( (int)n!=nRead ){ + *pzErrmsg = sqlite3_mprintf("error in fread()"); + return SQLITE_ERROR; + } + return SQLITE_OK; +} + +static int zipfileAppendData( + ZipfileTab *pTab, + const u8 *aWrite, + int nWrite +){ + if( nWrite>0 ){ + size_t n = nWrite; + fseek(pTab->pWriteFd, (long)pTab->szCurrent, SEEK_SET); + n = fwrite(aWrite, 1, nWrite, pTab->pWriteFd); + if( (int)n!=nWrite ){ + pTab->base.zErrMsg = sqlite3_mprintf("error in fwrite()"); + return SQLITE_ERROR; + } + pTab->szCurrent += nWrite; + } + return SQLITE_OK; +} + +/* +** Read and return a 16-bit little-endian unsigned integer from buffer aBuf. +*/ +static u16 zipfileGetU16(const u8 *aBuf){ + return (aBuf[1] << 8) + aBuf[0]; +} + +/* +** Read and return a 32-bit little-endian unsigned integer from buffer aBuf. +*/ +static u32 zipfileGetU32(const u8 *aBuf){ + if( aBuf==0 ) return 0; + return ((u32)(aBuf[3]) << 24) + + ((u32)(aBuf[2]) << 16) + + ((u32)(aBuf[1]) << 8) + + ((u32)(aBuf[0]) << 0); +} + +/* +** Write a 16-bit little endiate integer into buffer aBuf. +*/ +static void zipfilePutU16(u8 *aBuf, u16 val){ + aBuf[0] = val & 0xFF; + aBuf[1] = (val>>8) & 0xFF; +} + +/* +** Write a 32-bit little endiate integer into buffer aBuf. +*/ +static void zipfilePutU32(u8 *aBuf, u32 val){ + aBuf[0] = val & 0xFF; + aBuf[1] = (val>>8) & 0xFF; + aBuf[2] = (val>>16) & 0xFF; + aBuf[3] = (val>>24) & 0xFF; +} + +#define zipfileRead32(aBuf) ( aBuf+=4, zipfileGetU32(aBuf-4) ) +#define zipfileRead16(aBuf) ( aBuf+=2, zipfileGetU16(aBuf-2) ) + +#define zipfileWrite32(aBuf,val) { zipfilePutU32(aBuf,val); aBuf+=4; } +#define zipfileWrite16(aBuf,val) { zipfilePutU16(aBuf,val); aBuf+=2; } + +/* +** Magic numbers used to read CDS records. +*/ +#define ZIPFILE_CDS_NFILE_OFF 28 +#define ZIPFILE_CDS_SZCOMPRESSED_OFF 20 + +/* +** Decode the CDS record in buffer aBuf into (*pCDS). Return SQLITE_ERROR +** if the record is not well-formed, or SQLITE_OK otherwise. +*/ +static int zipfileReadCDS(u8 *aBuf, ZipfileCDS *pCDS){ + u8 *aRead = aBuf; + u32 sig = zipfileRead32(aRead); + int rc = SQLITE_OK; + if( sig!=ZIPFILE_SIGNATURE_CDS ){ + rc = SQLITE_ERROR; + }else{ + pCDS->iVersionMadeBy = zipfileRead16(aRead); + pCDS->iVersionExtract = zipfileRead16(aRead); + pCDS->flags = zipfileRead16(aRead); + pCDS->iCompression = zipfileRead16(aRead); + pCDS->mTime = zipfileRead16(aRead); + pCDS->mDate = zipfileRead16(aRead); + pCDS->crc32 = zipfileRead32(aRead); + pCDS->szCompressed = zipfileRead32(aRead); + pCDS->szUncompressed = zipfileRead32(aRead); + assert( aRead==&aBuf[ZIPFILE_CDS_NFILE_OFF] ); + pCDS->nFile = zipfileRead16(aRead); + pCDS->nExtra = zipfileRead16(aRead); + pCDS->nComment = zipfileRead16(aRead); + pCDS->iDiskStart = zipfileRead16(aRead); + pCDS->iInternalAttr = zipfileRead16(aRead); + pCDS->iExternalAttr = zipfileRead32(aRead); + pCDS->iOffset = zipfileRead32(aRead); + assert( aRead==&aBuf[ZIPFILE_CDS_FIXED_SZ] ); + } + + return rc; +} + +/* +** Decode the LFH record in buffer aBuf into (*pLFH). Return SQLITE_ERROR +** if the record is not well-formed, or SQLITE_OK otherwise. +*/ +static int zipfileReadLFH( + u8 *aBuffer, + ZipfileLFH *pLFH +){ + u8 *aRead = aBuffer; + int rc = SQLITE_OK; + + u32 sig = zipfileRead32(aRead); + if( sig!=ZIPFILE_SIGNATURE_LFH ){ + rc = SQLITE_ERROR; + }else{ + pLFH->iVersionExtract = zipfileRead16(aRead); + pLFH->flags = zipfileRead16(aRead); + pLFH->iCompression = zipfileRead16(aRead); + pLFH->mTime = zipfileRead16(aRead); + pLFH->mDate = zipfileRead16(aRead); + pLFH->crc32 = zipfileRead32(aRead); + pLFH->szCompressed = zipfileRead32(aRead); + pLFH->szUncompressed = zipfileRead32(aRead); + pLFH->nFile = zipfileRead16(aRead); + pLFH->nExtra = zipfileRead16(aRead); + } + return rc; +} + + +/* +** Buffer aExtra (size nExtra bytes) contains zip archive "extra" fields. +** Scan through this buffer to find an "extra-timestamp" field. If one +** exists, extract the 32-bit modification-timestamp from it and store +** the value in output parameter *pmTime. +** +** Zero is returned if no extra-timestamp record could be found (and so +** *pmTime is left unchanged), or non-zero otherwise. +** +** The general format of an extra field is: +** +** Header ID 2 bytes +** Data Size 2 bytes +** Data N bytes +*/ +static int zipfileScanExtra(u8 *aExtra, int nExtra, u32 *pmTime){ + int ret = 0; + u8 *p = aExtra; + u8 *pEnd = &aExtra[nExtra]; + + while( p<pEnd ){ + u16 id = zipfileRead16(p); + u16 nByte = zipfileRead16(p); + + switch( id ){ + case ZIPFILE_EXTRA_TIMESTAMP: { + u8 b = p[0]; + if( b & 0x01 ){ /* 0x01 -> modtime is present */ + *pmTime = zipfileGetU32(&p[1]); + ret = 1; + } + break; + } + } + + p += nByte; + } + return ret; +} + +/* +** Convert the standard MS-DOS timestamp stored in the mTime and mDate +** fields of the CDS structure passed as the only argument to a 32-bit +** UNIX seconds-since-the-epoch timestamp. Return the result. +** +** "Standard" MS-DOS time format: +** +** File modification time: +** Bits 00-04: seconds divided by 2 +** Bits 05-10: minute +** Bits 11-15: hour +** File modification date: +** Bits 00-04: day +** Bits 05-08: month (1-12) +** Bits 09-15: years from 1980 +** +** https://msdn.microsoft.com/en-us/library/9kkf9tah.aspx +*/ +static u32 zipfileMtime(ZipfileCDS *pCDS){ + int Y,M,D,X1,X2,A,B,sec,min,hr; + i64 JDsec; + Y = (1980 + ((pCDS->mDate >> 9) & 0x7F)); + M = ((pCDS->mDate >> 5) & 0x0F); + D = (pCDS->mDate & 0x1F); + sec = (pCDS->mTime & 0x1F)*2; + min = (pCDS->mTime >> 5) & 0x3F; + hr = (pCDS->mTime >> 11) & 0x1F; + if( M<=2 ){ + Y--; + M += 12; + } + X1 = 36525*(Y+4716)/100; + X2 = 306001*(M+1)/10000; + A = Y/100; + B = 2 - A + (A/4); + JDsec = (i64)((X1 + X2 + D + B - 1524.5)*86400) + hr*3600 + min*60 + sec; + return (u32)(JDsec - (i64)24405875*(i64)8640); +} + +/* +** The opposite of zipfileMtime(). This function populates the mTime and +** mDate fields of the CDS structure passed as the first argument according +** to the UNIX timestamp value passed as the second. +*/ +static void zipfileMtimeToDos(ZipfileCDS *pCds, u32 mUnixTime){ + /* Convert unix timestamp to JD (2440588 is noon on 1/1/1970) */ + i64 JD = (i64)2440588 + mUnixTime / (24*60*60); + + int A, B, C, D, E; + int yr, mon, day; + int hr, min, sec; + + A = (int)((JD - 1867216.25)/36524.25); + A = (int)(JD + 1 + A - (A/4)); + B = A + 1524; + C = (int)((B - 122.1)/365.25); + D = (36525*(C&32767))/100; + E = (int)((B-D)/30.6001); + + day = B - D - (int)(30.6001*E); + mon = (E<14 ? E-1 : E-13); + yr = mon>2 ? C-4716 : C-4715; + + hr = (mUnixTime % (24*60*60)) / (60*60); + min = (mUnixTime % (60*60)) / 60; + sec = (mUnixTime % 60); + + if( yr>=1980 ){ + pCds->mDate = (u16)(day + (mon << 5) + ((yr-1980) << 9)); + pCds->mTime = (u16)(sec/2 + (min<<5) + (hr<<11)); + }else{ + pCds->mDate = pCds->mTime = 0; + } + + assert( mUnixTime<315507600 + || mUnixTime==zipfileMtime(pCds) + || ((mUnixTime % 2) && mUnixTime-1==zipfileMtime(pCds)) + /* || (mUnixTime % 2) */ + ); +} + +/* +** If aBlob is not NULL, then it is a pointer to a buffer (nBlob bytes in +** size) containing an entire zip archive image. Or, if aBlob is NULL, +** then pFile is a file-handle open on a zip file. In either case, this +** function creates a ZipfileEntry object based on the zip archive entry +** for which the CDS record is at offset iOff. +** +** If successful, SQLITE_OK is returned and (*ppEntry) set to point to +** the new object. Otherwise, an SQLite error code is returned and the +** final value of (*ppEntry) undefined. +*/ +static int zipfileGetEntry( + ZipfileTab *pTab, /* Store any error message here */ + const u8 *aBlob, /* Pointer to in-memory file image */ + int nBlob, /* Size of aBlob[] in bytes */ + FILE *pFile, /* If aBlob==0, read from this file */ + i64 iOff, /* Offset of CDS record */ + ZipfileEntry **ppEntry /* OUT: Pointer to new object */ +){ + u8 *aRead; + char **pzErr = &pTab->base.zErrMsg; + int rc = SQLITE_OK; + + if( aBlob==0 ){ + aRead = pTab->aBuffer; + rc = zipfileReadData(pFile, aRead, ZIPFILE_CDS_FIXED_SZ, iOff, pzErr); + }else{ + aRead = (u8*)&aBlob[iOff]; + } + + if( rc==SQLITE_OK ){ + sqlite3_int64 nAlloc; + ZipfileEntry *pNew; + + int nFile = zipfileGetU16(&aRead[ZIPFILE_CDS_NFILE_OFF]); + int nExtra = zipfileGetU16(&aRead[ZIPFILE_CDS_NFILE_OFF+2]); + nExtra += zipfileGetU16(&aRead[ZIPFILE_CDS_NFILE_OFF+4]); + + nAlloc = sizeof(ZipfileEntry) + nExtra; + if( aBlob ){ + nAlloc += zipfileGetU32(&aRead[ZIPFILE_CDS_SZCOMPRESSED_OFF]); + } + + pNew = (ZipfileEntry*)sqlite3_malloc64(nAlloc); + if( pNew==0 ){ + rc = SQLITE_NOMEM; + }else{ + memset(pNew, 0, sizeof(ZipfileEntry)); + rc = zipfileReadCDS(aRead, &pNew->cds); + if( rc!=SQLITE_OK ){ + *pzErr = sqlite3_mprintf("failed to read CDS at offset %lld", iOff); + }else if( aBlob==0 ){ + rc = zipfileReadData( + pFile, aRead, nExtra+nFile, iOff+ZIPFILE_CDS_FIXED_SZ, pzErr + ); + }else{ + aRead = (u8*)&aBlob[iOff + ZIPFILE_CDS_FIXED_SZ]; + } + } + + if( rc==SQLITE_OK ){ + u32 *pt = &pNew->mUnixTime; + pNew->cds.zFile = sqlite3_mprintf("%.*s", nFile, aRead); + pNew->aExtra = (u8*)&pNew[1]; + memcpy(pNew->aExtra, &aRead[nFile], nExtra); + if( pNew->cds.zFile==0 ){ + rc = SQLITE_NOMEM; + }else if( 0==zipfileScanExtra(&aRead[nFile], pNew->cds.nExtra, pt) ){ + pNew->mUnixTime = zipfileMtime(&pNew->cds); + } + } + + if( rc==SQLITE_OK ){ + static const int szFix = ZIPFILE_LFH_FIXED_SZ; + ZipfileLFH lfh; + if( pFile ){ + rc = zipfileReadData(pFile, aRead, szFix, pNew->cds.iOffset, pzErr); + }else{ + aRead = (u8*)&aBlob[pNew->cds.iOffset]; + } + + if( rc==SQLITE_OK ) rc = zipfileReadLFH(aRead, &lfh); + if( rc==SQLITE_OK ){ + pNew->iDataOff = pNew->cds.iOffset + ZIPFILE_LFH_FIXED_SZ; + pNew->iDataOff += lfh.nFile + lfh.nExtra; + if( aBlob && pNew->cds.szCompressed ){ + pNew->aData = &pNew->aExtra[nExtra]; + memcpy(pNew->aData, &aBlob[pNew->iDataOff], pNew->cds.szCompressed); + } + }else{ + *pzErr = sqlite3_mprintf("failed to read LFH at offset %d", + (int)pNew->cds.iOffset + ); + } + } + + if( rc!=SQLITE_OK ){ + zipfileEntryFree(pNew); + }else{ + *ppEntry = pNew; + } + } + + return rc; +} + +/* +** Advance an ZipfileCsr to its next row of output. +*/ +static int zipfileNext(sqlite3_vtab_cursor *cur){ + ZipfileCsr *pCsr = (ZipfileCsr*)cur; + int rc = SQLITE_OK; + + if( pCsr->pFile ){ + i64 iEof = pCsr->eocd.iOffset + pCsr->eocd.nSize; + zipfileEntryFree(pCsr->pCurrent); + pCsr->pCurrent = 0; + if( pCsr->iNextOff>=iEof ){ + pCsr->bEof = 1; + }else{ + ZipfileEntry *p = 0; + ZipfileTab *pTab = (ZipfileTab*)(cur->pVtab); + rc = zipfileGetEntry(pTab, 0, 0, pCsr->pFile, pCsr->iNextOff, &p); + if( rc==SQLITE_OK ){ + pCsr->iNextOff += ZIPFILE_CDS_FIXED_SZ; + pCsr->iNextOff += (int)p->cds.nExtra + p->cds.nFile + p->cds.nComment; + } + pCsr->pCurrent = p; + } + }else{ + if( !pCsr->bNoop ){ + pCsr->pCurrent = pCsr->pCurrent->pNext; + } + if( pCsr->pCurrent==0 ){ + pCsr->bEof = 1; + } + } + + pCsr->bNoop = 0; + return rc; +} + +static void zipfileFree(void *p) { + sqlite3_free(p); +} + +/* +** Buffer aIn (size nIn bytes) contains compressed data. Uncompressed, the +** size is nOut bytes. This function uncompresses the data and sets the +** return value in context pCtx to the result (a blob). +** +** If an error occurs, an error code is left in pCtx instead. +*/ +static void zipfileInflate( + sqlite3_context *pCtx, /* Store result here */ + const u8 *aIn, /* Compressed data */ + int nIn, /* Size of buffer aIn[] in bytes */ + int nOut /* Expected output size */ +){ + u8 *aRes = sqlite3_malloc(nOut); + if( aRes==0 ){ + sqlite3_result_error_nomem(pCtx); + }else{ + int err; + z_stream str; + memset(&str, 0, sizeof(str)); + + str.next_in = (Byte*)aIn; + str.avail_in = nIn; + str.next_out = (Byte*)aRes; + str.avail_out = nOut; + + err = inflateInit2(&str, -15); + if( err!=Z_OK ){ + zipfileCtxErrorMsg(pCtx, "inflateInit2() failed (%d)", err); + }else{ + err = inflate(&str, Z_NO_FLUSH); + if( err!=Z_STREAM_END ){ + zipfileCtxErrorMsg(pCtx, "inflate() failed (%d)", err); + }else{ + sqlite3_result_blob(pCtx, aRes, nOut, zipfileFree); + aRes = 0; + } + } + sqlite3_free(aRes); + inflateEnd(&str); + } +} + +/* +** Buffer aIn (size nIn bytes) contains uncompressed data. This function +** compresses it and sets (*ppOut) to point to a buffer containing the +** compressed data. The caller is responsible for eventually calling +** sqlite3_free() to release buffer (*ppOut). Before returning, (*pnOut) +** is set to the size of buffer (*ppOut) in bytes. +** +** If no error occurs, SQLITE_OK is returned. Otherwise, an SQLite error +** code is returned and an error message left in virtual-table handle +** pTab. The values of (*ppOut) and (*pnOut) are left unchanged in this +** case. +*/ +static int zipfileDeflate( + const u8 *aIn, int nIn, /* Input */ + u8 **ppOut, int *pnOut, /* Output */ + char **pzErr /* OUT: Error message */ +){ + int rc = SQLITE_OK; + sqlite3_int64 nAlloc; + z_stream str; + u8 *aOut; + + memset(&str, 0, sizeof(str)); + str.next_in = (Bytef*)aIn; + str.avail_in = nIn; + deflateInit2(&str, 9, Z_DEFLATED, -15, 8, Z_DEFAULT_STRATEGY); + + nAlloc = deflateBound(&str, nIn); + aOut = (u8*)sqlite3_malloc64(nAlloc); + if( aOut==0 ){ + rc = SQLITE_NOMEM; + }else{ + int res; + str.next_out = aOut; + str.avail_out = nAlloc; + res = deflate(&str, Z_FINISH); + if( res==Z_STREAM_END ){ + *ppOut = aOut; + *pnOut = (int)str.total_out; + }else{ + sqlite3_free(aOut); + *pzErr = sqlite3_mprintf("zipfile: deflate() error"); + rc = SQLITE_ERROR; + } + deflateEnd(&str); + } + + return rc; +} + + +/* +** Return values of columns for the row at which the series_cursor +** is currently pointing. +*/ +static int zipfileColumn( + sqlite3_vtab_cursor *cur, /* The cursor */ + sqlite3_context *ctx, /* First argument to sqlite3_result_...() */ + int i /* Which column to return */ +){ + ZipfileCsr *pCsr = (ZipfileCsr*)cur; + ZipfileCDS *pCDS = &pCsr->pCurrent->cds; + int rc = SQLITE_OK; + switch( i ){ + case 0: /* name */ + sqlite3_result_text(ctx, pCDS->zFile, -1, SQLITE_TRANSIENT); + break; + case 1: /* mode */ + /* TODO: Whether or not the following is correct surely depends on + ** the platform on which the archive was created. */ + sqlite3_result_int(ctx, pCDS->iExternalAttr >> 16); + break; + case 2: { /* mtime */ + sqlite3_result_int64(ctx, pCsr->pCurrent->mUnixTime); + break; + } + case 3: { /* sz */ + if( sqlite3_vtab_nochange(ctx)==0 ){ + sqlite3_result_int64(ctx, pCDS->szUncompressed); + } + break; + } + case 4: /* rawdata */ + if( sqlite3_vtab_nochange(ctx) ) break; + case 5: { /* data */ + if( i==4 || pCDS->iCompression==0 || pCDS->iCompression==8 ){ + int sz = pCDS->szCompressed; + int szFinal = pCDS->szUncompressed; + if( szFinal>0 ){ + u8 *aBuf; + u8 *aFree = 0; + if( pCsr->pCurrent->aData ){ + aBuf = pCsr->pCurrent->aData; + }else{ + aBuf = aFree = sqlite3_malloc64(sz); + if( aBuf==0 ){ + rc = SQLITE_NOMEM; + }else{ + FILE *pFile = pCsr->pFile; + if( pFile==0 ){ + pFile = ((ZipfileTab*)(pCsr->base.pVtab))->pWriteFd; + } + rc = zipfileReadData(pFile, aBuf, sz, pCsr->pCurrent->iDataOff, + &pCsr->base.pVtab->zErrMsg + ); + } + } + if( rc==SQLITE_OK ){ + if( i==5 && pCDS->iCompression ){ + zipfileInflate(ctx, aBuf, sz, szFinal); + }else{ + sqlite3_result_blob(ctx, aBuf, sz, SQLITE_TRANSIENT); + } + } + sqlite3_free(aFree); + }else{ + /* Figure out if this is a directory or a zero-sized file. Consider + ** it to be a directory either if the mode suggests so, or if + ** the final character in the name is '/'. */ + u32 mode = pCDS->iExternalAttr >> 16; + if( !(mode & S_IFDIR) && pCDS->zFile[pCDS->nFile-1]!='/' ){ + sqlite3_result_blob(ctx, "", 0, SQLITE_STATIC); + } + } + } + break; + } + case 6: /* method */ + sqlite3_result_int(ctx, pCDS->iCompression); + break; + default: /* z */ + assert( i==7 ); + sqlite3_result_int64(ctx, pCsr->iId); + break; + } + + return rc; +} + +/* +** Return TRUE if the cursor is at EOF. +*/ +static int zipfileEof(sqlite3_vtab_cursor *cur){ + ZipfileCsr *pCsr = (ZipfileCsr*)cur; + return pCsr->bEof; +} + +/* +** If aBlob is not NULL, then it points to a buffer nBlob bytes in size +** containing an entire zip archive image. Or, if aBlob is NULL, then pFile +** is guaranteed to be a file-handle open on a zip file. +** +** This function attempts to locate the EOCD record within the zip archive +** and populate *pEOCD with the results of decoding it. SQLITE_OK is +** returned if successful. Otherwise, an SQLite error code is returned and +** an English language error message may be left in virtual-table pTab. +*/ +static int zipfileReadEOCD( + ZipfileTab *pTab, /* Return errors here */ + const u8 *aBlob, /* Pointer to in-memory file image */ + int nBlob, /* Size of aBlob[] in bytes */ + FILE *pFile, /* Read from this file if aBlob==0 */ + ZipfileEOCD *pEOCD /* Object to populate */ +){ + u8 *aRead = pTab->aBuffer; /* Temporary buffer */ + int nRead; /* Bytes to read from file */ + int rc = SQLITE_OK; + + memset(pEOCD, 0, sizeof(ZipfileEOCD)); + if( aBlob==0 ){ + i64 iOff; /* Offset to read from */ + i64 szFile; /* Total size of file in bytes */ + fseek(pFile, 0, SEEK_END); + szFile = (i64)ftell(pFile); + if( szFile==0 ){ + return SQLITE_OK; + } + nRead = (int)(MIN(szFile, ZIPFILE_BUFFER_SIZE)); + iOff = szFile - nRead; + rc = zipfileReadData(pFile, aRead, nRead, iOff, &pTab->base.zErrMsg); + }else{ + nRead = (int)(MIN(nBlob, ZIPFILE_BUFFER_SIZE)); + aRead = (u8*)&aBlob[nBlob-nRead]; + } + + if( rc==SQLITE_OK ){ + int i; + + /* Scan backwards looking for the signature bytes */ + for(i=nRead-20; i>=0; i--){ + if( aRead[i]==0x50 && aRead[i+1]==0x4b + && aRead[i+2]==0x05 && aRead[i+3]==0x06 + ){ + break; + } + } + if( i<0 ){ + pTab->base.zErrMsg = sqlite3_mprintf( + "cannot find end of central directory record" + ); + return SQLITE_ERROR; + } + + aRead += i+4; + pEOCD->iDisk = zipfileRead16(aRead); + pEOCD->iFirstDisk = zipfileRead16(aRead); + pEOCD->nEntry = zipfileRead16(aRead); + pEOCD->nEntryTotal = zipfileRead16(aRead); + pEOCD->nSize = zipfileRead32(aRead); + pEOCD->iOffset = zipfileRead32(aRead); + } + + return rc; +} + +/* +** Add object pNew to the linked list that begins at ZipfileTab.pFirstEntry +** and ends with pLastEntry. If argument pBefore is NULL, then pNew is added +** to the end of the list. Otherwise, it is added to the list immediately +** before pBefore (which is guaranteed to be a part of said list). +*/ +static void zipfileAddEntry( + ZipfileTab *pTab, + ZipfileEntry *pBefore, + ZipfileEntry *pNew +){ + assert( (pTab->pFirstEntry==0)==(pTab->pLastEntry==0) ); + assert( pNew->pNext==0 ); + if( pBefore==0 ){ + if( pTab->pFirstEntry==0 ){ + pTab->pFirstEntry = pTab->pLastEntry = pNew; + }else{ + assert( pTab->pLastEntry->pNext==0 ); + pTab->pLastEntry->pNext = pNew; + pTab->pLastEntry = pNew; + } + }else{ + ZipfileEntry **pp; + for(pp=&pTab->pFirstEntry; *pp!=pBefore; pp=&((*pp)->pNext)); + pNew->pNext = pBefore; + *pp = pNew; + } +} + +static int zipfileLoadDirectory(ZipfileTab *pTab, const u8 *aBlob, int nBlob){ + ZipfileEOCD eocd; + int rc; + int i; + i64 iOff; + + rc = zipfileReadEOCD(pTab, aBlob, nBlob, pTab->pWriteFd, &eocd); + iOff = eocd.iOffset; + for(i=0; rc==SQLITE_OK && i<eocd.nEntry; i++){ + ZipfileEntry *pNew = 0; + rc = zipfileGetEntry(pTab, aBlob, nBlob, pTab->pWriteFd, iOff, &pNew); + + if( rc==SQLITE_OK ){ + zipfileAddEntry(pTab, 0, pNew); + iOff += ZIPFILE_CDS_FIXED_SZ; + iOff += (int)pNew->cds.nExtra + pNew->cds.nFile + pNew->cds.nComment; + } + } + return rc; +} + +/* +** xFilter callback. +*/ +static int zipfileFilter( + sqlite3_vtab_cursor *cur, + int idxNum, const char *idxStr, + int argc, sqlite3_value **argv +){ + ZipfileTab *pTab = (ZipfileTab*)cur->pVtab; + ZipfileCsr *pCsr = (ZipfileCsr*)cur; + const char *zFile = 0; /* Zip file to scan */ + int rc = SQLITE_OK; /* Return Code */ + int bInMemory = 0; /* True for an in-memory zipfile */ + + zipfileResetCursor(pCsr); + + if( pTab->zFile ){ + zFile = pTab->zFile; + }else if( idxNum==0 ){ + zipfileCursorErr(pCsr, "zipfile() function requires an argument"); + return SQLITE_ERROR; + }else if( sqlite3_value_type(argv[0])==SQLITE_BLOB ){ + static const u8 aEmptyBlob = 0; + const u8 *aBlob = (const u8*)sqlite3_value_blob(argv[0]); + int nBlob = sqlite3_value_bytes(argv[0]); + assert( pTab->pFirstEntry==0 ); + if( aBlob==0 ){ + aBlob = &aEmptyBlob; + nBlob = 0; + } + rc = zipfileLoadDirectory(pTab, aBlob, nBlob); + pCsr->pFreeEntry = pTab->pFirstEntry; + pTab->pFirstEntry = pTab->pLastEntry = 0; + if( rc!=SQLITE_OK ) return rc; + bInMemory = 1; + }else{ + zFile = (const char*)sqlite3_value_text(argv[0]); + } + + if( 0==pTab->pWriteFd && 0==bInMemory ){ + pCsr->pFile = fopen(zFile, "rb"); + if( pCsr->pFile==0 ){ + zipfileCursorErr(pCsr, "cannot open file: %s", zFile); + rc = SQLITE_ERROR; + }else{ + rc = zipfileReadEOCD(pTab, 0, 0, pCsr->pFile, &pCsr->eocd); + if( rc==SQLITE_OK ){ + if( pCsr->eocd.nEntry==0 ){ + pCsr->bEof = 1; + }else{ + pCsr->iNextOff = pCsr->eocd.iOffset; + rc = zipfileNext(cur); + } + } + } + }else{ + pCsr->bNoop = 1; + pCsr->pCurrent = pCsr->pFreeEntry ? pCsr->pFreeEntry : pTab->pFirstEntry; + rc = zipfileNext(cur); + } + + return rc; +} + +/* +** xBestIndex callback. +*/ +static int zipfileBestIndex( + sqlite3_vtab *tab, + sqlite3_index_info *pIdxInfo +){ + int i; + int idx = -1; + int unusable = 0; + + for(i=0; i<pIdxInfo->nConstraint; i++){ + const struct sqlite3_index_constraint *pCons = &pIdxInfo->aConstraint[i]; + if( pCons->iColumn!=ZIPFILE_F_COLUMN_IDX ) continue; + if( pCons->usable==0 ){ + unusable = 1; + }else if( pCons->op==SQLITE_INDEX_CONSTRAINT_EQ ){ + idx = i; + } + } + pIdxInfo->estimatedCost = 1000.0; + if( idx>=0 ){ + pIdxInfo->aConstraintUsage[idx].argvIndex = 1; + pIdxInfo->aConstraintUsage[idx].omit = 1; + pIdxInfo->idxNum = 1; + }else if( unusable ){ + return SQLITE_CONSTRAINT; + } + return SQLITE_OK; +} + +static ZipfileEntry *zipfileNewEntry(const char *zPath){ + ZipfileEntry *pNew; + pNew = sqlite3_malloc(sizeof(ZipfileEntry)); + if( pNew ){ + memset(pNew, 0, sizeof(ZipfileEntry)); + pNew->cds.zFile = sqlite3_mprintf("%s", zPath); + if( pNew->cds.zFile==0 ){ + sqlite3_free(pNew); + pNew = 0; + } + } + return pNew; +} + +static int zipfileSerializeLFH(ZipfileEntry *pEntry, u8 *aBuf){ + ZipfileCDS *pCds = &pEntry->cds; + u8 *a = aBuf; + + pCds->nExtra = 9; + + /* Write the LFH itself */ + zipfileWrite32(a, ZIPFILE_SIGNATURE_LFH); + zipfileWrite16(a, pCds->iVersionExtract); + zipfileWrite16(a, pCds->flags); + zipfileWrite16(a, pCds->iCompression); + zipfileWrite16(a, pCds->mTime); + zipfileWrite16(a, pCds->mDate); + zipfileWrite32(a, pCds->crc32); + zipfileWrite32(a, pCds->szCompressed); + zipfileWrite32(a, pCds->szUncompressed); + zipfileWrite16(a, (u16)pCds->nFile); + zipfileWrite16(a, pCds->nExtra); + assert( a==&aBuf[ZIPFILE_LFH_FIXED_SZ] ); + + /* Add the file name */ + memcpy(a, pCds->zFile, (int)pCds->nFile); + a += (int)pCds->nFile; + + /* The "extra" data */ + zipfileWrite16(a, ZIPFILE_EXTRA_TIMESTAMP); + zipfileWrite16(a, 5); + *a++ = 0x01; + zipfileWrite32(a, pEntry->mUnixTime); + + return a-aBuf; +} + +static int zipfileAppendEntry( + ZipfileTab *pTab, + ZipfileEntry *pEntry, + const u8 *pData, + int nData +){ + u8 *aBuf = pTab->aBuffer; + int nBuf; + int rc; + + nBuf = zipfileSerializeLFH(pEntry, aBuf); + rc = zipfileAppendData(pTab, aBuf, nBuf); + if( rc==SQLITE_OK ){ + pEntry->iDataOff = pTab->szCurrent; + rc = zipfileAppendData(pTab, pData, nData); + } + + return rc; +} + +static int zipfileGetMode( + sqlite3_value *pVal, + int bIsDir, /* If true, default to directory */ + u32 *pMode, /* OUT: Mode value */ + char **pzErr /* OUT: Error message */ +){ + const char *z = (const char*)sqlite3_value_text(pVal); + u32 mode = 0; + if( z==0 ){ + mode = (bIsDir ? (S_IFDIR + 0755) : (S_IFREG + 0644)); + }else if( z[0]>='0' && z[0]<='9' ){ + mode = (unsigned int)sqlite3_value_int(pVal); + }else{ + const char zTemplate[11] = "-rwxrwxrwx"; + int i; + if( strlen(z)!=10 ) goto parse_error; + switch( z[0] ){ + case '-': mode |= S_IFREG; break; + case 'd': mode |= S_IFDIR; break; + case 'l': mode |= S_IFLNK; break; + default: goto parse_error; + } + for(i=1; i<10; i++){ + if( z[i]==zTemplate[i] ) mode |= 1 << (9-i); + else if( z[i]!='-' ) goto parse_error; + } + } + if( ((mode & S_IFDIR)==0)==bIsDir ){ + /* The "mode" attribute is a directory, but data has been specified. + ** Or vice-versa - no data but "mode" is a file or symlink. */ + *pzErr = sqlite3_mprintf("zipfile: mode does not match data"); + return SQLITE_CONSTRAINT; + } + *pMode = mode; + return SQLITE_OK; + + parse_error: + *pzErr = sqlite3_mprintf("zipfile: parse error in mode: %s", z); + return SQLITE_ERROR; +} + +/* +** Both (const char*) arguments point to nul-terminated strings. Argument +** nB is the value of strlen(zB). This function returns 0 if the strings are +** identical, ignoring any trailing '/' character in either path. */ +static int zipfileComparePath(const char *zA, const char *zB, int nB){ + int nA = (int)strlen(zA); + if( nA>0 && zA[nA-1]=='/' ) nA--; + if( nB>0 && zB[nB-1]=='/' ) nB--; + if( nA==nB && memcmp(zA, zB, nA)==0 ) return 0; + return 1; +} + +static int zipfileBegin(sqlite3_vtab *pVtab){ + ZipfileTab *pTab = (ZipfileTab*)pVtab; + int rc = SQLITE_OK; + + assert( pTab->pWriteFd==0 ); + if( pTab->zFile==0 || pTab->zFile[0]==0 ){ + pTab->base.zErrMsg = sqlite3_mprintf("zipfile: missing filename"); + return SQLITE_ERROR; + } + + /* Open a write fd on the file. Also load the entire central directory + ** structure into memory. During the transaction any new file data is + ** appended to the archive file, but the central directory is accumulated + ** in main-memory until the transaction is committed. */ + pTab->pWriteFd = fopen(pTab->zFile, "ab+"); + if( pTab->pWriteFd==0 ){ + pTab->base.zErrMsg = sqlite3_mprintf( + "zipfile: failed to open file %s for writing", pTab->zFile + ); + rc = SQLITE_ERROR; + }else{ + fseek(pTab->pWriteFd, 0, SEEK_END); + pTab->szCurrent = pTab->szOrig = (i64)ftell(pTab->pWriteFd); + rc = zipfileLoadDirectory(pTab, 0, 0); + } + + if( rc!=SQLITE_OK ){ + zipfileCleanupTransaction(pTab); + } + + return rc; +} + +/* +** Return the current time as a 32-bit timestamp in UNIX epoch format (like +** time(2)). +*/ +static u32 zipfileTime(void){ + sqlite3_vfs *pVfs = sqlite3_vfs_find(0); + u32 ret; + if( pVfs==0 ) return 0; + if( pVfs->iVersion>=2 && pVfs->xCurrentTimeInt64 ){ + i64 ms; + pVfs->xCurrentTimeInt64(pVfs, &ms); + ret = (u32)((ms/1000) - ((i64)24405875 * 8640)); + }else{ + double day; + pVfs->xCurrentTime(pVfs, &day); + ret = (u32)((day - 2440587.5) * 86400); + } + return ret; +} + +/* +** Return a 32-bit timestamp in UNIX epoch format. +** +** If the value passed as the only argument is either NULL or an SQL NULL, +** return the current time. Otherwise, return the value stored in (*pVal) +** cast to a 32-bit unsigned integer. +*/ +static u32 zipfileGetTime(sqlite3_value *pVal){ + if( pVal==0 || sqlite3_value_type(pVal)==SQLITE_NULL ){ + return zipfileTime(); + } + return (u32)sqlite3_value_int64(pVal); +} + +/* +** Unless it is NULL, entry pOld is currently part of the pTab->pFirstEntry +** linked list. Remove it from the list and free the object. +*/ +static void zipfileRemoveEntryFromList(ZipfileTab *pTab, ZipfileEntry *pOld){ + if( pOld ){ + ZipfileEntry **pp; + for(pp=&pTab->pFirstEntry; (*pp)!=pOld; pp=&((*pp)->pNext)); + *pp = (*pp)->pNext; + zipfileEntryFree(pOld); + } +} + +/* +** xUpdate method. +*/ +static int zipfileUpdate( + sqlite3_vtab *pVtab, + int nVal, + sqlite3_value **apVal, + sqlite_int64 *pRowid +){ + ZipfileTab *pTab = (ZipfileTab*)pVtab; + int rc = SQLITE_OK; /* Return Code */ + ZipfileEntry *pNew = 0; /* New in-memory CDS entry */ + + u32 mode = 0; /* Mode for new entry */ + u32 mTime = 0; /* Modification time for new entry */ + i64 sz = 0; /* Uncompressed size */ + const char *zPath = 0; /* Path for new entry */ + int nPath = 0; /* strlen(zPath) */ + const u8 *pData = 0; /* Pointer to buffer containing content */ + int nData = 0; /* Size of pData buffer in bytes */ + int iMethod = 0; /* Compression method for new entry */ + u8 *pFree = 0; /* Free this */ + char *zFree = 0; /* Also free this */ + ZipfileEntry *pOld = 0; + ZipfileEntry *pOld2 = 0; + int bUpdate = 0; /* True for an update that modifies "name" */ + int bIsDir = 0; + u32 iCrc32 = 0; + + if( pTab->pWriteFd==0 ){ + rc = zipfileBegin(pVtab); + if( rc!=SQLITE_OK ) return rc; + } + + /* If this is a DELETE or UPDATE, find the archive entry to delete. */ + if( sqlite3_value_type(apVal[0])!=SQLITE_NULL ){ + const char *zDelete = (const char*)sqlite3_value_text(apVal[0]); + int nDelete = (int)strlen(zDelete); + if( nVal>1 ){ + const char *zUpdate = (const char*)sqlite3_value_text(apVal[1]); + if( zUpdate && zipfileComparePath(zUpdate, zDelete, nDelete)!=0 ){ + bUpdate = 1; + } + } + for(pOld=pTab->pFirstEntry; 1; pOld=pOld->pNext){ + if( zipfileComparePath(pOld->cds.zFile, zDelete, nDelete)==0 ){ + break; + } + assert( pOld->pNext ); + } + } + + if( nVal>1 ){ + /* Check that "sz" and "rawdata" are both NULL: */ + if( sqlite3_value_type(apVal[5])!=SQLITE_NULL ){ + zipfileTableErr(pTab, "sz must be NULL"); + rc = SQLITE_CONSTRAINT; + } + if( sqlite3_value_type(apVal[6])!=SQLITE_NULL ){ + zipfileTableErr(pTab, "rawdata must be NULL"); + rc = SQLITE_CONSTRAINT; + } + + if( rc==SQLITE_OK ){ + if( sqlite3_value_type(apVal[7])==SQLITE_NULL ){ + /* data=NULL. A directory */ + bIsDir = 1; + }else{ + /* Value specified for "data", and possibly "method". This must be + ** a regular file or a symlink. */ + const u8 *aIn = sqlite3_value_blob(apVal[7]); + int nIn = sqlite3_value_bytes(apVal[7]); + int bAuto = sqlite3_value_type(apVal[8])==SQLITE_NULL; + + iMethod = sqlite3_value_int(apVal[8]); + sz = nIn; + pData = aIn; + nData = nIn; + if( iMethod!=0 && iMethod!=8 ){ + zipfileTableErr(pTab, "unknown compression method: %d", iMethod); + rc = SQLITE_CONSTRAINT; + }else{ + if( bAuto || iMethod ){ + int nCmp; + rc = zipfileDeflate(aIn, nIn, &pFree, &nCmp, &pTab->base.zErrMsg); + if( rc==SQLITE_OK ){ + if( iMethod || nCmp<nIn ){ + iMethod = 8; + pData = pFree; + nData = nCmp; + } + } + } + iCrc32 = crc32(0, aIn, nIn); + } + } + } + + if( rc==SQLITE_OK ){ + rc = zipfileGetMode(apVal[3], bIsDir, &mode, &pTab->base.zErrMsg); + } + + if( rc==SQLITE_OK ){ + zPath = (const char*)sqlite3_value_text(apVal[2]); + if( zPath==0 ) zPath = ""; + nPath = (int)strlen(zPath); + mTime = zipfileGetTime(apVal[4]); + } + + if( rc==SQLITE_OK && bIsDir ){ + /* For a directory, check that the last character in the path is a + ** '/'. This appears to be required for compatibility with info-zip + ** (the unzip command on unix). It does not create directories + ** otherwise. */ + if( nPath<=0 || zPath[nPath-1]!='/' ){ + zFree = sqlite3_mprintf("%s/", zPath); + zPath = (const char*)zFree; + if( zFree==0 ){ + rc = SQLITE_NOMEM; + nPath = 0; + }else{ + nPath = (int)strlen(zPath); + } + } + } + + /* Check that we're not inserting a duplicate entry -OR- updating an + ** entry with a path, thereby making it into a duplicate. */ + if( (pOld==0 || bUpdate) && rc==SQLITE_OK ){ + ZipfileEntry *p; + for(p=pTab->pFirstEntry; p; p=p->pNext){ + if( zipfileComparePath(p->cds.zFile, zPath, nPath)==0 ){ + switch( sqlite3_vtab_on_conflict(pTab->db) ){ + case SQLITE_IGNORE: { + goto zipfile_update_done; + } + case SQLITE_REPLACE: { + pOld2 = p; + break; + } + default: { + zipfileTableErr(pTab, "duplicate name: \"%s\"", zPath); + rc = SQLITE_CONSTRAINT; + break; + } + } + break; + } + } + } + + if( rc==SQLITE_OK ){ + /* Create the new CDS record. */ + pNew = zipfileNewEntry(zPath); + if( pNew==0 ){ + rc = SQLITE_NOMEM; + }else{ + pNew->cds.iVersionMadeBy = ZIPFILE_NEWENTRY_MADEBY; + pNew->cds.iVersionExtract = ZIPFILE_NEWENTRY_REQUIRED; + pNew->cds.flags = ZIPFILE_NEWENTRY_FLAGS; + pNew->cds.iCompression = (u16)iMethod; + zipfileMtimeToDos(&pNew->cds, mTime); + pNew->cds.crc32 = iCrc32; + pNew->cds.szCompressed = nData; + pNew->cds.szUncompressed = (u32)sz; + pNew->cds.iExternalAttr = (mode<<16); + pNew->cds.iOffset = (u32)pTab->szCurrent; + pNew->cds.nFile = (u16)nPath; + pNew->mUnixTime = (u32)mTime; + rc = zipfileAppendEntry(pTab, pNew, pData, nData); + zipfileAddEntry(pTab, pOld, pNew); + } + } + } + + if( rc==SQLITE_OK && (pOld || pOld2) ){ + ZipfileCsr *pCsr; + for(pCsr=pTab->pCsrList; pCsr; pCsr=pCsr->pCsrNext){ + if( pCsr->pCurrent && (pCsr->pCurrent==pOld || pCsr->pCurrent==pOld2) ){ + pCsr->pCurrent = pCsr->pCurrent->pNext; + pCsr->bNoop = 1; + } + } + + zipfileRemoveEntryFromList(pTab, pOld); + zipfileRemoveEntryFromList(pTab, pOld2); + } + +zipfile_update_done: + sqlite3_free(pFree); + sqlite3_free(zFree); + return rc; +} + +static int zipfileSerializeEOCD(ZipfileEOCD *p, u8 *aBuf){ + u8 *a = aBuf; + zipfileWrite32(a, ZIPFILE_SIGNATURE_EOCD); + zipfileWrite16(a, p->iDisk); + zipfileWrite16(a, p->iFirstDisk); + zipfileWrite16(a, p->nEntry); + zipfileWrite16(a, p->nEntryTotal); + zipfileWrite32(a, p->nSize); + zipfileWrite32(a, p->iOffset); + zipfileWrite16(a, 0); /* Size of trailing comment in bytes*/ + + return a-aBuf; +} + +static int zipfileAppendEOCD(ZipfileTab *pTab, ZipfileEOCD *p){ + int nBuf = zipfileSerializeEOCD(p, pTab->aBuffer); + assert( nBuf==ZIPFILE_EOCD_FIXED_SZ ); + return zipfileAppendData(pTab, pTab->aBuffer, nBuf); +} + +/* +** Serialize the CDS structure into buffer aBuf[]. Return the number +** of bytes written. +*/ +static int zipfileSerializeCDS(ZipfileEntry *pEntry, u8 *aBuf){ + u8 *a = aBuf; + ZipfileCDS *pCDS = &pEntry->cds; + + if( pEntry->aExtra==0 ){ + pCDS->nExtra = 9; + } + + zipfileWrite32(a, ZIPFILE_SIGNATURE_CDS); + zipfileWrite16(a, pCDS->iVersionMadeBy); + zipfileWrite16(a, pCDS->iVersionExtract); + zipfileWrite16(a, pCDS->flags); + zipfileWrite16(a, pCDS->iCompression); + zipfileWrite16(a, pCDS->mTime); + zipfileWrite16(a, pCDS->mDate); + zipfileWrite32(a, pCDS->crc32); + zipfileWrite32(a, pCDS->szCompressed); + zipfileWrite32(a, pCDS->szUncompressed); + assert( a==&aBuf[ZIPFILE_CDS_NFILE_OFF] ); + zipfileWrite16(a, pCDS->nFile); + zipfileWrite16(a, pCDS->nExtra); + zipfileWrite16(a, pCDS->nComment); + zipfileWrite16(a, pCDS->iDiskStart); + zipfileWrite16(a, pCDS->iInternalAttr); + zipfileWrite32(a, pCDS->iExternalAttr); + zipfileWrite32(a, pCDS->iOffset); + + memcpy(a, pCDS->zFile, pCDS->nFile); + a += pCDS->nFile; + + if( pEntry->aExtra ){ + int n = (int)pCDS->nExtra + (int)pCDS->nComment; + memcpy(a, pEntry->aExtra, n); + a += n; + }else{ + assert( pCDS->nExtra==9 ); + zipfileWrite16(a, ZIPFILE_EXTRA_TIMESTAMP); + zipfileWrite16(a, 5); + *a++ = 0x01; + zipfileWrite32(a, pEntry->mUnixTime); + } + + return a-aBuf; +} + +static int zipfileCommit(sqlite3_vtab *pVtab){ + ZipfileTab *pTab = (ZipfileTab*)pVtab; + int rc = SQLITE_OK; + if( pTab->pWriteFd ){ + i64 iOffset = pTab->szCurrent; + ZipfileEntry *p; + ZipfileEOCD eocd; + int nEntry = 0; + + /* Write out all entries */ + for(p=pTab->pFirstEntry; rc==SQLITE_OK && p; p=p->pNext){ + int n = zipfileSerializeCDS(p, pTab->aBuffer); + rc = zipfileAppendData(pTab, pTab->aBuffer, n); + nEntry++; + } + + /* Write out the EOCD record */ + eocd.iDisk = 0; + eocd.iFirstDisk = 0; + eocd.nEntry = (u16)nEntry; + eocd.nEntryTotal = (u16)nEntry; + eocd.nSize = (u32)(pTab->szCurrent - iOffset); + eocd.iOffset = (u32)iOffset; + rc = zipfileAppendEOCD(pTab, &eocd); + + zipfileCleanupTransaction(pTab); + } + return rc; +} + +static int zipfileRollback(sqlite3_vtab *pVtab){ + return zipfileCommit(pVtab); +} + +static ZipfileCsr *zipfileFindCursor(ZipfileTab *pTab, i64 iId){ + ZipfileCsr *pCsr; + for(pCsr=pTab->pCsrList; pCsr; pCsr=pCsr->pCsrNext){ + if( iId==pCsr->iId ) break; + } + return pCsr; +} + +static void zipfileFunctionCds( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + ZipfileCsr *pCsr; + ZipfileTab *pTab = (ZipfileTab*)sqlite3_user_data(context); + assert( argc>0 ); + + pCsr = zipfileFindCursor(pTab, sqlite3_value_int64(argv[0])); + if( pCsr ){ + ZipfileCDS *p = &pCsr->pCurrent->cds; + char *zRes = sqlite3_mprintf("{" + "\"version-made-by\" : %u, " + "\"version-to-extract\" : %u, " + "\"flags\" : %u, " + "\"compression\" : %u, " + "\"time\" : %u, " + "\"date\" : %u, " + "\"crc32\" : %u, " + "\"compressed-size\" : %u, " + "\"uncompressed-size\" : %u, " + "\"file-name-length\" : %u, " + "\"extra-field-length\" : %u, " + "\"file-comment-length\" : %u, " + "\"disk-number-start\" : %u, " + "\"internal-attr\" : %u, " + "\"external-attr\" : %u, " + "\"offset\" : %u }", + (u32)p->iVersionMadeBy, (u32)p->iVersionExtract, + (u32)p->flags, (u32)p->iCompression, + (u32)p->mTime, (u32)p->mDate, + (u32)p->crc32, (u32)p->szCompressed, + (u32)p->szUncompressed, (u32)p->nFile, + (u32)p->nExtra, (u32)p->nComment, + (u32)p->iDiskStart, (u32)p->iInternalAttr, + (u32)p->iExternalAttr, (u32)p->iOffset + ); + + if( zRes==0 ){ + sqlite3_result_error_nomem(context); + }else{ + sqlite3_result_text(context, zRes, -1, SQLITE_TRANSIENT); + sqlite3_free(zRes); + } + } +} + +/* +** xFindFunction method. +*/ +static int zipfileFindFunction( + sqlite3_vtab *pVtab, /* Virtual table handle */ + int nArg, /* Number of SQL function arguments */ + const char *zName, /* Name of SQL function */ + void (**pxFunc)(sqlite3_context*,int,sqlite3_value**), /* OUT: Result */ + void **ppArg /* OUT: User data for *pxFunc */ +){ + if( sqlite3_stricmp("zipfile_cds", zName)==0 ){ + *pxFunc = zipfileFunctionCds; + *ppArg = (void*)pVtab; + return 1; + } + return 0; +} + +typedef struct ZipfileBuffer ZipfileBuffer; +struct ZipfileBuffer { + u8 *a; /* Pointer to buffer */ + int n; /* Size of buffer in bytes */ + int nAlloc; /* Byte allocated at a[] */ +}; + +typedef struct ZipfileCtx ZipfileCtx; +struct ZipfileCtx { + int nEntry; + ZipfileBuffer body; + ZipfileBuffer cds; +}; + +static int zipfileBufferGrow(ZipfileBuffer *pBuf, int nByte){ + if( pBuf->n+nByte>pBuf->nAlloc ){ + u8 *aNew; + sqlite3_int64 nNew = pBuf->n ? pBuf->n*2 : 512; + int nReq = pBuf->n + nByte; + + while( nNew<nReq ) nNew = nNew*2; + aNew = sqlite3_realloc64(pBuf->a, nNew); + if( aNew==0 ) return SQLITE_NOMEM; + pBuf->a = aNew; + pBuf->nAlloc = (int)nNew; + } + return SQLITE_OK; +} + +/* +** xStep() callback for the zipfile() aggregate. This can be called in +** any of the following ways: +** +** SELECT zipfile(name,data) ... +** SELECT zipfile(name,mode,mtime,data) ... +** SELECT zipfile(name,mode,mtime,data,method) ... +*/ +static void zipfileStep(sqlite3_context *pCtx, int nVal, sqlite3_value **apVal){ + ZipfileCtx *p; /* Aggregate function context */ + ZipfileEntry e; /* New entry to add to zip archive */ + + sqlite3_value *pName = 0; + sqlite3_value *pMode = 0; + sqlite3_value *pMtime = 0; + sqlite3_value *pData = 0; + sqlite3_value *pMethod = 0; + + int bIsDir = 0; + u32 mode; + int rc = SQLITE_OK; + char *zErr = 0; + + int iMethod = -1; /* Compression method to use (0 or 8) */ + + const u8 *aData = 0; /* Possibly compressed data for new entry */ + int nData = 0; /* Size of aData[] in bytes */ + int szUncompressed = 0; /* Size of data before compression */ + u8 *aFree = 0; /* Free this before returning */ + u32 iCrc32 = 0; /* crc32 of uncompressed data */ + + char *zName = 0; /* Path (name) of new entry */ + int nName = 0; /* Size of zName in bytes */ + char *zFree = 0; /* Free this before returning */ + int nByte; + + memset(&e, 0, sizeof(e)); + p = (ZipfileCtx*)sqlite3_aggregate_context(pCtx, sizeof(ZipfileCtx)); + if( p==0 ) return; + + /* Martial the arguments into stack variables */ + if( nVal!=2 && nVal!=4 && nVal!=5 ){ + zErr = sqlite3_mprintf("wrong number of arguments to function zipfile()"); + rc = SQLITE_ERROR; + goto zipfile_step_out; + } + pName = apVal[0]; + if( nVal==2 ){ + pData = apVal[1]; + }else{ + pMode = apVal[1]; + pMtime = apVal[2]; + pData = apVal[3]; + if( nVal==5 ){ + pMethod = apVal[4]; + } + } + + /* Check that the 'name' parameter looks ok. */ + zName = (char*)sqlite3_value_text(pName); + nName = sqlite3_value_bytes(pName); + if( zName==0 ){ + zErr = sqlite3_mprintf("first argument to zipfile() must be non-NULL"); + rc = SQLITE_ERROR; + goto zipfile_step_out; + } + + /* Inspect the 'method' parameter. This must be either 0 (store), 8 (use + ** deflate compression) or NULL (choose automatically). */ + if( pMethod && SQLITE_NULL!=sqlite3_value_type(pMethod) ){ + iMethod = (int)sqlite3_value_int64(pMethod); + if( iMethod!=0 && iMethod!=8 ){ + zErr = sqlite3_mprintf("illegal method value: %d", iMethod); + rc = SQLITE_ERROR; + goto zipfile_step_out; + } + } + + /* Now inspect the data. If this is NULL, then the new entry must be a + ** directory. Otherwise, figure out whether or not the data should + ** be deflated or simply stored in the zip archive. */ + if( sqlite3_value_type(pData)==SQLITE_NULL ){ + bIsDir = 1; + iMethod = 0; + }else{ + aData = sqlite3_value_blob(pData); + szUncompressed = nData = sqlite3_value_bytes(pData); + iCrc32 = crc32(0, aData, nData); + if( iMethod<0 || iMethod==8 ){ + int nOut = 0; + rc = zipfileDeflate(aData, nData, &aFree, &nOut, &zErr); + if( rc!=SQLITE_OK ){ + goto zipfile_step_out; + } + if( iMethod==8 || nOut<nData ){ + aData = aFree; + nData = nOut; + iMethod = 8; + }else{ + iMethod = 0; + } + } + } + + /* Decode the "mode" argument. */ + rc = zipfileGetMode(pMode, bIsDir, &mode, &zErr); + if( rc ) goto zipfile_step_out; + + /* Decode the "mtime" argument. */ + e.mUnixTime = zipfileGetTime(pMtime); + + /* If this is a directory entry, ensure that there is exactly one '/' + ** at the end of the path. Or, if this is not a directory and the path + ** ends in '/' it is an error. */ + if( bIsDir==0 ){ + if( nName>0 && zName[nName-1]=='/' ){ + zErr = sqlite3_mprintf("non-directory name must not end with /"); + rc = SQLITE_ERROR; + goto zipfile_step_out; + } + }else{ + if( nName==0 || zName[nName-1]!='/' ){ + zName = zFree = sqlite3_mprintf("%s/", zName); + if( zName==0 ){ + rc = SQLITE_NOMEM; + goto zipfile_step_out; + } + nName = (int)strlen(zName); + }else{ + while( nName>1 && zName[nName-2]=='/' ) nName--; + } + } + + /* Assemble the ZipfileEntry object for the new zip archive entry */ + e.cds.iVersionMadeBy = ZIPFILE_NEWENTRY_MADEBY; + e.cds.iVersionExtract = ZIPFILE_NEWENTRY_REQUIRED; + e.cds.flags = ZIPFILE_NEWENTRY_FLAGS; + e.cds.iCompression = (u16)iMethod; + zipfileMtimeToDos(&e.cds, (u32)e.mUnixTime); + e.cds.crc32 = iCrc32; + e.cds.szCompressed = nData; + e.cds.szUncompressed = szUncompressed; + e.cds.iExternalAttr = (mode<<16); + e.cds.iOffset = p->body.n; + e.cds.nFile = (u16)nName; + e.cds.zFile = zName; + + /* Append the LFH to the body of the new archive */ + nByte = ZIPFILE_LFH_FIXED_SZ + e.cds.nFile + 9; + if( (rc = zipfileBufferGrow(&p->body, nByte)) ) goto zipfile_step_out; + p->body.n += zipfileSerializeLFH(&e, &p->body.a[p->body.n]); + + /* Append the data to the body of the new archive */ + if( nData>0 ){ + if( (rc = zipfileBufferGrow(&p->body, nData)) ) goto zipfile_step_out; + memcpy(&p->body.a[p->body.n], aData, nData); + p->body.n += nData; + } + + /* Append the CDS record to the directory of the new archive */ + nByte = ZIPFILE_CDS_FIXED_SZ + e.cds.nFile + 9; + if( (rc = zipfileBufferGrow(&p->cds, nByte)) ) goto zipfile_step_out; + p->cds.n += zipfileSerializeCDS(&e, &p->cds.a[p->cds.n]); + + /* Increment the count of entries in the archive */ + p->nEntry++; + + zipfile_step_out: + sqlite3_free(aFree); + sqlite3_free(zFree); + if( rc ){ + if( zErr ){ + sqlite3_result_error(pCtx, zErr, -1); + }else{ + sqlite3_result_error_code(pCtx, rc); + } + } + sqlite3_free(zErr); +} + +/* +** xFinalize() callback for zipfile aggregate function. +*/ +static void zipfileFinal(sqlite3_context *pCtx){ + ZipfileCtx *p; + ZipfileEOCD eocd; + sqlite3_int64 nZip; + u8 *aZip; + + p = (ZipfileCtx*)sqlite3_aggregate_context(pCtx, sizeof(ZipfileCtx)); + if( p==0 ) return; + if( p->nEntry>0 ){ + memset(&eocd, 0, sizeof(eocd)); + eocd.nEntry = (u16)p->nEntry; + eocd.nEntryTotal = (u16)p->nEntry; + eocd.nSize = p->cds.n; + eocd.iOffset = p->body.n; + + nZip = p->body.n + p->cds.n + ZIPFILE_EOCD_FIXED_SZ; + aZip = (u8*)sqlite3_malloc64(nZip); + if( aZip==0 ){ + sqlite3_result_error_nomem(pCtx); + }else{ + memcpy(aZip, p->body.a, p->body.n); + memcpy(&aZip[p->body.n], p->cds.a, p->cds.n); + zipfileSerializeEOCD(&eocd, &aZip[p->body.n + p->cds.n]); + sqlite3_result_blob(pCtx, aZip, (int)nZip, zipfileFree); + } + } + + sqlite3_free(p->body.a); + sqlite3_free(p->cds.a); +} + + +/* +** Register the "zipfile" virtual table. +*/ +static int zipfileRegister(sqlite3 *db){ + static sqlite3_module zipfileModule = { + 1, /* iVersion */ + zipfileConnect, /* xCreate */ + zipfileConnect, /* xConnect */ + zipfileBestIndex, /* xBestIndex */ + zipfileDisconnect, /* xDisconnect */ + zipfileDisconnect, /* xDestroy */ + zipfileOpen, /* xOpen - open a cursor */ + zipfileClose, /* xClose - close a cursor */ + zipfileFilter, /* xFilter - configure scan constraints */ + zipfileNext, /* xNext - advance a cursor */ + zipfileEof, /* xEof - check for end of scan */ + zipfileColumn, /* xColumn - read data */ + 0, /* xRowid - read data */ + zipfileUpdate, /* xUpdate */ + zipfileBegin, /* xBegin */ + 0, /* xSync */ + zipfileCommit, /* xCommit */ + zipfileRollback, /* xRollback */ + zipfileFindFunction, /* xFindMethod */ + 0, /* xRename */ + 0, /* xSavepoint */ + 0, /* xRelease */ + 0, /* xRollback */ + 0 /* xShadowName */ + }; + + int rc = sqlite3_create_module(db, "zipfile" , &zipfileModule, 0); + if( rc==SQLITE_OK ) rc = sqlite3_overload_function(db, "zipfile_cds", -1); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "zipfile", -1, SQLITE_UTF8, 0, 0, + zipfileStep, zipfileFinal + ); + } + assert( sizeof(i64)==8 ); + assert( sizeof(u32)==4 ); + assert( sizeof(u16)==2 ); + assert( sizeof(u8)==1 ); + return rc; +} +#else /* SQLITE_OMIT_VIRTUALTABLE */ +# define zipfileRegister(x) SQLITE_OK +#endif + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_zipfile_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + return zipfileRegister(db); +} diff --git a/ext/misc/zorder.c b/ext/misc/zorder.c new file mode 100644 index 0000000..c385d3c --- /dev/null +++ b/ext/misc/zorder.c @@ -0,0 +1,102 @@ +/* +** 2018-02-09 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +****************************************************************************** +** +** SQL functions for z-order (Morton code) transformations. +** +** zorder(X0,X0,..,xN) Generate an N+1 dimension Morton code +** +** unzorder(Z,N,I) Extract the I-th dimension from N-dimensional +** Morton code Z. +*/ +#include "sqlite3ext.h" +SQLITE_EXTENSION_INIT1 +#include <assert.h> +#include <string.h> + +/* +** Functions: zorder(X0,X1,....) +** +** Convert integers X0, X1, ... into morton code. +** +** The output is a signed 64-bit integer. If any argument is too large, +** an error is thrown. +*/ +static void zorderFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3_int64 z, x[63]; + int i, j; + z = 0; + for(i=0; i<argc; i++){ + x[i] = sqlite3_value_int64(argv[i]); + } + if( argc>0 ){ + for(i=0; i<63; i++){ + j = i%argc; + z |= (x[j]&1)<<i; + x[j] >>= 1; + } + } + sqlite3_result_int64(context, z); + for(i=0; i<argc; i++){ + if( x[i] ){ + sqlite3_result_error(context, "parameter too large", -1); + } + } +} + + +/* +** Functions: unzorder(Z,N,I) +** +** Assuming that Z is an N-dimensional Morton code, extract the I-th +** dimension. +*/ +static void unzorderFunc( + sqlite3_context *context, + int argc, + sqlite3_value **argv +){ + sqlite3_int64 z, n, i, x; + int j, k; + z = sqlite3_value_int64(argv[0]); + n = sqlite3_value_int64(argv[1]); + i = sqlite3_value_int64(argv[2]); + x = 0; + for(k=0, j=i; j<63; j+=n, k++){ + x |= ((z>>j)&1)<<k; + } + sqlite3_result_int64(context, x); +} + + +#ifdef _WIN32 +__declspec(dllexport) +#endif +int sqlite3_zorder_init( + sqlite3 *db, + char **pzErrMsg, + const sqlite3_api_routines *pApi +){ + int rc = SQLITE_OK; + SQLITE_EXTENSION_INIT2(pApi); + (void)pzErrMsg; /* Unused parameter */ + rc = sqlite3_create_function(db, "zorder", -1, SQLITE_UTF8, 0, + zorderFunc, 0, 0); + if( rc==SQLITE_OK ){ + rc = sqlite3_create_function(db, "unzorder", 3, SQLITE_UTF8, 0, + unzorderFunc, 0, 0); + } + return rc; +} |