/* 2022-07-22 The author disclaims copyright to this source code. In place of a legal notice, here is a blessing: * May you do good and not evil. * May you find forgiveness for yourself and forgive others. * May you share freely, never taking more than you give. *********************************************************************** This file implements the initializer for the sqlite3 "Worker API #1", a very basic DB access API intended to be scripted from a main window thread via Worker-style messages. Because of limitations in that type of communication, this API is minimalistic and only capable of serving relatively basic DB requests (e.g. it cannot process nested query loops concurrently). This file requires that the core C-style sqlite3 API and OO API #1 have been loaded. */ /** sqlite3.initWorker1API() implements a Worker-based wrapper around SQLite3 OO API #1, colloquially known as "Worker API #1". In order to permit this API to be loaded in worker threads without automatically registering onmessage handlers, initializing the worker API requires calling initWorker1API(). If this function is called from a non-worker thread then it throws an exception. It must only be called once per Worker. When initialized, it installs message listeners to receive Worker messages and then it posts a message in the form: ``` {type:'sqlite3-api', result:'worker1-ready'} ``` to let the client know that it has been initialized. Clients may optionally depend on this function not returning until initialization is complete, as the initialization is synchronous. In some contexts, however, listening for the above message is a better fit. Note that the worker-based interface can be slightly quirky because of its async nature. In particular, any number of messages may be posted to the worker before it starts handling any of them. If, e.g., an "open" operation fails, any subsequent messages will fail. The Promise-based wrapper for this API (`sqlite3-worker1-promiser.js`) is more comfortable to use in that regard. The documentation for the input and output worker messages for this API follows... ==================================================================== Common message format... Each message posted to the worker has an operation-independent envelope and operation-dependent arguments: ``` { type: string, // one of: 'open', 'close', 'exec', 'config-get' messageId: OPTIONAL arbitrary value. The worker will copy it as-is into response messages to assist in client-side dispatching. dbId: a db identifier string (returned by 'open') which tells the operation which database instance to work on. If not provided, the first-opened db is used. This is an "opaque" value, with no inherently useful syntax or information. Its value is subject to change with any given build of this API and cannot be used as a basis for anything useful beyond its one intended purpose. args: ...operation-dependent arguments... // the framework may add other properties for testing or debugging // purposes. } ``` Response messages, posted back to the main thread, look like: ``` { type: string. Same as above except for error responses, which have the type 'error', messageId: same value, if any, provided by the inbound message dbId: the id of the db which was operated on, if any, as returned by the corresponding 'open' operation. result: ...operation-dependent result... } ``` ==================================================================== Error responses Errors are reported messages in an operation-independent format: ``` { type: "error", messageId: ...as above..., dbId: ...as above... result: { operation: type of the triggering operation: 'open', 'close', ... message: ...error message text... errorClass: string. The ErrorClass.name property from the thrown exception. input: the message object which triggered the error. stack: _if available_, a stack trace array. } } ``` ==================================================================== "config-get" This operation fetches the serializable parts of the sqlite3 API configuration. Message format: ``` { type: "config-get", messageId: ...as above..., args: currently ignored and may be elided. } ``` Response: ``` { type: "config-get", messageId: ...as above..., result: { version: sqlite3.version object bigIntEnabled: bool. True if BigInt support is enabled. wasmfsOpfsDir: path prefix, if any, _intended_ for use with WASMFS OPFS persistent storage. wasmfsOpfsEnabled: true if persistent storage is enabled in the current environment. Only files stored under wasmfsOpfsDir will persist using that mechanism, however. It is legal to use the non-WASMFS OPFS VFS to open a database via a URI-style db filename. vfsList: result of sqlite3.capi.sqlite3_js_vfs_list() } } ``` ==================================================================== "open" a database Message format: ``` { type: "open", messageId: ...as above..., args:{ filename [=":memory:" or "" (unspecified)]: the db filename. See the sqlite3.oo1.DB constructor for peculiarities and transformations, vfs: sqlite3_vfs name. Ignored if filename is ":memory:" or "". This may change how the given filename is resolved. } } ``` Response: ``` { type: "open", messageId: ...as above..., result: { filename: db filename, possibly differing from the input. dbId: an opaque ID value which must be passed in the message envelope to other calls in this API to tell them which db to use. If it is not provided to future calls, they will default to operating on the least-recently-opened db. This property is, for API consistency's sake, also part of the containing message envelope. Only the `open` operation includes it in the `result` property. persistent: true if the given filename resides in the known-persistent storage, else false. vfs: name of the VFS the "main" db is using. } } ``` ==================================================================== "close" a database Message format: ``` { type: "close", messageId: ...as above... dbId: ...as above... args: OPTIONAL {unlink: boolean} } ``` If the `dbId` does not refer to an opened ID, this is a no-op. If the `args` object contains a truthy `unlink` value then the database will be unlinked (deleted) after closing it. The inability to close a db (because it's not opened) or delete its file does not trigger an error. Response: ``` { type: "close", messageId: ...as above..., result: { filename: filename of closed db, or undefined if no db was closed } } ``` ==================================================================== "exec" SQL All SQL execution is processed through the exec operation. It offers most of the features of the oo1.DB.exec() method, with a few limitations imposed by the state having to cross thread boundaries. Message format: ``` { type: "exec", messageId: ...as above... dbId: ...as above... args: string (SQL) or {... see below ...} } ``` Response: ``` { type: "exec", messageId: ...as above..., dbId: ...as above... result: { input arguments, possibly modified. See below. } } ``` The arguments are in the same form accepted by oo1.DB.exec(), with the exceptions noted below. A function-type args.callback property cannot cross the window/Worker boundary, so is not useful here. If args.callback is a string then it is assumed to be a message type key, in which case a callback function will be applied which posts each row result via: postMessage({type: thatKeyType, rowNumber: 1-based-#, row: theRow, columnNames: anArray }) And, at the end of the result set (whether or not any result rows were produced), it will post an identical message with (row=undefined, rowNumber=null) to alert the caller than the result set is completed. Note that a row value of `null` is a legal row result for certain arg.rowMode values. (Design note: we don't use (row=undefined, rowNumber=undefined) to indicate end-of-results because fetching those would be indistinguishable from fetching from an empty object unless the client used hasOwnProperty() (or similar) to distinguish "missing property" from "property with the undefined value". Similarly, `null` is a legal value for `row` in some case , whereas the db layer won't emit a result value of `undefined`.) The callback proxy must not recurse into this interface. An exec() call will tie up the Worker thread, causing any recursion attempt to wait until the first exec() is completed. The response is the input options object (or a synthesized one if passed only a string), noting that options.resultRows and options.columnNames may be populated by the call to db.exec(). */ self.sqlite3ApiBootstrap.initializers.push(function(sqlite3){ sqlite3.initWorker1API = function(){ 'use strict'; const toss = (...args)=>{throw new Error(args.join(' '))}; if('function' !== typeof importScripts){ toss("initWorker1API() must be run from a Worker thread."); } const self = this.self; const sqlite3 = this.sqlite3 || toss("Missing this.sqlite3 object."); const DB = sqlite3.oo1.DB; /** Returns the app-wide unique ID for the given db, creating one if needed. */ const getDbId = function(db){ let id = wState.idMap.get(db); if(id) return id; id = 'db#'+(++wState.idSeq)+'@'+db.pointer; /** ^^^ can't simply use db.pointer b/c closing/opening may re-use the same address, which could map pending messages to a wrong instance. */ wState.idMap.set(db, id); return id; }; /** Internal helper for managing Worker-level state. */ const wState = { /** Each opened DB is added to this.dbList, and the first entry in that list is the default db. As each db is closed, its entry is removed from the list. */ dbList: [], /** Sequence number of dbId generation. */ idSeq: 0, /** Map of DB instances to dbId. */ idMap: new WeakMap, /** Temp holder for "transferable" postMessage() state. */ xfer: [], open: function(opt){ const db = new DB(opt); this.dbs[getDbId(db)] = db; if(this.dbList.indexOf(db)<0) this.dbList.push(db); return db; }, close: function(db,alsoUnlink){ if(db){ delete this.dbs[getDbId(db)]; const filename = db.filename; const pVfs = sqlite3.wasm.sqlite3_wasm_db_vfs(db.pointer, 0); db.close(); const ddNdx = this.dbList.indexOf(db); if(ddNdx>=0) this.dbList.splice(ddNdx, 1); if(alsoUnlink && filename && pVfs){ sqlite3.wasm.sqlite3_wasm_vfs_unlink(pVfs, filename); } } }, /** Posts the given worker message value. If xferList is provided, it must be an array, in which case a copy of it passed as postMessage()'s second argument and xferList.length is set to 0. */ post: function(msg,xferList){ if(xferList && xferList.length){ self.postMessage( msg, Array.from(xferList) ); xferList.length = 0; }else{ self.postMessage(msg); } }, /** Map of DB IDs to DBs. */ dbs: Object.create(null), /** Fetch the DB for the given id. Throw if require=true and the id is not valid, else return the db or undefined. */ getDb: function(id,require=true){ return this.dbs[id] || (require ? toss("Unknown (or closed) DB ID:",id) : undefined); } }; /** Throws if the given db is falsy or not opened, else returns its argument. */ const affirmDbOpen = function(db = wState.dbList[0]){ return (db && db.pointer) ? db : toss("DB is not opened."); }; /** Extract dbId from the given message payload. */ const getMsgDb = function(msgData,affirmExists=true){ const db = wState.getDb(msgData.dbId,false) || wState.dbList[0]; return affirmExists ? affirmDbOpen(db) : db; }; const getDefaultDbId = function(){ return wState.dbList[0] && getDbId(wState.dbList[0]); }; const guessVfs = function(filename){ const m = /^file:.+(vfs=(\w+))/.exec(filename); return sqlite3.capi.sqlite3_vfs_find(m ? m[2] : 0); }; const isSpecialDbFilename = (n)=>{ return ""===n || ':'===n[0]; }; /** A level of "organizational abstraction" for the Worker1 API. Each method in this object must map directly to a Worker1 message type key. The onmessage() dispatcher attempts to dispatch all inbound messages to a method of this object, passing it the event.data part of the inbound event object. All methods must return a plain Object containing any result state, which the dispatcher may amend. All methods must throw on error. */ const wMsgHandler = { open: function(ev){ const oargs = Object.create(null), args = (ev.args || Object.create(null)); if(args.simulateError){ // undocumented internal testing option toss("Throwing because of simulateError flag."); } const rc = Object.create(null); const pDir = sqlite3.capi.sqlite3_wasmfs_opfs_dir(); let byteArray, pVfs; oargs.vfs = args.vfs; if(isSpecialDbFilename(args.filename)){ oargs.filename = args.filename || ""; }else{ oargs.filename = args.filename; byteArray = args.byteArray; if(byteArray) pVfs = guessVfs(args.filename); } if(pVfs){ /* 2022-11-02: this feature is as-yet untested except that sqlite3_wasm_vfs_create_file() has been tested from the browser dev console. */ let pMem; try{ pMem = sqlite3.wasm.allocFromTypedArray(byteArray); const rc = sqlite3.wasm.sqlite3_wasm_vfs_create_file( pVfs, oargs.filename, pMem, byteArray.byteLength ); if(rc) sqlite3.SQLite3Error.toss(rc); }catch(e){ throw new sqlite3.SQLite3Error( e.name+' creating '+args.filename+": "+e.message, { cause: e } ); }finally{ if(pMem) sqlite3.wasm.dealloc(pMem); } } const db = wState.open(oargs); rc.filename = db.filename; rc.persistent = (!!pDir && db.filename.startsWith(pDir+'/')) || !!sqlite3.capi.sqlite3_js_db_uses_vfs(db.pointer, "opfs"); rc.dbId = getDbId(db); rc.vfs = db.dbVfsName(); return rc; }, close: function(ev){ const db = getMsgDb(ev,false); const response = { filename: db && db.filename }; if(db){ const doUnlink = ((ev.args && 'object'===typeof ev.args) ? !!ev.args.unlink : false); wState.close(db, doUnlink); } return response; }, exec: function(ev){ const rc = ( 'string'===typeof ev.args ) ? {sql: ev.args} : (ev.args || Object.create(null)); if('stmt'===rc.rowMode){ toss("Invalid rowMode for 'exec': stmt mode", "does not work in the Worker API."); }else if(!rc.sql){ toss("'exec' requires input SQL."); } const db = getMsgDb(ev); if(rc.callback || Array.isArray(rc.resultRows)){ // Part of a copy-avoidance optimization for blobs db._blobXfer = wState.xfer; } const theCallback = rc.callback; let rowNumber = 0; const hadColNames = !!rc.columnNames; if('string' === typeof theCallback){ if(!hadColNames) rc.columnNames = []; /* Treat this as a worker message type and post each row as a message of that type. */ rc.callback = function(row,stmt){ wState.post({ type: theCallback, columnNames: rc.columnNames, rowNumber: ++rowNumber, row: row }, wState.xfer); } } try { db.exec(rc); if(rc.callback instanceof Function){ rc.callback = theCallback; /* Post a sentinel message to tell the client that the end of the result set has been reached (possibly with zero rows). */ wState.post({ type: theCallback, columnNames: rc.columnNames, rowNumber: null /*null to distinguish from "property not set"*/, row: undefined /*undefined because null is a legal row value for some rowType values, but undefined is not*/ }); } }finally{ delete db._blobXfer; if(rc.callback) rc.callback = theCallback; } return rc; }/*exec()*/, 'config-get': function(){ const rc = Object.create(null), src = sqlite3.config; [ 'wasmfsOpfsDir', 'bigIntEnabled' ].forEach(function(k){ if(Object.getOwnPropertyDescriptor(src, k)) rc[k] = src[k]; }); rc.wasmfsOpfsEnabled = !!sqlite3.capi.sqlite3_wasmfs_opfs_dir(); rc.version = sqlite3.version; rc.vfsList = sqlite3.capi.sqlite3_js_vfs_list(); rc.opfsEnabled = !!sqlite3.opfs; return rc; }, /** Exports the database to a byte array, as per sqlite3_serialize(). Response is an object: { byteArray: Uint8Array (db file contents), filename: the current db filename, mimetype: 'application/x-sqlite3' } */ export: function(ev){ const db = getMsgDb(ev); const response = { byteArray: sqlite3.capi.sqlite3_js_db_export(db.pointer), filename: db.filename, mimetype: 'application/x-sqlite3' }; wState.xfer.push(response.byteArray.buffer); return response; }/*export()*/, toss: function(ev){ toss("Testing worker exception"); }, 'opfs-tree': async function(ev){ if(!sqlite3.opfs) toss("OPFS support is unavailable."); const response = await sqlite3.opfs.treeList(); return response; } }/*wMsgHandler*/; self.onmessage = async function(ev){ ev = ev.data; let result, dbId = ev.dbId, evType = ev.type; const arrivalTime = performance.now(); try { if(wMsgHandler.hasOwnProperty(evType) && wMsgHandler[evType] instanceof Function){ result = await wMsgHandler[evType](ev); }else{ toss("Unknown db worker message type:",ev.type); } }catch(err){ evType = 'error'; result = { operation: ev.type, message: err.message, errorClass: err.name, input: ev }; if(err.stack){ result.stack = ('string'===typeof err.stack) ? err.stack.split(/\n\s*/) : err.stack; } if(0) console.warn("Worker is propagating an exception to main thread.", "Reporting it _here_ for the stack trace:",err,result); } if(!dbId){ dbId = result.dbId/*from 'open' cmd*/ || getDefaultDbId(); } // Timing info is primarily for use in testing this API. It's not part of // the public API. arrivalTime = when the worker got the message. wState.post({ type: evType, dbId: dbId, messageId: ev.messageId, workerReceivedTime: arrivalTime, workerRespondTime: performance.now(), departureTime: ev.departureTime, // TODO: move the timing bits into... //timing:{ // departure: ev.departureTime, // workerReceived: arrivalTime, // workerResponse: performance.now(); //}, result: result }, wState.xfer); }; self.postMessage({type:'sqlite3-api',result:'worker1-ready'}); }.bind({self, sqlite3}); });