Fossil SCM

Update the built-in SQLite to an alpha of version 3.7.4.

drh 2010-11-15 20:07 trunk

Commit ea442f3704397b15ad58038a80a04bd160eda1bb

Parent bb045dbdbd42326…

2 files changed +2542 -849 +51 -11

M src/sqlite3.c

+2542 -849

		--- src/sqlite3.c
		+++ src/sqlite3.c
		@@ -1,12 +1,12 @@
1	1	/******************************************************************************
2	2	** This file is an amalgamation of many separate C source files from SQLite
3		-** version 3.7.3. By combining all the individual C code files into this
	3	+** version 3.7.4. By combining all the individual C code files into this
4	4	** single large file, the entire code can be compiled as a one translation
5	5	** unit. This allows many compilers to do optimizations that would not be
6	6	** possible if the files were compiled separately. Performance improvements
7		-** of 5% are more are commonly seen when SQLite is compiled as a single
	7	+** of 5% or more are commonly seen when SQLite is compiled as a single
8	8	** translation unit.
9	9	**
10	10	** This file is all you need to compile SQLite. To use SQLite in other
11	11	** programs, you need this file and the "sqlite3.h" header file that defines
12	12	** the programming interface to the SQLite library. (If you do not have
		@@ -648,13 +648,13 @@
648	648	**
649	649	** See also: [sqlite3_libversion()],
650	650	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
651	651	** [sqlite_version()] and [sqlite_source_id()].
652	652	*/
653		-#define SQLITE_VERSION "3.7.3"
654		-#define SQLITE_VERSION_NUMBER 3007003
655		-#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e"
	653	+#define SQLITE_VERSION "3.7.4"
	654	+#define SQLITE_VERSION_NUMBER 3007004
	655	+#define SQLITE_SOURCE_ID "2010-11-15 16:29:31 136c2ac24ee1663bc0904bce1a619ecef3d11c1c"
656	656
657	657	/*
658	658	** CAPI3REF: Run-Time Library Version Numbers
659	659	** KEYWORDS: sqlite3_version, sqlite3_sourceid
660	660	**
		@@ -3273,11 +3273,14 @@
3273	3273	** ^If the fourth parameter is negative, the length of the string is
3274	3274	** the number of bytes up to the first zero terminator.
3275	3275	**
3276	3276	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
3277	3277	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
3278		-** string after SQLite has finished with it. ^If the fifth argument is
	3278	+** string after SQLite has finished with it. ^The destructor is called
	3279	+** to dispose of the BLOB or string even if the call to sqlite3_bind_blob(),
	3280	+** sqlite3_bind_text(), or sqlite3_bind_text16() fails.
	3281	+** ^If the fifth argument is
3279	3282	** the special value [SQLITE_STATIC], then SQLite assumes that the
3280	3283	** information is in static, unmanaged space and does not need to be freed.
3281	3284	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
3282	3285	** SQLite makes its own private copy of the data immediately, before
3283	3286	** the sqlite3_bind_*() routine returns.
		@@ -3913,16 +3916,19 @@
3913	3916	** parameters. ^An aggregate SQL function requires an implementation of xStep
3914	3917	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3915	3918	** SQL function or aggregate, pass NULL poiners for all three function
3916	3919	** callbacks.
3917	3920	**
3918		-** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3919		-** then it is invoked when the function is deleted, either by being
3920		-** overloaded or when the database connection closes.
3921		-** ^When the destructure callback of the tenth parameter is invoked, it
3922		-** is passed a single argument which is a copy of the pointer which was
3923		-** the fifth parameter to sqlite3_create_function_v2().
	3921	+** ^(If the tenth parameter to sqlite3_create_function_v2() is not NULL,
	3922	+** then it is destructor for the application data pointer.
	3923	+** The destructor is invoked when the function is deleted, either by being
	3924	+** overloaded or when the database connection closes.)^
	3925	+** ^The destructor is also invoked if the call to
	3926	+** sqlite3_create_function_v2() fails.
	3927	+** ^When the destructor callback of the tenth parameter is invoked, it
	3928	+** is passed a single argument which is a copy of the application data
	3929	+** pointer which was the fifth parameter to sqlite3_create_function_v2().
3924	3930	**
3925	3931	** ^It is permitted to register multiple implementations of the same
3926	3932	** functions with the same name but with either differing numbers of
3927	3933	** arguments or differing preferred text encodings. ^SQLite will use
3928	3934	** the implementation that most closely matches the way in which the
		@@ -4381,10 +4387,19 @@
4381	4387	** with the addition that the xDestroy callback is invoked on pArg when
4382	4388	** the collating function is deleted.
4383	4389	** ^Collating functions are deleted when they are overridden by later
4384	4390	** calls to the collation creation functions or when the
4385	4391	** [database connection] is closed using [sqlite3_close()].
	4392	+**
	4393	+** ^The xDestroy callback is <u>not</u> called if the
	4394	+** sqlite3_create_collation_v2() function fails. Applications that invoke
	4395	+** sqlite3_create_collation_v2() with a non-NULL xDestroy argument should
	4396	+** check the return code and dispose of the application data pointer
	4397	+** themselves rather than expecting SQLite to deal with it for them.
	4398	+** This is different from every other SQLite interface. The inconsistency
	4399	+** is unfortunate but cannot be changed without breaking backwards
	4400	+** compatibility.
4386	4401	**
4387	4402	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
4388	4403	*/
4389	4404	SQLITE_API int sqlite3_create_collation(
4390	4405	sqlite3*,
		@@ -5136,11 +5151,13 @@
5136	5151	** when a new virtual table is be being created or reinitialized.
5137	5152	**
5138	5153	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
5139	5154	** is a pointer to a destructor for the pClientData. ^SQLite will
5140	5155	** invoke the destructor function (if it is not NULL) when SQLite
5141		-** no longer needs the pClientData pointer. ^The sqlite3_create_module()
	5156	+** no longer needs the pClientData pointer. ^The destructor will also
	5157	+** be invoked if the call to sqlite3_create_module_v2() fails.
	5158	+** ^The sqlite3_create_module()
5142	5159	** interface is equivalent to sqlite3_create_module_v2() with a NULL
5143	5160	** destructor.
5144	5161	*/
5145	5162	SQLITE_API int sqlite3_create_module(
5146	5163	sqlite3 db, / SQLite connection to register module with */
		@@ -5319,10 +5336,33 @@
5319	5336	sqlite3_int64 iRow,
5320	5337	int flags,
5321	5338	sqlite3_blob **ppBlob
5322	5339	);
5323	5340
	5341	+/*
	5342	+** CAPI3REF: Move a BLOB Handle to a New Row
	5343	+**
	5344	+** ^This function is used to move an existing blob handle so that it points
	5345	+** to a different row of the same database table. ^The new row is identified
	5346	+** by the rowid value passed as the second argument. Only the row can be
	5347	+** changed. ^The database, table and column on which the blob handle is open
	5348	+** remain the same. Moving an existing blob handle to a new row can be
	5349	+** faster than closing the existing handle and opening a new one.
	5350	+**
	5351	+** ^(The new row must meet the same criteria as for [sqlite3_blob_open()] -
	5352	+** it must exist and there must be either a blob or text value stored in
	5353	+** the nominated column.)^ ^If the new row is not present in the table, or if
	5354	+** it does not contain a blob or text value, or if another error occurs, an
	5355	+** SQLite error code is returned and the blob handle is considered aborted.
	5356	+** ^All subsequent calls to [sqlite3_blob_read()], [sqlite3_blob_write()] or
	5357	+** [sqlite3_blob_reopen()] on an aborted blob handle immediately return
	5358	+** SQLITE_ABORT.
	5359	+**
	5360	+** ^This function sets the database handle error code and message.
	5361	+*/
	5362	+SQLITE_API SQLITE_EXPERIMENTAL int sqlite3_blob_reopen(sqlite3_blob *, sqlite3_int64);
	5363	+
5324	5364	/*
5325	5365	** CAPI3REF: Close A BLOB Handle
5326	5366	**
5327	5367	** ^Closes an open [BLOB handle].
5328	5368	**
		@@ -9890,10 +9930,13 @@
9890	9930	Expr pOn; / The ON clause of a join */
9891	9931	IdList pUsing; / The USING clause of a join */
9892	9932	Bitmask colUsed; /* Bit N (1<<N) set if column N of pTab is used */
9893	9933	char zIndex; / Identifier from "INDEXED BY <zIndex>" clause */
9894	9934	Index pIndex; / Index structure corresponding to zIndex, if any */
	9935	+#ifndef SQLITE_OMIT_EXPLAIN
	9936	+ int iSelectId; /* If pSelect!=0, the id of the sub-select in EQP */
	9937	+#endif
9895	9938	} a[1]; /* One entry for each identifier on the list */
9896	9939	};
9897	9940
9898	9941	/*
9899	9942	** Permitted values of the SrcList.a.jointype field
		@@ -9922,10 +9965,11 @@
9922	9965	** case that more than one of these conditions is true.
9923	9966	*/
9924	9967	struct WherePlan {
9925	9968	u32 wsFlags; /* WHERE_* flags that describe the strategy */
9926	9969	u32 nEq; /* Number of == constraints */
	9970	+ double nRow; /* Estimated number of rows (for EQP) */
9927	9971	union {
9928	9972	Index pIdx; / Index when WHERE_INDEXED is true */
9929	9973	struct WhereTerm pTerm; / WHERE clause term for OR-search */
9930	9974	sqlite3_index_info pVtabIdx; / Virtual table index to use */
9931	9975	} u;
		@@ -10276,10 +10320,15 @@
10276	10320	Table *apVtabLock; / Pointer to virtual tables needing locking */
10277	10321	#endif
10278	10322	int nHeight; /* Expression tree height of current sub-select */
10279	10323	Table pZombieTab; / List of Table objects to delete after code gen */
10280	10324	TriggerPrg pTriggerPrg; / Linked list of coded triggers */
	10325	+
	10326	+#ifndef SQLITE_OMIT_EXPLAIN
	10327	+ int iSelectId;
	10328	+ int iNextSelectId;
	10329	+#endif
10281	10330	};
10282	10331
10283	10332	#ifdef SQLITE_OMIT_VIRTUALTABLE
10284	10333	#define IN_DECLARE_VTAB 0
10285	10334	#else
		@@ -27311,13 +27360,28 @@
27311	27360	if( flags & (SQLITE_OPEN_WAL\|SQLITE_OPEN_MAIN_JOURNAL) ){
27312	27361	char zDb[MAX_PATHNAME+1]; /* Database file path */
27313	27362	int nDb; /* Number of valid bytes in zDb */
27314	27363	struct stat sStat; /* Output of stat() on database file */
27315	27364
27316		- nDb = sqlite3Strlen30(zPath) - ((flags & SQLITE_OPEN_WAL) ? 4 : 8);
	27365	+ /* zPath is a path to a WAL or journal file. The following block derives
	27366	+ ** the path to the associated database file from zPath. This block handles
	27367	+ ** the following naming conventions:
	27368	+ **
	27369	+ ** "<path to db>-journal"
	27370	+ ** "<path to db>-wal"
	27371	+ ** "<path to db>-journal-NNNN"
	27372	+ ** "<path to db>-wal-NNNN"
	27373	+ **
	27374	+ ** where NNNN is a 4 digit decimal number. The NNNN naming schemes are
	27375	+ ** used by the test_multiplex.c module.
	27376	+ */
	27377	+ nDb = sqlite3Strlen30(zPath) - 1;
	27378	+ while( nDb>0 && zPath[nDb]!='l' ) nDb--;
	27379	+ nDb -= ((flags & SQLITE_OPEN_WAL) ? 3 : 7);
27317	27380	memcpy(zDb, zPath, nDb);
27318	27381	zDb[nDb] = '\0';
	27382	+
27319	27383	if( 0==stat(zDb, &sStat) ){
27320	27384	*pMode = sStat.st_mode & 0777;
27321	27385	}else{
27322	27386	rc = SQLITE_IOERR_FSTAT;
27323	27387	}
		@@ -34519,10 +34583,11 @@
34519	34583	# define sqlite3WalSavepointUndo(y,z) 0
34520	34584	# define sqlite3WalFrames(u,v,w,x,y,z) 0
34521	34585	# define sqlite3WalCheckpoint(u,v,w,x) 0
34522	34586	# define sqlite3WalCallback(z) 0
34523	34587	# define sqlite3WalExclusiveMode(y,z) 0
	34588	+# define sqlite3WalHeapMemory(z) 0
34524	34589	#else
34525	34590
34526	34591	#define WAL_SAVEPOINT_NDATA 4
34527	34592
34528	34593	/* Connection to a write-ahead log (WAL) file.
		@@ -34529,11 +34594,11 @@
34529	34594	** There is one object of this type for each pager.
34530	34595	*/
34531	34596	typedef struct Wal Wal;
34532	34597
34533	34598	/* Open and close a connection to a write-ahead log. */
34534		-SQLITE_PRIVATE int sqlite3WalOpen(sqlite3_vfs, sqlite3_file, const char zName, Wal*);
	34599	+SQLITE_PRIVATE int sqlite3WalOpen(sqlite3_vfs, sqlite3_file, const char zName, int, Wal*);
34535	34600	SQLITE_PRIVATE int sqlite3WalClose(Wal pWal, int sync_flags, int, u8 );
34536	34601
34537	34602	/* Used by readers to open (lock) and close (unlock) a snapshot. A
34538	34603	** snapshot is like a read-transaction. It is the state of the database
34539	34604	** at an instant in time. sqlite3WalOpenSnapshot gets a read lock and
		@@ -34585,10 +34650,16 @@
34585	34650
34586	34651	/* Tell the wal layer that an EXCLUSIVE lock has been obtained (or released)
34587	34652	** by the pager layer on the database file.
34588	34653	*/
34589	34654	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op);
	34655	+
	34656	+/* Return true if the argument is non-NULL and the WAL module is using
	34657	+** heap-memory for the wal-index. Otherwise, if the argument is NULL or the
	34658	+** WAL module is using shared-memory, return false.
	34659	+*/
	34660	+SQLITE_PRIVATE int sqlite3WalHeapMemory(Wal *pWal);
34590	34661
34591	34662	#endif /* ifndef SQLITE_OMIT_WAL */
34592	34663	#endif /* _WAL_H_ */
34593	34664
34594	34665	/************ End of wal.h ***********************************************/
		@@ -35496,11 +35567,13 @@
35496	35567	break;
35497	35568	}
35498	35569
35499	35570	return 1;
35500	35571	}
	35572	+#endif /* ifndef NDEBUG */
35501	35573
	35574	+#ifdef SQLITE_DEBUG
35502	35575	/*
35503	35576	** Return a pointer to a human readable string in a static buffer
35504	35577	** containing the state of the Pager object passed as an argument. This
35505	35578	** is intended to be used within debuggers. For example, as an alternative
35506	35579	** to "print *pPager" in gdb:
		@@ -35620,11 +35693,11 @@
35620	35693	** UNKNOWN_LOCK for an explanation of this.
35621	35694	*/
35622	35695	static int pagerUnlockDb(Pager *pPager, int eLock){
35623	35696	int rc = SQLITE_OK;
35624	35697
35625		- assert( !pPager->exclusiveMode );
	35698	+ assert( !pPager->exclusiveMode \|\| pPager->eLock==eLock );
35626	35699	assert( eLock==NO_LOCK \|\| eLock==SHARED_LOCK );
35627	35700	assert( eLock!=NO_LOCK \|\| pagerUseWal(pPager)==0 );
35628	35701	if( isOpen(pPager->fd) ){
35629	35702	assert( pPager->eLock>=eLock );
35630	35703	rc = sqlite3OsUnlock(pPager->fd, eLock);
		@@ -39094,11 +39167,11 @@
39094	39167	if( rc==SQLITE_OK ){
39095	39168	if( nPage==0 ){
39096	39169	sqlite3BeginBenignMalloc();
39097	39170	if( pagerLockDb(pPager, RESERVED_LOCK)==SQLITE_OK ){
39098	39171	sqlite3OsDelete(pVfs, pPager->zJournal, 0);
39099		- pagerUnlockDb(pPager, SHARED_LOCK);
	39172	+ if( !pPager->exclusiveMode ) pagerUnlockDb(pPager, SHARED_LOCK);
39100	39173	}
39101	39174	sqlite3EndBenignMalloc();
39102	39175	}else{
39103	39176	/* The journal file exists and no other connection has a reserved
39104	39177	** or greater lock on the database file. Now check that there is
		@@ -40920,11 +40993,12 @@
40920	40993	assert( eMode==PAGER_LOCKINGMODE_QUERY
40921	40994	\|\| eMode==PAGER_LOCKINGMODE_NORMAL
40922	40995	\|\| eMode==PAGER_LOCKINGMODE_EXCLUSIVE );
40923	40996	assert( PAGER_LOCKINGMODE_QUERY<0 );
40924	40997	assert( PAGER_LOCKINGMODE_NORMAL>=0 && PAGER_LOCKINGMODE_EXCLUSIVE>=0 );
40925		- if( eMode>=0 && !pPager->tempFile ){
	40998	+ assert( pPager->exclusiveMode \|\| 0==sqlite3WalHeapMemory(pPager->pWal) );
	40999	+ if( eMode>=0 && !pPager->tempFile && !sqlite3WalHeapMemory(pPager->pWal) ){
40926	41000	pPager->exclusiveMode = (u8)eMode;
40927	41001	}
40928	41002	return (int)pPager->exclusiveMode;
40929	41003	}
40930	41004
		@@ -41107,12 +41181,64 @@
41107	41181	** Return true if the underlying VFS for the given pager supports the
41108	41182	** primitives necessary for write-ahead logging.
41109	41183	*/
41110	41184	SQLITE_PRIVATE int sqlite3PagerWalSupported(Pager *pPager){
41111	41185	const sqlite3_io_methods *pMethods = pPager->fd->pMethods;
41112		- return pMethods->iVersion>=2 && pMethods->xShmMap!=0;
	41186	+ return pPager->exclusiveMode \|\| (pMethods->iVersion>=2 && pMethods->xShmMap);
41113	41187	}
	41188	+
	41189	+/*
	41190	+** Attempt to take an exclusive lock on the database file. If a PENDING lock
	41191	+** is obtained instead, immediately release it.
	41192	+*/
	41193	+static int pagerExclusiveLock(Pager *pPager){
	41194	+ int rc; /* Return code */
	41195	+
	41196	+ assert( pPager->eLock==SHARED_LOCK \|\| pPager->eLock==EXCLUSIVE_LOCK );
	41197	+ rc = pagerLockDb(pPager, EXCLUSIVE_LOCK);
	41198	+ if( rc!=SQLITE_OK ){
	41199	+ /* If the attempt to grab the pending lock failed, release the
	41200	+ ** exclusive lock that may have been obtained instead. */
	41201	+ pagerUnlockDb(pPager, SHARED_LOCK);
	41202	+ }
	41203	+
	41204	+ return rc;
	41205	+}
	41206	+
	41207	+/*
	41208	+** Call sqlite3WalOpen() to open the WAL handle. If the pager is in
	41209	+** exclusive-locking mode when this function is called, take an EXCLUSIVE
	41210	+** lock on the database file and use heap-memory to store the wal-index
	41211	+** in. Otherwise, use the normal shared-memory.
	41212	+*/
	41213	+static int pagerOpenWal(Pager *pPager){
	41214	+ int rc = SQLITE_OK;
	41215	+
	41216	+ assert( pPager->pWal==0 && pPager->tempFile==0 );
	41217	+ assert( pPager->eLock==SHARED_LOCK \|\| pPager->eLock==EXCLUSIVE_LOCK \|\| pPager->noReadlock);
	41218	+
	41219	+ /* If the pager is already in exclusive-mode, the WAL module will use
	41220	+ ** heap-memory for the wal-index instead of the VFS shared-memory
	41221	+ ** implementation. Take the exclusive lock now, before opening the WAL
	41222	+ ** file, to make sure this is safe.
	41223	+ */
	41224	+ if( pPager->exclusiveMode ){
	41225	+ rc = pagerExclusiveLock(pPager);
	41226	+ }
	41227	+
	41228	+ /* Open the connection to the log file. If this operation fails,
	41229	+ ** (e.g. due to malloc() failure), return an error code.
	41230	+ */
	41231	+ if( rc==SQLITE_OK ){
	41232	+ rc = sqlite3WalOpen(pPager->pVfs,
	41233	+ pPager->fd, pPager->zWal, pPager->exclusiveMode, &pPager->pWal
	41234	+ );
	41235	+ }
	41236	+
	41237	+ return rc;
	41238	+}
	41239	+
41114	41240
41115	41241	/*
41116	41242	** The caller must be holding a SHARED lock on the database file to call
41117	41243	** this function.
41118	41244	**
		@@ -41143,15 +41269,11 @@
41143	41269	if( !sqlite3PagerWalSupported(pPager) ) return SQLITE_CANTOPEN;
41144	41270
41145	41271	/* Close any rollback journal previously open */
41146	41272	sqlite3OsClose(pPager->jfd);
41147	41273
41148		- /* Open the connection to the log file. If this operation fails,
41149		- ** (e.g. due to malloc() failure), unlock the database file and
41150		- ** return an error code.
41151		- */
41152		- rc = sqlite3WalOpen(pPager->pVfs, pPager->fd, pPager->zWal, &pPager->pWal);
	41274	+ rc = pagerOpenWal(pPager);
41153	41275	if( rc==SQLITE_OK ){
41154	41276	pPager->journalMode = PAGER_JOURNALMODE_WAL;
41155	41277	pPager->eState = PAGER_OPEN;
41156	41278	}
41157	41279	}else{
		@@ -41186,30 +41308,25 @@
41186	41308	rc = sqlite3OsAccess(
41187	41309	pPager->pVfs, pPager->zWal, SQLITE_ACCESS_EXISTS, &logexists
41188	41310	);
41189	41311	}
41190	41312	if( rc==SQLITE_OK && logexists ){
41191		- rc = sqlite3WalOpen(pPager->pVfs, pPager->fd,
41192		- pPager->zWal, &pPager->pWal);
	41313	+ rc = pagerOpenWal(pPager);
41193	41314	}
41194	41315	}
41195	41316
41196	41317	/* Checkpoint and close the log. Because an EXCLUSIVE lock is held on
41197	41318	** the database file, the log and log-summary files will be deleted.
41198	41319	*/
41199	41320	if( rc==SQLITE_OK && pPager->pWal ){
41200		- rc = pagerLockDb(pPager, EXCLUSIVE_LOCK);
	41321	+ rc = pagerExclusiveLock(pPager);
41201	41322	if( rc==SQLITE_OK ){
41202	41323	rc = sqlite3WalClose(pPager->pWal,
41203		- (pPager->noSync ? 0 : pPager->sync_flags),
41204		- pPager->pageSize, (u8*)pPager->pTmpSpace
	41324	+ (pPager->noSync ? 0 : pPager->sync_flags),
	41325	+ pPager->pageSize, (u8*)pPager->pTmpSpace
41205	41326	);
41206	41327	pPager->pWal = 0;
41207		- }else{
41208		- /* If we cannot get an EXCLUSIVE lock, downgrade the PENDING lock
41209		- ** that we did get back to SHARED. */
41210		- pagerUnlockDb(pPager, SQLITE_LOCK_SHARED);
41211	41328	}
41212	41329	}
41213	41330	return rc;
41214	41331	}
41215	41332
		@@ -41661,10 +41778,17 @@
41661	41778	#ifdef SQLITE_DEBUG
41662	41779	u8 lockError; /* True if a locking error has occurred */
41663	41780	#endif
41664	41781	};
41665	41782
	41783	+/*
	41784	+** Candidate values for Wal.exclusiveMode.
	41785	+*/
	41786	+#define WAL_NORMAL_MODE 0
	41787	+#define WAL_EXCLUSIVE_MODE 1
	41788	+#define WAL_HEAPMEMORY_MODE 2
	41789	+
41666	41790	/*
41667	41791	** Each page of the wal-index mapping contains a hash-table made up of
41668	41792	** an array of HASHTABLE_NSLOT elements of the following type.
41669	41793	*/
41670	41794	typedef u16 ht_slot;
		@@ -41747,13 +41871,18 @@
41747	41871	pWal->nWiData = iPage+1;
41748	41872	}
41749	41873
41750	41874	/* Request a pointer to the required page from the VFS */
41751	41875	if( pWal->apWiData[iPage]==0 ){
41752		- rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ,
41753		- pWal->writeLock, (void volatile **)&pWal->apWiData[iPage]
41754		- );
	41876	+ if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){
	41877	+ pWal->apWiData[iPage] = (u32 volatile *)sqlite3MallocZero(WALINDEX_PGSZ);
	41878	+ if( !pWal->apWiData[iPage] ) rc = SQLITE_NOMEM;
	41879	+ }else{
	41880	+ rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ,
	41881	+ pWal->writeLock, (void volatile **)&pWal->apWiData[iPage]
	41882	+ );
	41883	+ }
41755	41884	}
41756	41885
41757	41886	*ppPage = pWal->apWiData[iPage];
41758	41887	assert( iPage==0 \|\| *ppPage \|\| rc!=SQLITE_OK );
41759	41888	return rc;
		@@ -41831,10 +41960,16 @@
41831	41960	}
41832	41961
41833	41962	aOut[0] = s1;
41834	41963	aOut[1] = s2;
41835	41964	}
	41965	+
	41966	+static void walShmBarrier(Wal *pWal){
	41967	+ if( pWal->exclusiveMode!=WAL_HEAPMEMORY_MODE ){
	41968	+ sqlite3OsShmBarrier(pWal->pDbFd);
	41969	+ }
	41970	+}
41836	41971
41837	41972	/*
41838	41973	** Write the header information in pWal->hdr into the wal-index.
41839	41974	**
41840	41975	** The checksum on pWal->hdr is updated before it is written.
		@@ -41846,11 +41981,11 @@
41846	41981	assert( pWal->writeLock );
41847	41982	pWal->hdr.isInit = 1;
41848	41983	pWal->hdr.iVersion = WALINDEX_MAX_VERSION;
41849	41984	walChecksumBytes(1, (u8*)&pWal->hdr, nCksum, 0, pWal->hdr.aCksum);
41850	41985	memcpy((void )&aHdr[1], (void )&pWal->hdr, sizeof(WalIndexHdr));
41851		- sqlite3OsShmBarrier(pWal->pDbFd);
	41986	+ walShmBarrier(pWal);
41852	41987	memcpy((void )&aHdr[0], (void )&pWal->hdr, sizeof(WalIndexHdr));
41853	41988	}
41854	41989
41855	41990	/*
41856	41991	** This function encodes a single frame header and writes it to a buffer
		@@ -42418,11 +42553,19 @@
42418	42553
42419	42554	/*
42420	42555	** Close an open wal-index.
42421	42556	*/
42422	42557	static void walIndexClose(Wal *pWal, int isDelete){
42423		- sqlite3OsShmUnmap(pWal->pDbFd, isDelete);
	42558	+ if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){
	42559	+ int i;
	42560	+ for(i=0; i<pWal->nWiData; i++){
	42561	+ sqlite3_free((void *)pWal->apWiData[i]);
	42562	+ pWal->apWiData[i] = 0;
	42563	+ }
	42564	+ }else{
	42565	+ sqlite3OsShmUnmap(pWal->pDbFd, isDelete);
	42566	+ }
42424	42567	}
42425	42568
42426	42569	/*
42427	42570	** Open a connection to the WAL file zWalName. The database file must
42428	42571	** already be opened on connection pDbFd. The buffer that zWalName points
		@@ -42440,10 +42583,11 @@
42440	42583	*/
42441	42584	SQLITE_PRIVATE int sqlite3WalOpen(
42442	42585	sqlite3_vfs pVfs, / vfs module to open wal and wal-index */
42443	42586	sqlite3_file pDbFd, / The open database file */
42444	42587	const char zWalName, / Name of the WAL file */
	42588	+ int bNoShm, /* True to run in heap-memory mode */
42445	42589	Wal *ppWal / OUT: Allocated Wal handle */
42446	42590	){
42447	42591	int rc; /* Return Code */
42448	42592	Wal pRet; / Object to allocate and return */
42449	42593	int flags; /* Flags passed to OsOpen() */
		@@ -42473,10 +42617,11 @@
42473	42617	pRet->pVfs = pVfs;
42474	42618	pRet->pWalFd = (sqlite3_file *)&pRet[1];
42475	42619	pRet->pDbFd = pDbFd;
42476	42620	pRet->readLock = -1;
42477	42621	pRet->zWalName = zWalName;
	42622	+ pRet->exclusiveMode = (bNoShm ? WAL_HEAPMEMORY_MODE: WAL_NORMAL_MODE);
42478	42623
42479	42624	/* Open file handle on the write-ahead log file. */
42480	42625	flags = (SQLITE_OPEN_READWRITE\|SQLITE_OPEN_CREATE\|SQLITE_OPEN_WAL);
42481	42626	rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, flags, &flags);
42482	42627	if( rc==SQLITE_OK && flags&SQLITE_OPEN_READONLY ){
		@@ -42906,11 +43051,13 @@
42906	43051	**
42907	43052	** The EXCLUSIVE lock is not released before returning.
42908	43053	*/
42909	43054	rc = sqlite3OsLock(pWal->pDbFd, SQLITE_LOCK_EXCLUSIVE);
42910	43055	if( rc==SQLITE_OK ){
42911		- pWal->exclusiveMode = 1;
	43056	+ if( pWal->exclusiveMode==WAL_NORMAL_MODE ){
	43057	+ pWal->exclusiveMode = WAL_EXCLUSIVE_MODE;
	43058	+ }
42912	43059	rc = sqlite3WalCheckpoint(pWal, sync_flags, nBuf, zBuf);
42913	43060	if( rc==SQLITE_OK ){
42914	43061	isDelete = 1;
42915	43062	}
42916	43063	}
		@@ -42962,11 +43109,11 @@
42962	43109	** Memory barriers are used to prevent the compiler or the hardware from
42963	43110	** reordering the reads and writes.
42964	43111	*/
42965	43112	aHdr = walIndexHdr(pWal);
42966	43113	memcpy(&h1, (void *)&aHdr[0], sizeof(h1));
42967		- sqlite3OsShmBarrier(pWal->pDbFd);
	43114	+ walShmBarrier(pWal);
42968	43115	memcpy(&h2, (void *)&aHdr[1], sizeof(h2));
42969	43116
42970	43117	if( memcmp(&h1, &h2, sizeof(h1))!=0 ){
42971	43118	return 1; /* Dirty read */
42972	43119	}
		@@ -43163,11 +43310,11 @@
43163	43310	if( !useWal && pInfo->nBackfill==pWal->hdr.mxFrame ){
43164	43311	/* The WAL has been completely backfilled (or it is empty).
43165	43312	** and can be safely ignored.
43166	43313	*/
43167	43314	rc = walLockShared(pWal, WAL_READ_LOCK(0));
43168		- sqlite3OsShmBarrier(pWal->pDbFd);
	43315	+ walShmBarrier(pWal);
43169	43316	if( rc==SQLITE_OK ){
43170	43317	if( memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr)) ){
43171	43318	/* It is not safe to allow the reader to continue here if frames
43172	43319	** may have been appended to the log before READ_LOCK(0) was obtained.
43173	43320	** When holding READ_LOCK(0), the reader ignores the entire log file,
		@@ -43257,11 +43404,11 @@
43257	43404	** date before proceeding. That would not be possible without somehow
43258	43405	** blocking writers. It only guarantees that a dangerous checkpoint or
43259	43406	** log-wrap (either of which would require an exclusive lock on
43260	43407	** WAL_READ_LOCK(mxI)) has not occurred since the snapshot was valid.
43261	43408	*/
43262		- sqlite3OsShmBarrier(pWal->pDbFd);
	43409	+ walShmBarrier(pWal);
43263	43410	if( pInfo->aReadMark[mxI]!=mxReadMark
43264	43411	\|\| memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr))
43265	43412	){
43266	43413	walUnlockShared(pWal, WAL_READ_LOCK(mxI));
43267	43414	return WAL_RETRY;
		@@ -43900,17 +44047,18 @@
43900	44047	** WAL is already in exclusive-locking mode - meaning that this
43901	44048	** routine is a no-op. The pager must already hold the exclusive lock
43902	44049	** on the main database file before invoking this operation.
43903	44050	**
43904	44051	** If op is negative, then do a dry-run of the op==1 case but do
43905		-** not actually change anything. The pager uses this to see if it
	44052	+** not actually change anything. The pager uses this to see if it
43906	44053	** should acquire the database exclusive lock prior to invoking
43907	44054	** the op==1 case.
43908	44055	*/
43909	44056	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op){
43910	44057	int rc;
43911	44058	assert( pWal->writeLock==0 );
	44059	+ assert( pWal->exclusiveMode!=WAL_HEAPMEMORY_MODE \|\| op==-1 );
43912	44060
43913	44061	/* pWal->readLock is usually set, but might be -1 if there was a
43914	44062	** prior error while attempting to acquire are read-lock. This cannot
43915	44063	** happen if the connection is actually in exclusive mode (as no xShmLock
43916	44064	** locks are taken in this case). Nor should the pager attempt to
		@@ -43939,10 +44087,19 @@
43939	44087	}else{
43940	44088	rc = pWal->exclusiveMode==0;
43941	44089	}
43942	44090	return rc;
43943	44091	}
	44092	+
	44093	+/*
	44094	+** Return true if the argument is non-NULL and the WAL module is using
	44095	+** heap-memory for the wal-index. Otherwise, if the argument is NULL or the
	44096	+** WAL module is using shared-memory, return false.
	44097	+*/
	44098	+SQLITE_PRIVATE int sqlite3WalHeapMemory(Wal *pWal){
	44099	+ return (pWal && pWal->exclusiveMode==WAL_HEAPMEMORY_MODE );
	44100	+}
43944	44101
43945	44102	#endif /* #ifndef SQLITE_OMIT_WAL */
43946	44103
43947	44104	/************ End of wal.c ***********************************************/
43948	44105	/************ Begin file btmutex.c ***************************************/
		@@ -48119,20 +48276,21 @@
48119	48276	**
48120	48277	** This will release the write lock on the database file. If there
48121	48278	** are no active cursors, it also releases the read lock.
48122	48279	*/
48123	48280	SQLITE_PRIVATE int sqlite3BtreeCommitPhaseTwo(Btree *p){
48124		- BtShared *pBt = p->pBt;
48125	48281
	48282	+ if( p->inTrans==TRANS_NONE ) return SQLITE_OK;
48126	48283	sqlite3BtreeEnter(p);
48127	48284	btreeIntegrity(p);
48128	48285
48129	48286	/* If the handle has a write-transaction open, commit the shared-btrees
48130	48287	** transaction and set the shared state to TRANS_READ.
48131	48288	*/
48132	48289	if( p->inTrans==TRANS_WRITE ){
48133	48290	int rc;
	48291	+ BtShared *pBt = p->pBt;
48134	48292	assert( pBt->inTransaction==TRANS_WRITE );
48135	48293	assert( pBt->nTransaction>0 );
48136	48294	rc = sqlite3PagerCommitPhaseTwo(pBt->pPager);
48137	48295	if( rc!=SQLITE_OK ){
48138	48296	sqlite3BtreeLeave(p);
		@@ -53047,12 +53205,11 @@
53047	53205	** sqlite3BtreePutData()).
53048	53206	*/
53049	53207	SQLITE_PRIVATE void sqlite3BtreeCacheOverflow(BtCursor *pCur){
53050	53208	assert( cursorHoldsMutex(pCur) );
53051	53209	assert( sqlite3_mutex_held(pCur->pBtree->db->mutex) );
53052		- assert(!pCur->isIncrblobHandle);
53053		- assert(!pCur->aOverflow);
	53210	+ invalidateOverflowCache(pCur);
53054	53211	pCur->isIncrblobHandle = 1;
53055	53212	}
53056	53213	#endif
53057	53214
53058	53215	/*
		@@ -56081,16 +56238,14 @@
56081	56238	pMem->flags = MEM_Int;
56082	56239	pMem->u.i = pOp->p2; /* P2 */
56083	56240	pMem->type = SQLITE_INTEGER;
56084	56241	pMem++;
56085	56242
56086		- if( p->explain==1 ){
56087		- pMem->flags = MEM_Int;
56088		- pMem->u.i = pOp->p3; /* P3 */
56089		- pMem->type = SQLITE_INTEGER;
56090		- pMem++;
56091		- }
	56243	+ pMem->flags = MEM_Int;
	56244	+ pMem->u.i = pOp->p3; /* P3 */
	56245	+ pMem->type = SQLITE_INTEGER;
	56246	+ pMem++;
56092	56247
56093	56248	if( sqlite3VdbeMemGrow(pMem, 32, 0) ){ /* P4 */
56094	56249	assert( p->db->mallocFailed );
56095	56250	return SQLITE_ERROR;
56096	56251	}
		@@ -56131,11 +56286,11 @@
56131	56286	pMem->flags = MEM_Null; /* Comment */
56132	56287	pMem->type = SQLITE_NULL;
56133	56288	}
56134	56289	}
56135	56290
56136		- p->nResColumn = 8 - 5*(p->explain-1);
	56291	+ p->nResColumn = 8 - 4*(p->explain-1);
56137	56292	p->rc = SQLITE_OK;
56138	56293	rc = SQLITE_ROW;
56139	56294	}
56140	56295	return rc;
56141	56296	}
		@@ -59078,10 +59233,12 @@
59078	59233	}
59079	59234	sqlite3Error(p->db, rc, 0);
59080	59235	rc = sqlite3ApiExit(p->db, rc);
59081	59236	}
59082	59237	sqlite3_mutex_leave(p->db->mutex);
	59238	+ }else if( xDel!=SQLITE_STATIC && xDel!=SQLITE_TRANSIENT ){
	59239	+ xDel((void*)zData);
59083	59240	}
59084	59241	return rc;
59085	59242	}
59086	59243
59087	59244
		@@ -66071,15 +66228,86 @@
66071	66228	typedef struct Incrblob Incrblob;
66072	66229	struct Incrblob {
66073	66230	int flags; /* Copy of "flags" passed to sqlite3_blob_open() */
66074	66231	int nByte; /* Size of open blob, in bytes */
66075	66232	int iOffset; /* Byte offset of blob in cursor data */
	66233	+ int iCol; /* Table column this handle is open on */
66076	66234	BtCursor pCsr; / Cursor pointing at blob row */
66077	66235	sqlite3_stmt pStmt; / Statement holding cursor open */
66078	66236	sqlite3 db; / The associated database */
66079	66237	};
66080	66238
	66239	+
	66240	+/*
	66241	+** This function is used by both blob_open() and blob_reopen(). It seeks
	66242	+** the b-tree cursor associated with blob handle p to point to row iRow.
	66243	+** If successful, SQLITE_OK is returned and subsequent calls to
	66244	+** sqlite3_blob_read() or sqlite3_blob_write() access the specified row.
	66245	+**
	66246	+** If an error occurs, or if the specified row does not exist or does not
	66247	+** contain a value of type TEXT or BLOB in the column nominated when the
	66248	+** blob handle was opened, then an error code is returned and *pzErr may
	66249	+** be set to point to a buffer containing an error message. It is the
	66250	+** responsibility of the caller to free the error message buffer using
	66251	+** sqlite3DbFree().
	66252	+**
	66253	+** If an error does occur, then the b-tree cursor is closed. All subsequent
	66254	+** calls to sqlite3_blob_read(), blob_write() or blob_reopen() will
	66255	+** immediately return SQLITE_ABORT.
	66256	+*/
	66257	+static int blobSeekToRow(Incrblob p, sqlite3_int64 iRow, char *pzErr){
	66258	+ int rc; /* Error code */
	66259	+ char zErr = 0; / Error message */
	66260	+ Vdbe v = (Vdbe )p->pStmt;
	66261	+
	66262	+ /* Set the value of the SQL statements only variable to integer iRow.
	66263	+ ** This is done directly instead of using sqlite3_bind_int64() to avoid
	66264	+ ** triggering asserts related to mutexes.
	66265	+ */
	66266	+ assert( v->aVar[0].flags&MEM_Int );
	66267	+ v->aVar[0].u.i = iRow;
	66268	+
	66269	+ rc = sqlite3_step(p->pStmt);
	66270	+ if( rc==SQLITE_ROW ){
	66271	+ u32 type = v->apCsr[0]->aType[p->iCol];
	66272	+ if( type<12 ){
	66273	+ zErr = sqlite3MPrintf(p->db, "cannot open value of type %s",
	66274	+ type==0?"null": type==7?"real": "integer"
	66275	+ );
	66276	+ rc = SQLITE_ERROR;
	66277	+ sqlite3_finalize(p->pStmt);
	66278	+ p->pStmt = 0;
	66279	+ }else{
	66280	+ p->iOffset = v->apCsr[0]->aOffset[p->iCol];
	66281	+ p->nByte = sqlite3VdbeSerialTypeLen(type);
	66282	+ p->pCsr = v->apCsr[0]->pCursor;
	66283	+ sqlite3BtreeEnterCursor(p->pCsr);
	66284	+ sqlite3BtreeCacheOverflow(p->pCsr);
	66285	+ sqlite3BtreeLeaveCursor(p->pCsr);
	66286	+ }
	66287	+ }
	66288	+
	66289	+ if( rc==SQLITE_ROW ){
	66290	+ rc = SQLITE_OK;
	66291	+ }else if( p->pStmt ){
	66292	+ rc = sqlite3_finalize(p->pStmt);
	66293	+ p->pStmt = 0;
	66294	+ if( rc==SQLITE_OK ){
	66295	+ zErr = sqlite3MPrintf(p->db, "no such rowid: %lld", iRow);
	66296	+ rc = SQLITE_ERROR;
	66297	+ }else{
	66298	+ zErr = sqlite3MPrintf(p->db, "%s", sqlite3_errmsg(p->db));
	66299	+ }
	66300	+ }
	66301	+
	66302	+ assert( rc!=SQLITE_OK \|\| zErr==0 );
	66303	+ assert( rc!=SQLITE_ROW && rc!=SQLITE_DONE );
	66304	+
	66305	+ *pzErr = zErr;
	66306	+ return rc;
	66307	+}
	66308	+
66081	66309	/*
66082	66310	** Open a blob handle.
66083	66311	*/
66084	66312	SQLITE_API int sqlite3_blob_open(
66085	66313	sqlite3* db, /* The database connection */
		@@ -66116,33 +66344,39 @@
66116	66344	/* One of the following two instructions is replaced by an OP_Noop. */
66117	66345	{OP_OpenRead, 0, 0, 0}, /* 3: Open cursor 0 for reading */
66118	66346	{OP_OpenWrite, 0, 0, 0}, /* 4: Open cursor 0 for read/write */
66119	66347
66120	66348	{OP_Variable, 1, 1, 1}, /* 5: Push the rowid to the stack */
66121		- {OP_NotExists, 0, 9, 1}, /* 6: Seek the cursor */
	66349	+ {OP_NotExists, 0, 10, 1}, /* 6: Seek the cursor */
66122	66350	{OP_Column, 0, 0, 1}, /* 7 */
66123	66351	{OP_ResultRow, 1, 0, 0}, /* 8 */
66124		- {OP_Close, 0, 0, 0}, /* 9 */
66125		- {OP_Halt, 0, 0, 0}, /* 10 */
	66352	+ {OP_Goto, 0, 5, 0}, /* 9 */
	66353	+ {OP_Close, 0, 0, 0}, /* 10 */
	66354	+ {OP_Halt, 0, 0, 0}, /* 11 */
66126	66355	};
66127	66356
66128		- Vdbe *v = 0;
66129	66357	int rc = SQLITE_OK;
66130	66358	char *zErr = 0;
66131	66359	Table *pTab;
66132		- Parse *pParse;
	66360	+ Parse *pParse = 0;
	66361	+ Incrblob *pBlob = 0;
66133	66362
	66363	+ flags = !!flags; /* flags = (flags ? 1 : 0); */
66134	66364	*ppBlob = 0;
	66365	+
66135	66366	sqlite3_mutex_enter(db->mutex);
	66367	+
	66368	+ pBlob = (Incrblob *)sqlite3DbMallocZero(db, sizeof(Incrblob));
	66369	+ if( !pBlob ) goto blob_open_out;
66136	66370	pParse = sqlite3StackAllocRaw(db, sizeof(*pParse));
66137		- if( pParse==0 ){
66138		- rc = SQLITE_NOMEM;
66139		- goto blob_open_out;
66140		- }
	66371	+ if( !pParse ) goto blob_open_out;
	66372	+
66141	66373	do {
66142	66374	memset(pParse, 0, sizeof(Parse));
66143	66375	pParse->db = db;
	66376	+ sqlite3DbFree(db, zErr);
	66377	+ zErr = 0;
66144	66378
66145	66379	sqlite3BtreeEnterAll(db);
66146	66380	pTab = sqlite3LocateTable(pParse, 0, zTable, zDb);
66147	66381	if( pTab && IsVirtual(pTab) ){
66148	66382	pTab = 0;
		@@ -66164,11 +66398,11 @@
66164	66398	sqlite3BtreeLeaveAll(db);
66165	66399	goto blob_open_out;
66166	66400	}
66167	66401
66168	66402	/* Now search pTab for the exact column. */
66169		- for(iCol=0; iCol < pTab->nCol; iCol++) {
	66403	+ for(iCol=0; iCol<pTab->nCol; iCol++) {
66170	66404	if( sqlite3StrICmp(pTab->aCol[iCol].zName, zColumn)==0 ){
66171	66405	break;
66172	66406	}
66173	66407	}
66174	66408	if( iCol==pTab->nCol ){
		@@ -66218,15 +66452,18 @@
66218	66452	sqlite3BtreeLeaveAll(db);
66219	66453	goto blob_open_out;
66220	66454	}
66221	66455	}
66222	66456
66223		- v = sqlite3VdbeCreate(db);
66224		- if( v ){
	66457	+ pBlob->pStmt = (sqlite3_stmt *)sqlite3VdbeCreate(db);
	66458	+ assert( pBlob->pStmt \|\| db->mallocFailed );
	66459	+ if( pBlob->pStmt ){
	66460	+ Vdbe v = (Vdbe )pBlob->pStmt;
66225	66461	int iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
	66462	+
66226	66463	sqlite3VdbeAddOpList(v, sizeof(openBlob)/sizeof(VdbeOpList), openBlob);
66227		- flags = !!flags; /* flags = (flags ? 1 : 0); */
	66464	+
66228	66465
66229	66466	/* Configure the OP_Transaction */
66230	66467	sqlite3VdbeChangeP1(v, 0, iDb);
66231	66468	sqlite3VdbeChangeP2(v, 0, flags);
66232	66469
		@@ -66265,69 +66502,29 @@
66265	66502	if( !db->mallocFailed ){
66266	66503	sqlite3VdbeMakeReady(v, 1, 1, 1, 0, 0, 0);
66267	66504	}
66268	66505	}
66269	66506
	66507	+ pBlob->flags = flags;
	66508	+ pBlob->iCol = iCol;
	66509	+ pBlob->db = db;
66270	66510	sqlite3BtreeLeaveAll(db);
66271	66511	if( db->mallocFailed ){
66272	66512	goto blob_open_out;
66273	66513	}
66274		-
66275		- sqlite3_bind_int64((sqlite3_stmt *)v, 1, iRow);
66276		- rc = sqlite3_step((sqlite3_stmt *)v);
66277		- if( rc!=SQLITE_ROW ){
66278		- nAttempt++;
66279		- rc = sqlite3_finalize((sqlite3_stmt *)v);
66280		- sqlite3DbFree(db, zErr);
66281		- zErr = sqlite3MPrintf(db, sqlite3_errmsg(db));
66282		- v = 0;
66283		- }
66284		- } while( nAttempt<5 && rc==SQLITE_SCHEMA );
66285		-
66286		- if( rc==SQLITE_ROW ){
66287		- /* The row-record has been opened successfully. Check that the
66288		- ** column in question contains text or a blob. If it contains
66289		- ** text, it is up to the caller to get the encoding right.
66290		- */
66291		- Incrblob *pBlob;
66292		- u32 type = v->apCsr[0]->aType[iCol];
66293		-
66294		- if( type<12 ){
66295		- sqlite3DbFree(db, zErr);
66296		- zErr = sqlite3MPrintf(db, "cannot open value of type %s",
66297		- type==0?"null": type==7?"real": "integer"
66298		- );
66299		- rc = SQLITE_ERROR;
66300		- goto blob_open_out;
66301		- }
66302		- pBlob = (Incrblob *)sqlite3DbMallocZero(db, sizeof(Incrblob));
66303		- if( db->mallocFailed ){
66304		- sqlite3DbFree(db, pBlob);
66305		- goto blob_open_out;
66306		- }
66307		- pBlob->flags = flags;
66308		- pBlob->pCsr = v->apCsr[0]->pCursor;
66309		- sqlite3BtreeEnterCursor(pBlob->pCsr);
66310		- sqlite3BtreeCacheOverflow(pBlob->pCsr);
66311		- sqlite3BtreeLeaveCursor(pBlob->pCsr);
66312		- pBlob->pStmt = (sqlite3_stmt *)v;
66313		- pBlob->iOffset = v->apCsr[0]->aOffset[iCol];
66314		- pBlob->nByte = sqlite3VdbeSerialTypeLen(type);
66315		- pBlob->db = db;
66316		- ppBlob = (sqlite3_blob )pBlob;
66317		- rc = SQLITE_OK;
66318		- }else if( rc==SQLITE_OK ){
66319		- sqlite3DbFree(db, zErr);
66320		- zErr = sqlite3MPrintf(db, "no such rowid: %lld", iRow);
66321		- rc = SQLITE_ERROR;
66322		- }
	66514	+ sqlite3_bind_int64(pBlob->pStmt, 1, iRow);
	66515	+ rc = blobSeekToRow(pBlob, iRow, &zErr);
	66516	+ } while( (++nAttempt)<5 && rc==SQLITE_SCHEMA );
66323	66517
66324	66518	blob_open_out:
66325		- if( v && (rc!=SQLITE_OK \|\| db->mallocFailed) ){
66326		- sqlite3VdbeFinalize(v);
	66519	+ if( rc==SQLITE_OK && db->mallocFailed==0 ){
	66520	+ ppBlob = (sqlite3_blob )pBlob;
	66521	+ }else{
	66522	+ if( pBlob && pBlob->pStmt ) sqlite3VdbeFinalize((Vdbe *)pBlob->pStmt);
	66523	+ sqlite3DbFree(db, pBlob);
66327	66524	}
66328		- sqlite3Error(db, rc, zErr);
	66525	+ sqlite3Error(db, rc, (zErr ? "%s" : 0), zErr);
66329	66526	sqlite3DbFree(db, zErr);
66330	66527	sqlite3StackFree(db, pParse);
66331	66528	rc = sqlite3ApiExit(db, rc);
66332	66529	sqlite3_mutex_leave(db->mutex);
66333	66530	return rc;
		@@ -66376,11 +66573,11 @@
66376	66573
66377	66574	if( n<0 \|\| iOffset<0 \|\| (iOffset+n)>p->nByte ){
66378	66575	/* Request is out of range. Return a transient error. */
66379	66576	rc = SQLITE_ERROR;
66380	66577	sqlite3Error(db, SQLITE_ERROR, 0);
66381		- } else if( v==0 ){
	66578	+ }else if( v==0 ){
66382	66579	/* If there is no statement handle, then the blob-handle has
66383	66580	** already been invalidated. Return SQLITE_ABORT in this case.
66384	66581	*/
66385	66582	rc = SQLITE_ABORT;
66386	66583	}else{
		@@ -66426,10 +66623,49 @@
66426	66623	*/
66427	66624	SQLITE_API int sqlite3_blob_bytes(sqlite3_blob *pBlob){
66428	66625	Incrblob p = (Incrblob )pBlob;
66429	66626	return p ? p->nByte : 0;
66430	66627	}
	66628	+
	66629	+/*
	66630	+** Move an existing blob handle to point to a different row of the same
	66631	+** database table.
	66632	+**
	66633	+** If an error occurs, or if the specified row does not exist or does not
	66634	+** contain a blob or text value, then an error code is returned and the
	66635	+** database handle error code and message set. If this happens, then all
	66636	+** subsequent calls to sqlite3_blob_xxx() functions (except blob_close())
	66637	+** immediately return SQLITE_ABORT.
	66638	+*/
	66639	+SQLITE_API int sqlite3_blob_reopen(sqlite3_blob *pBlob, sqlite3_int64 iRow){
	66640	+ int rc;
	66641	+ Incrblob p = (Incrblob )pBlob;
	66642	+ sqlite3 *db;
	66643	+
	66644	+ if( p==0 ) return SQLITE_MISUSE_BKPT;
	66645	+ db = p->db;
	66646	+ sqlite3_mutex_enter(db->mutex);
	66647	+
	66648	+ if( p->pStmt==0 ){
	66649	+ /* If there is no statement handle, then the blob-handle has
	66650	+ ** already been invalidated. Return SQLITE_ABORT in this case.
	66651	+ */
	66652	+ rc = SQLITE_ABORT;
	66653	+ }else{
	66654	+ char *zErr;
	66655	+ rc = blobSeekToRow(p, iRow, &zErr);
	66656	+ if( rc!=SQLITE_OK ){
	66657	+ sqlite3Error(db, rc, (zErr ? "%s" : 0), zErr);
	66658	+ sqlite3DbFree(db, zErr);
	66659	+ }
	66660	+ assert( rc!=SQLITE_SCHEMA );
	66661	+ }
	66662	+
	66663	+ rc = sqlite3ApiExit(db, rc);
	66664	+ sqlite3_mutex_leave(db->mutex);
	66665	+ return rc;
	66666	+}
66431	66667
66432	66668	#endif /* #ifndef SQLITE_OMIT_INCRBLOB */
66433	66669
66434	66670	/************ End of vdbeblob.c ******************************************/
66435	66671	/************ Begin file journal.c ***************************************/
		@@ -68757,10 +68993,13 @@
68757	68993	Expr pRight, / Right operand */
68758	68994	const Token pToken / Argument token */
68759	68995	){
68760	68996	Expr *p = sqlite3ExprAlloc(pParse->db, op, pToken, 1);
68761	68997	sqlite3ExprAttachSubtrees(pParse->db, p, pLeft, pRight);
	68998	+ if( p ) {
	68999	+ sqlite3ExprCheckHeight(pParse, p->nHeight);
	69000	+ }
68762	69001	return p;
68763	69002	}
68764	69003
68765	69004	/*
68766	69005	** Join two expressions using an AND operator. If either expression is
		@@ -69869,10 +70108,20 @@
69869	70108	int mem = ++pParse->nMem;
69870	70109	sqlite3VdbeAddOp1(v, OP_If, mem);
69871	70110	testAddr = sqlite3VdbeAddOp2(v, OP_Integer, 1, mem);
69872	70111	assert( testAddr>0 \|\| pParse->db->mallocFailed );
69873	70112	}
	70113	+
	70114	+#ifndef SQLITE_OMIT_EXPLAIN
	70115	+ if( pParse->explain==2 ){
	70116	+ char *zMsg = sqlite3MPrintf(
	70117	+ pParse->db, "EXECUTE %s%s SUBQUERY %d", testAddr?"":"CORRELATED ",
	70118	+ pExpr->op==TK_IN?"LIST":"SCALAR", pParse->iNextSelectId
	70119	+ );
	70120	+ sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
	70121	+ }
	70122	+#endif
69874	70123
69875	70124	switch( pExpr->op ){
69876	70125	case TK_IN: {
69877	70126	char affinity; /* Affinity of the LHS of the IN */
69878	70127	KeyInfo keyInfo; /* Keyinfo for the generated table */
		@@ -84056,10 +84305,31 @@
84056	84305	int (extended_result_codes)(sqlite3,int);
84057	84306	int (limit)(sqlite3,int,int);
84058	84307	sqlite3_stmt (next_stmt)(sqlite3,sqlite3_stmt);
84059	84308	const char (sql)(sqlite3_stmt*);
84060	84309	int (status)(int,int,int*,int);
	84310	+ int (backup_finish)(sqlite3_backup);
	84311	+ sqlite3_backup (backup_init)(sqlite3,const char,sqlite3,const char);
	84312	+ int (backup_pagecount)(sqlite3_backup);
	84313	+ int (backup_remaining)(sqlite3_backup);
	84314	+ int (backup_step)(sqlite3_backup,int);
	84315	+ const char (compileoption_get)(int);
	84316	+ int (compileoption_used)(const char);
	84317	+ int (create_function_v2)(sqlite3,const char,int,int,void,void (xFunc)(sqlite3_context,int,sqlite3_value*),void (xStep)(sqlite3_context,int,sqlite3_value),void (xFinal)(sqlite3_context),void(xDestroy)(void*));
	84318	+ int (db_config)(sqlite3,int,...);
	84319	+ sqlite3_mutex (db_mutex)(sqlite3*);
	84320	+ int (db_status)(sqlite3,int,int,int,int);
	84321	+ int (extended_errcode)(sqlite3);
	84322	+ void (log)(int,const char,...);
	84323	+ sqlite3_int64 (*soft_heap_limit64)(sqlite3_int64);
	84324	+ const char (sourceid)(void);
	84325	+ int (stmt_status)(sqlite3_stmt,int,int);
	84326	+ int (strnicmp)(const char,const char*,int);
	84327	+ int (unlock_notify)(sqlite3,void()(void,int),void);
	84328	+ int (wal_autocheckpoint)(sqlite3,int);
	84329	+ int (wal_checkpoint)(sqlite3,const char*);
	84330	+ void (wal_hook)(sqlite3,int()(void,sqlite3,const char,int),void);
84061	84331	};
84062	84332
84063	84333	/*
84064	84334	** The following macros redefine the API routines so that they are
84065	84335	** redirected throught the global sqlite3_api structure.
		@@ -84235,10 +84505,31 @@
84235	84505	#define sqlite3_extended_result_codes sqlite3_api->extended_result_codes
84236	84506	#define sqlite3_limit sqlite3_api->limit
84237	84507	#define sqlite3_next_stmt sqlite3_api->next_stmt
84238	84508	#define sqlite3_sql sqlite3_api->sql
84239	84509	#define sqlite3_status sqlite3_api->status
	84510	+#define sqlite3_backup_finish sqlite3_api->backup_finish
	84511	+#define sqlite3_backup_init sqlite3_api->backup_init
	84512	+#define sqlite3_backup_pagecount sqlite3_api->backup_pagecount
	84513	+#define sqlite3_backup_remaining sqlite3_api->backup_remaining
	84514	+#define sqlite3_backup_step sqlite3_api->backup_step
	84515	+#define sqlite3_compileoption_get sqlite3_api->compileoption_get
	84516	+#define sqlite3_compileoption_used sqlite3_api->compileoption_used
	84517	+#define sqlite3_create_function_v2 sqlite3_api->create_function_v2
	84518	+#define sqlite3_db_config sqlite3_api->db_config
	84519	+#define sqlite3_db_mutex sqlite3_api->db_mutex
	84520	+#define sqlite3_db_status sqlite3_api->db_status
	84521	+#define sqlite3_extended_errcode sqlite3_api->extended_errcode
	84522	+#define sqlite3_log sqlite3_api->log
	84523	+#define sqlite3_soft_heap_limit64 sqlite3_api->soft_heap_limit64
	84524	+#define sqlite3_sourceid sqlite3_api->sourceid
	84525	+#define sqlite3_stmt_status sqlite3_api->stmt_status
	84526	+#define sqlite3_strnicmp sqlite3_api->strnicmp
	84527	+#define sqlite3_unlock_notify sqlite3_api->unlock_notify
	84528	+#define sqlite3_wal_autocheckpoint sqlite3_api->wal_autocheckpoint
	84529	+#define sqlite3_wal_checkpoint sqlite3_api->wal_checkpoint
	84530	+#define sqlite3_wal_hook sqlite3_api->wal_hook
84240	84531	#endif /* SQLITE_CORE */
84241	84532
84242	84533	#define SQLITE_EXTENSION_INIT1 const sqlite3_api_routines *sqlite3_api = 0;
84243	84534	#define SQLITE_EXTENSION_INIT2(v) sqlite3_api = v;
84244	84535
		@@ -84552,10 +84843,50 @@
84552	84843	sqlite3_extended_result_codes,
84553	84844	sqlite3_limit,
84554	84845	sqlite3_next_stmt,
84555	84846	sqlite3_sql,
84556	84847	sqlite3_status,
	84848	+
	84849	+ /*
	84850	+ ** Added for 3.7.4
	84851	+ */
	84852	+ sqlite3_backup_finish,
	84853	+ sqlite3_backup_init,
	84854	+ sqlite3_backup_pagecount,
	84855	+ sqlite3_backup_remaining,
	84856	+ sqlite3_backup_step,
	84857	+#ifndef SQLITE_OMIT_COMPILEOPTION_DIAGS
	84858	+ sqlite3_compileoption_get,
	84859	+ sqlite3_compileoption_used,
	84860	+#else
	84861	+ 0,
	84862	+ 0,
	84863	+#endif
	84864	+ sqlite3_create_function_v2,
	84865	+ sqlite3_db_config,
	84866	+ sqlite3_db_mutex,
	84867	+ sqlite3_db_status,
	84868	+ sqlite3_extended_errcode,
	84869	+ sqlite3_log,
	84870	+ sqlite3_soft_heap_limit64,
	84871	+ sqlite3_sourceid,
	84872	+ sqlite3_stmt_status,
	84873	+ sqlite3_strnicmp,
	84874	+#ifdef SQLITE_ENABLE_UNLOCK_NOTIFY
	84875	+ sqlite3_unlock_notify,
	84876	+#else
	84877	+ 0,
	84878	+#endif
	84879	+#ifndef SQLITE_OMIT_WAL
	84880	+ sqlite3_wal_autocheckpoint,
	84881	+ sqlite3_wal_checkpoint,
	84882	+ sqlite3_wal_hook,
	84883	+#else
	84884	+ 0,
	84885	+ 0,
	84886	+ 0,
	84887	+#endif
84557	84888	};
84558	84889
84559	84890	/*
84560	84891	** Attempt to load an SQLite extension library contained in the file
84561	84892	** zFile. The entry point is zProc. zProc may be 0 in which case a
		@@ -86976,17 +87307,17 @@
86976	87307
86977	87308	#ifndef SQLITE_OMIT_EXPLAIN
86978	87309	if( rc==SQLITE_OK && pParse->pVdbe && pParse->explain ){
86979	87310	static const char * const azColName[] = {
86980	87311	"addr", "opcode", "p1", "p2", "p3", "p4", "p5", "comment",
86981		- "order", "from", "detail"
	87312	+ "selectid", "order", "from", "detail"
86982	87313	};
86983	87314	int iFirst, mx;
86984	87315	if( pParse->explain==2 ){
86985		- sqlite3VdbeSetNumCols(pParse->pVdbe, 3);
	87316	+ sqlite3VdbeSetNumCols(pParse->pVdbe, 4);
86986	87317	iFirst = 8;
86987		- mx = 11;
	87318	+ mx = 12;
86988	87319	}else{
86989	87320	sqlite3VdbeSetNumCols(pParse->pVdbe, 8);
86990	87321	iFirst = 0;
86991	87322	mx = 8;
86992	87323	}
		@@ -87981,10 +88312,92 @@
87981	88312	}
87982	88313	}
87983	88314	return pInfo;
87984	88315	}
87985	88316
	88317	+#ifndef SQLITE_OMIT_COMPOUND_SELECT
	88318	+/*
	88319	+** Name of the connection operator, used for error messages.
	88320	+*/
	88321	+static const char *selectOpName(int id){
	88322	+ char *z;
	88323	+ switch( id ){
	88324	+ case TK_ALL: z = "UNION ALL"; break;
	88325	+ case TK_INTERSECT: z = "INTERSECT"; break;
	88326	+ case TK_EXCEPT: z = "EXCEPT"; break;
	88327	+ default: z = "UNION"; break;
	88328	+ }
	88329	+ return z;
	88330	+}
	88331	+#endif /* SQLITE_OMIT_COMPOUND_SELECT */
	88332	+
	88333	+#ifndef SQLITE_OMIT_EXPLAIN
	88334	+/*
	88335	+** Unless an "EXPLAIN QUERY PLAN" command is being processed, this function
	88336	+** is a no-op. Otherwise, it adds a single row of output to the EQP result,
	88337	+** where the caption is of the form:
	88338	+**
	88339	+** "USE TEMP B-TREE FOR xxx"
	88340	+**
	88341	+** where xxx is one of "DISTINCT", "ORDER BY" or "GROUP BY". Exactly which
	88342	+** is determined by the zUsage argument.
	88343	+*/
	88344	+static void explainTempTable(Parse pParse, const char zUsage){
	88345	+ if( pParse->explain==2 ){
	88346	+ Vdbe *v = pParse->pVdbe;
	88347	+ char *zMsg = sqlite3MPrintf(pParse->db, "USE TEMP B-TREE FOR %s", zUsage);
	88348	+ sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
	88349	+ }
	88350	+}
	88351	+
	88352	+/*
	88353	+** Unless an "EXPLAIN QUERY PLAN" command is being processed, this function
	88354	+** is a no-op. Otherwise, it adds a single row of output to the EQP result,
	88355	+** where the caption is of one of the two forms:
	88356	+**
	88357	+** "COMPOSITE SUBQUERIES iSub1 and iSub2 (op)"
	88358	+** "COMPOSITE SUBQUERIES iSub1 and iSub2 USING TEMP B-TREE (op)"
	88359	+**
	88360	+** where iSub1 and iSub2 are the integers passed as the corresponding
	88361	+** function parameters, and op is the text representation of the parameter
	88362	+** of the same name. The parameter "op" must be one of TK_UNION, TK_EXCEPT,
	88363	+** TK_INTERSECT or TK_ALL. The first form is used if argument bUseTmp is
	88364	+** false, or the second form if it is true.
	88365	+*/
	88366	+static void explainComposite(
	88367	+ Parse pParse, / Parse context */
	88368	+ int op, /* One of TK_UNION, TK_EXCEPT etc. */
	88369	+ int iSub1, /* Subquery id 1 */
	88370	+ int iSub2, /* Subquery id 2 */
	88371	+ int bUseTmp /* True if a temp table was used */
	88372	+){
	88373	+ assert( op==TK_UNION \|\| op==TK_EXCEPT \|\| op==TK_INTERSECT \|\| op==TK_ALL );
	88374	+ if( pParse->explain==2 ){
	88375	+ Vdbe *v = pParse->pVdbe;
	88376	+ char *zMsg = sqlite3MPrintf(
	88377	+ pParse->db, "COMPOUND SUBQUERIES %d AND %d %s(%s)", iSub1, iSub2,
	88378	+ bUseTmp?"USING TEMP B-TREE ":"", selectOpName(op)
	88379	+ );
	88380	+ sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
	88381	+ }
	88382	+}
	88383	+
	88384	+/*
	88385	+** Assign expression b to lvalue a. A second, no-op, version of this macro
	88386	+** is provided when SQLITE_OMIT_EXPLAIN is defined. This allows the code
	88387	+** in sqlite3Select() to assign values to structure member variables that
	88388	+** only exist if SQLITE_OMIT_EXPLAIN is not defined without polluting the
	88389	+** code with #ifndef directives.
	88390	+*/
	88391	+# define explainSetInteger(a, b) a = b
	88392	+
	88393	+#else
	88394	+/* No-op versions of the explainXXX() functions and macros. */
	88395	+# define explainTempTable(y,z)
	88396	+# define explainComposite(v,w,x,y,z)
	88397	+# define explainSetInteger(y,z)
	88398	+#endif
87986	88399
87987	88400	/*
87988	88401	** If the inner loop was generated using a non-null pOrderBy argument,
87989	88402	** then the results were placed in a sorter. After the loop is terminated
87990	88403	** we need to run the sorter and output the results. The following
		@@ -88328,26 +88741,10 @@
88328	88741	}
88329	88742	}
88330	88743	generateColumnTypes(pParse, pTabList, pEList);
88331	88744	}
88332	88745
88333		-#ifndef SQLITE_OMIT_COMPOUND_SELECT
88334		-/*
88335		-** Name of the connection operator, used for error messages.
88336		-*/
88337		-static const char *selectOpName(int id){
88338		- char *z;
88339		- switch( id ){
88340		- case TK_ALL: z = "UNION ALL"; break;
88341		- case TK_INTERSECT: z = "INTERSECT"; break;
88342		- case TK_EXCEPT: z = "EXCEPT"; break;
88343		- default: z = "UNION"; break;
88344		- }
88345		- return z;
88346		-}
88347		-#endif /* SQLITE_OMIT_COMPOUND_SELECT */
88348		-
88349	88746	/*
88350	88747	** Given a an expression list (which is really the list of expressions
88351	88748	** that form the result set of a SELECT statement) compute appropriate
88352	88749	** column names for a table that would hold the expression list.
88353	88750	**
		@@ -88506,10 +88903,11 @@
88506	88903	/* The sqlite3ResultSetOfSelect() is only used n contexts where lookaside
88507	88904	** is disabled */
88508	88905	assert( db->lookaside.bEnabled==0 );
88509	88906	pTab->nRef = 1;
88510	88907	pTab->zName = 0;
	88908	+ pTab->nRowEst = 1000000;
88511	88909	selectColumnsFromExprList(pParse, pSelect->pEList, &pTab->nCol, &pTab->aCol);
88512	88910	selectAddColumnTypeAndCollation(pParse, pTab->nCol, pTab->aCol, pSelect);
88513	88911	pTab->iPKey = -1;
88514	88912	if( db->mallocFailed ){
88515	88913	sqlite3DeleteTable(db, pTab);
		@@ -88676,10 +89074,14 @@
88676	89074	Select pPrior; / Another SELECT immediately to our left */
88677	89075	Vdbe v; / Generate code to this VDBE */
88678	89076	SelectDest dest; /* Alternative data destination */
88679	89077	Select pDelete = 0; / Chain of simple selects to delete */
88680	89078	sqlite3 db; / Database connection */
	89079	+#ifndef SQLITE_OMIT_EXPLAIN
	89080	+ int iSub1; /* EQP id of left-hand query */
	89081	+ int iSub2; /* EQP id of right-hand query */
	89082	+#endif
88681	89083
88682	89084	/* Make sure there is no ORDER BY or LIMIT clause on prior SELECTs. Only
88683	89085	** the last (right-most) SELECT in the series may have an ORDER BY or LIMIT.
88684	89086	*/
88685	89087	assert( p && p->pPrior ); /* Calling function guarantees this much */
		@@ -88736,10 +89138,11 @@
88736	89138	case TK_ALL: {
88737	89139	int addr = 0;
88738	89140	assert( !pPrior->pLimit );
88739	89141	pPrior->pLimit = p->pLimit;
88740	89142	pPrior->pOffset = p->pOffset;
	89143	+ explainSetInteger(iSub1, pParse->iNextSelectId);
88741	89144	rc = sqlite3Select(pParse, pPrior, &dest);
88742	89145	p->pLimit = 0;
88743	89146	p->pOffset = 0;
88744	89147	if( rc ){
88745	89148	goto multi_select_end;
		@@ -88749,10 +89152,11 @@
88749	89152	p->iOffset = pPrior->iOffset;
88750	89153	if( p->iLimit ){
88751	89154	addr = sqlite3VdbeAddOp1(v, OP_IfZero, p->iLimit);
88752	89155	VdbeComment((v, "Jump ahead if LIMIT reached"));
88753	89156	}
	89157	+ explainSetInteger(iSub2, pParse->iNextSelectId);
88754	89158	rc = sqlite3Select(pParse, p, &dest);
88755	89159	testcase( rc!=SQLITE_OK );
88756	89160	pDelete = p->pPrior;
88757	89161	p->pPrior = pPrior;
88758	89162	if( addr ){
		@@ -88796,10 +89200,11 @@
88796	89200
88797	89201	/* Code the SELECT statements to our left
88798	89202	*/
88799	89203	assert( !pPrior->pOrderBy );
88800	89204	sqlite3SelectDestInit(&uniondest, priorOp, unionTab);
	89205	+ explainSetInteger(iSub1, pParse->iNextSelectId);
88801	89206	rc = sqlite3Select(pParse, pPrior, &uniondest);
88802	89207	if( rc ){
88803	89208	goto multi_select_end;
88804	89209	}
88805	89210
		@@ -88815,10 +89220,11 @@
88815	89220	pLimit = p->pLimit;
88816	89221	p->pLimit = 0;
88817	89222	pOffset = p->pOffset;
88818	89223	p->pOffset = 0;
88819	89224	uniondest.eDest = op;
	89225	+ explainSetInteger(iSub2, pParse->iNextSelectId);
88820	89226	rc = sqlite3Select(pParse, p, &uniondest);
88821	89227	testcase( rc!=SQLITE_OK );
88822	89228	/* Query flattening in sqlite3Select() might refill p->pOrderBy.
88823	89229	** Be sure to delete p->pOrderBy, therefore, to avoid a memory leak. */
88824	89230	sqlite3ExprListDelete(db, p->pOrderBy);
		@@ -88880,10 +89286,11 @@
88880	89286	assert( p->pEList );
88881	89287
88882	89288	/* Code the SELECTs to our left into temporary table "tab1".
88883	89289	*/
88884	89290	sqlite3SelectDestInit(&intersectdest, SRT_Union, tab1);
	89291	+ explainSetInteger(iSub1, pParse->iNextSelectId);
88885	89292	rc = sqlite3Select(pParse, pPrior, &intersectdest);
88886	89293	if( rc ){
88887	89294	goto multi_select_end;
88888	89295	}
88889	89296
		@@ -88896,10 +89303,11 @@
88896	89303	pLimit = p->pLimit;
88897	89304	p->pLimit = 0;
88898	89305	pOffset = p->pOffset;
88899	89306	p->pOffset = 0;
88900	89307	intersectdest.iParm = tab2;
	89308	+ explainSetInteger(iSub2, pParse->iNextSelectId);
88901	89309	rc = sqlite3Select(pParse, p, &intersectdest);
88902	89310	testcase( rc!=SQLITE_OK );
88903	89311	pDelete = p->pPrior;
88904	89312	p->pPrior = pPrior;
88905	89313	sqlite3ExprDelete(db, p->pLimit);
		@@ -88931,10 +89339,12 @@
88931	89339	sqlite3VdbeAddOp2(v, OP_Close, tab2, 0);
88932	89340	sqlite3VdbeAddOp2(v, OP_Close, tab1, 0);
88933	89341	break;
88934	89342	}
88935	89343	}
	89344	+
	89345	+ explainComposite(pParse, p->op, iSub1, iSub2, p->op!=TK_ALL);
88936	89346
88937	89347	/* Compute collating sequences used by
88938	89348	** temporary tables needed to implement the compound select.
88939	89349	** Attach the KeyInfo structure to all temporary tables.
88940	89350	**
		@@ -89275,10 +89685,14 @@
89275	89685	KeyInfo pKeyMerge; / Comparison information for merging rows */
89276	89686	sqlite3 db; / Database connection */
89277	89687	ExprList pOrderBy; / The ORDER BY clause */
89278	89688	int nOrderBy; /* Number of terms in the ORDER BY clause */
89279	89689	int aPermute; / Mapping from ORDER BY terms to result set columns */
	89690	+#ifndef SQLITE_OMIT_EXPLAIN
	89691	+ int iSub1; /* EQP id of left-hand query */
	89692	+ int iSub2; /* EQP id of right-hand query */
	89693	+#endif
89280	89694
89281	89695	assert( p->pOrderBy!=0 );
89282	89696	assert( pKeyDup==0 ); /* "Managed" code needs this. Ticket #3382. */
89283	89697	db = pParse->db;
89284	89698	v = pParse->pVdbe;
		@@ -89428,10 +89842,11 @@
89428	89842	/* Generate a coroutine to evaluate the SELECT statement to the
89429	89843	** left of the compound operator - the "A" select.
89430	89844	*/
89431	89845	VdbeNoopComment((v, "Begin coroutine for left SELECT"));
89432	89846	pPrior->iLimit = regLimitA;
	89847	+ explainSetInteger(iSub1, pParse->iNextSelectId);
89433	89848	sqlite3Select(pParse, pPrior, &destA);
89434	89849	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofA);
89435	89850	sqlite3VdbeAddOp1(v, OP_Yield, regAddrA);
89436	89851	VdbeNoopComment((v, "End coroutine for left SELECT"));
89437	89852
		@@ -89442,10 +89857,11 @@
89442	89857	VdbeNoopComment((v, "Begin coroutine for right SELECT"));
89443	89858	savedLimit = p->iLimit;
89444	89859	savedOffset = p->iOffset;
89445	89860	p->iLimit = regLimitB;
89446	89861	p->iOffset = 0;
	89862	+ explainSetInteger(iSub2, pParse->iNextSelectId);
89447	89863	sqlite3Select(pParse, p, &destB);
89448	89864	p->iLimit = savedLimit;
89449	89865	p->iOffset = savedOffset;
89450	89866	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofB);
89451	89867	sqlite3VdbeAddOp1(v, OP_Yield, regAddrB);
		@@ -89572,10 +89988,11 @@
89572	89988	}
89573	89989	p->pPrior = pPrior;
89574	89990
89575	89991	/*** TBD: Insert subroutine calls to close cursors on incomplete
89576	89992	** subqueries **/
	89993	+ explainComposite(pParse, p->op, iSub1, iSub2, 0);
89577	89994	return SQLITE_OK;
89578	89995	}
89579	89996	#endif
89580	89997
89581	89998	#if !defined(SQLITE_OMIT_SUBQUERY) \|\| !defined(SQLITE_OMIT_VIEW)
		@@ -90305,10 +90722,11 @@
90305	90722	pTab->nRef = 1;
90306	90723	pTab->zName = sqlite3MPrintf(db, "sqlite_subquery_%p_", (void*)pTab);
90307	90724	while( pSel->pPrior ){ pSel = pSel->pPrior; }
90308	90725	selectColumnsFromExprList(pParse, pSel->pEList, &pTab->nCol, &pTab->aCol);
90309	90726	pTab->iPKey = -1;
	90727	+ pTab->nRowEst = 1000000;
90310	90728	pTab->tabFlags \|= TF_Ephemeral;
90311	90729	#endif
90312	90730	}else{
90313	90731	/* An ordinary table or view name in the FROM clause */
90314	90732	assert( pFrom->pTab==0 );
		@@ -90797,10 +91215,15 @@
90797	91215	int rc = 1; /* Value to return from this function */
90798	91216	int addrSortIndex; /* Address of an OP_OpenEphemeral instruction */
90799	91217	AggInfo sAggInfo; /* Information used by aggregate queries */
90800	91218	int iEnd; /* Address of the end of the query */
90801	91219	sqlite3 db; / The database connection */
	91220	+
	91221	+#ifndef SQLITE_OMIT_EXPLAIN
	91222	+ int iRestoreSelectId = pParse->iSelectId;
	91223	+ pParse->iSelectId = pParse->iNextSelectId++;
	91224	+#endif
90802	91225
90803	91226	db = pParse->db;
90804	91227	if( p==0 \|\| db->mallocFailed \|\| pParse->nErr ){
90805	91228	return 1;
90806	91229	}
		@@ -90869,10 +91292,11 @@
90869	91292	}
90870	91293	i = -1;
90871	91294	}else{
90872	91295	sqlite3SelectDestInit(&dest, SRT_EphemTab, pItem->iCursor);
90873	91296	assert( pItem->isPopulated==0 );
	91297	+ explainSetInteger(pItem->iSelectId, pParse->iNextSelectId);
90874	91298	sqlite3Select(pParse, pSub, &dest);
90875	91299	pItem->isPopulated = 1;
90876	91300	}
90877	91301	if( /pParse->nErr \|\|/ db->mallocFailed ){
90878	91302	goto select_end;
		@@ -90904,14 +91328,16 @@
90904	91328	pRight = pLoop;
90905	91329	}
90906	91330	mxSelect = db->aLimit[SQLITE_LIMIT_COMPOUND_SELECT];
90907	91331	if( mxSelect && cnt>mxSelect ){
90908	91332	sqlite3ErrorMsg(pParse, "too many terms in compound SELECT");
90909		- return 1;
	91333	+ goto select_end;
90910	91334	}
90911	91335	}
90912		- return multiSelect(pParse, p, pDest);
	91336	+ rc = multiSelect(pParse, p, pDest);
	91337	+ explainSetInteger(pParse->iSelectId, iRestoreSelectId);
	91338	+ return rc;
90913	91339	}
90914	91340	#endif
90915	91341
90916	91342	/* If possible, rewrite the query to use GROUP BY instead of DISTINCT.
90917	91343	** GROUP BY might use an index, DISTINCT never does.
		@@ -90919,11 +91345,10 @@
90919	91345	assert( p->pGroupBy==0 \|\| (p->selFlags & SF_Aggregate)!=0 );
90920	91346	if( (p->selFlags & (SF_Distinct\|SF_Aggregate))==SF_Distinct ){
90921	91347	p->pGroupBy = sqlite3ExprListDup(db, p->pEList, 0);
90922	91348	pGroupBy = p->pGroupBy;
90923	91349	p->selFlags &= ~SF_Distinct;
90924		- isDistinct = 0;
90925	91350	}
90926	91351
90927	91352	/* If there is both a GROUP BY and an ORDER BY clause and they are
90928	91353	** identical, then disable the ORDER BY clause since the GROUP BY
90929	91354	** will cause elements to come out in the correct order. This is
		@@ -90966,11 +91391,11 @@
90966	91391	iEnd = sqlite3VdbeMakeLabel(v);
90967	91392	computeLimitRegisters(pParse, p, iEnd);
90968	91393
90969	91394	/* Open a virtual index to use for the distinct set.
90970	91395	*/
90971		- if( isDistinct ){
	91396	+ if( p->selFlags & SF_Distinct ){
90972	91397	KeyInfo *pKeyInfo;
90973	91398	assert( isAgg \|\| pGroupBy );
90974	91399	distinct = pParse->nTab++;
90975	91400	pKeyInfo = keyInfoFromExprList(pParse, p->pEList);
90976	91401	sqlite3VdbeAddOp4(v, OP_OpenEphemeral, distinct, 0, 0,
		@@ -91125,10 +91550,13 @@
91125	91550	int regBase;
91126	91551	int regRecord;
91127	91552	int nCol;
91128	91553	int nGroupBy;
91129	91554
	91555	+ explainTempTable(pParse,
	91556	+ isDistinct && !(p->selFlags&SF_Distinct)?"DISTINCT":"GROUP BY");
	91557	+
91130	91558	groupBySort = 1;
91131	91559	nGroupBy = pGroupBy->nExpr;
91132	91560	nCol = nGroupBy + 1;
91133	91561	j = nGroupBy+1;
91134	91562	for(i=0; i<sAggInfo.nColumn; i++){
		@@ -91385,15 +91813,20 @@
91385	91813	sqlite3ExprListDelete(db, pDel);
91386	91814	}
91387	91815	sqlite3VdbeResolveLabel(v, addrEnd);
91388	91816
91389	91817	} /* endif aggregate query */
	91818	+
	91819	+ if( distinct>=0 ){
	91820	+ explainTempTable(pParse, "DISTINCT");
	91821	+ }
91390	91822
91391	91823	/* If there is an ORDER BY clause, then we need to sort the results
91392	91824	** and send them to the callback one by one.
91393	91825	*/
91394	91826	if( pOrderBy ){
	91827	+ explainTempTable(pParse, "ORDER BY");
91395	91828	generateSortTail(pParse, p, v, pEList->nExpr, pDest);
91396	91829	}
91397	91830
91398	91831	/* Jump here to skip this query
91399	91832	*/
		@@ -91406,10 +91839,11 @@
91406	91839
91407	91840	/* Control jumps to here if an error is encountered above, or upon
91408	91841	** successful coding of the SELECT.
91409	91842	*/
91410	91843	select_end:
	91844	+ explainSetInteger(pParse->iSelectId, iRestoreSelectId);
91411	91845
91412	91846	/* Identify column names if results of the SELECT are to be output.
91413	91847	*/
91414	91848	if( rc==SQLITE_OK && pDest->eDest==SRT_Output ){
91415	91849	generateColumnNames(pParse, pTabList, pEList);
		@@ -94495,11 +94929,11 @@
94495	94929	pParse->pNewTable->nCol = 0;
94496	94930	pParse->pNewTable->aCol = 0;
94497	94931	}
94498	94932	db->pVTab = 0;
94499	94933	}else{
94500		- sqlite3Error(db, SQLITE_ERROR, zErr);
	94934	+ sqlite3Error(db, SQLITE_ERROR, (zErr ? "%s" : 0), zErr);
94501	94935	sqlite3DbFree(db, zErr);
94502	94936	rc = SQLITE_ERROR;
94503	94937	}
94504	94938	pParse->declareVtab = 0;
94505	94939
		@@ -94957,11 +95391,10 @@
94957	95391	** cost of pursuing that strategy.
94958	95392	*/
94959	95393	struct WhereCost {
94960	95394	WherePlan plan; /* The lookup strategy */
94961	95395	double rCost; /* Overall cost of pursuing this search strategy */
94962		- double nRow; /* Estimated number of output rows */
94963	95396	Bitmask used; /* Bitmask of cursors used by this plan */
94964	95397	};
94965	95398
94966	95399	/*
94967	95400	** Bitmasks for the operators that indices are able to exploit. An
		@@ -95000,11 +95433,11 @@
95000	95433	#define WHERE_COLUMN_EQ 0x00010000 /* x=EXPR or x IN (...) or x IS NULL */
95001	95434	#define WHERE_COLUMN_RANGE 0x00020000 /* x<EXPR and/or x>EXPR */
95002	95435	#define WHERE_COLUMN_IN 0x00040000 /* x IN (...) */
95003	95436	#define WHERE_COLUMN_NULL 0x00080000 /* x IS NULL */
95004	95437	#define WHERE_INDEXED 0x000f0000 /* Anything that uses an index */
95005		-#define WHERE_NOT_FULLSCAN 0x000f3000 /* Does not do a full table scan */
	95438	+#define WHERE_NOT_FULLSCAN 0x100f3000 /* Does not do a full table scan */
95006	95439	#define WHERE_IN_ABLE 0x000f1000 /* Able to support an IN operator */
95007	95440	#define WHERE_TOP_LIMIT 0x00100000 /* x<EXPR or x<=EXPR constraint */
95008	95441	#define WHERE_BTM_LIMIT 0x00200000 /* x>EXPR or x>=EXPR constraint */
95009	95442	#define WHERE_IDX_ONLY 0x00800000 /* Use index only - omit table */
95010	95443	#define WHERE_ORDERBY 0x01000000 /* Output will appear in correct order */
		@@ -96346,12 +96779,13 @@
96346	96779	const int iCur = pSrc->iCursor; /* The cursor of the table to be accessed */
96347	96780	const Bitmask maskSrc = getMask(pWC->pMaskSet, iCur); /* Bitmask for pSrc */
96348	96781	WhereTerm * const pWCEnd = &pWC->a[pWC->nTerm]; /* End of pWC->a[] */
96349	96782	WhereTerm pTerm; / A single term of the WHERE clause */
96350	96783
96351		- /* No OR-clause optimization allowed if the NOT INDEXED clause is used */
96352		- if( pSrc->notIndexed ){
	96784	+ /* No OR-clause optimization allowed if the INDEXED BY or NOT INDEXED clauses
	96785	+ ** are used */
	96786	+ if( pSrc->notIndexed \|\| pSrc->pIndex!=0 ){
96353	96787	return;
96354	96788	}
96355	96789
96356	96790	/* Search the WHERE clause terms for a usable WO_OR term. */
96357	96791	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
		@@ -96385,11 +96819,11 @@
96385	96819	bestIndex(pParse, &tempWC, pSrc, notReady, notValid, 0, &sTermCost);
96386	96820	}else{
96387	96821	continue;
96388	96822	}
96389	96823	rTotal += sTermCost.rCost;
96390		- nRow += sTermCost.nRow;
	96824	+ nRow += sTermCost.plan.nRow;
96391	96825	used \|= sTermCost.used;
96392	96826	if( rTotal>=pCost->rCost ) break;
96393	96827	}
96394	96828
96395	96829	/* If there is an ORDER BY clause, increase the scan cost to account
		@@ -96404,12 +96838,12 @@
96404	96838	** less than the current cost stored in pCost, replace the contents
96405	96839	** of pCost. */
96406	96840	WHERETRACE(("... multi-index OR cost=%.9g nrow=%.9g\n", rTotal, nRow));
96407	96841	if( rTotal<pCost->rCost ){
96408	96842	pCost->rCost = rTotal;
96409		- pCost->nRow = nRow;
96410	96843	pCost->used = used;
	96844	+ pCost->plan.nRow = nRow;
96411	96845	pCost->plan.wsFlags = flags;
96412	96846	pCost->plan.u.pTerm = pTerm;
96413	96847	}
96414	96848	}
96415	96849	}
		@@ -96489,11 +96923,11 @@
96489	96923	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
96490	96924	if( termCanDriveIndex(pTerm, pSrc, notReady) ){
96491	96925	WHERETRACE(("auto-index reduces cost from %.2f to %.2f\n",
96492	96926	pCost->rCost, costTempIdx));
96493	96927	pCost->rCost = costTempIdx;
96494		- pCost->nRow = logN + 1;
	96928	+ pCost->plan.nRow = logN + 1;
96495	96929	pCost->plan.wsFlags = WHERE_TEMP_INDEX;
96496	96930	pCost->used = pTerm->prereqRight;
96497	96931	break;
96498	96932	}
96499	96933	}
		@@ -97562,15 +97996,15 @@
97562	97996
97563	97997	/* If this index is the best we have seen so far, then record this
97564	97998	** index and its cost in the pCost structure.
97565	97999	*/
97566	98000	if( (!pIdx \|\| wsFlags)
97567		- && (cost<pCost->rCost \|\| (cost<=pCost->rCost && nRow<pCost->nRow))
	98001	+ && (cost<pCost->rCost \|\| (cost<=pCost->rCost && nRow<pCost->plan.nRow))
97568	98002	){
97569	98003	pCost->rCost = cost;
97570		- pCost->nRow = nRow;
97571	98004	pCost->used = used;
	98005	+ pCost->plan.nRow = nRow;
97572	98006	pCost->plan.wsFlags = (wsFlags&wsFlagMask);
97573	98007	pCost->plan.nEq = nEq;
97574	98008	pCost->plan.u.pIdx = pIdx;
97575	98009	}
97576	98010
		@@ -97894,10 +98328,162 @@
97894	98328	}
97895	98329	}
97896	98330	*pzAff = zAff;
97897	98331	return regBase;
97898	98332	}
	98333	+
	98334	+#ifndef SQLITE_OMIT_EXPLAIN
	98335	+/*
	98336	+** This routine is a helper for explainIndexRange() below
	98337	+**
	98338	+** pStr holds the text of an expression that we are building up one term
	98339	+** at a time. This routine adds a new term to the end of the expression.
	98340	+** Terms are separated by AND so add the "AND" text for second and subsequent
	98341	+** terms only.
	98342	+*/
	98343	+static void explainAppendTerm(
	98344	+ StrAccum pStr, / The text expression being built */
	98345	+ int iTerm, /* Index of this term. First is zero */
	98346	+ const char zColumn, / Name of the column */
	98347	+ const char zOp / Name of the operator */
	98348	+){
	98349	+ if( iTerm ) sqlite3StrAccumAppend(pStr, " AND ", 5);
	98350	+ sqlite3StrAccumAppend(pStr, zColumn, -1);
	98351	+ sqlite3StrAccumAppend(pStr, zOp, 1);
	98352	+ sqlite3StrAccumAppend(pStr, "?", 1);
	98353	+}
	98354	+
	98355	+/*
	98356	+** Argument pLevel describes a strategy for scanning table pTab. This
	98357	+** function returns a pointer to a string buffer containing a description
	98358	+** of the subset of table rows scanned by the strategy in the form of an
	98359	+** SQL expression. Or, if all rows are scanned, NULL is returned.
	98360	+**
	98361	+** For example, if the query:
	98362	+**
	98363	+** SELECT * FROM t1 WHERE a=1 AND b>2;
	98364	+**
	98365	+** is run and there is an index on (a, b), then this function returns a
	98366	+** string similar to:
	98367	+**
	98368	+** "a=? AND b>?"
	98369	+**
	98370	+** The returned pointer points to memory obtained from sqlite3DbMalloc().
	98371	+** It is the responsibility of the caller to free the buffer when it is
	98372	+** no longer required.
	98373	+*/
	98374	+static char explainIndexRange(sqlite3 db, WhereLevel pLevel, Table pTab){
	98375	+ WherePlan *pPlan = &pLevel->plan;
	98376	+ Index *pIndex = pPlan->u.pIdx;
	98377	+ int nEq = pPlan->nEq;
	98378	+ int i, j;
	98379	+ Column *aCol = pTab->aCol;
	98380	+ int *aiColumn = pIndex->aiColumn;
	98381	+ StrAccum txt;
	98382	+
	98383	+ if( nEq==0 && (pPlan->wsFlags & (WHERE_BTM_LIMIT\|WHERE_TOP_LIMIT))==0 ){
	98384	+ return 0;
	98385	+ }
	98386	+ sqlite3StrAccumInit(&txt, 0, 0, SQLITE_MAX_LENGTH);
	98387	+ txt.db = db;
	98388	+ sqlite3StrAccumAppend(&txt, " (", 2);
	98389	+ for(i=0; i<nEq; i++){
	98390	+ explainAppendTerm(&txt, i, aCol[aiColumn[i]].zName, "=");
	98391	+ }
	98392	+
	98393	+ j = i;
	98394	+ if( pPlan->wsFlags&WHERE_BTM_LIMIT ){
	98395	+ explainAppendTerm(&txt, i++, aCol[aiColumn[j]].zName, ">");
	98396	+ }
	98397	+ if( pPlan->wsFlags&WHERE_TOP_LIMIT ){
	98398	+ explainAppendTerm(&txt, i, aCol[aiColumn[j]].zName, "<");
	98399	+ }
	98400	+ sqlite3StrAccumAppend(&txt, ")", 1);
	98401	+ return sqlite3StrAccumFinish(&txt);
	98402	+}
	98403	+
	98404	+/*
	98405	+** This function is a no-op unless currently processing an EXPLAIN QUERY PLAN
	98406	+** command. If the query being compiled is an EXPLAIN QUERY PLAN, a single
	98407	+** record is added to the output to describe the table scan strategy in
	98408	+** pLevel.
	98409	+*/
	98410	+static void explainOneScan(
	98411	+ Parse pParse, / Parse context */
	98412	+ SrcList pTabList, / Table list this loop refers to */
	98413	+ WhereLevel pLevel, / Scan to write OP_Explain opcode for */
	98414	+ int iLevel, /* Value for "level" column of output */
	98415	+ int iFrom, /* Value for "from" column of output */
	98416	+ u16 wctrlFlags /* Flags passed to sqlite3WhereBegin() */
	98417	+){
	98418	+ if( pParse->explain==2 ){
	98419	+ u32 flags = pLevel->plan.wsFlags;
	98420	+ struct SrcList_item *pItem = &pTabList->a[pLevel->iFrom];
	98421	+ Vdbe v = pParse->pVdbe; / VM being constructed */
	98422	+ sqlite3 db = pParse->db; / Database handle */
	98423	+ char zMsg; / Text to add to EQP output */
	98424	+ sqlite3_int64 nRow; /* Expected number of rows visited by scan */
	98425	+ int iId = pParse->iSelectId; /* Select id (left-most output column) */
	98426	+ int isSearch; /* True for a SEARCH. False for SCAN. */
	98427	+
	98428	+ if( (flags&WHERE_MULTI_OR) \|\| (wctrlFlags&WHERE_ONETABLE_ONLY) ) return;
	98429	+
	98430	+ isSearch = (pLevel->plan.nEq>0 \|\| flags&(WHERE_BTM_LIMIT\|WHERE_TOP_LIMIT));
	98431	+
	98432	+ zMsg = sqlite3MPrintf(db, "%s", isSearch?"SEARCH":"SCAN");
	98433	+ if( pItem->pSelect ){
	98434	+ zMsg = sqlite3MAppendf(db, zMsg, "%s SUBQUERY %d", zMsg,pItem->iSelectId);
	98435	+ }else{
	98436	+ zMsg = sqlite3MAppendf(db, zMsg, "%s TABLE %s", zMsg, pItem->zName);
	98437	+ }
	98438	+
	98439	+ if( pItem->zAlias ){
	98440	+ zMsg = sqlite3MAppendf(db, zMsg, "%s AS %s", zMsg, pItem->zAlias);
	98441	+ }
	98442	+ if( (flags & WHERE_INDEXED)!=0 ){
	98443	+ char *zWhere = explainIndexRange(db, pLevel, pItem->pTab);
	98444	+ zMsg = sqlite3MAppendf(db, zMsg, "%s USING %s%sINDEX%s%s%s", zMsg,
	98445	+ ((flags & WHERE_TEMP_INDEX)?"AUTOMATIC ":""),
	98446	+ ((flags & WHERE_IDX_ONLY)?"COVERING ":""),
	98447	+ ((flags & WHERE_TEMP_INDEX)?"":" "),
	98448	+ ((flags & WHERE_TEMP_INDEX)?"": pLevel->plan.u.pIdx->zName),
	98449	+ zWhere
	98450	+ );
	98451	+ sqlite3DbFree(db, zWhere);
	98452	+ }else if( flags & (WHERE_ROWID_EQ\|WHERE_ROWID_RANGE) ){
	98453	+ zMsg = sqlite3MAppendf(db, zMsg, "%s USING INTEGER PRIMARY KEY", zMsg);
	98454	+
	98455	+ if( flags&WHERE_ROWID_EQ ){
	98456	+ zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid=?)", zMsg);
	98457	+ }else if( flags&WHERE_BTM_LIMIT && flags&WHERE_TOP_LIMIT ){
	98458	+ zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid>? AND rowid<?)", zMsg);
	98459	+ }else if( flags&WHERE_BTM_LIMIT ){
	98460	+ zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid>?)", zMsg);
	98461	+ }else if( flags&WHERE_TOP_LIMIT ){
	98462	+ zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid<?)", zMsg);
	98463	+ }
	98464	+ }
	98465	+#ifndef SQLITE_OMIT_VIRTUALTABLE
	98466	+ else if( (flags & WHERE_VIRTUALTABLE)!=0 ){
	98467	+ sqlite3_index_info *pVtabIdx = pLevel->plan.u.pVtabIdx;
	98468	+ zMsg = sqlite3MAppendf(db, zMsg, "%s VIRTUAL TABLE INDEX %d:%s", zMsg,
	98469	+ pVtabIdx->idxNum, pVtabIdx->idxStr);
	98470	+ }
	98471	+#endif
	98472	+ if( wctrlFlags&(WHERE_ORDERBY_MIN\|WHERE_ORDERBY_MAX) ){
	98473	+ nRow = 1;
	98474	+ }else{
	98475	+ nRow = (sqlite3_int64)pLevel->plan.nRow;
	98476	+ }
	98477	+ zMsg = sqlite3MAppendf(db, zMsg, "%s (~%lld rows)", zMsg, nRow);
	98478	+ sqlite3VdbeAddOp4(v, OP_Explain, iId, iLevel, iFrom, zMsg, P4_DYNAMIC);
	98479	+ }
	98480	+}
	98481	+#else
	98482	+# define explainOneScan(u,v,w,x,y,z)
	98483	+#endif /* SQLITE_OMIT_EXPLAIN */
	98484	+
97899	98485
97900	98486	/*
97901	98487	** Generate code for the start of the iLevel-th loop in the WHERE clause
97902	98488	** implementation described by pWInfo.
97903	98489	*/
		@@ -98436,10 +99022,13 @@
98436	99022	/* Loop through table entries that match term pOrTerm. */
98437	99023	pSubWInfo = sqlite3WhereBegin(pParse, pOrTab, pOrTerm->pExpr, 0,
98438	99024	WHERE_OMIT_OPEN \| WHERE_OMIT_CLOSE \|
98439	99025	WHERE_FORCE_TABLE \| WHERE_ONETABLE_ONLY);
98440	99026	if( pSubWInfo ){
	99027	+ explainOneScan(
	99028	+ pParse, pOrTab, &pSubWInfo->a[0], iLevel, pLevel->iFrom, 0
	99029	+ );
98441	99030	if( (wctrlFlags & WHERE_DUPLICATES_OK)==0 ){
98442	99031	int iSet = ((ii==pOrWc->nTerm-1)?-1:ii);
98443	99032	int r;
98444	99033	r = sqlite3ExprCodeGetColumn(pParse, pTabItem->pTab, -1, iCur,
98445	99034	regRowid);
		@@ -98831,10 +99420,11 @@
98831	99420	int nUnconstrained; /* Number tables without INDEXED BY */
98832	99421	Bitmask notIndexed; /* Mask of tables that cannot use an index */
98833	99422
98834	99423	memset(&bestPlan, 0, sizeof(bestPlan));
98835	99424	bestPlan.rCost = SQLITE_BIG_DBL;
	99425	+ WHERETRACE(("* Begin search for loop %d *\n", i));
98836	99426
98837	99427	/* Loop through the remaining entries in the FROM clause to find the
98838	99428	** next nested loop. The loop tests all FROM clause entries
98839	99429	** either once or twice.
98840	99430	**
		@@ -98895,10 +99485,12 @@
98895	99485	}
98896	99486	mask = (isOptimal ? m : notReady);
98897	99487	pOrderBy = ((i==0 && ppOrderBy )?*ppOrderBy:0);
98898	99488	if( pTabItem->pIndex==0 ) nUnconstrained++;
98899	99489
	99490	+ WHERETRACE(("=== trying table %d with isOptimal=%d ===\n",
	99491	+ j, isOptimal));
98900	99492	assert( pTabItem->pTab );
98901	99493	#ifndef SQLITE_OMIT_VIRTUALTABLE
98902	99494	if( IsVirtual(pTabItem->pTab) ){
98903	99495	sqlite3_index_info **pp = &pWInfo->a[j].pIdxInfo;
98904	99496	bestVirtualIndex(pParse, pWC, pTabItem, mask, notReady, pOrderBy,
		@@ -98945,24 +99537,27 @@
98945	99537	&& (bestJ<0 \|\| (notIndexed&m)!=0 /* (2) */
98946	99538	\|\| (sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0)
98947	99539	&& (nUnconstrained==0 \|\| pTabItem->pIndex==0 /* (3) */
98948	99540	\|\| NEVER((sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0))
98949	99541	&& (bestJ<0 \|\| sCost.rCost<bestPlan.rCost /* (4) */
98950		- \|\| (sCost.rCost<=bestPlan.rCost && sCost.nRow<bestPlan.nRow))
	99542	+ \|\| (sCost.rCost<=bestPlan.rCost
	99543	+ && sCost.plan.nRow<bestPlan.plan.nRow))
98951	99544	){
98952		- WHERETRACE(("... best so far with cost=%g and nRow=%g\n",
98953		- sCost.rCost, sCost.nRow));
	99545	+ WHERETRACE(("=== table %d is best so far"
	99546	+ " with cost=%g and nRow=%g\n",
	99547	+ j, sCost.rCost, sCost.plan.nRow));
98954	99548	bestPlan = sCost;
98955	99549	bestJ = j;
98956	99550	}
98957	99551	if( doNotReorder ) break;
98958	99552	}
98959	99553	}
98960	99554	assert( bestJ>=0 );
98961	99555	assert( notReady & getMask(pMaskSet, pTabList->a[bestJ].iCursor) );
98962		- WHERETRACE(("*** Optimizer selects table %d for loop %d\n", bestJ,
98963		- pLevel-pWInfo->a));
	99556	+ WHERETRACE(("*** Optimizer selects table %d for loop %d"
	99557	+ " with cost=%g and nRow=%g\n",
	99558	+ bestJ, pLevel-pWInfo->a, bestPlan.rCost, bestPlan.plan.nRow));
98964	99559	if( (bestPlan.plan.wsFlags & WHERE_ORDERBY)!=0 ){
98965	99560	*ppOrderBy = 0;
98966	99561	}
98967	99562	andFlags &= bestPlan.plan.wsFlags;
98968	99563	pLevel->plan = bestPlan.plan;
		@@ -98973,11 +99568,13 @@
98973	99568	}else{
98974	99569	pLevel->iIdxCur = -1;
98975	99570	}
98976	99571	notReady &= ~getMask(pMaskSet, pTabList->a[bestJ].iCursor);
98977	99572	pLevel->iFrom = (u8)bestJ;
98978		- if( bestPlan.nRow>=(double)1 ) pParse->nQueryLoop *= bestPlan.nRow;
	99573	+ if( bestPlan.plan.nRow>=(double)1 ){
	99574	+ pParse->nQueryLoop *= bestPlan.plan.nRow;
	99575	+ }
98979	99576
98980	99577	/* Check that if the table scanned by this loop iteration had an
98981	99578	** INDEXED BY clause attached to it, that the named index is being
98982	99579	** used for the scan. If not, then query compilation has failed.
98983	99580	** Return an error.
		@@ -99025,41 +99622,10 @@
99025	99622	notReady = ~(Bitmask)0;
99026	99623	for(i=0, pLevel=pWInfo->a; i<nTabList; i++, pLevel++){
99027	99624	Table pTab; / Table to open */
99028	99625	int iDb; /* Index of database containing table/index */
99029	99626
99030		-#ifndef SQLITE_OMIT_EXPLAIN
99031		- if( pParse->explain==2 ){
99032		- char *zMsg;
99033		- struct SrcList_item *pItem = &pTabList->a[pLevel->iFrom];
99034		- zMsg = sqlite3MPrintf(db, "TABLE %s", pItem->zName);
99035		- if( pItem->zAlias ){
99036		- zMsg = sqlite3MAppendf(db, zMsg, "%s AS %s", zMsg, pItem->zAlias);
99037		- }
99038		- if( (pLevel->plan.wsFlags & WHERE_TEMP_INDEX)!=0 ){
99039		- zMsg = sqlite3MAppendf(db, zMsg, "%s WITH AUTOMATIC INDEX", zMsg);
99040		- }else if( (pLevel->plan.wsFlags & WHERE_INDEXED)!=0 ){
99041		- zMsg = sqlite3MAppendf(db, zMsg, "%s WITH INDEX %s",
99042		- zMsg, pLevel->plan.u.pIdx->zName);
99043		- }else if( pLevel->plan.wsFlags & WHERE_MULTI_OR ){
99044		- zMsg = sqlite3MAppendf(db, zMsg, "%s VIA MULTI-INDEX UNION", zMsg);
99045		- }else if( pLevel->plan.wsFlags & (WHERE_ROWID_EQ\|WHERE_ROWID_RANGE) ){
99046		- zMsg = sqlite3MAppendf(db, zMsg, "%s USING PRIMARY KEY", zMsg);
99047		- }
99048		-#ifndef SQLITE_OMIT_VIRTUALTABLE
99049		- else if( (pLevel->plan.wsFlags & WHERE_VIRTUALTABLE)!=0 ){
99050		- sqlite3_index_info *pVtabIdx = pLevel->plan.u.pVtabIdx;
99051		- zMsg = sqlite3MAppendf(db, zMsg, "%s VIRTUAL TABLE INDEX %d:%s", zMsg,
99052		- pVtabIdx->idxNum, pVtabIdx->idxStr);
99053		- }
99054		-#endif
99055		- if( pLevel->plan.wsFlags & WHERE_ORDERBY ){
99056		- zMsg = sqlite3MAppendf(db, zMsg, "%s ORDER BY", zMsg);
99057		- }
99058		- sqlite3VdbeAddOp4(v, OP_Explain, i, pLevel->iFrom, 0, zMsg, P4_DYNAMIC);
99059		- }
99060		-#endif /* SQLITE_OMIT_EXPLAIN */
99061	99627	pTabItem = &pTabList->a[pLevel->iFrom];
99062	99628	pTab = pTabItem->pTab;
99063	99629	pLevel->iTabCur = pTabItem->iCursor;
99064	99630	iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
99065	99631	if( (pTab->tabFlags & TF_Ephemeral)!=0 \|\| pTab->pSelect ){
		@@ -99114,12 +99680,14 @@
99114	99680	** loop below generates code for a single nested loop of the VM
99115	99681	** program.
99116	99682	*/
99117	99683	notReady = ~(Bitmask)0;
99118	99684	for(i=0; i<nTabList; i++){
	99685	+ WhereLevel *pLevel = &pWInfo->a[i];
	99686	+ explainOneScan(pParse, pTabList, pLevel, i, pLevel->iFrom, wctrlFlags);
99119	99687	notReady = codeOneLoopStart(pWInfo, i, wctrlFlags, notReady);
99120		- pWInfo->iContinue = pWInfo->a[i].addrCont;
	99688	+ pWInfo->iContinue = pLevel->addrCont;
99121	99689	}
99122	99690
99123	99691	#ifdef SQLITE_TEST /* For testing and debugging use only */
99124	99692	/* Record in the query plan information about the current table
99125	99693	** and the index used to access it (if any). If the table itself
		@@ -105517,17 +106085,16 @@
105517	106085	}
105518	106086	}
105519	106087	}
105520	106088
105521	106089	pColl = sqlite3FindCollSeq(db, (u8)enc2, zName, 1);
105522		- if( pColl ){
105523		- pColl->xCmp = xCompare;
105524		- pColl->pUser = pCtx;
105525		- pColl->xDel = xDel;
105526		- pColl->enc = (u8)(enc2 \| (enc & SQLITE_UTF16_ALIGNED));
105527		- pColl->type = collType;
105528		- }
	106090	+ if( pColl==0 ) return SQLITE_NOMEM;
	106091	+ pColl->xCmp = xCompare;
	106092	+ pColl->pUser = pCtx;
	106093	+ pColl->xDel = xDel;
	106094	+ pColl->enc = (u8)(enc2 \| (enc & SQLITE_UTF16_ALIGNED));
	106095	+ pColl->type = collType;
105529	106096	sqlite3Error(db, SQLITE_OK, 0);
105530	106097	return SQLITE_OK;
105531	106098	}
105532	106099
105533	106100
		@@ -107491,12 +108058,18 @@
107491	108058	#ifndef SQLITE_AMALGAMATION
107492	108059	/*
107493	108060	** Macros indicating that conditional expressions are always true or
107494	108061	** false.
107495	108062	*/
	108063	+#ifdef SQLITE_COVERAGE_TEST
	108064	+# define ALWAYS(x) (1)
	108065	+# define NEVER(X) (0)
	108066	+#else
107496	108067	# define ALWAYS(x) (x)
107497	108068	# define NEVER(X) (x)
	108069	+#endif
	108070	+
107498	108071	/*
107499	108072	** Internal types used by SQLite.
107500	108073	*/
107501	108074	typedef unsigned char u8; /* 1-byte (or larger) unsigned integer */
107502	108075	typedef short int i16; /* 2-byte (or larger) signed integer */
		@@ -107510,12 +108083,16 @@
107510	108083
107511	108084	typedef struct Fts3Table Fts3Table;
107512	108085	typedef struct Fts3Cursor Fts3Cursor;
107513	108086	typedef struct Fts3Expr Fts3Expr;
107514	108087	typedef struct Fts3Phrase Fts3Phrase;
107515		-typedef struct Fts3SegReader Fts3SegReader;
	108088	+typedef struct Fts3PhraseToken Fts3PhraseToken;
	108089	+
107516	108090	typedef struct Fts3SegFilter Fts3SegFilter;
	108091	+typedef struct Fts3DeferredToken Fts3DeferredToken;
	108092	+typedef struct Fts3SegReader Fts3SegReader;
	108093	+typedef struct Fts3SegReaderArray Fts3SegReaderArray;
107517	108094
107518	108095	/*
107519	108096	** A connection to a fulltext index is an instance of the following
107520	108097	** structure. The xCreate and xConnect methods create an instance
107521	108098	** of this structure and xDestroy and xDisconnect free that instance.
		@@ -107532,26 +108109,18 @@
107532	108109	sqlite3_tokenizer pTokenizer; / tokenizer for inserts and queries */
107533	108110
107534	108111	/* Precompiled statements used by the implementation. Each of these
107535	108112	** statements is run and reset within a single virtual table API call.
107536	108113	*/
107537		- sqlite3_stmt *aStmt[25];
107538		-
107539		- /* Pointer to string containing the SQL:
107540		- **
107541		- ** "SELECT block FROM %_segments WHERE blockid BETWEEN ? AND ?
107542		- ** ORDER BY blockid"
107543		- */
107544		- char *zSelectLeaves;
107545		- int nLeavesStmt; /* Valid statements in aLeavesStmt */
107546		- int nLeavesTotal; /* Total number of prepared leaves stmts */
107547		- int nLeavesAlloc; /* Allocated size of aLeavesStmt */
107548		- sqlite3_stmt *aLeavesStmt; / Array of prepared zSelectLeaves stmts */
	108114	+ sqlite3_stmt *aStmt[24];
107549	108115
107550	108116	int nNodeSize; /* Soft limit for node size */
107551		- u8 bHasContent; /* True if %_content table exists */
	108117	+ u8 bHasStat; /* True if %_stat table exists */
107552	108118	u8 bHasDocsize; /* True if %_docsize table exists */
	108119	+ int nPgsz; /* Page size for host database */
	108120	+ char zSegmentsTbl; / Name of %_segments table */
	108121	+ sqlite3_blob pSegments; / Blob handle open on %_segments table */
107553	108122
107554	108123	/* The following hash table is used to buffer pending index updates during
107555	108124	** transactions. Variable nPendingData estimates the memory size of the
107556	108125	** pending data, including hash table overhead, but not malloc overhead.
107557	108126	** When nPendingData exceeds nMaxPendingData, the buffer is flushed
		@@ -107574,17 +108143,24 @@
107574	108143	i16 eSearch; /* Search strategy (see below) */
107575	108144	u8 isEof; /* True if at End Of Results */
107576	108145	u8 isRequireSeek; /* True if must seek pStmt to %_content row */
107577	108146	sqlite3_stmt pStmt; / Prepared statement in use by the cursor */
107578	108147	Fts3Expr pExpr; / Parsed MATCH query string */
	108148	+ Fts3DeferredToken pDeferred; / Deferred search tokens, if any */
107579	108149	sqlite3_int64 iPrevId; /* Previous id read from aDoclist */
107580	108150	char pNextId; / Pointer into the body of aDoclist */
107581	108151	char aDoclist; / List of docids for full-text queries */
107582	108152	int nDoclist; /* Size of buffer at aDoclist */
107583	108153	int isMatchinfoNeeded; /* True when aMatchinfo[] needs filling in */
107584	108154	u32 aMatchinfo; / Information about most recent match */
	108155	+ int eEvalmode; /* An FTS3_EVAL_XX constant */
	108156	+ int nRowAvg; /* Average size of database rows, in pages */
107585	108157	};
	108158	+
	108159	+#define FTS3_EVAL_FILTER 0
	108160	+#define FTS3_EVAL_NEXT 1
	108161	+#define FTS3_EVAL_MATCHINFO 2
107586	108162
107587	108163	/*
107588	108164	** The Fts3Cursor.eSearch member is always set to one of the following.
107589	108165	** Actualy, Fts3Cursor.eSearch can be greater than or equal to
107590	108166	** FTS3_FULLTEXT_SEARCH. If so, then Fts3Cursor.eSearch - 2 is the index
		@@ -107604,22 +108180,34 @@
107604	108180	#define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */
107605	108181
107606	108182	/*
107607	108183	** A "phrase" is a sequence of one or more tokens that must match in
107608	108184	** sequence. A single token is the base case and the most common case.
107609		-** For a sequence of tokens contained in "...", nToken will be the number
107610		-** of tokens in the string.
	108185	+** For a sequence of tokens contained in double-quotes (i.e. "one two three")
	108186	+** nToken will be the number of tokens in the string.
	108187	+**
	108188	+** The nDocMatch and nMatch variables contain data that may be used by the
	108189	+** matchinfo() function. They are populated when the full-text index is
	108190	+** queried for hits on the phrase. If one or more tokens in the phrase
	108191	+** are deferred, the nDocMatch and nMatch variables are populated based
	108192	+** on the assumption that the
107611	108193	*/
	108194	+struct Fts3PhraseToken {
	108195	+ char z; / Text of the token */
	108196	+ int n; /* Number of bytes in buffer z */
	108197	+ int isPrefix; /* True if token ends with a "" character /
	108198	+ int bFulltext; /* True if full-text index was used */
	108199	+ Fts3SegReaderArray pArray; / Segment-reader for this token */
	108200	+ Fts3DeferredToken pDeferred; / Deferred token object for this token */
	108201	+};
	108202	+
107612	108203	struct Fts3Phrase {
	108204	+ /* Variables populated by fts3_expr.c when parsing a MATCH expression */
107613	108205	int nToken; /* Number of tokens in the phrase */
107614	108206	int iColumn; /* Index of column this phrase must match */
107615	108207	int isNot; /* Phrase prefixed by unary not (-) operator */
107616		- struct PhraseToken {
107617		- char z; / Text of the token */
107618		- int n; /* Number of bytes in buffer pointed to by z */
107619		- int isPrefix; /* True if token ends in with a "" character /
107620		- } aToken[1]; /* One entry for each token in the phrase */
	108208	+ Fts3PhraseToken aToken[1]; /* One entry for each token in the phrase */
107621	108209	};
107622	108210
107623	108211	/*
107624	108212	** A tree of these objects forms the RHS of a MATCH operator.
107625	108213	**
		@@ -107665,15 +108253,10 @@
107665	108253	#define FTSQUERY_AND 3
107666	108254	#define FTSQUERY_OR 4
107667	108255	#define FTSQUERY_PHRASE 5
107668	108256
107669	108257
107670		-/* fts3_init.c */
107671		-SQLITE_PRIVATE int sqlite3Fts3DeleteVtab(int, sqlite3_vtab *);
107672		-SQLITE_PRIVATE int sqlite3Fts3InitVtab(int, sqlite3, void, int, const charconst,
107673		- sqlite3_vtab , char );
107674		-
107675	108258	/* fts3_write.c */
107676	108259	SQLITE_PRIVATE int sqlite3Fts3UpdateMethod(sqlite3_vtab,int,sqlite3_value,sqlite3_int64);
107677	108260	SQLITE_PRIVATE int sqlite3Fts3PendingTermsFlush(Fts3Table *);
107678	108261	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *);
107679	108262	SQLITE_PRIVATE int sqlite3Fts3Optimize(Fts3Table *);
		@@ -107683,15 +108266,24 @@
107683	108266	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table , Fts3SegReader );
107684	108267	SQLITE_PRIVATE int sqlite3Fts3SegReaderIterate(
107685	108268	Fts3Table , Fts3SegReader , int, Fts3SegFilter ,
107686	108269	int ()(Fts3Table , void , char , int, char , int), void
107687	108270	);
107688		-SQLITE_PRIVATE int sqlite3Fts3ReadBlock(Fts3Table, sqlite3_int64, char const, int);
	108271	+SQLITE_PRIVATE int sqlite3Fts3SegReaderCost(Fts3Cursor , Fts3SegReader , int *);
107689	108272	SQLITE_PRIVATE int sqlite3Fts3AllSegdirs(Fts3Table, sqlite3_stmt *);
107690	108273	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeLocal(Fts3Cursor, u32);
107691	108274	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeGlobal(Fts3Cursor, u32);
107692	108275	SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *);
	108276	+SQLITE_PRIVATE int sqlite3Fts3ReadBlock(Fts3Table, sqlite3_int64, char , int);
	108277	+
	108278	+SQLITE_PRIVATE void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *);
	108279	+SQLITE_PRIVATE int sqlite3Fts3DeferToken(Fts3Cursor , Fts3PhraseToken , int);
	108280	+SQLITE_PRIVATE int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *);
	108281	+SQLITE_PRIVATE void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *);
	108282	+SQLITE_PRIVATE char sqlite3Fts3DeferredDoclist(Fts3DeferredToken , int *);
	108283	+
	108284	+SQLITE_PRIVATE void sqlite3Fts3SegmentsClose(Fts3Table *);
107693	108285
107694	108286	/* Flags allowed as part of the 4th argument to SegmentReaderIterate() */
107695	108287	#define FTS3_SEGMENT_REQUIRE_POS 0x00000001
107696	108288	#define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002
107697	108289	#define FTS3_SEGMENT_COLUMN_FILTER 0x00000004
		@@ -107711,19 +108303,21 @@
107711	108303	SQLITE_PRIVATE int sqlite3Fts3GetVarint32(const char , int );
107712	108304	SQLITE_PRIVATE int sqlite3Fts3VarintLen(sqlite3_uint64);
107713	108305	SQLITE_PRIVATE void sqlite3Fts3Dequote(char *);
107714	108306
107715	108307	SQLITE_PRIVATE char sqlite3Fts3FindPositions(Fts3Expr , sqlite3_int64, int);
107716		-SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Table , Fts3Expr );
	108308	+SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Cursor , Fts3Expr );
	108309	+SQLITE_PRIVATE int sqlite3Fts3ExprLoadFtDoclist(Fts3Cursor , Fts3Expr , char *, int );
107717	108310	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr , Fts3Expr , int);
107718	108311
107719	108312	/* fts3_tokenizer.c */
107720	108313	SQLITE_PRIVATE const char sqlite3Fts3NextToken(const char , int *);
107721	108314	SQLITE_PRIVATE int sqlite3Fts3InitHashTable(sqlite3 , Fts3Hash , const char *);
107722		-SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(Fts3Hash *pHash,
107723		- const char , sqlite3_tokenizer , const char , char *
	108315	+SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(Fts3Hash pHash, const char ,
	108316	+ sqlite3_tokenizer , char
107724	108317	);
	108318	+SQLITE_PRIVATE int sqlite3Fts3IsIdChar(char);
107725	108319
107726	108320	/* fts3_snippet.c */
107727	108321	SQLITE_PRIVATE void sqlite3Fts3Offsets(sqlite3_context, Fts3Cursor);
107728	108322	SQLITE_PRIVATE void sqlite3Fts3Snippet(sqlite3_context , Fts3Cursor , const char *,
107729	108323	const char , const char , int, int
		@@ -107875,20 +108469,17 @@
107875	108469	static int fts3DisconnectMethod(sqlite3_vtab *pVtab){
107876	108470	Fts3Table p = (Fts3Table )pVtab;
107877	108471	int i;
107878	108472
107879	108473	assert( p->nPendingData==0 );
	108474	+ assert( p->pSegments==0 );
107880	108475
107881	108476	/* Free any prepared statements held */
107882	108477	for(i=0; i<SizeofArray(p->aStmt); i++){
107883	108478	sqlite3_finalize(p->aStmt[i]);
107884	108479	}
107885		- for(i=0; i<p->nLeavesStmt; i++){
107886		- sqlite3_finalize(p->aLeavesStmt[i]);
107887		- }
107888		- sqlite3_free(p->zSelectLeaves);
107889		- sqlite3_free(p->aLeavesStmt);
	108480	+ sqlite3_free(p->zSegmentsTbl);
107890	108481
107891	108482	/* Invoke the tokenizer destructor to free the tokenizer. */
107892	108483	p->pTokenizer->pModule->xDestroy(p->pTokenizer);
107893	108484
107894	108485	sqlite3_free(p);
		@@ -107895,11 +108486,11 @@
107895	108486	return SQLITE_OK;
107896	108487	}
107897	108488
107898	108489	/*
107899	108490	** Construct one or more SQL statements from the format string given
107900		-** and then evaluate those statements. The success code is writting
	108491	+** and then evaluate those statements. The success code is written
107901	108492	** into *pRc.
107902	108493	**
107903	108494	** If *pRc is initially non-zero then this routine is a no-op.
107904	108495	*/
107905	108496	static void fts3DbExec(
		@@ -107947,37 +108538,42 @@
107947	108538
107948	108539	/*
107949	108540	** Invoke sqlite3_declare_vtab() to declare the schema for the FTS3 table
107950	108541	** passed as the first argument. This is done as part of the xConnect()
107951	108542	** and xCreate() methods.
	108543	+**
	108544	+** If *pRc is non-zero when this function is called, it is a no-op.
	108545	+** Otherwise, if an error occurs, an SQLite error code is stored in *pRc
	108546	+** before returning.
107952	108547	*/
107953		-static int fts3DeclareVtab(Fts3Table *p){
107954		- int i; /* Iterator variable */
107955		- int rc; /* Return code */
107956		- char zSql; / SQL statement passed to declare_vtab() */
107957		- char zCols; / List of user defined columns */
107958		-
107959		- /* Create a list of user columns for the virtual table */
107960		- zCols = sqlite3_mprintf("%Q, ", p->azColumn[0]);
107961		- for(i=1; zCols && i<p->nColumn; i++){
107962		- zCols = sqlite3_mprintf("%z%Q, ", zCols, p->azColumn[i]);
107963		- }
107964		-
107965		- /* Create the whole "CREATE TABLE" statement to pass to SQLite */
107966		- zSql = sqlite3_mprintf(
107967		- "CREATE TABLE x(%s %Q HIDDEN, docid HIDDEN)", zCols, p->zName
107968		- );
107969		-
107970		- if( !zCols \|\| !zSql ){
107971		- rc = SQLITE_NOMEM;
107972		- }else{
107973		- rc = sqlite3_declare_vtab(p->db, zSql);
107974		- }
107975		-
107976		- sqlite3_free(zSql);
107977		- sqlite3_free(zCols);
107978		- return rc;
	108548	+static void fts3DeclareVtab(int pRc, Fts3Table p){
	108549	+ if( *pRc==SQLITE_OK ){
	108550	+ int i; /* Iterator variable */
	108551	+ int rc; /* Return code */
	108552	+ char zSql; / SQL statement passed to declare_vtab() */
	108553	+ char zCols; / List of user defined columns */
	108554	+
	108555	+ /* Create a list of user columns for the virtual table */
	108556	+ zCols = sqlite3_mprintf("%Q, ", p->azColumn[0]);
	108557	+ for(i=1; zCols && i<p->nColumn; i++){
	108558	+ zCols = sqlite3_mprintf("%z%Q, ", zCols, p->azColumn[i]);
	108559	+ }
	108560	+
	108561	+ /* Create the whole "CREATE TABLE" statement to pass to SQLite */
	108562	+ zSql = sqlite3_mprintf(
	108563	+ "CREATE TABLE x(%s %Q HIDDEN, docid HIDDEN)", zCols, p->zName
	108564	+ );
	108565	+ if( !zCols \|\| !zSql ){
	108566	+ rc = SQLITE_NOMEM;
	108567	+ }else{
	108568	+ rc = sqlite3_declare_vtab(p->db, zSql);
	108569	+ }
	108570	+
	108571	+ sqlite3_free(zSql);
	108572	+ sqlite3_free(zCols);
	108573	+ *pRc = rc;
	108574	+ }
107979	108575	}
107980	108576
107981	108577	/*
107982	108578	** Create the backing store tables (%_content, %_segments and %_segdir)
107983	108579	** required by the FTS3 table passed as the only argument. This is done
		@@ -107992,25 +108588,23 @@
107992	108588	int i; /* Iterator variable */
107993	108589	char zContentCols; / Columns of %_content table */
107994	108590	sqlite3 db = p->db; / The database connection */
107995	108591
107996	108592	/* Create a list of user columns for the content table */
107997		- if( p->bHasContent ){
107998		- zContentCols = sqlite3_mprintf("docid INTEGER PRIMARY KEY");
107999		- for(i=0; zContentCols && i<p->nColumn; i++){
108000		- char *z = p->azColumn[i];
108001		- zContentCols = sqlite3_mprintf("%z, 'c%d%q'", zContentCols, i, z);
108002		- }
108003		- if( zContentCols==0 ) rc = SQLITE_NOMEM;
108004		-
108005		- /* Create the content table */
108006		- fts3DbExec(&rc, db,
108007		- "CREATE TABLE %Q.'%q_content'(%s)",
108008		- p->zDb, p->zName, zContentCols
108009		- );
108010		- sqlite3_free(zContentCols);
108011		- }
	108593	+ zContentCols = sqlite3_mprintf("docid INTEGER PRIMARY KEY");
	108594	+ for(i=0; zContentCols && i<p->nColumn; i++){
	108595	+ char *z = p->azColumn[i];
	108596	+ zContentCols = sqlite3_mprintf("%z, 'c%d%q'", zContentCols, i, z);
	108597	+ }
	108598	+ if( zContentCols==0 ) rc = SQLITE_NOMEM;
	108599	+
	108600	+ /* Create the content table */
	108601	+ fts3DbExec(&rc, db,
	108602	+ "CREATE TABLE %Q.'%q_content'(%s)",
	108603	+ p->zDb, p->zName, zContentCols
	108604	+ );
	108605	+ sqlite3_free(zContentCols);
108012	108606	/* Create other tables */
108013	108607	fts3DbExec(&rc, db,
108014	108608	"CREATE TABLE %Q.'%q_segments'(blockid INTEGER PRIMARY KEY, block BLOB);",
108015	108609	p->zDb, p->zName
108016	108610	);
		@@ -108029,10 +108623,12 @@
108029	108623	if( p->bHasDocsize ){
108030	108624	fts3DbExec(&rc, db,
108031	108625	"CREATE TABLE %Q.'%q_docsize'(docid INTEGER PRIMARY KEY, size BLOB);",
108032	108626	p->zDb, p->zName
108033	108627	);
	108628	+ }
	108629	+ if( p->bHasStat ){
108034	108630	fts3DbExec(&rc, db,
108035	108631	"CREATE TABLE %Q.'%q_stat'(id INTEGER PRIMARY KEY, value BLOB);",
108036	108632	p->zDb, p->zName
108037	108633	);
108038	108634	}
		@@ -108072,10 +108668,70 @@
108072	108668	rc = sqlite3_exec(db, zSql, fts3TableExistsCallback, &res, 0);
108073	108669	sqlite3_free(zSql);
108074	108670	*pResult = (u8)(res & 0xff);
108075	108671	if( rc!=SQLITE_ABORT ) *pRc = rc;
108076	108672	}
	108673	+
	108674	+/*
	108675	+** Store the current database page-size in bytes in p->nPgsz.
	108676	+**
	108677	+** If *pRc is non-zero when this function is called, it is a no-op.
	108678	+** Otherwise, if an error occurs, an SQLite error code is stored in *pRc
	108679	+** before returning.
	108680	+*/
	108681	+static void fts3DatabasePageSize(int pRc, Fts3Table p){
	108682	+ if( *pRc==SQLITE_OK ){
	108683	+ int rc; /* Return code */
	108684	+ char zSql; / SQL text "PRAGMA %Q.page_size" */
	108685	+ sqlite3_stmt pStmt; / Compiled "PRAGMA %Q.page_size" statement */
	108686	+
	108687	+ zSql = sqlite3_mprintf("PRAGMA %Q.page_size", p->zDb);
	108688	+ if( !zSql ){
	108689	+ rc = SQLITE_NOMEM;
	108690	+ }else{
	108691	+ rc = sqlite3_prepare(p->db, zSql, -1, &pStmt, 0);
	108692	+ if( rc==SQLITE_OK ){
	108693	+ sqlite3_step(pStmt);
	108694	+ p->nPgsz = sqlite3_column_int(pStmt, 0);
	108695	+ rc = sqlite3_finalize(pStmt);
	108696	+ }
	108697	+ }
	108698	+ assert( p->nPgsz>0 \|\| rc!=SQLITE_OK );
	108699	+ sqlite3_free(zSql);
	108700	+ *pRc = rc;
	108701	+ }
	108702	+}
	108703	+
	108704	+/*
	108705	+** "Special" FTS4 arguments are column specifications of the following form:
	108706	+**
	108707	+** <key> = <value>
	108708	+**
	108709	+** There may not be whitespace surrounding the "=" character. The <value>
	108710	+** term may be quoted, but the <key> may not.
	108711	+*/
	108712	+static int fts3IsSpecialColumn(
	108713	+ const char *z,
	108714	+ int *pnKey,
	108715	+ char **pzValue
	108716	+){
	108717	+ char *zValue;
	108718	+ const char *zCsr = z;
	108719	+
	108720	+ while( *zCsr!='=' ){
	108721	+ if( *zCsr=='\0' ) return 0;
	108722	+ zCsr++;
	108723	+ }
	108724	+
	108725	+ *pnKey = zCsr-z;
	108726	+ zValue = sqlite3_mprintf("%s", &zCsr[1]);
	108727	+ if( zValue ){
	108728	+ sqlite3Fts3Dequote(zValue);
	108729	+ }
	108730	+ *pzValue = zValue;
	108731	+ return 1;
	108732	+}
108077	108733
108078	108734	/*
108079	108735	** This function is the implementation of both the xConnect and xCreate
108080	108736	** methods of the FTS3 virtual table.
108081	108737	**
		@@ -108094,48 +108750,103 @@
108094	108750	const char * const argv, / xCreate/xConnect argument array */
108095	108751	sqlite3_vtab *ppVTab, / Write the resulting vtab structure here */
108096	108752	char *pzErr / Write any error message here */
108097	108753	){
108098	108754	Fts3Hash pHash = (Fts3Hash )pAux;
108099		- Fts3Table p; / Pointer to allocated vtab */
108100		- int rc; /* Return code */
	108755	+ Fts3Table p = 0; / Pointer to allocated vtab */
	108756	+ int rc = SQLITE_OK; /* Return code */
108101	108757	int i; /* Iterator variable */
108102	108758	int nByte; /* Size of allocation used for p /
108103	108759	int iCol; /* Column index */
108104	108760	int nString = 0; /* Bytes required to hold all column names */
108105	108761	int nCol = 0; /* Number of columns in the FTS table */
108106	108762	char zCsr; / Space for holding column names */
108107	108763	int nDb; /* Bytes required to hold database name */
108108	108764	int nName; /* Bytes required to hold table name */
108109		-
108110		- const char zTokenizer = 0; / Name of tokenizer to use */
	108765	+ int isFts4 = (argv[0][3]=='4'); /* True for FTS4, false for FTS3 */
	108766	+ int bNoDocsize = 0; /* True to omit %_docsize table */
	108767	+ const char *aCol; / Array of column names */
108111	108768	sqlite3_tokenizer pTokenizer = 0; / Tokenizer for this table */
	108769	+
	108770	+ assert( strlen(argv[0])==4 );
	108771	+ assert( (sqlite3_strnicmp(argv[0], "fts4", 4)==0 && isFts4)
	108772	+ \|\| (sqlite3_strnicmp(argv[0], "fts3", 4)==0 && !isFts4)
	108773	+ );
108112	108774
108113	108775	nDb = (int)strlen(argv[1]) + 1;
108114	108776	nName = (int)strlen(argv[2]) + 1;
108115		- for(i=3; i<argc; i++){
	108777	+
	108778	+ aCol = (const char *)sqlite3_malloc(sizeof(const char ) * (argc-2) );
	108779	+ if( !aCol ) return SQLITE_NOMEM;
	108780	+ memset(aCol, 0, sizeof(const char ) (argc-2));
	108781	+
	108782	+ /* Loop through all of the arguments passed by the user to the FTS3/4
	108783	+ ** module (i.e. all the column names and special arguments). This loop
	108784	+ ** does the following:
	108785	+ **
	108786	+ ** + Figures out the number of columns the FTSX table will have, and
	108787	+ ** the number of bytes of space that must be allocated to store copies
	108788	+ ** of the column names.
	108789	+ **
	108790	+ ** + If there is a tokenizer specification included in the arguments,
	108791	+ ** initializes the tokenizer pTokenizer.
	108792	+ */
	108793	+ for(i=3; rc==SQLITE_OK && i<argc; i++){
108116	108794	char const *z = argv[i];
108117		- rc = sqlite3Fts3InitTokenizer(pHash, z, &pTokenizer, &zTokenizer, pzErr);
108118		- if( rc!=SQLITE_OK ){
108119		- return rc;
	108795	+ int nKey;
	108796	+ char *zVal;
	108797	+
	108798	+ /* Check if this is a tokenizer specification */
	108799	+ if( !pTokenizer
	108800	+ && strlen(z)>8
	108801	+ && 0==sqlite3_strnicmp(z, "tokenize", 8)
	108802	+ && 0==sqlite3Fts3IsIdChar(z[8])
	108803	+ ){
	108804	+ rc = sqlite3Fts3InitTokenizer(pHash, &z[9], &pTokenizer, pzErr);
108120	108805	}
108121		- if( z!=zTokenizer ){
	108806	+
	108807	+ /* Check if it is an FTS4 special argument. */
	108808	+ else if( isFts4 && fts3IsSpecialColumn(z, &nKey, &zVal) ){
	108809	+ if( !zVal ){
	108810	+ rc = SQLITE_NOMEM;
	108811	+ goto fts3_init_out;
	108812	+ }
	108813	+ if( nKey==9 && 0==sqlite3_strnicmp(z, "matchinfo", 9) ){
	108814	+ if( strlen(zVal)==4 && 0==sqlite3_strnicmp(zVal, "fts3", 4) ){
	108815	+ bNoDocsize = 1;
	108816	+ }else{
	108817	+ *pzErr = sqlite3_mprintf("unrecognized matchinfo: %s", zVal);
	108818	+ rc = SQLITE_ERROR;
	108819	+ }
	108820	+ }else{
	108821	+ *pzErr = sqlite3_mprintf("unrecognized parameter: %s", z);
	108822	+ rc = SQLITE_ERROR;
	108823	+ }
	108824	+ sqlite3_free(zVal);
	108825	+ }
	108826	+
	108827	+ /* Otherwise, the argument is a column name. */
	108828	+ else {
108122	108829	nString += (int)(strlen(z) + 1);
	108830	+ aCol[nCol++] = z;
108123	108831	}
108124	108832	}
108125		- nCol = argc - 3 - (zTokenizer!=0);
108126		- if( zTokenizer==0 ){
108127		- rc = sqlite3Fts3InitTokenizer(pHash, 0, &pTokenizer, 0, pzErr);
108128		- if( rc!=SQLITE_OK ){
108129		- return rc;
108130		- }
108131		- assert( pTokenizer );
108132		- }
	108833	+ if( rc!=SQLITE_OK ) goto fts3_init_out;
108133	108834
108134	108835	if( nCol==0 ){
	108836	+ assert( nString==0 );
	108837	+ aCol[0] = "content";
	108838	+ nString = 8;
108135	108839	nCol = 1;
108136	108840	}
	108841	+
	108842	+ if( pTokenizer==0 ){
	108843	+ rc = sqlite3Fts3InitTokenizer(pHash, "simple", &pTokenizer, pzErr);
	108844	+ if( rc!=SQLITE_OK ) goto fts3_init_out;
	108845	+ }
	108846	+ assert( pTokenizer );
	108847	+
108137	108848
108138	108849	/* Allocate and populate the Fts3Table structure. */
108139	108850	nByte = sizeof(Fts3Table) + /* Fts3Table */
108140	108851	nCol * sizeof(char ) + / azColumn */
108141	108852	nName + /* zName */
		@@ -108145,77 +108856,70 @@
108145	108856	if( p==0 ){
108146	108857	rc = SQLITE_NOMEM;
108147	108858	goto fts3_init_out;
108148	108859	}
108149	108860	memset(p, 0, nByte);
108150		-
108151	108861	p->db = db;
108152	108862	p->nColumn = nCol;
108153	108863	p->nPendingData = 0;
108154	108864	p->azColumn = (char **)&p[1];
108155	108865	p->pTokenizer = pTokenizer;
108156	108866	p->nNodeSize = 1000;
108157	108867	p->nMaxPendingData = FTS3_MAX_PENDING_DATA;
108158		- zCsr = (char *)&p->azColumn[nCol];
108159		-
	108868	+ p->bHasDocsize = (isFts4 && bNoDocsize==0);
	108869	+ p->bHasStat = isFts4;
108160	108870	fts3HashInit(&p->pendingTerms, FTS3_HASH_STRING, 1);
108161	108871
108162	108872	/* Fill in the zName and zDb fields of the vtab structure. */
	108873	+ zCsr = (char *)&p->azColumn[nCol];
108163	108874	p->zName = zCsr;
108164	108875	memcpy(zCsr, argv[2], nName);
108165	108876	zCsr += nName;
108166	108877	p->zDb = zCsr;
108167	108878	memcpy(zCsr, argv[1], nDb);
108168	108879	zCsr += nDb;
108169	108880
108170	108881	/* Fill in the azColumn array */
108171		- iCol = 0;
108172		- for(i=3; i<argc; i++){
108173		- if( argv[i]!=zTokenizer ){
108174		- char *z;
108175		- int n;
108176		- z = (char *)sqlite3Fts3NextToken(argv[i], &n);
108177		- memcpy(zCsr, z, n);
108178		- zCsr[n] = '\0';
108179		- sqlite3Fts3Dequote(zCsr);
108180		- p->azColumn[iCol++] = zCsr;
108181		- zCsr += n+1;
108182		- assert( zCsr <= &((char *)p)[nByte] );
108183		- }
108184		- }
108185		- if( iCol==0 ){
108186		- assert( nCol==1 );
108187		- p->azColumn[0] = "content";
	108882	+ for(iCol=0; iCol<nCol; iCol++){
	108883	+ char *z;
	108884	+ int n;
	108885	+ z = (char *)sqlite3Fts3NextToken(aCol[iCol], &n);
	108886	+ memcpy(zCsr, z, n);
	108887	+ zCsr[n] = '\0';
	108888	+ sqlite3Fts3Dequote(zCsr);
	108889	+ p->azColumn[iCol] = zCsr;
	108890	+ zCsr += n+1;
	108891	+ assert( zCsr <= &((char *)p)[nByte] );
108188	108892	}
108189	108893
108190	108894	/* If this is an xCreate call, create the underlying tables in the
108191	108895	** database. TODO: For xConnect(), it could verify that said tables exist.
108192	108896	*/
108193	108897	if( isCreate ){
108194		- p->bHasContent = 1;
108195		- p->bHasDocsize = argv[0][3]=='4';
108196	108898	rc = fts3CreateTables(p);
108197		- }else{
108198		- rc = SQLITE_OK;
108199		- fts3TableExists(&rc, db, argv[1], argv[2], "_content", &p->bHasContent);
108200		- fts3TableExists(&rc, db, argv[1], argv[2], "_docsize", &p->bHasDocsize);
108201		- }
108202		- if( rc!=SQLITE_OK ) goto fts3_init_out;
108203		-
108204		- rc = fts3DeclareVtab(p);
108205		- if( rc!=SQLITE_OK ) goto fts3_init_out;
108206		-
108207		- *ppVTab = &p->base;
	108899	+ }
	108900	+
	108901	+ /* Figure out the page-size for the database. This is required in order to
	108902	+ ** estimate the cost of loading large doclists from the database (see
	108903	+ ** function sqlite3Fts3SegReaderCost() for details).
	108904	+ */
	108905	+ fts3DatabasePageSize(&rc, p);
	108906	+
	108907	+ /* Declare the table schema to SQLite. */
	108908	+ fts3DeclareVtab(&rc, p);
108208	108909
108209	108910	fts3_init_out:
108210		- assert( p \|\| (pTokenizer && rc!=SQLITE_OK) );
	108911	+
	108912	+ sqlite3_free(aCol);
108211	108913	if( rc!=SQLITE_OK ){
108212	108914	if( p ){
108213	108915	fts3DisconnectMethod((sqlite3_vtab *)p);
108214		- }else{
	108916	+ }else if( pTokenizer ){
108215	108917	pTokenizer->pModule->xDestroy(pTokenizer);
108216	108918	}
	108919	+ }else{
	108920	+ *ppVTab = &p->base;
108217	108921	}
108218	108922	return rc;
108219	108923	}
108220	108924
108221	108925	/*
		@@ -108323,14 +109027,16 @@
108323	109027
108324	109028	/*
108325	109029	** Close the cursor. For additional information see the documentation
108326	109030	** on the xClose method of the virtual table interface.
108327	109031	*/
108328		-static int fulltextClose(sqlite3_vtab_cursor *pCursor){
	109032	+static int fts3CloseMethod(sqlite3_vtab_cursor *pCursor){
108329	109033	Fts3Cursor pCsr = (Fts3Cursor )pCursor;
	109034	+ assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 );
108330	109035	sqlite3_finalize(pCsr->pStmt);
108331	109036	sqlite3Fts3ExprFree(pCsr->pExpr);
	109037	+ sqlite3Fts3FreeDeferredTokens(pCsr);
108332	109038	sqlite3_free(pCsr->aDoclist);
108333	109039	sqlite3_free(pCsr->aMatchinfo);
108334	109040	sqlite3_free(pCsr);
108335	109041	return SQLITE_OK;
108336	109042	}
		@@ -108365,134 +109071,186 @@
108365	109071	return SQLITE_OK;
108366	109072	}
108367	109073	}
108368	109074
108369	109075	/*
108370		-** Advance the cursor to the next row in the %_content table that
108371		-** matches the search criteria. For a MATCH search, this will be
108372		-** the next row that matches. For a full-table scan, this will be
108373		-** simply the next row in the %_content table. For a docid lookup,
108374		-** this routine simply sets the EOF flag.
	109076	+** This function is used to process a single interior node when searching
	109077	+** a b-tree for a term or term prefix. The node data is passed to this
	109078	+** function via the zNode/nNode parameters. The term to search for is
	109079	+** passed in zTerm/nTerm.
108375	109080	**
108376		-** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned
108377		-** even if we reach end-of-file. The fts3EofMethod() will be called
108378		-** subsequently to determine whether or not an EOF was hit.
	109081	+** If piFirst is not NULL, then this function sets *piFirst to the blockid
	109082	+** of the child node that heads the sub-tree that may contain the term.
	109083	+**
	109084	+** If piLast is not NULL, then *piLast is set to the right-most child node
	109085	+** that heads a sub-tree that may contain a term for which zTerm/nTerm is
	109086	+** a prefix.
	109087	+**
	109088	+** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK.
108379	109089	*/
108380		-static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){
	109090	+static int fts3ScanInteriorNode(
	109091	+ Fts3Table p, / Virtual table handle */
	109092	+ const char zTerm, / Term to select leaves for */
	109093	+ int nTerm, /* Size of term zTerm in bytes */
	109094	+ const char zNode, / Buffer containing segment interior node */
	109095	+ int nNode, /* Size of buffer at zNode */
	109096	+ sqlite3_int64 piFirst, / OUT: Selected child node */
	109097	+ sqlite3_int64 piLast / OUT: Selected child node */
	109098	+){
108381	109099	int rc = SQLITE_OK; /* Return code */
108382		- Fts3Cursor pCsr = (Fts3Cursor )pCursor;
108383		-
108384		- if( pCsr->aDoclist==0 ){
108385		- if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){
108386		- pCsr->isEof = 1;
108387		- rc = sqlite3_reset(pCsr->pStmt);
108388		- }
108389		- }else if( pCsr->pNextId>=&pCsr->aDoclist[pCsr->nDoclist] ){
108390		- pCsr->isEof = 1;
108391		- }else{
108392		- sqlite3_reset(pCsr->pStmt);
108393		- fts3GetDeltaVarint(&pCsr->pNextId, &pCsr->iPrevId);
108394		- pCsr->isRequireSeek = 1;
108395		- pCsr->isMatchinfoNeeded = 1;
108396		- }
108397		- return rc;
108398		-}
108399		-
108400		-
108401		-/*
108402		-** The buffer pointed to by argument zNode (size nNode bytes) contains the
108403		-** root node of a b-tree segment. The segment is guaranteed to be at least
108404		-** one level high (i.e. the root node is not also a leaf). If successful,
108405		-** this function locates the leaf node of the segment that may contain the
108406		-** term specified by arguments zTerm and nTerm and writes its block number
108407		-** to *piLeaf.
108408		-**
108409		-** It is possible that the returned leaf node does not contain the specified
108410		-** term. However, if the segment does contain said term, it is stored on
108411		-** the identified leaf node. Because this function only inspects interior
108412		-** segment nodes (and never loads leaf nodes into memory), it is not possible
108413		-** to be sure.
	109100	+ const char zCsr = zNode; / Cursor to iterate through node */
	109101	+ const char zEnd = &zCsr[nNode];/ End of interior node buffer */
	109102	+ char zBuffer = 0; / Buffer to load terms into */
	109103	+ int nAlloc = 0; /* Size of allocated buffer */
	109104	+ int isFirstTerm = 1; /* True when processing first term on page */
	109105	+ sqlite3_int64 iChild; /* Block id of child node to descend to */
	109106	+
	109107	+ /* Skip over the 'height' varint that occurs at the start of every
	109108	+ ** interior node. Then load the blockid of the left-child of the b-tree
	109109	+ ** node into variable iChild.
	109110	+ **
	109111	+ ** Even if the data structure on disk is corrupted, this (reading two
	109112	+ ** varints from the buffer) does not risk an overread. If zNode is a
	109113	+ ** root node, then the buffer comes from a SELECT statement. SQLite does
	109114	+ ** not make this guarantee explicitly, but in practice there are always
	109115	+ ** either more than 20 bytes of allocated space following the nNode bytes of
	109116	+ ** contents, or two zero bytes. Or, if the node is read from the %_segments
	109117	+ ** table, then there are always 20 bytes of zeroed padding following the
	109118	+ ** nNode bytes of content (see sqlite3Fts3ReadBlock() for details).
	109119	+ */
	109120	+ zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
	109121	+ zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
	109122	+ if( zCsr>=zEnd ){
	109123	+ return SQLITE_CORRUPT;
	109124	+ }
	109125	+
	109126	+ while( zCsr<zEnd && (piFirst \|\| piLast) ){
	109127	+ int cmp; /* memcmp() result */
	109128	+ int nSuffix; /* Size of term suffix */
	109129	+ int nPrefix = 0; /* Size of term prefix */
	109130	+ int nBuffer; /* Total term size */
	109131	+
	109132	+ /* Load the next term on the node into zBuffer. Use realloc() to expand
	109133	+ ** the size of zBuffer if required. */
	109134	+ if( !isFirstTerm ){
	109135	+ zCsr += sqlite3Fts3GetVarint32(zCsr, &nPrefix);
	109136	+ }
	109137	+ isFirstTerm = 0;
	109138	+ zCsr += sqlite3Fts3GetVarint32(zCsr, &nSuffix);
	109139	+
	109140	+ if( nPrefix<0 \|\| nSuffix<0 \|\| &zCsr[nSuffix]>zEnd ){
	109141	+ rc = SQLITE_CORRUPT;
	109142	+ goto finish_scan;
	109143	+ }
	109144	+ if( nPrefix+nSuffix>nAlloc ){
	109145	+ char *zNew;
	109146	+ nAlloc = (nPrefix+nSuffix) * 2;
	109147	+ zNew = (char *)sqlite3_realloc(zBuffer, nAlloc);
	109148	+ if( !zNew ){
	109149	+ rc = SQLITE_NOMEM;
	109150	+ goto finish_scan;
	109151	+ }
	109152	+ zBuffer = zNew;
	109153	+ }
	109154	+ memcpy(&zBuffer[nPrefix], zCsr, nSuffix);
	109155	+ nBuffer = nPrefix + nSuffix;
	109156	+ zCsr += nSuffix;
	109157	+
	109158	+ /* Compare the term we are searching for with the term just loaded from
	109159	+ ** the interior node. If the specified term is greater than or equal
	109160	+ ** to the term from the interior node, then all terms on the sub-tree
	109161	+ ** headed by node iChild are smaller than zTerm. No need to search
	109162	+ ** iChild.
	109163	+ **
	109164	+ ** If the interior node term is larger than the specified term, then
	109165	+ ** the tree headed by iChild may contain the specified term.
	109166	+ */
	109167	+ cmp = memcmp(zTerm, zBuffer, (nBuffer>nTerm ? nTerm : nBuffer));
	109168	+ if( piFirst && (cmp<0 \|\| (cmp==0 && nBuffer>nTerm)) ){
	109169	+ *piFirst = iChild;
	109170	+ piFirst = 0;
	109171	+ }
	109172	+
	109173	+ if( piLast && cmp<0 ){
	109174	+ *piLast = iChild;
	109175	+ piLast = 0;
	109176	+ }
	109177	+
	109178	+ iChild++;
	109179	+ };
	109180	+
	109181	+ if( piFirst ) *piFirst = iChild;
	109182	+ if( piLast ) *piLast = iChild;
	109183	+
	109184	+ finish_scan:
	109185	+ sqlite3_free(zBuffer);
	109186	+ return rc;
	109187	+}
	109188	+
	109189	+
	109190	+/*
	109191	+** The buffer pointed to by argument zNode (size nNode bytes) contains an
	109192	+** interior node of a b-tree segment. The zTerm buffer (size nTerm bytes)
	109193	+** contains a term. This function searches the sub-tree headed by the zNode
	109194	+** node for the range of leaf nodes that may contain the specified term
	109195	+** or terms for which the specified term is a prefix.
	109196	+**
	109197	+** If piLeaf is not NULL, then *piLeaf is set to the blockid of the
	109198	+** left-most leaf node in the tree that may contain the specified term.
	109199	+** If piLeaf2 is not NULL, then *piLeaf2 is set to the blockid of the
	109200	+** right-most leaf node that may contain a term for which the specified
	109201	+** term is a prefix.
	109202	+**
	109203	+** It is possible that the range of returned leaf nodes does not contain
	109204	+** the specified term or any terms for which it is a prefix. However, if the
	109205	+** segment does contain any such terms, they are stored within the identified
	109206	+** range. Because this function only inspects interior segment nodes (and
	109207	+** never loads leaf nodes into memory), it is not possible to be sure.
108414	109208	**
108415	109209	** If an error occurs, an error code other than SQLITE_OK is returned.
108416	109210	*/
108417	109211	static int fts3SelectLeaf(
108418	109212	Fts3Table p, / Virtual table handle */
108419	109213	const char zTerm, / Term to select leaves for */
108420	109214	int nTerm, /* Size of term zTerm in bytes */
108421	109215	const char zNode, / Buffer containing segment interior node */
108422	109216	int nNode, /* Size of buffer at zNode */
108423		- sqlite3_int64 piLeaf / Selected leaf node */
108424		-){
108425		- int rc = SQLITE_OK; /* Return code */
108426		- const char zCsr = zNode; / Cursor to iterate through node */
108427		- const char zEnd = &zCsr[nNode];/ End of interior node buffer */
108428		- char zBuffer = 0; / Buffer to load terms into */
108429		- int nAlloc = 0; /* Size of allocated buffer */
108430		-
108431		- while( 1 ){
108432		- int isFirstTerm = 1; /* True when processing first term on page */
108433		- int iHeight; /* Height of this node in tree */
108434		- sqlite3_int64 iChild; /* Block id of child node to descend to */
108435		- int nBlock; /* Size of child node in bytes */
108436		-
108437		- zCsr += sqlite3Fts3GetVarint32(zCsr, &iHeight);
108438		- zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
108439		-
108440		- while( zCsr<zEnd ){
108441		- int cmp; /* memcmp() result */
108442		- int nSuffix; /* Size of term suffix */
108443		- int nPrefix = 0; /* Size of term prefix */
108444		- int nBuffer; /* Total term size */
108445		-
108446		- /* Load the next term on the node into zBuffer */
108447		- if( !isFirstTerm ){
108448		- zCsr += sqlite3Fts3GetVarint32(zCsr, &nPrefix);
108449		- }
108450		- isFirstTerm = 0;
108451		- zCsr += sqlite3Fts3GetVarint32(zCsr, &nSuffix);
108452		- if( nPrefix+nSuffix>nAlloc ){
108453		- char *zNew;
108454		- nAlloc = (nPrefix+nSuffix) * 2;
108455		- zNew = (char *)sqlite3_realloc(zBuffer, nAlloc);
108456		- if( !zNew ){
108457		- sqlite3_free(zBuffer);
108458		- return SQLITE_NOMEM;
108459		- }
108460		- zBuffer = zNew;
108461		- }
108462		- memcpy(&zBuffer[nPrefix], zCsr, nSuffix);
108463		- nBuffer = nPrefix + nSuffix;
108464		- zCsr += nSuffix;
108465		-
108466		- /* Compare the term we are searching for with the term just loaded from
108467		- ** the interior node. If the specified term is greater than or equal
108468		- ** to the term from the interior node, then all terms on the sub-tree
108469		- ** headed by node iChild are smaller than zTerm. No need to search
108470		- ** iChild.
108471		- **
108472		- ** If the interior node term is larger than the specified term, then
108473		- ** the tree headed by iChild may contain the specified term.
108474		- */
108475		- cmp = memcmp(zTerm, zBuffer, (nBuffer>nTerm ? nTerm : nBuffer));
108476		- if( cmp<0 \|\| (cmp==0 && nBuffer>nTerm) ) break;
108477		- iChild++;
108478		- };
108479		-
108480		- /* If (iHeight==1), the children of this interior node are leaves. The
108481		- ** specified term may be present on leaf node iChild.
108482		- */
108483		- if( iHeight==1 ){
108484		- *piLeaf = iChild;
108485		- break;
108486		- }
108487		-
108488		- /* Descend to interior node iChild. */
108489		- rc = sqlite3Fts3ReadBlock(p, iChild, &zCsr, &nBlock);
108490		- if( rc!=SQLITE_OK ) break;
108491		- zEnd = &zCsr[nBlock];
108492		- }
108493		- sqlite3_free(zBuffer);
	109217	+ sqlite3_int64 piLeaf, / Selected leaf node */
	109218	+ sqlite3_int64 piLeaf2 / Selected leaf node */
	109219	+){
	109220	+ int rc; /* Return code */
	109221	+ int iHeight; /* Height of this node in tree */
	109222	+
	109223	+ assert( piLeaf \|\| piLeaf2 );
	109224	+
	109225	+ sqlite3Fts3GetVarint32(zNode, &iHeight);
	109226	+ rc = fts3ScanInteriorNode(p, zTerm, nTerm, zNode, nNode, piLeaf, piLeaf2);
	109227	+ assert( !piLeaf2 \|\| !piLeaf \|\| rc!=SQLITE_OK \|\| (piLeaf<=piLeaf2) );
	109228	+
	109229	+ if( rc==SQLITE_OK && iHeight>1 ){
	109230	+ char zBlob = 0; / Blob read from %_segments table */
	109231	+ int nBlob; /* Size of zBlob in bytes */
	109232	+
	109233	+ if( piLeaf && piLeaf2 && (piLeaf!=piLeaf2) ){
	109234	+ rc = sqlite3Fts3ReadBlock(p, *piLeaf, &zBlob, &nBlob);
	109235	+ if( rc==SQLITE_OK ){
	109236	+ rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, 0);
	109237	+ }
	109238	+ sqlite3_free(zBlob);
	109239	+ piLeaf = 0;
	109240	+ zBlob = 0;
	109241	+ }
	109242	+
	109243	+ if( rc==SQLITE_OK ){
	109244	+ rc = sqlite3Fts3ReadBlock(p, piLeaf ? piLeaf : piLeaf2, &zBlob, &nBlob);
	109245	+ }
	109246	+ if( rc==SQLITE_OK ){
	109247	+ rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, piLeaf2);
	109248	+ }
	109249	+ sqlite3_free(zBlob);
	109250	+ }
	109251	+
108494	109252	return rc;
108495	109253	}
108496	109254
108497	109255	/*
108498	109256	** This function is used to create delta-encoded serialized lists of FTS3
		@@ -108720,24 +109478,48 @@
108720	109478	*pp2 = p2 + 1;
108721	109479	}
108722	109480
108723	109481	/*
108724	109482	** nToken==1 searches for adjacent positions.
	109483	+**
	109484	+** This function is used to merge two position lists into one. When it is
	109485	+** called, pp1 and pp2 must both point to position lists. A position-list is
	109486	+** the part of a doclist that follows each document id. For example, if a row
	109487	+** contains:
	109488	+**
	109489	+** 'a b c'\|'x y z'\|'a b b a'
	109490	+**
	109491	+** Then the position list for this row for token 'b' would consist of:
	109492	+**
	109493	+** 0x02 0x01 0x02 0x03 0x03 0x00
	109494	+**
	109495	+** When this function returns, both pp1 and pp2 are left pointing to the
	109496	+** byte following the 0x00 terminator of their respective position lists.
	109497	+**
	109498	+** If isSaveLeft is 0, an entry is added to the output position list for
	109499	+** each position in *pp2 for which there exists one or more positions in
	109500	+** pp1 so that (pos(pp2)>pos(pp1) && pos(pp2)-pos(*pp1)<=nToken). i.e.
	109501	+** when the pp1 token appears before the pp2 token, but not more than nToken
	109502	+** slots before it.
108725	109503	*/
108726	109504	static int fts3PoslistPhraseMerge(
108727		- char *pp, / Output buffer */
	109505	+ char *pp, / IN/OUT: Preallocated output buffer */
108728	109506	int nToken, /* Maximum difference in token positions */
108729	109507	int isSaveLeft, /* Save the left position */
108730		- char *pp1, / Left input list */
108731		- char *pp2 / Right input list */
	109508	+ int isExact, /* If pp1 is exactly nTokens before pp2 */
	109509	+ char *pp1, / IN/OUT: Left input list */
	109510	+ char *pp2 / IN/OUT: Right input list */
108732	109511	){
108733	109512	char p = (pp ? pp : 0);
108734	109513	char p1 = pp1;
108735	109514	char p2 = pp2;
108736		-
108737	109515	int iCol1 = 0;
108738	109516	int iCol2 = 0;
	109517	+
	109518	+ /* Never set both isSaveLeft and isExact for the same invocation. */
	109519	+ assert( isSaveLeft==0 \|\| isExact==0 );
	109520	+
108739	109521	assert( p1!=0 && p2!=0 );
108740	109522	if( *p1==POS_COLUMN ){
108741	109523	p1++;
108742	109524	p1 += sqlite3Fts3GetVarint32(p1, &iCol1);
108743	109525	}
		@@ -108762,11 +109544,13 @@
108762	109544	assert( p2!=POS_END && p2!=POS_COLUMN );
108763	109545	fts3GetDeltaVarint(&p1, &iPos1); iPos1 -= 2;
108764	109546	fts3GetDeltaVarint(&p2, &iPos2); iPos2 -= 2;
108765	109547
108766	109548	while( 1 ){
108767		- if( iPos2>iPos1 && iPos2<=iPos1+nToken ){
	109549	+ if( iPos2==iPos1+nToken
	109550	+ \|\| (isExact==0 && iPos2>iPos1 && iPos2<=iPos1+nToken)
	109551	+ ){
108768	109552	sqlite3_int64 iSave;
108769	109553	if( !pp ){
108770	109554	fts3PoslistCopy(0, &p2);
108771	109555	fts3PoslistCopy(0, &p1);
108772	109556	*pp1 = p1;
		@@ -108845,25 +109629,25 @@
108845	109629	){
108846	109630	char p1 = pp1;
108847	109631	char p2 = pp2;
108848	109632
108849	109633	if( !pp ){
108850		- if( fts3PoslistPhraseMerge(0, nRight, 0, pp1, pp2) ) return 1;
	109634	+ if( fts3PoslistPhraseMerge(0, nRight, 0, 0, pp1, pp2) ) return 1;
108851	109635	*pp1 = p1;
108852	109636	*pp2 = p2;
108853		- return fts3PoslistPhraseMerge(0, nLeft, 0, pp2, pp1);
	109637	+ return fts3PoslistPhraseMerge(0, nLeft, 0, 0, pp2, pp1);
108854	109638	}else{
108855	109639	char *pTmp1 = aTmp;
108856	109640	char *pTmp2;
108857	109641	char *aTmp2;
108858	109642	int res = 1;
108859	109643
108860		- fts3PoslistPhraseMerge(&pTmp1, nRight, 0, pp1, pp2);
	109644	+ fts3PoslistPhraseMerge(&pTmp1, nRight, 0, 0, pp1, pp2);
108861	109645	aTmp2 = pTmp2 = pTmp1;
108862	109646	*pp1 = p1;
108863	109647	*pp2 = p2;
108864		- fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, pp2, pp1);
	109648	+ fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, 0, pp2, pp1);
108865	109649	if( pTmp1!=aTmp && pTmp2!=aTmp2 ){
108866	109650	fts3PoslistMerge(pp, &aTmp, &aTmp2);
108867	109651	}else if( pTmp1!=aTmp ){
108868	109652	fts3PoslistCopy(pp, &aTmp);
108869	109653	}else if( pTmp2!=aTmp2 ){
		@@ -108905,11 +109689,12 @@
108905	109689	char aBuffer, / Pre-allocated output buffer */
108906	109690	int pnBuffer, / OUT: Bytes written to aBuffer */
108907	109691	char a1, / Buffer containing first doclist */
108908	109692	int n1, /* Size of buffer a1 */
108909	109693	char a2, / Buffer containing second doclist */
108910		- int n2 /* Size of buffer a2 */
	109694	+ int n2, /* Size of buffer a2 */
	109695	+ int pnDoc / OUT: Number of docids in output */
108911	109696	){
108912	109697	sqlite3_int64 i1 = 0;
108913	109698	sqlite3_int64 i2 = 0;
108914	109699	sqlite3_int64 iPrev = 0;
108915	109700
		@@ -108916,10 +109701,11 @@
108916	109701	char *p = aBuffer;
108917	109702	char *p1 = a1;
108918	109703	char *p2 = a2;
108919	109704	char *pEnd1 = &a1[n1];
108920	109705	char *pEnd2 = &a2[n2];
	109706	+ int nDoc = 0;
108921	109707
108922	109708	assert( mergetype==MERGE_OR \|\| mergetype==MERGE_POS_OR
108923	109709	\|\| mergetype==MERGE_AND \|\| mergetype==MERGE_NOT
108924	109710	\|\| mergetype==MERGE_PHRASE \|\| mergetype==MERGE_POS_PHRASE
108925	109711	\|\| mergetype==MERGE_NEAR \|\| mergetype==MERGE_POS_NEAR
		@@ -108959,10 +109745,11 @@
108959	109745	while( p1 && p2 ){
108960	109746	if( i1==i2 ){
108961	109747	fts3PutDeltaVarint(&p, &iPrev, i1);
108962	109748	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108963	109749	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
	109750	+ nDoc++;
108964	109751	}else if( i1<i2 ){
108965	109752	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108966	109753	}else{
108967	109754	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
108968	109755	}
		@@ -108989,13 +109776,15 @@
108989	109776	while( p1 && p2 ){
108990	109777	if( i1==i2 ){
108991	109778	char *pSave = p;
108992	109779	sqlite3_int64 iPrevSave = iPrev;
108993	109780	fts3PutDeltaVarint(&p, &iPrev, i1);
108994		- if( 0==fts3PoslistPhraseMerge(ppPos, 1, 0, &p1, &p2) ){
	109781	+ if( 0==fts3PoslistPhraseMerge(ppPos, nParam1, 0, 1, &p1, &p2) ){
108995	109782	p = pSave;
108996	109783	iPrev = iPrevSave;
	109784	+ }else{
	109785	+ nDoc++;
108997	109786	}
108998	109787	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108999	109788	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
109000	109789	}else if( i1<i2 ){
109001	109790	fts3PoslistCopy(0, &p1);
		@@ -109044,10 +109833,11 @@
109044	109833	sqlite3_free(aTmp);
109045	109834	break;
109046	109835	}
109047	109836	}
109048	109837
	109838	+ if( pnDoc ) *pnDoc = nDoc;
109049	109839	*pnBuffer = (int)(p-aBuffer);
109050	109840	return SQLITE_OK;
109051	109841	}
109052	109842
109053	109843	/*
		@@ -109082,20 +109872,20 @@
109082	109872	for(i=0; i<SizeofArray(pTS->aaOutput); i++){
109083	109873	if( pTS->aaOutput[i] ){
109084	109874	if( !aOut ){
109085	109875	aOut = pTS->aaOutput[i];
109086	109876	nOut = pTS->anOutput[i];
109087		- pTS->aaOutput[0] = 0;
	109877	+ pTS->aaOutput[i] = 0;
109088	109878	}else{
109089	109879	int nNew = nOut + pTS->anOutput[i];
109090	109880	char *aNew = sqlite3_malloc(nNew);
109091	109881	if( !aNew ){
109092	109882	sqlite3_free(aOut);
109093	109883	return SQLITE_NOMEM;
109094	109884	}
109095	109885	fts3DoclistMerge(mergetype, 0, 0,
109096		- aNew, &nNew, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut
	109886	+ aNew, &nNew, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut, 0
109097	109887	);
109098	109888	sqlite3_free(pTS->aaOutput[i]);
109099	109889	sqlite3_free(aOut);
109100	109890	pTS->aaOutput[i] = 0;
109101	109891	aOut = aNew;
		@@ -109162,12 +109952,12 @@
109162	109952	if( aMerge!=aDoclist ){
109163	109953	sqlite3_free(aMerge);
109164	109954	}
109165	109955	return SQLITE_NOMEM;
109166	109956	}
109167		- fts3DoclistMerge(mergetype, 0, 0,
109168		- aNew, &nNew, pTS->aaOutput[iOut], pTS->anOutput[iOut], aMerge, nMerge
	109957	+ fts3DoclistMerge(mergetype, 0, 0, aNew, &nNew,
	109958	+ pTS->aaOutput[iOut], pTS->anOutput[iOut], aMerge, nMerge, 0
109169	109959	);
109170	109960
109171	109961	if( iOut>0 ) sqlite3_free(aMerge);
109172	109962	sqlite3_free(pTS->aaOutput[iOut]);
109173	109963	pTS->aaOutput[iOut] = 0;
		@@ -109180,10 +109970,165 @@
109180	109970	}
109181	109971	}
109182	109972	}
109183	109973	return SQLITE_OK;
109184	109974	}
	109975	+
	109976	+static int fts3DeferredTermSelect(
	109977	+ Fts3DeferredToken pToken, / Phrase token */
	109978	+ int isTermPos, /* True to include positions */
	109979	+ int pnOut, / OUT: Size of list */
	109980	+ char *ppOut / OUT: Body of list */
	109981	+){
	109982	+ char *aSource;
	109983	+ int nSource;
	109984	+
	109985	+ aSource = sqlite3Fts3DeferredDoclist(pToken, &nSource);
	109986	+ if( !aSource ){
	109987	+ *pnOut = 0;
	109988	+ *ppOut = 0;
	109989	+ }else if( isTermPos ){
	109990	+ *ppOut = sqlite3_malloc(nSource);
	109991	+ if( !*ppOut ) return SQLITE_NOMEM;
	109992	+ memcpy(*ppOut, aSource, nSource);
	109993	+ *pnOut = nSource;
	109994	+ }else{
	109995	+ sqlite3_int64 docid;
	109996	+ *pnOut = sqlite3Fts3GetVarint(aSource, &docid);
	109997	+ ppOut = sqlite3_malloc(pnOut);
	109998	+ if( !*ppOut ) return SQLITE_NOMEM;
	109999	+ sqlite3Fts3PutVarint(*ppOut, docid);
	110000	+ }
	110001	+
	110002	+ return SQLITE_OK;
	110003	+}
	110004	+
	110005	+/*
	110006	+** An Fts3SegReaderArray is used to store an array of Fts3SegReader objects.
	110007	+** Elements are added to the array using fts3SegReaderArrayAdd().
	110008	+*/
	110009	+struct Fts3SegReaderArray {
	110010	+ int nSegment; /* Number of valid entries in apSegment[] */
	110011	+ int nAlloc; /* Allocated size of apSegment[] */
	110012	+ int nCost; /* The cost of executing SegReaderIterate() */
	110013	+ Fts3SegReader apSegment[1]; / Array of seg-reader objects */
	110014	+};
	110015	+
	110016	+
	110017	+/*
	110018	+** Free an Fts3SegReaderArray object. Also free all seg-readers in the
	110019	+** array (using sqlite3Fts3SegReaderFree()).
	110020	+*/
	110021	+static void fts3SegReaderArrayFree(Fts3SegReaderArray *pArray){
	110022	+ if( pArray ){
	110023	+ int i;
	110024	+ for(i=0; i<pArray->nSegment; i++){
	110025	+ sqlite3Fts3SegReaderFree(0, pArray->apSegment[i]);
	110026	+ }
	110027	+ sqlite3_free(pArray);
	110028	+ }
	110029	+}
	110030	+
	110031	+static int fts3SegReaderArrayAdd(
	110032	+ Fts3SegReaderArray **ppArray,
	110033	+ Fts3SegReader *pNew
	110034	+){
	110035	+ Fts3SegReaderArray pArray = ppArray;
	110036	+
	110037	+ if( !pArray \|\| pArray->nAlloc==pArray->nSegment ){
	110038	+ int nNew = (pArray ? pArray->nAlloc+16 : 16);
	110039	+ pArray = (Fts3SegReaderArray *)sqlite3_realloc(pArray,
	110040	+ sizeof(Fts3SegReaderArray) + (nNew-1) * sizeof(Fts3SegReader*)
	110041	+ );
	110042	+ if( !pArray ){
	110043	+ sqlite3Fts3SegReaderFree(0, pNew);
	110044	+ return SQLITE_NOMEM;
	110045	+ }
	110046	+ if( nNew==16 ){
	110047	+ pArray->nSegment = 0;
	110048	+ pArray->nCost = 0;
	110049	+ }
	110050	+ pArray->nAlloc = nNew;
	110051	+ *ppArray = pArray;
	110052	+ }
	110053	+
	110054	+ pArray->apSegment[pArray->nSegment++] = pNew;
	110055	+ return SQLITE_OK;
	110056	+}
	110057	+
	110058	+static int fts3TermSegReaderArray(
	110059	+ Fts3Cursor pCsr, / Virtual table cursor handle */
	110060	+ const char zTerm, / Term to query for */
	110061	+ int nTerm, /* Size of zTerm in bytes */
	110062	+ int isPrefix, /* True for a prefix search */
	110063	+ Fts3SegReaderArray *ppArray / OUT: Allocated seg-reader array */
	110064	+){
	110065	+ Fts3Table p = (Fts3Table )pCsr->base.pVtab;
	110066	+ int rc; /* Return code */
	110067	+ Fts3SegReaderArray pArray = 0; / Array object to build */
	110068	+ Fts3SegReader pReader = 0; / Seg-reader to add to pArray */
	110069	+ sqlite3_stmt pStmt = 0; / SQL statement to scan %_segdir table */
	110070	+ int iAge = 0; /* Used to assign ages to segments */
	110071	+
	110072	+ /* Allocate a seg-reader to scan the pending terms, if any. */
	110073	+ rc = sqlite3Fts3SegReaderPending(p, zTerm, nTerm, isPrefix, &pReader);
	110074	+ if( rc==SQLITE_OK && pReader ) {
	110075	+ rc = fts3SegReaderArrayAdd(&pArray, pReader);
	110076	+ }
	110077	+
	110078	+ /* Loop through the entire %_segdir table. For each segment, create a
	110079	+ ** Fts3SegReader to iterate through the subset of the segment leaves
	110080	+ ** that may contain a term that matches zTerm/nTerm. For non-prefix
	110081	+ ** searches, this is always a single leaf. For prefix searches, this
	110082	+ ** may be a contiguous block of leaves.
	110083	+ */
	110084	+ if( rc==SQLITE_OK ){
	110085	+ rc = sqlite3Fts3AllSegdirs(p, &pStmt);
	110086	+ }
	110087	+ while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){
	110088	+ Fts3SegReader *pNew = 0;
	110089	+ int nRoot = sqlite3_column_bytes(pStmt, 4);
	110090	+ char const *zRoot = sqlite3_column_blob(pStmt, 4);
	110091	+ if( sqlite3_column_int64(pStmt, 1)==0 ){
	110092	+ /* The entire segment is stored on the root node (which must be a
	110093	+ ** leaf). Do not bother inspecting any data in this case, just
	110094	+ ** create a Fts3SegReader to scan the single leaf.
	110095	+ */
	110096	+ rc = sqlite3Fts3SegReaderNew(p, iAge, 0, 0, 0, zRoot, nRoot, &pNew);
	110097	+ }else{
	110098	+ sqlite3_int64 i1; /* First leaf that may contain zTerm */
	110099	+ sqlite3_int64 i2; /* Final leaf that may contain zTerm */
	110100	+ rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &i1, (isPrefix?&i2:0));
	110101	+ if( isPrefix==0 ) i2 = i1;
	110102	+ if( rc==SQLITE_OK ){
	110103	+ rc = sqlite3Fts3SegReaderNew(p, iAge, i1, i2, 0, 0, 0, &pNew);
	110104	+ }
	110105	+ }
	110106	+ assert( (pNew==0)==(rc!=SQLITE_OK) );
	110107	+
	110108	+ /* If a new Fts3SegReader was allocated, add it to the array. */
	110109	+ if( rc==SQLITE_OK ){
	110110	+ rc = fts3SegReaderArrayAdd(&pArray, pNew);
	110111	+ }
	110112	+ if( rc==SQLITE_OK ){
	110113	+ rc = sqlite3Fts3SegReaderCost(pCsr, pNew, &pArray->nCost);
	110114	+ }
	110115	+ iAge++;
	110116	+ }
	110117	+
	110118	+ if( rc==SQLITE_DONE ){
	110119	+ rc = sqlite3_reset(pStmt);
	110120	+ }else{
	110121	+ sqlite3_reset(pStmt);
	110122	+ }
	110123	+ if( rc!=SQLITE_OK ){
	110124	+ fts3SegReaderArrayFree(pArray);
	110125	+ pArray = 0;
	110126	+ }
	110127	+ *ppArray = pArray;
	110128	+ return rc;
	110129	+}
109185	110130
109186	110131	/*
109187	110132	** This function retreives the doclist for the specified term (or term
109188	110133	** prefix) from the database.
109189	110134	**
		@@ -109194,145 +110139,150 @@
109194	110139	** in the database without the found length specifier at the start of on-disk
109195	110140	** doclists.
109196	110141	*/
109197	110142	static int fts3TermSelect(
109198	110143	Fts3Table p, / Virtual table handle */
	110144	+ Fts3PhraseToken pTok, / Token to query for */
109199	110145	int iColumn, /* Column to query (or -ve for all columns) */
109200		- const char zTerm, / Term to query for */
109201		- int nTerm, /* Size of zTerm in bytes */
109202		- int isPrefix, /* True for a prefix search */
109203	110146	int isReqPos, /* True to include position lists in output */
109204	110147	int pnOut, / OUT: Size of buffer at ppOut /
109205	110148	char *ppOut / OUT: Malloced result buffer */
109206	110149	){
109207		- int i;
109208		- TermSelect tsc;
109209		- Fts3SegFilter filter; /* Segment term filter configuration */
109210		- Fts3SegReader *apSegment; / Array of segments to read data from */
109211		- int nSegment = 0; /* Size of apSegment array */
109212		- int nAlloc = 16; /* Allocated size of segment array */
109213	110150	int rc; /* Return code */
109214		- sqlite3_stmt pStmt = 0; / SQL statement to scan %_segdir table */
109215		- int iAge = 0; /* Used to assign ages to segments */
109216		-
109217		- apSegment = (Fts3SegReader *)sqlite3_malloc(sizeof(Fts3SegReader)*nAlloc);
109218		- if( !apSegment ) return SQLITE_NOMEM;
109219		- rc = sqlite3Fts3SegReaderPending(p, zTerm, nTerm, isPrefix, &apSegment[0]);
109220		- if( rc!=SQLITE_OK ) goto finished;
109221		- if( apSegment[0] ){
109222		- nSegment = 1;
109223		- }
109224		-
109225		- /* Loop through the entire %_segdir table. For each segment, create a
109226		- ** Fts3SegReader to iterate through the subset of the segment leaves
109227		- ** that may contain a term that matches zTerm/nTerm. For non-prefix
109228		- ** searches, this is always a single leaf. For prefix searches, this
109229		- ** may be a contiguous block of leaves.
109230		- **
109231		- ** The code in this loop does not actually load any leaves into memory
109232		- ** (unless the root node happens to be a leaf). It simply examines the
109233		- ** b-tree structure to determine which leaves need to be inspected.
109234		- */
109235		- rc = sqlite3Fts3AllSegdirs(p, &pStmt);
109236		- while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){
109237		- Fts3SegReader *pNew = 0;
109238		- int nRoot = sqlite3_column_bytes(pStmt, 4);
109239		- char const *zRoot = sqlite3_column_blob(pStmt, 4);
109240		- if( sqlite3_column_int64(pStmt, 1)==0 ){
109241		- /* The entire segment is stored on the root node (which must be a
109242		- ** leaf). Do not bother inspecting any data in this case, just
109243		- ** create a Fts3SegReader to scan the single leaf.
109244		- */
109245		- rc = sqlite3Fts3SegReaderNew(p, iAge, 0, 0, 0, zRoot, nRoot, &pNew);
109246		- }else{
109247		- int rc2; /* Return value of sqlite3Fts3ReadBlock() */
109248		- sqlite3_int64 i1; /* Blockid of leaf that may contain zTerm */
109249		- rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &i1);
109250		- if( rc==SQLITE_OK ){
109251		- sqlite3_int64 i2 = sqlite3_column_int64(pStmt, 2);
109252		- rc = sqlite3Fts3SegReaderNew(p, iAge, i1, i2, 0, 0, 0, &pNew);
109253		- }
109254		-
109255		- /* The following call to ReadBlock() serves to reset the SQL statement
109256		- ** used to retrieve blocks of data from the %_segments table. If it is
109257		- ** not reset here, then it may remain classified as an active statement
109258		- ** by SQLite, which may lead to "DROP TABLE" or "DETACH" commands
109259		- ** failing.
109260		- */
109261		- rc2 = sqlite3Fts3ReadBlock(p, 0, 0, 0);
109262		- if( rc==SQLITE_OK ){
109263		- rc = rc2;
109264		- }
109265		- }
109266		- iAge++;
109267		-
109268		- /* If a new Fts3SegReader was allocated, add it to the apSegment array. */
109269		- assert( pNew!=0 \|\| rc!=SQLITE_OK );
109270		- if( pNew ){
109271		- if( nSegment==nAlloc ){
109272		- Fts3SegReader **pArray;
109273		- nAlloc += 16;
109274		- pArray = (Fts3SegReader **)sqlite3_realloc(
109275		- apSegment, nAllocsizeof(Fts3SegReader )
109276		- );
109277		- if( !pArray ){
109278		- sqlite3Fts3SegReaderFree(p, pNew);
109279		- rc = SQLITE_NOMEM;
109280		- goto finished;
109281		- }
109282		- apSegment = pArray;
109283		- }
109284		- apSegment[nSegment++] = pNew;
109285		- }
109286		- }
109287		- if( rc!=SQLITE_DONE ){
109288		- assert( rc!=SQLITE_OK );
109289		- goto finished;
109290		- }
109291		-
	110151	+ Fts3SegReaderArray pArray; / Seg-reader array for this term */
	110152	+ TermSelect tsc; /* Context object for fts3TermSelectCb() */
	110153	+ Fts3SegFilter filter; /* Segment term filter configuration */
	110154	+
	110155	+ pArray = pTok->pArray;
109292	110156	memset(&tsc, 0, sizeof(TermSelect));
109293	110157	tsc.isReqPos = isReqPos;
109294	110158
109295	110159	filter.flags = FTS3_SEGMENT_IGNORE_EMPTY
109296		- \| (isPrefix ? FTS3_SEGMENT_PREFIX : 0)
	110160	+ \| (pTok->isPrefix ? FTS3_SEGMENT_PREFIX : 0)
109297	110161	\| (isReqPos ? FTS3_SEGMENT_REQUIRE_POS : 0)
109298	110162	\| (iColumn<p->nColumn ? FTS3_SEGMENT_COLUMN_FILTER : 0);
109299	110163	filter.iCol = iColumn;
109300		- filter.zTerm = zTerm;
109301		- filter.nTerm = nTerm;
	110164	+ filter.zTerm = pTok->z;
	110165	+ filter.nTerm = pTok->n;
109302	110166
109303		- rc = sqlite3Fts3SegReaderIterate(p, apSegment, nSegment, &filter,
109304		- fts3TermSelectCb, (void *)&tsc
	110167	+ rc = sqlite3Fts3SegReaderIterate(p, pArray->apSegment, pArray->nSegment,
	110168	+ &filter, fts3TermSelectCb, (void *)&tsc
109305	110169	);
109306	110170	if( rc==SQLITE_OK ){
109307	110171	rc = fts3TermSelectMerge(&tsc);
109308	110172	}
109309	110173
109310	110174	if( rc==SQLITE_OK ){
109311	110175	*ppOut = tsc.aaOutput[0];
109312	110176	*pnOut = tsc.anOutput[0];
109313	110177	}else{
	110178	+ int i;
109314	110179	for(i=0; i<SizeofArray(tsc.aaOutput); i++){
109315	110180	sqlite3_free(tsc.aaOutput[i]);
109316	110181	}
109317	110182	}
109318	110183
109319		-finished:
109320		- sqlite3_reset(pStmt);
109321		- for(i=0; i<nSegment; i++){
109322		- sqlite3Fts3SegReaderFree(p, apSegment[i]);
109323		- }
109324		- sqlite3_free(apSegment);
	110184	+ fts3SegReaderArrayFree(pArray);
	110185	+ pTok->pArray = 0;
109325	110186	return rc;
109326	110187	}
109327	110188
	110189	+/*
	110190	+** This function counts the total number of docids in the doclist stored
	110191	+** in buffer aList[], size nList bytes.
	110192	+**
	110193	+** If the isPoslist argument is true, then it is assumed that the doclist
	110194	+** contains a position-list following each docid. Otherwise, it is assumed
	110195	+** that the doclist is simply a list of docids stored as delta encoded
	110196	+** varints.
	110197	+*/
	110198	+static int fts3DoclistCountDocids(int isPoslist, char *aList, int nList){
	110199	+ int nDoc = 0; /* Return value */
	110200	+ if( aList ){
	110201	+ char aEnd = &aList[nList]; / Pointer to one byte after EOF */
	110202	+ char p = aList; / Cursor */
	110203	+ if( !isPoslist ){
	110204	+ /* The number of docids in the list is the same as the number of
	110205	+ ** varints. In FTS3 a varint consists of a single byte with the 0x80
	110206	+ ** bit cleared and zero or more bytes with the 0x80 bit set. So to
	110207	+ ** count the varints in the buffer, just count the number of bytes
	110208	+ ** with the 0x80 bit clear. */
	110209	+ while( p<aEnd ) nDoc += (((*p++)&0x80)==0);
	110210	+ }else{
	110211	+ while( p<aEnd ){
	110212	+ nDoc++;
	110213	+ while( (p++)&0x80 ); / Skip docid varint */
	110214	+ fts3PoslistCopy(0, &p); /* Skip over position list */
	110215	+ }
	110216	+ }
	110217	+ }
	110218	+
	110219	+ return nDoc;
	110220	+}
	110221	+
	110222	+/*
	110223	+** Call sqlite3Fts3DeferToken() for each token in the expression pExpr.
	110224	+*/
	110225	+static int fts3DeferExpression(Fts3Cursor pCsr, Fts3Expr pExpr){
	110226	+ int rc = SQLITE_OK;
	110227	+ if( pExpr ){
	110228	+ rc = fts3DeferExpression(pCsr, pExpr->pLeft);
	110229	+ if( rc==SQLITE_OK ){
	110230	+ rc = fts3DeferExpression(pCsr, pExpr->pRight);
	110231	+ }
	110232	+ if( pExpr->eType==FTSQUERY_PHRASE ){
	110233	+ int iCol = pExpr->pPhrase->iColumn;
	110234	+ int i;
	110235	+ for(i=0; rc==SQLITE_OK && i<pExpr->pPhrase->nToken; i++){
	110236	+ Fts3PhraseToken *pToken = &pExpr->pPhrase->aToken[i];
	110237	+ if( pToken->pDeferred==0 ){
	110238	+ rc = sqlite3Fts3DeferToken(pCsr, pToken, iCol);
	110239	+ }
	110240	+ }
	110241	+ }
	110242	+ }
	110243	+ return rc;
	110244	+}
	110245	+
	110246	+/*
	110247	+** This function removes the position information from a doclist. When
	110248	+** called, buffer aList (size *pnList bytes) contains a doclist that includes
	110249	+** position information. This function removes the position information so
	110250	+** that aList contains only docids, and adjusts *pnList to reflect the new
	110251	+** (possibly reduced) size of the doclist.
	110252	+*/
	110253	+static void fts3DoclistStripPositions(
	110254	+ char aList, / IN/OUT: Buffer containing doclist */
	110255	+ int pnList / IN/OUT: Size of doclist in bytes */
	110256	+){
	110257	+ if( aList ){
	110258	+ char aEnd = &aList[pnList]; /* Pointer to one byte after EOF */
	110259	+ char p = aList; / Input cursor */
	110260	+ char pOut = aList; / Output cursor */
	110261	+
	110262	+ while( p<aEnd ){
	110263	+ sqlite3_int64 delta;
	110264	+ p += sqlite3Fts3GetVarint(p, &delta);
	110265	+ fts3PoslistCopy(0, &p);
	110266	+ pOut += sqlite3Fts3PutVarint(pOut, delta);
	110267	+ }
	110268	+
	110269	+ *pnList = (pOut - aList);
	110270	+ }
	110271	+}
109328	110272
109329	110273	/*
109330	110274	** Return a DocList corresponding to the phrase *pPhrase.
	110275	+**
	110276	+** If this function returns SQLITE_OK, but *pnOut is set to a negative value,
	110277	+** then no tokens in the phrase were looked up in the full-text index. This
	110278	+** is only possible when this function is called from within xFilter(). The
	110279	+** caller should assume that all documents match the phrase. The actual
	110280	+** filtering will take place in xNext().
109331	110281	*/
109332	110282	static int fts3PhraseSelect(
109333		- Fts3Table p, / Virtual table handle */
	110283	+ Fts3Cursor pCsr, / Virtual table cursor handle */
109334	110284	Fts3Phrase pPhrase, / Phrase to return a doclist for */
109335	110285	int isReqPos, /* True if output should contain positions */
109336	110286	char *paOut, / OUT: Pointer to malloc'd result buffer */
109337	110287	int pnOut / OUT: Size of buffer at paOut /
109338	110288	){
		@@ -109340,54 +110290,156 @@
109340	110290	int nOut = 0;
109341	110291	int rc = SQLITE_OK;
109342	110292	int ii;
109343	110293	int iCol = pPhrase->iColumn;
109344	110294	int isTermPos = (pPhrase->nToken>1 \|\| isReqPos);
	110295	+ Fts3Table p = (Fts3Table )pCsr->base.pVtab;
	110296	+ int isFirst = 1;
	110297	+
	110298	+ int iPrevTok = 0;
	110299	+ int nDoc = 0;
	110300	+
	110301	+ /* If this is an xFilter() evaluation, create a segment-reader for each
	110302	+ ** phrase token. Or, if this is an xNext() or snippet/offsets/matchinfo
	110303	+ ** evaluation, only create segment-readers if there are no Fts3DeferredToken
	110304	+ ** objects attached to the phrase-tokens.
	110305	+ */
	110306	+ for(ii=0; ii<pPhrase->nToken; ii++){
	110307	+ Fts3PhraseToken *pTok = &pPhrase->aToken[ii];
	110308	+ if( pTok->pArray==0 ){
	110309	+ if( (pCsr->eEvalmode==FTS3_EVAL_FILTER)
	110310	+ \|\| (pCsr->eEvalmode==FTS3_EVAL_NEXT && pCsr->pDeferred==0)
	110311	+ \|\| (pCsr->eEvalmode==FTS3_EVAL_MATCHINFO && pTok->bFulltext)
	110312	+ ){
	110313	+ rc = fts3TermSegReaderArray(
	110314	+ pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pArray
	110315	+ );
	110316	+ if( rc!=SQLITE_OK ) return rc;
	110317	+ }
	110318	+ }
	110319	+ }
109345	110320
109346	110321	for(ii=0; ii<pPhrase->nToken; ii++){
109347		- struct PhraseToken *pTok = &pPhrase->aToken[ii];
109348		- char z = pTok->z; / Next token of the phrase */
109349		- int n = pTok->n; /* Size of z in bytes */
109350		- int isPrefix = pTok->isPrefix;/* True if token is a prefix */
	110322	+ Fts3PhraseToken pTok; / Token to find doclist for */
	110323	+ int iTok; /* The token being queried this iteration */
109351	110324	char pList; / Pointer to token doclist */
109352	110325	int nList; /* Size of buffer at pList */
109353	110326
109354		- rc = fts3TermSelect(p, iCol, z, n, isPrefix, isTermPos, &nList, &pList);
	110327	+ /* Select a token to process. If this is an xFilter() call, then tokens
	110328	+ ** are processed in order from least to most costly. Otherwise, tokens
	110329	+ ** are processed in the order in which they occur in the phrase.
	110330	+ */
	110331	+ if( pCsr->eEvalmode==FTS3_EVAL_MATCHINFO ){
	110332	+ assert( isReqPos );
	110333	+ iTok = ii;
	110334	+ pTok = &pPhrase->aToken[iTok];
	110335	+ if( pTok->bFulltext==0 ) continue;
	110336	+ }else if( pCsr->eEvalmode==FTS3_EVAL_NEXT \|\| isReqPos ){
	110337	+ iTok = ii;
	110338	+ pTok = &pPhrase->aToken[iTok];
	110339	+ }else{
	110340	+ int nMinCost = 0x7FFFFFFF;
	110341	+ int jj;
	110342	+
	110343	+ /* Find the remaining token with the lowest cost. */
	110344	+ for(jj=0; jj<pPhrase->nToken; jj++){
	110345	+ Fts3SegReaderArray *pArray = pPhrase->aToken[jj].pArray;
	110346	+ if( pArray && pArray->nCost<nMinCost ){
	110347	+ iTok = jj;
	110348	+ nMinCost = pArray->nCost;
	110349	+ }
	110350	+ }
	110351	+ pTok = &pPhrase->aToken[iTok];
	110352	+
	110353	+ /* This branch is taken if it is determined that loading the doclist
	110354	+ ** for the next token would require more IO than loading all documents
	110355	+ ** currently identified by doclist pOut/nOut. No further doclists will
	110356	+ ** be loaded from the full-text index for this phrase.
	110357	+ */
	110358	+ if( nMinCost>nDoc && ii>0 ){
	110359	+ rc = fts3DeferExpression(pCsr, pCsr->pExpr);
	110360	+ break;
	110361	+ }
	110362	+ }
	110363	+
	110364	+ if( pCsr->eEvalmode==FTS3_EVAL_NEXT && pTok->pDeferred ){
	110365	+ rc = fts3DeferredTermSelect(pTok->pDeferred, isTermPos, &nList, &pList);
	110366	+ }else{
	110367	+ assert( pTok->pArray );
	110368	+ rc = fts3TermSelect(p, pTok, iCol, isTermPos, &nList, &pList);
	110369	+ pTok->bFulltext = 1;
	110370	+ }
	110371	+ assert( rc!=SQLITE_OK \|\| pCsr->eEvalmode \|\| pTok->pArray==0 );
109355	110372	if( rc!=SQLITE_OK ) break;
109356	110373
109357		- if( ii==0 ){
	110374	+ if( isFirst ){
109358	110375	pOut = pList;
109359	110376	nOut = nList;
	110377	+ if( pCsr->eEvalmode==FTS3_EVAL_FILTER && pPhrase->nToken>1 ){
	110378	+ nDoc = fts3DoclistCountDocids(1, pOut, nOut);
	110379	+ }
	110380	+ isFirst = 0;
	110381	+ iPrevTok = iTok;
109360	110382	}else{
109361		- /* Merge the new term list and the current output. If this is the
109362		- ** last term in the phrase, and positions are not required in the
109363		- ** output of this function, the positions can be dropped as part
109364		- ** of this merge. Either way, the result of this merge will be
109365		- ** smaller than nList bytes. The code in fts3DoclistMerge() is written
109366		- ** so that it is safe to use pList as the output as well as an input
109367		- ** in this case.
	110383	+ /* Merge the new term list and the current output. */
	110384	+ char aLeft, aRight;
	110385	+ int nLeft, nRight;
	110386	+ int nDist;
	110387	+ int mt;
	110388	+
	110389	+ /* If this is the final token of the phrase, and positions were not
	110390	+ ** requested by the caller, use MERGE_PHRASE instead of POS_PHRASE.
	110391	+ ** This drops the position information from the output list.
109368	110392	*/
109369		- int mergetype = MERGE_POS_PHRASE;
109370		- if( ii==pPhrase->nToken-1 && !isReqPos ){
109371		- mergetype = MERGE_PHRASE;
	110393	+ mt = MERGE_POS_PHRASE;
	110394	+ if( ii==pPhrase->nToken-1 && !isReqPos ) mt = MERGE_PHRASE;
	110395	+
	110396	+ assert( iPrevTok!=iTok );
	110397	+ if( iPrevTok<iTok ){
	110398	+ aLeft = pOut;
	110399	+ nLeft = nOut;
	110400	+ aRight = pList;
	110401	+ nRight = nList;
	110402	+ nDist = iTok-iPrevTok;
	110403	+ iPrevTok = iTok;
	110404	+ }else{
	110405	+ aRight = pOut;
	110406	+ nRight = nOut;
	110407	+ aLeft = pList;
	110408	+ nLeft = nList;
	110409	+ nDist = iPrevTok-iTok;
109372	110410	}
109373		- fts3DoclistMerge(mergetype, 0, 0, pList, &nOut, pOut, nOut, pList, nList);
109374		- sqlite3_free(pOut);
109375		- pOut = pList;
	110411	+ pOut = aRight;
	110412	+ fts3DoclistMerge(
	110413	+ mt, nDist, 0, pOut, &nOut, aLeft, nLeft, aRight, nRight, &nDoc
	110414	+ );
	110415	+ sqlite3_free(aLeft);
109376	110416	}
109377	110417	assert( nOut==0 \|\| pOut!=0 );
109378	110418	}
109379	110419
109380	110420	if( rc==SQLITE_OK ){
	110421	+ if( ii!=pPhrase->nToken ){
	110422	+ assert( pCsr->eEvalmode==FTS3_EVAL_FILTER && isReqPos==0 );
	110423	+ fts3DoclistStripPositions(pOut, &nOut);
	110424	+ }
109381	110425	*paOut = pOut;
109382	110426	*pnOut = nOut;
109383	110427	}else{
109384	110428	sqlite3_free(pOut);
109385	110429	}
109386	110430	return rc;
109387	110431	}
109388	110432
	110433	+/*
	110434	+** This function merges two doclists according to the requirements of a
	110435	+** NEAR operator.
	110436	+**
	110437	+** Both input doclists must include position information. The output doclist
	110438	+** includes position information if the first argument to this function
	110439	+** is MERGE_POS_NEAR, or does not if it is MERGE_NEAR.
	110440	+*/
109389	110441	static int fts3NearMerge(
109390	110442	int mergetype, /* MERGE_POS_NEAR or MERGE_NEAR */
109391	110443	int nNear, /* Parameter to NEAR operator */
109392	110444	int nTokenLeft, /* Number of tokens in LHS phrase arg */
109393	110445	char aLeft, / Doclist for LHS (incl. positions) */
		@@ -109396,21 +110448,21 @@
109396	110448	char aRight, / As aLeft */
109397	110449	int nRight, /* As nRight */
109398	110450	char *paOut, / OUT: Results of merge (malloced) */
109399	110451	int pnOut / OUT: Sized of output buffer */
109400	110452	){
109401		- char *aOut;
109402		- int rc;
	110453	+ char aOut; / Buffer to write output doclist to */
	110454	+ int rc; /* Return code */
109403	110455
109404	110456	assert( mergetype==MERGE_POS_NEAR \|\| MERGE_NEAR );
109405	110457
109406	110458	aOut = sqlite3_malloc(nLeft+nRight+1);
109407	110459	if( aOut==0 ){
109408	110460	rc = SQLITE_NOMEM;
109409	110461	}else{
109410	110462	rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft,
109411		- aOut, pnOut, aLeft, nLeft, aRight, nRight
	110463	+ aOut, pnOut, aLeft, nLeft, aRight, nRight, 0
109412	110464	);
109413	110465	if( rc!=SQLITE_OK ){
109414	110466	sqlite3_free(aOut);
109415	110467	aOut = 0;
109416	110468	}
		@@ -109418,21 +110470,36 @@
109418	110470
109419	110471	*paOut = aOut;
109420	110472	return rc;
109421	110473	}
109422	110474
	110475	+/*
	110476	+** This function is used as part of the processing for the snippet() and
	110477	+** offsets() functions.
	110478	+**
	110479	+** Both pLeft and pRight are expression nodes of type FTSQUERY_PHRASE. Both
	110480	+** have their respective doclists (including position information) loaded
	110481	+** in Fts3Expr.aDoclist/nDoclist. This function removes all entries from
	110482	+** each doclist that are not within nNear tokens of a corresponding entry
	110483	+** in the other doclist.
	110484	+*/
109423	110485	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr pLeft, Fts3Expr pRight, int nNear){
109424		- int rc;
	110486	+ int rc; /* Return code */
	110487	+
	110488	+ assert( pLeft->eType==FTSQUERY_PHRASE );
	110489	+ assert( pRight->eType==FTSQUERY_PHRASE );
	110490	+ assert( pLeft->isLoaded && pRight->isLoaded );
	110491	+
109425	110492	if( pLeft->aDoclist==0 \|\| pRight->aDoclist==0 ){
109426	110493	sqlite3_free(pLeft->aDoclist);
109427	110494	sqlite3_free(pRight->aDoclist);
109428	110495	pRight->aDoclist = 0;
109429	110496	pLeft->aDoclist = 0;
109430	110497	rc = SQLITE_OK;
109431	110498	}else{
109432		- char *aOut;
109433		- int nOut;
	110499	+ char aOut; / Buffer in which to assemble new doclist */
	110500	+ int nOut; /* Size of buffer aOut in bytes */
109434	110501
109435	110502	rc = fts3NearMerge(MERGE_POS_NEAR, nNear,
109436	110503	pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
109437	110504	pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
109438	110505	&aOut, &nOut
		@@ -109451,19 +110518,158 @@
109451	110518	pLeft->aDoclist = aOut;
109452	110519	pLeft->nDoclist = nOut;
109453	110520	}
109454	110521	return rc;
109455	110522	}
	110523	+
109456	110524
109457	110525	/*
109458		-** Evaluate the full-text expression pExpr against fts3 table pTab. Store
109459		-** the resulting doclist in paOut and pnOut. This routine mallocs for
109460		-** the space needed to store the output. The caller is responsible for
	110526	+** Allocate an Fts3SegReaderArray for each token in the expression pExpr.
	110527	+** The allocated objects are stored in the Fts3PhraseToken.pArray member
	110528	+** variables of each token structure.
	110529	+*/
	110530	+static int fts3ExprAllocateSegReaders(
	110531	+ Fts3Cursor pCsr, / FTS3 table */
	110532	+ Fts3Expr pExpr, / Expression to create seg-readers for */
	110533	+ int pnExpr / OUT: Number of AND'd expressions */
	110534	+){
	110535	+ int rc = SQLITE_OK; /* Return code */
	110536	+
	110537	+ assert( pCsr->eEvalmode==FTS3_EVAL_FILTER );
	110538	+ if( pnExpr && pExpr->eType!=FTSQUERY_AND ){
	110539	+ (*pnExpr)++;
	110540	+ pnExpr = 0;
	110541	+ }
	110542	+
	110543	+ if( pExpr->eType==FTSQUERY_PHRASE ){
	110544	+ Fts3Phrase *pPhrase = pExpr->pPhrase;
	110545	+ int ii;
	110546	+
	110547	+ for(ii=0; rc==SQLITE_OK && ii<pPhrase->nToken; ii++){
	110548	+ Fts3PhraseToken *pTok = &pPhrase->aToken[ii];
	110549	+ if( pTok->pArray==0 ){
	110550	+ rc = fts3TermSegReaderArray(
	110551	+ pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pArray
	110552	+ );
	110553	+ }
	110554	+ }
	110555	+ }else{
	110556	+ rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pLeft, pnExpr);
	110557	+ if( rc==SQLITE_OK ){
	110558	+ rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pRight, pnExpr);
	110559	+ }
	110560	+ }
	110561	+ return rc;
	110562	+}
	110563	+
	110564	+/*
	110565	+** Free the Fts3SegReaderArray objects associated with each token in the
	110566	+** expression pExpr. In other words, this function frees the resources
	110567	+** allocated by fts3ExprAllocateSegReaders().
	110568	+*/
	110569	+static void fts3ExprFreeSegReaders(Fts3Expr *pExpr){
	110570	+ if( pExpr ){
	110571	+ Fts3Phrase *pPhrase = pExpr->pPhrase;
	110572	+ if( pPhrase ){
	110573	+ int kk;
	110574	+ for(kk=0; kk<pPhrase->nToken; kk++){
	110575	+ fts3SegReaderArrayFree(pPhrase->aToken[kk].pArray);
	110576	+ pPhrase->aToken[kk].pArray = 0;
	110577	+ }
	110578	+ }
	110579	+ fts3ExprFreeSegReaders(pExpr->pLeft);
	110580	+ fts3ExprFreeSegReaders(pExpr->pRight);
	110581	+ }
	110582	+}
	110583	+
	110584	+/*
	110585	+** Return the sum of the costs of all tokens in the expression pExpr. This
	110586	+** function must be called after Fts3SegReaderArrays have been allocated
	110587	+** for all tokens using fts3ExprAllocateSegReaders().
	110588	+*/
	110589	+int fts3ExprCost(Fts3Expr *pExpr){
	110590	+ int nCost; /* Return value */
	110591	+ if( pExpr->eType==FTSQUERY_PHRASE ){
	110592	+ Fts3Phrase *pPhrase = pExpr->pPhrase;
	110593	+ int ii;
	110594	+ nCost = 0;
	110595	+ for(ii=0; ii<pPhrase->nToken; ii++){
	110596	+ nCost += pPhrase->aToken[ii].pArray->nCost;
	110597	+ }
	110598	+ }else{
	110599	+ nCost = fts3ExprCost(pExpr->pLeft) + fts3ExprCost(pExpr->pRight);
	110600	+ }
	110601	+ return nCost;
	110602	+}
	110603	+
	110604	+/*
	110605	+** The following is a helper function (and type) for fts3EvalExpr(). It
	110606	+** must be called after Fts3SegReaders have been allocated for every token
	110607	+** in the expression. See the context it is called from in fts3EvalExpr()
	110608	+** for further explanation.
	110609	+*/
	110610	+typedef struct ExprAndCost ExprAndCost;
	110611	+struct ExprAndCost {
	110612	+ Fts3Expr *pExpr;
	110613	+ int nCost;
	110614	+};
	110615	+static void fts3ExprAssignCosts(
	110616	+ Fts3Expr pExpr, / Expression to create seg-readers for */
	110617	+ ExprAndCost *ppExprCost / OUT: Write to ppExprCost /
	110618	+){
	110619	+ if( pExpr->eType==FTSQUERY_AND ){
	110620	+ fts3ExprAssignCosts(pExpr->pLeft, ppExprCost);
	110621	+ fts3ExprAssignCosts(pExpr->pRight, ppExprCost);
	110622	+ }else{
	110623	+ (*ppExprCost)->pExpr = pExpr;
	110624	+ (*ppExprCost)->nCost = fts3ExprCost(pExpr);;
	110625	+ (*ppExprCost)++;
	110626	+ }
	110627	+}
	110628	+
	110629	+/*
	110630	+** Evaluate the full-text expression pExpr against FTS3 table pTab. Store
	110631	+** the resulting doclist in paOut and pnOut. This routine mallocs for
	110632	+** the space needed to store the output. The caller is responsible for
109461	110633	** freeing the space when it has finished.
	110634	+**
	110635	+** This function is called in two distinct contexts:
	110636	+**
	110637	+** * From within the virtual table xFilter() method. In this case, the
	110638	+** output doclist contains entries for all rows in the table, based on
	110639	+** data read from the full-text index.
	110640	+**
	110641	+** In this case, if the query expression contains one or more tokens that
	110642	+** are very common, then the returned doclist may contain a superset of
	110643	+** the documents that actually match the expression.
	110644	+**
	110645	+** * From within the virtual table xNext() method. This call is only made
	110646	+** if the call from within xFilter() found that there were very common
	110647	+** tokens in the query expression and did return a superset of the
	110648	+** matching documents. In this case the returned doclist contains only
	110649	+** entries that correspond to the current row of the table. Instead of
	110650	+** reading the data for each token from the full-text index, the data is
	110651	+** already available in-memory in the Fts3PhraseToken.pDeferred structures.
	110652	+** See fts3EvalDeferred() for how it gets there.
	110653	+**
	110654	+** In the first case above, Fts3Cursor.doDeferred==0. In the second (if it is
	110655	+** required) Fts3Cursor.doDeferred==1.
	110656	+**
	110657	+** If the SQLite invokes the snippet(), offsets() or matchinfo() function
	110658	+** as part of a SELECT on an FTS3 table, this function is called on each
	110659	+** individual phrase expression in the query. If there were very common tokens
	110660	+** found in the xFilter() call, then this function is called once for phrase
	110661	+** for each row visited, and the returned doclist contains entries for the
	110662	+** current row only. Otherwise, if there were no very common tokens, then this
	110663	+** function is called once only for each phrase in the query and the returned
	110664	+** doclist contains entries for all rows of the table.
	110665	+**
	110666	+** Fts3Cursor.doDeferred==1 when this function is called on phrases as a
	110667	+** result of a snippet(), offsets() or matchinfo() invocation.
109462	110668	*/
109463		-static int evalFts3Expr(
109464		- Fts3Table p, / Virtual table handle */
	110669	+static int fts3EvalExpr(
	110670	+ Fts3Cursor p, / Virtual table cursor handle */
109465	110671	Fts3Expr pExpr, / Parsed fts3 expression */
109466	110672	char *paOut, / OUT: Pointer to malloc'd result buffer */
109467	110673	int pnOut, / OUT: Size of buffer at paOut /
109468	110674	int isReqPos /* Require positions in output buffer */
109469	110675	){
		@@ -109472,37 +110678,101 @@
109472	110678	/* Zero the output parameters. */
109473	110679	*paOut = 0;
109474	110680	*pnOut = 0;
109475	110681
109476	110682	if( pExpr ){
109477		- assert( pExpr->eType==FTSQUERY_PHRASE
109478		- \|\| pExpr->eType==FTSQUERY_NEAR
109479		- \|\| isReqPos==0
	110683	+ assert( pExpr->eType==FTSQUERY_NEAR \|\| pExpr->eType==FTSQUERY_OR
	110684	+ \|\| pExpr->eType==FTSQUERY_AND \|\| pExpr->eType==FTSQUERY_NOT
	110685	+ \|\| pExpr->eType==FTSQUERY_PHRASE
109480	110686	);
	110687	+ assert( pExpr->eType==FTSQUERY_PHRASE \|\| isReqPos==0 );
	110688	+
109481	110689	if( pExpr->eType==FTSQUERY_PHRASE ){
109482		- rc = fts3PhraseSelect(p, pExpr->pPhrase,
	110690	+ rc = fts3PhraseSelect(p, pExpr->pPhrase,
109483	110691	isReqPos \|\| (pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR),
109484	110692	paOut, pnOut
109485	110693	);
	110694	+ fts3ExprFreeSegReaders(pExpr);
	110695	+ }else if( p->eEvalmode==FTS3_EVAL_FILTER && pExpr->eType==FTSQUERY_AND ){
	110696	+ ExprAndCost aExpr = 0; / Array of AND'd expressions and costs */
	110697	+ int nExpr = 0; /* Size of aExpr[] */
	110698	+ char aRet = 0; / Doclist to return to caller */
	110699	+ int nRet = 0; /* Length of aRet[] in bytes */
	110700	+ int nDoc = 0x7FFFFFFF;
	110701	+
	110702	+ assert( !isReqPos );
	110703	+
	110704	+ rc = fts3ExprAllocateSegReaders(p, pExpr, &nExpr);
	110705	+ if( rc==SQLITE_OK ){
	110706	+ assert( nExpr>1 );
	110707	+ aExpr = sqlite3_malloc(sizeof(ExprAndCost) * nExpr);
	110708	+ if( !aExpr ) rc = SQLITE_NOMEM;
	110709	+ }
	110710	+ if( rc==SQLITE_OK ){
	110711	+ int ii; /* Used to iterate through expressions */
	110712	+
	110713	+ fts3ExprAssignCosts(pExpr, &aExpr);
	110714	+ aExpr -= nExpr;
	110715	+ for(ii=0; ii<nExpr; ii++){
	110716	+ char *aNew;
	110717	+ int nNew;
	110718	+ int jj;
	110719	+ ExprAndCost *pBest = 0;
	110720	+
	110721	+ for(jj=0; jj<nExpr; jj++){
	110722	+ ExprAndCost *pCand = &aExpr[jj];
	110723	+ if( pCand->pExpr && (pBest==0 \|\| pCand->nCost<pBest->nCost) ){
	110724	+ pBest = pCand;
	110725	+ }
	110726	+ }
	110727	+
	110728	+ if( pBest->nCost>nDoc ){
	110729	+ rc = fts3DeferExpression(p, p->pExpr);
	110730	+ break;
	110731	+ }else{
	110732	+ rc = fts3EvalExpr(p, pBest->pExpr, &aNew, &nNew, 0);
	110733	+ if( rc!=SQLITE_OK ) break;
	110734	+ pBest->pExpr = 0;
	110735	+ if( ii==0 ){
	110736	+ aRet = aNew;
	110737	+ nRet = nNew;
	110738	+ nDoc = fts3DoclistCountDocids(0, aRet, nRet);
	110739	+ }else{
	110740	+ fts3DoclistMerge(
	110741	+ MERGE_AND, 0, 0, aRet, &nRet, aRet, nRet, aNew, nNew, &nDoc
	110742	+ );
	110743	+ sqlite3_free(aNew);
	110744	+ }
	110745	+ }
	110746	+ }
	110747	+ }
	110748	+
	110749	+ *paOut = aRet;
	110750	+ *pnOut = nRet;
	110751	+ sqlite3_free(aExpr);
	110752	+ fts3ExprFreeSegReaders(pExpr);
	110753	+
109486	110754	}else{
109487	110755	char *aLeft;
109488	110756	char *aRight;
109489	110757	int nLeft;
109490	110758	int nRight;
109491	110759
109492		- if( 0==(rc = evalFts3Expr(p, pExpr->pRight, &aRight, &nRight, isReqPos))
109493		- && 0==(rc = evalFts3Expr(p, pExpr->pLeft, &aLeft, &nLeft, isReqPos))
	110760	+ assert( pExpr->eType==FTSQUERY_NEAR
	110761	+ \|\| pExpr->eType==FTSQUERY_OR
	110762	+ \|\| pExpr->eType==FTSQUERY_NOT
	110763	+ \|\| (pExpr->eType==FTSQUERY_AND && p->eEvalmode==FTS3_EVAL_NEXT)
	110764	+ );
	110765	+
	110766	+ if( 0==(rc = fts3EvalExpr(p, pExpr->pRight, &aRight, &nRight, isReqPos))
	110767	+ && 0==(rc = fts3EvalExpr(p, pExpr->pLeft, &aLeft, &nLeft, isReqPos))
109494	110768	){
109495		- assert( pExpr->eType==FTSQUERY_NEAR \|\| pExpr->eType==FTSQUERY_OR
109496		- \|\| pExpr->eType==FTSQUERY_AND \|\| pExpr->eType==FTSQUERY_NOT
109497		- );
109498	110769	switch( pExpr->eType ){
109499	110770	case FTSQUERY_NEAR: {
109500	110771	Fts3Expr *pLeft;
109501	110772	Fts3Expr *pRight;
109502		- int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR;
109503		-
	110773	+ int mergetype = MERGE_NEAR;
109504	110774	if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
109505	110775	mergetype = MERGE_POS_NEAR;
109506	110776	}
109507	110777	pLeft = pExpr->pLeft;
109508	110778	while( pLeft->eType==FTSQUERY_NEAR ){
		@@ -109527,21 +110797,21 @@
109527	110797	** so that a buffer of zero bytes is never allocated - this can
109528	110798	** cause fts3DoclistMerge() to incorrectly return SQLITE_NOMEM.
109529	110799	*/
109530	110800	char *aBuffer = sqlite3_malloc(nRight+nLeft+1);
109531	110801	rc = fts3DoclistMerge(MERGE_OR, 0, 0, aBuffer, pnOut,
109532		- aLeft, nLeft, aRight, nRight
	110802	+ aLeft, nLeft, aRight, nRight, 0
109533	110803	);
109534	110804	*paOut = aBuffer;
109535	110805	sqlite3_free(aLeft);
109536	110806	break;
109537	110807	}
109538	110808
109539	110809	default: {
109540	110810	assert( FTSQUERY_NOT==MERGE_NOT && FTSQUERY_AND==MERGE_AND );
109541	110811	fts3DoclistMerge(pExpr->eType, 0, 0, aLeft, pnOut,
109542		- aLeft, nLeft, aRight, nRight
	110812	+ aLeft, nLeft, aRight, nRight, 0
109543	110813	);
109544	110814	*paOut = aLeft;
109545	110815	break;
109546	110816	}
109547	110817	}
		@@ -109548,10 +110818,92 @@
109548	110818	}
109549	110819	sqlite3_free(aRight);
109550	110820	}
109551	110821	}
109552	110822
	110823	+ return rc;
	110824	+}
	110825	+
	110826	+/*
	110827	+** This function is called from within xNext() for each row visited by
	110828	+** an FTS3 query. If evaluating the FTS3 query expression within xFilter()
	110829	+** was able to determine the exact set of matching rows, this function sets
	110830	+** *pbRes to true and returns SQLITE_IO immediately.
	110831	+**
	110832	+** Otherwise, if evaluating the query expression within xFilter() returned a
	110833	+** superset of the matching documents instead of an exact set (this happens
	110834	+** when the query includes very common tokens and it is deemed too expensive to
	110835	+** load their doclists from disk), this function tests if the current row
	110836	+** really does match the FTS3 query.
	110837	+**
	110838	+** If an error occurs, an SQLite error code is returned. Otherwise, SQLITE_OK
	110839	+** is returned and *pbRes is set to true if the current row matches the
	110840	+** FTS3 query (and should be included in the results returned to SQLite), or
	110841	+** false otherwise.
	110842	+*/
	110843	+static int fts3EvalDeferred(
	110844	+ Fts3Cursor pCsr, / FTS3 cursor pointing at row to test */
	110845	+ int pbRes / OUT: Set to true if row is a match */
	110846	+){
	110847	+ int rc = SQLITE_OK;
	110848	+ if( pCsr->pDeferred==0 ){
	110849	+ *pbRes = 1;
	110850	+ }else{
	110851	+ rc = fts3CursorSeek(0, pCsr);
	110852	+ if( rc==SQLITE_OK ){
	110853	+ sqlite3Fts3FreeDeferredDoclists(pCsr);
	110854	+ rc = sqlite3Fts3CacheDeferredDoclists(pCsr);
	110855	+ }
	110856	+ if( rc==SQLITE_OK ){
	110857	+ char *a = 0;
	110858	+ int n = 0;
	110859	+ rc = fts3EvalExpr(pCsr, pCsr->pExpr, &a, &n, 0);
	110860	+ assert( n>=0 );
	110861	+ *pbRes = (n>0);
	110862	+ sqlite3_free(a);
	110863	+ }
	110864	+ }
	110865	+ return rc;
	110866	+}
	110867	+
	110868	+/*
	110869	+** Advance the cursor to the next row in the %_content table that
	110870	+** matches the search criteria. For a MATCH search, this will be
	110871	+** the next row that matches. For a full-table scan, this will be
	110872	+** simply the next row in the %_content table. For a docid lookup,
	110873	+** this routine simply sets the EOF flag.
	110874	+**
	110875	+** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned
	110876	+** even if we reach end-of-file. The fts3EofMethod() will be called
	110877	+** subsequently to determine whether or not an EOF was hit.
	110878	+*/
	110879	+static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){
	110880	+ int res;
	110881	+ int rc = SQLITE_OK; /* Return code */
	110882	+ Fts3Cursor pCsr = (Fts3Cursor )pCursor;
	110883	+
	110884	+ pCsr->eEvalmode = FTS3_EVAL_NEXT;
	110885	+ do {
	110886	+ if( pCsr->aDoclist==0 ){
	110887	+ if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){
	110888	+ pCsr->isEof = 1;
	110889	+ rc = sqlite3_reset(pCsr->pStmt);
	110890	+ break;
	110891	+ }
	110892	+ pCsr->iPrevId = sqlite3_column_int64(pCsr->pStmt, 0);
	110893	+ }else{
	110894	+ if( pCsr->pNextId>=&pCsr->aDoclist[pCsr->nDoclist] ){
	110895	+ pCsr->isEof = 1;
	110896	+ break;
	110897	+ }
	110898	+ sqlite3_reset(pCsr->pStmt);
	110899	+ fts3GetDeltaVarint(&pCsr->pNextId, &pCsr->iPrevId);
	110900	+ pCsr->isRequireSeek = 1;
	110901	+ pCsr->isMatchinfoNeeded = 1;
	110902	+ }
	110903	+ }while( SQLITE_OK==(rc = fts3EvalDeferred(pCsr, &res)) && res==0 );
	110904	+
109553	110905	return rc;
109554	110906	}
109555	110907
109556	110908	/*
109557	110909	** This is the xFilter interface for the virtual table. See
		@@ -109567,15 +110919,10 @@
109567	110919	** If idxNum>=FTS3_FULLTEXT_SEARCH then use the full text index. The
109568	110920	** column on the left-hand side of the MATCH operator is column
109569	110921	** number idxNum-FTS3_FULLTEXT_SEARCH, 0 indexed. argv[0] is the right-hand
109570	110922	** side of the MATCH operator.
109571	110923	*/
109572		-/* TODO(shess) Upgrade the cursor initialization and destruction to
109573		-** account for fts3FilterMethod() being called multiple times on the
109574		-** same cursor. The current solution is very fragile. Apply fix to
109575		-** fts3 as appropriate.
109576		-*/
109577	110924	static int fts3FilterMethod(
109578	110925	sqlite3_vtab_cursor pCursor, / The cursor used for this query */
109579	110926	int idxNum, /* Strategy index */
109580	110927	const char idxStr, / Unused */
109581	110928	int nVal, /* Number of elements in apVal */
		@@ -109594,35 +110941,19 @@
109594	110941	UNUSED_PARAMETER(nVal);
109595	110942
109596	110943	assert( idxNum>=0 && idxNum<=(FTS3_FULLTEXT_SEARCH+p->nColumn) );
109597	110944	assert( nVal==0 \|\| nVal==1 );
109598	110945	assert( (nVal==0)==(idxNum==FTS3_FULLSCAN_SEARCH) );
	110946	+ assert( p->pSegments==0 );
109599	110947
109600	110948	/* In case the cursor has been used before, clear it now. */
109601	110949	sqlite3_finalize(pCsr->pStmt);
109602	110950	sqlite3_free(pCsr->aDoclist);
109603	110951	sqlite3Fts3ExprFree(pCsr->pExpr);
109604	110952	memset(&pCursor[1], 0, sizeof(Fts3Cursor)-sizeof(sqlite3_vtab_cursor));
109605	110953
109606		- /* Compile a SELECT statement for this cursor. For a full-table-scan, the
109607		- ** statement loops through all rows of the %_content table. For a
109608		- ** full-text query or docid lookup, the statement retrieves a single
109609		- ** row by docid.
109610		- */
109611		- zSql = sqlite3_mprintf(azSql[idxNum==FTS3_FULLSCAN_SEARCH], p->zDb, p->zName);
109612		- if( !zSql ){
109613		- rc = SQLITE_NOMEM;
109614		- }else{
109615		- rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0);
109616		- sqlite3_free(zSql);
109617		- }
109618		- if( rc!=SQLITE_OK ) return rc;
109619		- pCsr->eSearch = (i16)idxNum;
109620		-
109621		- if( idxNum==FTS3_DOCID_SEARCH ){
109622		- rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]);
109623		- }else if( idxNum!=FTS3_FULLSCAN_SEARCH ){
	110954	+ if( idxNum!=FTS3_DOCID_SEARCH && idxNum!=FTS3_FULLSCAN_SEARCH ){
109624	110955	int iCol = idxNum-FTS3_FULLTEXT_SEARCH;
109625	110956	const char zQuery = (const char )sqlite3_value_text(apVal[0]);
109626	110957
109627	110958	if( zQuery==0 && sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
109628	110959	return SQLITE_NOMEM;
		@@ -109640,14 +110971,33 @@
109640	110971	}
109641	110972
109642	110973	rc = sqlite3Fts3ReadLock(p);
109643	110974	if( rc!=SQLITE_OK ) return rc;
109644	110975
109645		- rc = evalFts3Expr(p, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0);
	110976	+ rc = fts3EvalExpr(pCsr, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0);
	110977	+ sqlite3Fts3SegmentsClose(p);
	110978	+ if( rc!=SQLITE_OK ) return rc;
109646	110979	pCsr->pNextId = pCsr->aDoclist;
109647	110980	pCsr->iPrevId = 0;
109648	110981	}
	110982	+
	110983	+ /* Compile a SELECT statement for this cursor. For a full-table-scan, the
	110984	+ ** statement loops through all rows of the %_content table. For a
	110985	+ ** full-text query or docid lookup, the statement retrieves a single
	110986	+ ** row by docid.
	110987	+ */
	110988	+ zSql = sqlite3_mprintf(azSql[idxNum==FTS3_FULLSCAN_SEARCH], p->zDb, p->zName);
	110989	+ if( !zSql ){
	110990	+ rc = SQLITE_NOMEM;
	110991	+ }else{
	110992	+ rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0);
	110993	+ sqlite3_free(zSql);
	110994	+ }
	110995	+ if( rc==SQLITE_OK && idxNum==FTS3_DOCID_SEARCH ){
	110996	+ rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]);
	110997	+ }
	110998	+ pCsr->eSearch = (i16)idxNum;
109649	110999
109650	111000	if( rc!=SQLITE_OK ) return rc;
109651	111001	return fts3NextMethod(pCursor);
109652	111002	}
109653	111003
		@@ -109668,10 +111018,15 @@
109668	111018	static int fts3RowidMethod(sqlite3_vtab_cursor pCursor, sqlite_int64 pRowid){
109669	111019	Fts3Cursor pCsr = (Fts3Cursor ) pCursor;
109670	111020	if( pCsr->aDoclist ){
109671	111021	*pRowid = pCsr->iPrevId;
109672	111022	}else{
	111023	+ /* This branch runs if the query is implemented using a full-table scan
	111024	+ ** (not using the full-text index). In this case grab the rowid from the
	111025	+ ** SELECT statement.
	111026	+ */
	111027	+ assert( pCsr->isRequireSeek==0 );
109673	111028	*pRowid = sqlite3_column_int64(pCsr->pStmt, 0);
109674	111029	}
109675	111030	return SQLITE_OK;
109676	111031	}
109677	111032
		@@ -109730,11 +111085,13 @@
109730	111085	/*
109731	111086	** Implementation of xSync() method. Flush the contents of the pending-terms
109732	111087	** hash-table to the database.
109733	111088	*/
109734	111089	static int fts3SyncMethod(sqlite3_vtab *pVtab){
109735		- return sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab);
	111090	+ int rc = sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab);
	111091	+ sqlite3Fts3SegmentsClose((Fts3Table *)pVtab);
	111092	+ return rc;
109736	111093	}
109737	111094
109738	111095	/*
109739	111096	** Implementation of xBegin() method. This is a no-op.
109740	111097	*/
		@@ -109768,12 +111125,31 @@
109768	111125	** Load the doclist associated with expression pExpr to pExpr->aDoclist.
109769	111126	** The loaded doclist contains positions as well as the document ids.
109770	111127	** This is used by the matchinfo(), snippet() and offsets() auxillary
109771	111128	** functions.
109772	111129	*/
109773		-SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Table pTab, Fts3Expr pExpr){
109774		- return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
	111130	+SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Cursor pCsr, Fts3Expr pExpr){
	111131	+ int rc;
	111132	+ assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase );
	111133	+ assert( pCsr->eEvalmode==FTS3_EVAL_NEXT );
	111134	+ rc = fts3EvalExpr(pCsr, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
	111135	+ return rc;
	111136	+}
	111137	+
	111138	+SQLITE_PRIVATE int sqlite3Fts3ExprLoadFtDoclist(
	111139	+ Fts3Cursor *pCsr,
	111140	+ Fts3Expr *pExpr,
	111141	+ char **paDoclist,
	111142	+ int *pnDoclist
	111143	+){
	111144	+ int rc;
	111145	+ assert( pCsr->eEvalmode==FTS3_EVAL_NEXT );
	111146	+ assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase );
	111147	+ pCsr->eEvalmode = FTS3_EVAL_MATCHINFO;
	111148	+ rc = fts3EvalExpr(pCsr, pExpr, paDoclist, pnDoclist, 1);
	111149	+ pCsr->eEvalmode = FTS3_EVAL_NEXT;
	111150	+ return rc;
109775	111151	}
109776	111152
109777	111153	/*
109778	111154	** After ExprLoadDoclist() (see above) has been called, this function is
109779	111155	** used to iterate/search through the position lists that make up the doclist
		@@ -109835,11 +111211,11 @@
109835	111211	*/
109836	111212	static int fts3FunctionArg(
109837	111213	sqlite3_context pContext, / SQL function call context */
109838	111214	const char zFunc, / Function name */
109839	111215	sqlite3_value pVal, / argv[0] passed to function */
109840		- Fts3Cursor *ppCsr / OUT: Store cursor handle here */
	111216	+ Fts3Cursor *ppCsr / OUT: Store cursor handle here */
109841	111217	){
109842	111218	Fts3Cursor *pRet;
109843	111219	if( sqlite3_value_type(pVal)!=SQLITE_BLOB
109844	111220	\|\| sqlite3_value_bytes(pVal)!=sizeof(Fts3Cursor *)
109845	111221	){
		@@ -109961,17 +111337,11 @@
109961	111337	sqlite3_context pContext, / SQLite function call context */
109962	111338	int nVal, /* Size of argument array */
109963	111339	sqlite3_value *apVal / Array of arguments */
109964	111340	){
109965	111341	Fts3Cursor pCsr; / Cursor handle passed through apVal[0] */
109966		-
109967		- if( nVal!=1 ){
109968		- sqlite3_result_error(pContext,
109969		- "wrong number of arguments to function matchinfo()", -1);
109970		- return;
109971		- }
109972		-
	111342	+ assert( nVal==1 );
109973	111343	if( SQLITE_OK==fts3FunctionArg(pContext, "matchinfo", apVal[0], &pCsr) ){
109974	111344	sqlite3Fts3Matchinfo(pContext, pCsr);
109975	111345	}
109976	111346	}
109977	111347
		@@ -110030,16 +111400,17 @@
110030	111400
110031	111401	fts3DbExec(&rc, db,
110032	111402	"ALTER TABLE %Q.'%q_content' RENAME TO '%q_content';",
110033	111403	p->zDb, p->zName, zName
110034	111404	);
110035		- if( rc==SQLITE_ERROR ) rc = SQLITE_OK;
110036	111405	if( p->bHasDocsize ){
110037	111406	fts3DbExec(&rc, db,
110038	111407	"ALTER TABLE %Q.'%q_docsize' RENAME TO '%q_docsize';",
110039	111408	p->zDb, p->zName, zName
110040	111409	);
	111410	+ }
	111411	+ if( p->bHasStat ){
110041	111412	fts3DbExec(&rc, db,
110042	111413	"ALTER TABLE %Q.'%q_stat' RENAME TO '%q_stat';",
110043	111414	p->zDb, p->zName, zName
110044	111415	);
110045	111416	}
		@@ -110060,11 +111431,11 @@
110060	111431	/* xConnect */ fts3ConnectMethod,
110061	111432	/* xBestIndex */ fts3BestIndexMethod,
110062	111433	/* xDisconnect */ fts3DisconnectMethod,
110063	111434	/* xDestroy */ fts3DestroyMethod,
110064	111435	/* xOpen */ fts3OpenMethod,
110065		- /* xClose */ fulltextClose,
	111436	+ /* xClose */ fts3CloseMethod,
110066	111437	/* xFilter */ fts3FilterMethod,
110067	111438	/* xNext */ fts3NextMethod,
110068	111439	/* xEof */ fts3EofMethod,
110069	111440	/* xColumn */ fts3ColumnMethod,
110070	111441	/* xRowid */ fts3RowidMethod,
		@@ -110087,23 +111458,24 @@
110087	111458	sqlite3Fts3HashClear(pHash);
110088	111459	sqlite3_free(pHash);
110089	111460	}
110090	111461
110091	111462	/*
110092		-** The fts3 built-in tokenizers - "simple" and "porter" - are implemented
110093		-** in files fts3_tokenizer1.c and fts3_porter.c respectively. The following
110094		-** two forward declarations are for functions declared in these files
110095		-** used to retrieve the respective implementations.
	111463	+** The fts3 built-in tokenizers - "simple", "porter" and "icu"- are
	111464	+** implemented in files fts3_tokenizer1.c, fts3_porter.c and fts3_icu.c
	111465	+** respectively. The following three forward declarations are for functions
	111466	+** declared in these files used to retrieve the respective implementations.
110096	111467	**
110097	111468	** Calling sqlite3Fts3SimpleTokenizerModule() sets the value pointed
110098	111469	** to by the argument to point to the "simple" tokenizer implementation.
110099		-** Function ...PorterTokenizerModule() sets *pModule to point to the
110100		-** porter tokenizer/stemmer implementation.
	111470	+** And so on.
110101	111471	*/
110102	111472	SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule);
110103	111473	SQLITE_PRIVATE void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule);
	111474	+#ifdef SQLITE_ENABLE_ICU
110104	111475	SQLITE_PRIVATE void sqlite3Fts3IcuTokenizerModule(sqlite3_tokenizer_module const**ppModule);
	111476	+#endif
110105	111477
110106	111478	/*
110107	111479	** Initialise the fts3 extension. If this extension is built as part
110108	111480	** of the sqlite library, then this function is called directly by
110109	111481	** SQLite. If fts3 is built as a dynamically loadable extension, this
		@@ -110155,11 +111527,11 @@
110155	111527	*/
110156	111528	if( SQLITE_OK==rc
110157	111529	&& SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
110158	111530	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))
110159	111531	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
110160		- && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1))
	111532	+ && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", 1))
110161	111533	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
110162	111534	){
110163	111535	rc = sqlite3_create_module_v2(
110164	111536	db, "fts3", &fts3Module, (void *)pHash, hashDestroy
110165	111537	);
		@@ -110297,10 +111669,22 @@
110297	111669	** negative values).
110298	111670	*/
110299	111671	static int fts3isspace(char c){
110300	111672	return c==' ' \|\| c=='\t' \|\| c=='\n' \|\| c=='\r' \|\| c=='\v' \|\| c=='\f';
110301	111673	}
	111674	+
	111675	+/*
	111676	+** Allocate nByte bytes of memory using sqlite3_malloc(). If successful,
	111677	+** zero the memory before returning a pointer to it. If unsuccessful,
	111678	+** return NULL.
	111679	+*/
	111680	+static void *fts3MallocZero(int nByte){
	111681	+ void *pRet = sqlite3_malloc(nByte);
	111682	+ if( pRet ) memset(pRet, 0, nByte);
	111683	+ return pRet;
	111684	+}
	111685	+
110302	111686
110303	111687	/*
110304	111688	** Extract the next token from buffer z (length n) using the tokenizer
110305	111689	** and other information (column names etc.) in pParse. Create an Fts3Expr
110306	111690	** structure of type FTSQUERY_PHRASE containing a phrase consisting of this
		@@ -110335,15 +111719,14 @@
110335	111719	pCursor->pTokenizer = pTokenizer;
110336	111720	rc = pModule->xNext(pCursor, &zToken, &nToken, &iStart, &iEnd, &iPosition);
110337	111721
110338	111722	if( rc==SQLITE_OK ){
110339	111723	nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase) + nToken;
110340		- pRet = (Fts3Expr *)sqlite3_malloc(nByte);
	111724	+ pRet = (Fts3Expr *)fts3MallocZero(nByte);
110341	111725	if( !pRet ){
110342	111726	rc = SQLITE_NOMEM;
110343	111727	}else{
110344		- memset(pRet, 0, nByte);
110345	111728	pRet->eType = FTSQUERY_PHRASE;
110346	111729	pRet->pPhrase = (Fts3Phrase *)&pRet[1];
110347	111730	pRet->pPhrase->nToken = 1;
110348	111731	pRet->pPhrase->iColumn = iCol;
110349	111732	pRet->pPhrase->aToken[0].n = nToken;
		@@ -110415,20 +111798,21 @@
110415	111798	const char *zToken;
110416	111799	int nToken, iBegin, iEnd, iPos;
110417	111800	rc = pModule->xNext(pCursor, &zToken, &nToken, &iBegin, &iEnd, &iPos);
110418	111801	if( rc==SQLITE_OK ){
110419	111802	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
110420		- p = fts3ReallocOrFree(p, nByte+ii*sizeof(struct PhraseToken));
	111803	+ p = fts3ReallocOrFree(p, nByte+ii*sizeof(Fts3PhraseToken));
110421	111804	zTemp = fts3ReallocOrFree(zTemp, nTemp + nToken);
110422	111805	if( !p \|\| !zTemp ){
110423	111806	goto no_mem;
110424	111807	}
110425	111808	if( ii==0 ){
110426	111809	memset(p, 0, nByte);
110427	111810	p->pPhrase = (Fts3Phrase *)&p[1];
110428	111811	}
110429	111812	p->pPhrase = (Fts3Phrase *)&p[1];
	111813	+ memset(&p->pPhrase->aToken[ii], 0, sizeof(Fts3PhraseToken));
110430	111814	p->pPhrase->nToken = ii+1;
110431	111815	p->pPhrase->aToken[ii].n = nToken;
110432	111816	memcpy(&zTemp[nTemp], zToken, nToken);
110433	111817	nTemp += nToken;
110434	111818	if( iEnd<nInput && zInput[iEnd]=='*' ){
		@@ -110446,11 +111830,11 @@
110446	111830	if( rc==SQLITE_DONE ){
110447	111831	int jj;
110448	111832	char *zNew = NULL;
110449	111833	int nNew = 0;
110450	111834	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
110451		- nByte += (p?(p->pPhrase->nToken-1):0) * sizeof(struct PhraseToken);
	111835	+ nByte += (p?(p->pPhrase->nToken-1):0) * sizeof(Fts3PhraseToken);
110452	111836	p = fts3ReallocOrFree(p, nByte + nTemp);
110453	111837	if( !p ){
110454	111838	goto no_mem;
110455	111839	}
110456	111840	if( zTemp ){
		@@ -110564,15 +111948,14 @@
110564	111948	*/
110565	111949	cNext = zInput[nKey];
110566	111950	if( fts3isspace(cNext)
110567	111951	\|\| cNext=='"' \|\| cNext=='(' \|\| cNext==')' \|\| cNext==0
110568	111952	){
110569		- pRet = (Fts3Expr *)sqlite3_malloc(sizeof(Fts3Expr));
	111953	+ pRet = (Fts3Expr *)fts3MallocZero(sizeof(Fts3Expr));
110570	111954	if( !pRet ){
110571	111955	return SQLITE_NOMEM;
110572	111956	}
110573		- memset(pRet, 0, sizeof(Fts3Expr));
110574	111957	pRet->eType = pKey->eType;
110575	111958	pRet->nNear = nNear;
110576	111959	*ppExpr = pRet;
110577	111960	*pnConsumed = (int)((zInput - z) + nKey);
110578	111961	return SQLITE_OK;
		@@ -110744,17 +112127,16 @@
110744	112127
110745	112128	if( !sqlite3_fts3_enable_parentheses
110746	112129	&& p->eType==FTSQUERY_PHRASE && p->pPhrase->isNot
110747	112130	){
110748	112131	/* Create an implicit NOT operator. */
110749		- Fts3Expr *pNot = sqlite3_malloc(sizeof(Fts3Expr));
	112132	+ Fts3Expr *pNot = fts3MallocZero(sizeof(Fts3Expr));
110750	112133	if( !pNot ){
110751	112134	sqlite3Fts3ExprFree(p);
110752	112135	rc = SQLITE_NOMEM;
110753	112136	goto exprparse_out;
110754	112137	}
110755		- memset(pNot, 0, sizeof(Fts3Expr));
110756	112138	pNot->eType = FTSQUERY_NOT;
110757	112139	pNot->pRight = p;
110758	112140	if( pNotBranch ){
110759	112141	pNot->pLeft = pNotBranch;
110760	112142	}
		@@ -110778,17 +112160,16 @@
110778	112160
110779	112161	if( isPhrase && !isRequirePhrase ){
110780	112162	/* Insert an implicit AND operator. */
110781	112163	Fts3Expr *pAnd;
110782	112164	assert( pRet && pPrev );
110783		- pAnd = sqlite3_malloc(sizeof(Fts3Expr));
	112165	+ pAnd = fts3MallocZero(sizeof(Fts3Expr));
110784	112166	if( !pAnd ){
110785	112167	sqlite3Fts3ExprFree(p);
110786	112168	rc = SQLITE_NOMEM;
110787	112169	goto exprparse_out;
110788	112170	}
110789		- memset(pAnd, 0, sizeof(Fts3Expr));
110790	112171	pAnd->eType = FTSQUERY_AND;
110791	112172	insertBinaryOperator(&pRet, pPrev, pAnd);
110792	112173	pPrev = pAnd;
110793	112174	}
110794	112175
		@@ -112234,11 +113615,11 @@
112234	113615	}
112235	113616
112236	113617	sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
112237	113618	}
112238	113619
112239		-static int fts3IsIdChar(char c){
	113620	+SQLITE_PRIVATE int sqlite3Fts3IsIdChar(char c){
112240	113621	static const char isFtsIdChar[] = {
112241	113622	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x */
112242	113623	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 1x */
112243	113624	0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 2x */
112244	113625	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 3x */
		@@ -112272,13 +113653,13 @@
112272	113653	while( *z2 && z2[0]!=']' ) z2++;
112273	113654	if( *z2 ) z2++;
112274	113655	break;
112275	113656
112276	113657	default:
112277		- if( fts3IsIdChar(*z1) ){
	113658	+ if( sqlite3Fts3IsIdChar(*z1) ){
112278	113659	z2 = &z1[1];
112279		- while( fts3IsIdChar(*z2) ) z2++;
	113660	+ while( sqlite3Fts3IsIdChar(*z2) ) z2++;
112280	113661	}else{
112281	113662	z1++;
112282	113663	}
112283	113664	}
112284	113665	}
		@@ -112287,42 +113668,30 @@
112287	113668	return z1;
112288	113669	}
112289	113670
112290	113671	SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(
112291	113672	Fts3Hash pHash, / Tokenizer hash table */
112292		- const char zArg, / Possible tokenizer specification */
	113673	+ const char zArg, / Tokenizer name */
112293	113674	sqlite3_tokenizer *ppTok, / OUT: Tokenizer (if applicable) */
112294		- const char *pzTokenizer, / OUT: Set to zArg if is tokenizer */
112295	113675	char *pzErr / OUT: Set to malloced error message */
112296	113676	){
112297	113677	int rc;
112298	113678	char z = (char )zArg;
112299	113679	int n;
112300	113680	char *zCopy;
112301	113681	char zEnd; / Pointer to nul-term of zCopy */
112302	113682	sqlite3_tokenizer_module *m;
112303	113683
112304		- if( !z ){
112305		- zCopy = sqlite3_mprintf("simple");
112306		- }else{
112307		- if( sqlite3_strnicmp(z, "tokenize", 8) \|\| fts3IsIdChar(z[8])){
112308		- return SQLITE_OK;
112309		- }
112310		- zCopy = sqlite3_mprintf("%s", &z[8]);
112311		- *pzTokenizer = zArg;
112312		- }
112313		- if( !zCopy ){
112314		- return SQLITE_NOMEM;
112315		- }
112316		-
	113684	+ zCopy = sqlite3_mprintf("%s", zArg);
	113685	+ if( !zCopy ) return SQLITE_NOMEM;
112317	113686	zEnd = &zCopy[strlen(zCopy)];
112318	113687
112319	113688	z = (char *)sqlite3Fts3NextToken(zCopy, &n);
112320	113689	z[n] = '\0';
112321	113690	sqlite3Fts3Dequote(z);
112322	113691
112323		- m = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash, z, (int)strlen(z)+1);
	113692	+ m = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash,z,(int)strlen(z)+1);
112324	113693	if( !m ){
112325	113694	*pzErr = sqlite3_mprintf("unknown tokenizer: %s", z);
112326	113695	rc = SQLITE_ERROR;
112327	113696	}else{
112328	113697	char const **aArg = 0;
		@@ -112887,10 +114256,22 @@
112887	114256	*/
112888	114257
112889	114258	#if !defined(SQLITE_CORE) \|\| defined(SQLITE_ENABLE_FTS3)
112890	114259
112891	114260
	114261	+/*
	114262	+** When full-text index nodes are loaded from disk, the buffer that they
	114263	+** are loaded into has the following number of bytes of padding at the end
	114264	+** of it. i.e. if a full-text index node is 900 bytes in size, then a buffer
	114265	+** of 920 bytes is allocated for it.
	114266	+**
	114267	+** This means that if we have a pointer into a buffer containing node data,
	114268	+** it is always safe to read up to two varints from it without risking an
	114269	+** overread, even if the node data is corrupted.
	114270	+*/
	114271	+#define FTS3_NODE_PADDING (FTS3_VARINT_MAX*2)
	114272	+
112892	114273	typedef struct PendingList PendingList;
112893	114274	typedef struct SegmentNode SegmentNode;
112894	114275	typedef struct SegmentWriter SegmentWriter;
112895	114276
112896	114277	/*
		@@ -112905,19 +114286,31 @@
112905	114286	sqlite3_int64 iLastDocid;
112906	114287	sqlite3_int64 iLastCol;
112907	114288	sqlite3_int64 iLastPos;
112908	114289	};
112909	114290
	114291	+
	114292	+/*
	114293	+** Each cursor has a (possibly empty) linked list of the following objects.
	114294	+*/
	114295	+struct Fts3DeferredToken {
	114296	+ Fts3PhraseToken pToken; / Pointer to corresponding expr token */
	114297	+ int iCol; /* Column token must occur in */
	114298	+ Fts3DeferredToken pNext; / Next in list of deferred tokens */
	114299	+ PendingList pList; / Doclist is assembled here */
	114300	+};
	114301	+
112910	114302	/*
112911	114303	** An instance of this structure is used to iterate through the terms on
112912	114304	** a contiguous set of segment b-tree leaf nodes. Although the details of
112913	114305	** this structure are only manipulated by code in this file, opaque handles
112914	114306	** of type Fts3SegReader* are also used by code in fts3.c to iterate through
112915	114307	** terms when querying the full-text index. See functions:
112916	114308	**
112917	114309	** sqlite3Fts3SegReaderNew()
112918	114310	** sqlite3Fts3SegReaderFree()
	114311	+** sqlite3Fts3SegReaderCost()
112919	114312	** sqlite3Fts3SegReaderIterate()
112920	114313	**
112921	114314	** Methods used to manipulate Fts3SegReader structures:
112922	114315	**
112923	114316	** fts3SegReaderNext()
		@@ -112924,34 +114317,38 @@
112924	114317	** fts3SegReaderFirstDocid()
112925	114318	** fts3SegReaderNextDocid()
112926	114319	*/
112927	114320	struct Fts3SegReader {
112928	114321	int iIdx; /* Index within level, or 0x7FFFFFFF for PT */
112929		- sqlite3_int64 iStartBlock;
112930		- sqlite3_int64 iEndBlock;
112931		- sqlite3_stmt pStmt; / SQL Statement to access leaf nodes */
	114322	+
	114323	+ sqlite3_int64 iStartBlock; /* Rowid of first leaf block to traverse */
	114324	+ sqlite3_int64 iLeafEndBlock; /* Rowid of final leaf block to traverse */
	114325	+ sqlite3_int64 iEndBlock; /* Rowid of final block in segment (or 0) */
	114326	+ sqlite3_int64 iCurrentBlock; /* Current leaf block (or 0) */
	114327	+
112932	114328	char aNode; / Pointer to node data (or NULL) */
112933	114329	int nNode; /* Size of buffer at aNode (or 0) */
112934		- int nTermAlloc; /* Allocated size of zTerm buffer */
112935	114330	Fts3HashElem **ppNextElem;
112936	114331
112937	114332	/* Variables set by fts3SegReaderNext(). These may be read directly
112938	114333	** by the caller. They are valid from the time SegmentReaderNew() returns
112939	114334	** until SegmentReaderNext() returns something other than SQLITE_OK
112940	114335	** (i.e. SQLITE_DONE).
112941	114336	*/
112942	114337	int nTerm; /* Number of bytes in current term */
112943	114338	char zTerm; / Pointer to current term */
	114339	+ int nTermAlloc; /* Allocated size of zTerm buffer */
112944	114340	char aDoclist; / Pointer to doclist of current entry */
112945	114341	int nDoclist; /* Size of doclist in current entry */
112946	114342
112947	114343	/* The following variables are used to iterate through the current doclist */
112948	114344	char *pOffsetList;
112949	114345	sqlite3_int64 iDocid;
112950	114346	};
112951	114347
112952	114348	#define fts3SegReaderIsPending(p) ((p)->ppNextElem!=0)
	114349	+#define fts3SegReaderIsRootOnly(p) ((p)->aNode==(char *)&(p)[1])
112953	114350
112954	114351	/*
112955	114352	** An instance of this structure is used to create a segment b-tree in the
112956	114353	** database. The internal details of this type are only accessed by the
112957	114354	** following functions:
		@@ -113016,16 +114413,15 @@
113016	114413	#define SQL_SELECT_LEVEL_COUNT 14
113017	114414	#define SQL_SELECT_SEGDIR_COUNT_MAX 15
113018	114415	#define SQL_DELETE_SEGDIR_BY_LEVEL 16
113019	114416	#define SQL_DELETE_SEGMENTS_RANGE 17
113020	114417	#define SQL_CONTENT_INSERT 18
113021		-#define SQL_GET_BLOCK 19
113022		-#define SQL_DELETE_DOCSIZE 20
113023		-#define SQL_REPLACE_DOCSIZE 21
113024		-#define SQL_SELECT_DOCSIZE 22
113025		-#define SQL_SELECT_DOCTOTAL 23
113026		-#define SQL_REPLACE_DOCTOTAL 24
	114418	+#define SQL_DELETE_DOCSIZE 19
	114419	+#define SQL_REPLACE_DOCSIZE 20
	114420	+#define SQL_SELECT_DOCSIZE 21
	114421	+#define SQL_SELECT_DOCTOTAL 22
	114422	+#define SQL_REPLACE_DOCTOTAL 23
113027	114423
113028	114424	/*
113029	114425	** This function is used to obtain an SQLite prepared statement handle
113030	114426	** for the statement identified by the second argument. If successful,
113031	114427	** *pp is set to the requested statement handle and SQLITE_OK returned.
		@@ -113066,16 +114462,15 @@
113066	114462	/* 15 / "SELECT count(), max(level) FROM %Q.'%q_segdir'",
113067	114463
113068	114464	/* 16 */ "DELETE FROM %Q.'%q_segdir' WHERE level = ?",
113069	114465	/* 17 */ "DELETE FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ?",
113070	114466	/* 18 */ "INSERT INTO %Q.'%q_content' VALUES(%z)",
113071		-/* 19 */ "SELECT block FROM %Q.'%q_segments' WHERE blockid = ?",
113072		-/* 20 */ "DELETE FROM %Q.'%q_docsize' WHERE docid = ?",
113073		-/* 21 */ "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)",
113074		-/* 22 */ "SELECT size FROM %Q.'%q_docsize' WHERE docid=?",
113075		-/* 23 */ "SELECT value FROM %Q.'%q_stat' WHERE id=0",
113076		-/* 24 */ "REPLACE INTO %Q.'%q_stat' VALUES(0,?)",
	114467	+/* 19 */ "DELETE FROM %Q.'%q_docsize' WHERE docid = ?",
	114468	+/* 20 */ "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)",
	114469	+/* 21 */ "SELECT size FROM %Q.'%q_docsize' WHERE docid=?",
	114470	+/* 22 */ "SELECT value FROM %Q.'%q_stat' WHERE id=0",
	114471	+/* 23 */ "REPLACE INTO %Q.'%q_stat' VALUES(0,?)",
113077	114472	};
113078	114473	int rc = SQLITE_OK;
113079	114474	sqlite3_stmt *pStmt;
113080	114475
113081	114476	assert( SizeofArray(azSql)==SizeofArray(p->aStmt) );
		@@ -113146,49 +114541,10 @@
113146	114541	}
113147	114542	*pRC = rc;
113148	114543	}
113149	114544
113150	114545
113151		-/*
113152		-** Read a single block from the %_segments table. If the specified block
113153		-** does not exist, return SQLITE_CORRUPT. If some other error (malloc, IO
113154		-** etc.) occurs, return the appropriate SQLite error code.
113155		-**
113156		-** Otherwise, if successful, set *pzBlock to point to a buffer containing
113157		-** the block read from the database, and *pnBlock to the size of the read
113158		-** block in bytes.
113159		-**
113160		-** WARNING: The returned buffer is only valid until the next call to
113161		-** sqlite3Fts3ReadBlock().
113162		-*/
113163		-SQLITE_PRIVATE int sqlite3Fts3ReadBlock(
113164		- Fts3Table *p,
113165		- sqlite3_int64 iBlock,
113166		- char const **pzBlock,
113167		- int *pnBlock
113168		-){
113169		- sqlite3_stmt *pStmt;
113170		- int rc = fts3SqlStmt(p, SQL_GET_BLOCK, &pStmt, 0);
113171		- if( rc!=SQLITE_OK ) return rc;
113172		- sqlite3_reset(pStmt);
113173		-
113174		- if( pzBlock ){
113175		- sqlite3_bind_int64(pStmt, 1, iBlock);
113176		- rc = sqlite3_step(pStmt);
113177		- if( rc!=SQLITE_ROW ){
113178		- return (rc==SQLITE_DONE ? SQLITE_CORRUPT : rc);
113179		- }
113180		-
113181		- *pnBlock = sqlite3_column_bytes(pStmt, 0);
113182		- pzBlock = (char )sqlite3_column_blob(pStmt, 0);
113183		- if( sqlite3_column_type(pStmt, 0)!=SQLITE_BLOB ){
113184		- return SQLITE_CORRUPT;
113185		- }
113186		- }
113187		- return SQLITE_OK;
113188		-}
113189		-
113190	114546	/*
113191	114547	** This function ensures that the caller has obtained a shared-cache
113192	114548	** table-lock on the %_content table. This is required before reading
113193	114549	** data from the fts3 table. If this lock is not acquired first, then
113194	114550	** the caller may end up holding read-locks on the %_segments and %_segdir
		@@ -113353,14 +114709,14 @@
113353	114709	** p->iPrevDocid, and the column is specified by argument iCol.
113354	114710	**
113355	114711	** If successful, SQLITE_OK is returned. Otherwise, an SQLite error code.
113356	114712	*/
113357	114713	static int fts3PendingTermsAdd(
113358		- Fts3Table p, / FTS table into which text will be inserted */
113359		- const char zText, / Text of document to be inseted */
113360		- int iCol, /* Column number into which text is inserted */
113361		- u32 pnWord / OUT: Number of tokens inserted */
	114714	+ Fts3Table p, / Table into which text will be inserted */
	114715	+ const char zText, / Text of document to be inserted */
	114716	+ int iCol, /* Column into which text is being inserted */
	114717	+ u32 pnWord / OUT: Number of tokens inserted */
113362	114718	){
113363	114719	int rc;
113364	114720	int iStart;
113365	114721	int iEnd;
113366	114722	int iPos;
		@@ -113441,10 +114797,13 @@
113441	114797	}
113442	114798	p->iPrevDocid = iDocid;
113443	114799	return SQLITE_OK;
113444	114800	}
113445	114801
	114802	+/*
	114803	+** Discard the contents of the pending-terms hash table.
	114804	+*/
113446	114805	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *p){
113447	114806	Fts3HashElem *pElem;
113448	114807	for(pElem=fts3HashFirst(&p->pendingTerms); pElem; pElem=fts3HashNext(pElem)){
113449	114808	sqlite3_free(fts3HashData(pElem));
113450	114809	}
		@@ -113468,10 +114827,11 @@
113468	114827	int rc = fts3PendingTermsAdd(p, zText, i-2, &aSz[i-2]);
113469	114828	if( rc!=SQLITE_OK ){
113470	114829	return rc;
113471	114830	}
113472	114831	}
	114832	+ aSz[p->nColumn] += sqlite3_value_bytes(apVal[i]);
113473	114833	}
113474	114834	return SQLITE_OK;
113475	114835	}
113476	114836
113477	114837	/*
		@@ -113555,10 +114915,12 @@
113555	114915	fts3SqlExec(&rc, p, SQL_DELETE_ALL_CONTENT, 0);
113556	114916	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGMENTS, 0);
113557	114917	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGDIR, 0);
113558	114918	if( p->bHasDocsize ){
113559	114919	fts3SqlExec(&rc, p, SQL_DELETE_ALL_DOCSIZE, 0);
	114920	+ }
	114921	+ if( p->bHasStat ){
113560	114922	fts3SqlExec(&rc, p, SQL_DELETE_ALL_STAT, 0);
113561	114923	}
113562	114924	return rc;
113563	114925	}
113564	114926
		@@ -113565,11 +114927,11 @@
113565	114927	/*
113566	114928	** The first element in the apVal[] array is assumed to contain the docid
113567	114929	** (an integer) of a row about to be deleted. Remove all terms from the
113568	114930	** full-text index.
113569	114931	*/
113570		-static void fts3DeleteTerms(
	114932	+static void fts3DeleteTerms(
113571	114933	int pRC, / Result code */
113572	114934	Fts3Table p, / The FTS table to delete from */
113573	114935	sqlite3_value *apVal, / apVal[] contains the docid to be deleted */
113574	114936	u32 aSz / Sizes of deleted document written here */
113575	114937	){
		@@ -113587,10 +114949,11 @@
113587	114949	if( rc!=SQLITE_OK ){
113588	114950	sqlite3_reset(pSelect);
113589	114951	*pRC = rc;
113590	114952	return;
113591	114953	}
	114954	+ aSz[p->nColumn] += sqlite3_column_bytes(pSelect, i);
113592	114955	}
113593	114956	}
113594	114957	rc = sqlite3_reset(pSelect);
113595	114958	}else{
113596	114959	sqlite3_reset(pSelect);
		@@ -113648,17 +115011,98 @@
113648	115011	}
113649	115012	}
113650	115013
113651	115014	return rc;
113652	115015	}
	115016	+
	115017	+/*
	115018	+** The %_segments table is declared as follows:
	115019	+**
	115020	+** CREATE TABLE %_segments(blockid INTEGER PRIMARY KEY, block BLOB)
	115021	+**
	115022	+** This function reads data from a single row of the %_segments table. The
	115023	+** specific row is identified by the iBlockid parameter. If paBlob is not
	115024	+** NULL, then a buffer is allocated using sqlite3_malloc() and populated
	115025	+** with the contents of the blob stored in the "block" column of the
	115026	+** identified table row is. Whether or not paBlob is NULL, *pnBlob is set
	115027	+** to the size of the blob in bytes before returning.
	115028	+**
	115029	+** If an error occurs, or the table does not contain the specified row,
	115030	+** an SQLite error code is returned. Otherwise, SQLITE_OK is returned. If
	115031	+** paBlob is non-NULL, then it is the responsibility of the caller to
	115032	+** eventually free the returned buffer.
	115033	+**
	115034	+** This function may leave an open sqlite3_blob* handle in the
	115035	+** Fts3Table.pSegments variable. This handle is reused by subsequent calls
	115036	+** to this function. The handle may be closed by calling the
	115037	+** sqlite3Fts3SegmentsClose() function. Reusing a blob handle is a handy
	115038	+** performance improvement, but the blob handle should always be closed
	115039	+** before control is returned to the user (to prevent a lock being held
	115040	+** on the database file for longer than necessary). Thus, any virtual table
	115041	+** method (xFilter etc.) that may directly or indirectly call this function
	115042	+** must call sqlite3Fts3SegmentsClose() before returning.
	115043	+*/
	115044	+SQLITE_PRIVATE int sqlite3Fts3ReadBlock(
	115045	+ Fts3Table p, / FTS3 table handle */
	115046	+ sqlite3_int64 iBlockid, /* Access the row with blockid=$iBlockid */
	115047	+ char *paBlob, / OUT: Blob data in malloc'd buffer */
	115048	+ int pnBlob / OUT: Size of blob data */
	115049	+){
	115050	+ int rc; /* Return code */
	115051	+
	115052	+ /* pnBlob must be non-NULL. paBlob may be NULL or non-NULL. */
	115053	+ assert( pnBlob);
	115054	+
	115055	+ if( p->pSegments ){
	115056	+ rc = sqlite3_blob_reopen(p->pSegments, iBlockid);
	115057	+ }else{
	115058	+ if( 0==p->zSegmentsTbl ){
	115059	+ p->zSegmentsTbl = sqlite3_mprintf("%s_segments", p->zName);
	115060	+ if( 0==p->zSegmentsTbl ) return SQLITE_NOMEM;
	115061	+ }
	115062	+ rc = sqlite3_blob_open(
	115063	+ p->db, p->zDb, p->zSegmentsTbl, "block", iBlockid, 0, &p->pSegments
	115064	+ );
	115065	+ }
	115066	+
	115067	+ if( rc==SQLITE_OK ){
	115068	+ int nByte = sqlite3_blob_bytes(p->pSegments);
	115069	+ if( paBlob ){
	115070	+ char *aByte = sqlite3_malloc(nByte + FTS3_NODE_PADDING);
	115071	+ if( !aByte ){
	115072	+ rc = SQLITE_NOMEM;
	115073	+ }else{
	115074	+ rc = sqlite3_blob_read(p->pSegments, aByte, nByte, 0);
	115075	+ memset(&aByte[nByte], 0, FTS3_NODE_PADDING);
	115076	+ if( rc!=SQLITE_OK ){
	115077	+ sqlite3_free(aByte);
	115078	+ aByte = 0;
	115079	+ }
	115080	+ }
	115081	+ *paBlob = aByte;
	115082	+ }
	115083	+ *pnBlob = nByte;
	115084	+ }
	115085	+
	115086	+ return rc;
	115087	+}
	115088	+
	115089	+/*
	115090	+** Close the blob handle at p->pSegments, if it is open. See comments above
	115091	+** the sqlite3Fts3ReadBlock() function for details.
	115092	+*/
	115093	+SQLITE_PRIVATE void sqlite3Fts3SegmentsClose(Fts3Table *p){
	115094	+ sqlite3_blob_close(p->pSegments);
	115095	+ p->pSegments = 0;
	115096	+}
113653	115097
113654	115098	/*
113655	115099	** Move the iterator passed as the first argument to the next term in the
113656	115100	** segment. If successful, SQLITE_OK is returned. If there is no next term,
113657	115101	** SQLITE_DONE. Otherwise, an SQLite error code.
113658	115102	*/
113659		-static int fts3SegReaderNext(Fts3SegReader *pReader){
	115103	+static int fts3SegReaderNext(Fts3Table p, Fts3SegReader pReader){
113660	115104	char pNext; / Cursor variable */
113661	115105	int nPrefix; /* Number of bytes in term prefix */
113662	115106	int nSuffix; /* Number of bytes in term suffix */
113663	115107
113664	115108	if( !pReader->aDoclist ){
		@@ -113666,11 +115110,12 @@
113666	115110	}else{
113667	115111	pNext = &pReader->aDoclist[pReader->nDoclist];
113668	115112	}
113669	115113
113670	115114	if( !pNext \|\| pNext>=&pReader->aNode[pReader->nNode] ){
113671		- int rc;
	115115	+ int rc; /* Return code from Fts3ReadBlock() */
	115116	+
113672	115117	if( fts3SegReaderIsPending(pReader) ){
113673	115118	Fts3HashElem pElem = (pReader->ppNextElem);
113674	115119	if( pElem==0 ){
113675	115120	pReader->aNode = 0;
113676	115121	}else{
		@@ -113682,26 +115127,40 @@
113682	115127	pReader->ppNextElem++;
113683	115128	assert( pReader->aNode );
113684	115129	}
113685	115130	return SQLITE_OK;
113686	115131	}
113687		- if( !pReader->pStmt ){
113688		- pReader->aNode = 0;
	115132	+
	115133	+ if( !fts3SegReaderIsRootOnly(pReader) ){
	115134	+ sqlite3_free(pReader->aNode);
	115135	+ }
	115136	+ pReader->aNode = 0;
	115137	+
	115138	+ /* If iCurrentBlock>=iLeafEndBlock, this is an EOF condition. All leaf
	115139	+ ** blocks have already been traversed. */
	115140	+ assert( pReader->iCurrentBlock<=pReader->iLeafEndBlock );
	115141	+ if( pReader->iCurrentBlock>=pReader->iLeafEndBlock ){
113689	115142	return SQLITE_OK;
113690	115143	}
113691		- rc = sqlite3_step(pReader->pStmt);
113692		- if( rc!=SQLITE_ROW ){
113693		- pReader->aNode = 0;
113694		- return (rc==SQLITE_DONE ? SQLITE_OK : rc);
113695		- }
113696		- pReader->nNode = sqlite3_column_bytes(pReader->pStmt, 0);
113697		- pReader->aNode = (char *)sqlite3_column_blob(pReader->pStmt, 0);
	115144	+
	115145	+ rc = sqlite3Fts3ReadBlock(
	115146	+ p, ++pReader->iCurrentBlock, &pReader->aNode, &pReader->nNode
	115147	+ );
	115148	+ if( rc!=SQLITE_OK ) return rc;
113698	115149	pNext = pReader->aNode;
113699	115150	}
113700	115151
	115152	+ /* Because of the FTS3_NODE_PADDING bytes of padding, the following is
	115153	+ ** safe (no risk of overread) even if the node data is corrupted.
	115154	+ */
113701	115155	pNext += sqlite3Fts3GetVarint32(pNext, &nPrefix);
113702	115156	pNext += sqlite3Fts3GetVarint32(pNext, &nSuffix);
	115157	+ if( nPrefix<0 \|\| nSuffix<=0
	115158	+ \|\| &pNext[nSuffix]>&pReader->aNode[pReader->nNode]
	115159	+ ){
	115160	+ return SQLITE_CORRUPT;
	115161	+ }
113703	115162
113704	115163	if( nPrefix+nSuffix>pReader->nTermAlloc ){
113705	115164	int nNew = (nPrefix+nSuffix)*2;
113706	115165	char *zNew = sqlite3_realloc(pReader->zTerm, nNew);
113707	115166	if( !zNew ){
		@@ -113712,13 +115171,22 @@
113712	115171	}
113713	115172	memcpy(&pReader->zTerm[nPrefix], pNext, nSuffix);
113714	115173	pReader->nTerm = nPrefix+nSuffix;
113715	115174	pNext += nSuffix;
113716	115175	pNext += sqlite3Fts3GetVarint32(pNext, &pReader->nDoclist);
113717		- assert( pNext<&pReader->aNode[pReader->nNode] );
113718	115176	pReader->aDoclist = pNext;
113719	115177	pReader->pOffsetList = 0;
	115178	+
	115179	+ /* Check that the doclist does not appear to extend past the end of the
	115180	+ ** b-tree node. And that the final byte of the doclist is 0x00. If either
	115181	+ ** of these statements is untrue, then the data structure is corrupt.
	115182	+ */
	115183	+ if( &pReader->aDoclist[pReader->nDoclist]>&pReader->aNode[pReader->nNode]
	115184	+ \|\| pReader->aDoclist[pReader->nDoclist-1]
	115185	+ ){
	115186	+ return SQLITE_CORRUPT;
	115187	+ }
113720	115188	return SQLITE_OK;
113721	115189	}
113722	115190
113723	115191	/*
113724	115192	** Set the SegReader to point to the first docid in the doclist associated
		@@ -113776,30 +115244,105 @@
113776	115244	sqlite3_int64 iDelta;
113777	115245	pReader->pOffsetList = p + sqlite3Fts3GetVarint(p, &iDelta);
113778	115246	pReader->iDocid += iDelta;
113779	115247	}
113780	115248	}
	115249	+
	115250	+/*
	115251	+** This function is called to estimate the amount of data that will be
	115252	+** loaded from the disk If SegReaderIterate() is called on this seg-reader,
	115253	+** in units of average document size.
	115254	+**
	115255	+** This can be used as follows: If the caller has a small doclist that
	115256	+** contains references to N documents, and is considering merging it with
	115257	+** a large doclist (size X "average documents"), it may opt not to load
	115258	+** the large doclist if X>N.
	115259	+*/
	115260	+SQLITE_PRIVATE int sqlite3Fts3SegReaderCost(
	115261	+ Fts3Cursor pCsr, / FTS3 cursor handle */
	115262	+ Fts3SegReader pReader, / Segment-reader handle */
	115263	+ int pnCost / IN/OUT: Number of bytes read */
	115264	+){
	115265	+ Fts3Table p = (Fts3Table)pCsr->base.pVtab;
	115266	+ int rc = SQLITE_OK; /* Return code */
	115267	+ int nCost = 0; /* Cost in bytes to return */
	115268	+ int pgsz = p->nPgsz; /* Database page size */
	115269	+
	115270	+ /* If this seg-reader is reading the pending-terms table, or if all data
	115271	+ ** for the segment is stored on the root page of the b-tree, then the cost
	115272	+ ** is zero. In this case all required data is already in main memory.
	115273	+ */
	115274	+ if( p->bHasStat
	115275	+ && !fts3SegReaderIsPending(pReader)
	115276	+ && !fts3SegReaderIsRootOnly(pReader)
	115277	+ ){
	115278	+ int nBlob = 0;
	115279	+ sqlite3_int64 iBlock;
	115280	+
	115281	+ if( pCsr->nRowAvg==0 ){
	115282	+ /* The average document size, which is required to calculate the cost
	115283	+ ** of each doclist, has not yet been determined. Read the required
	115284	+ ** data from the %_stat table to calculate it.
	115285	+ **
	115286	+ ** Entry 0 of the %_stat table is a blob containing (nCol+1) FTS3
	115287	+ ** varints, where nCol is the number of columns in the FTS3 table.
	115288	+ ** The first varint is the number of documents currently stored in
	115289	+ ** the table. The following nCol varints contain the total amount of
	115290	+ ** data stored in all rows of each column of the table, from left
	115291	+ ** to right.
	115292	+ */
	115293	+ sqlite3_stmt *pStmt;
	115294	+ rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
	115295	+ if( rc ) return rc;
	115296	+ if( sqlite3_step(pStmt)==SQLITE_ROW ){
	115297	+ sqlite3_int64 nDoc = 0;
	115298	+ sqlite3_int64 nByte = 0;
	115299	+ const char *a = sqlite3_column_blob(pStmt, 0);
	115300	+ if( a ){
	115301	+ const char *pEnd = &a[sqlite3_column_bytes(pStmt, 0)];
	115302	+ a += sqlite3Fts3GetVarint(a, &nDoc);
	115303	+ while( a<pEnd ){
	115304	+ a += sqlite3Fts3GetVarint(a, &nByte);
	115305	+ }
	115306	+ }
	115307	+
	115308	+ pCsr->nRowAvg = (((nByte / nDoc) + pgsz - 1) / pgsz);
	115309	+ }
	115310	+ rc = sqlite3_reset(pStmt);
	115311	+ if( rc!=SQLITE_OK \|\| pCsr->nRowAvg==0 ) return rc;
	115312	+ }
	115313	+
	115314	+ /* Assume that a blob flows over onto overflow pages if it is larger
	115315	+ ** than (pgsz-35) bytes in size (the file-format documentation
	115316	+ ** confirms this).
	115317	+ */
	115318	+ for(iBlock=pReader->iStartBlock; iBlock<=pReader->iLeafEndBlock; iBlock++){
	115319	+ rc = sqlite3Fts3ReadBlock(p, iBlock, 0, &nBlob);
	115320	+ if( rc!=SQLITE_OK ) break;
	115321	+ if( (nBlob+35)>pgsz ){
	115322	+ int nOvfl = (nBlob + 34)/pgsz;
	115323	+ nCost += ((nOvfl + pCsr->nRowAvg - 1)/pCsr->nRowAvg);
	115324	+ }
	115325	+ }
	115326	+ }
	115327	+
	115328	+ *pnCost += nCost;
	115329	+ return rc;
	115330	+}
113781	115331
113782	115332	/*
113783	115333	** Free all allocations associated with the iterator passed as the
113784	115334	** second argument.
113785	115335	*/
113786	115336	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table p, Fts3SegReader pReader){
113787		- if( pReader ){
113788		- if( pReader->pStmt ){
113789		- /* Move the leaf-range SELECT statement to the aLeavesStmt[] array,
113790		- ** so that it can be reused when required by another query.
113791		- */
113792		- assert( p->nLeavesStmt<p->nLeavesTotal );
113793		- sqlite3_reset(pReader->pStmt);
113794		- p->aLeavesStmt[p->nLeavesStmt++] = pReader->pStmt;
113795		- }
113796		- if( !fts3SegReaderIsPending(pReader) ){
113797		- sqlite3_free(pReader->zTerm);
113798		- }
113799		- sqlite3_free(pReader);
113800		- }
	115337	+ if( pReader && !fts3SegReaderIsPending(pReader) ){
	115338	+ sqlite3_free(pReader->zTerm);
	115339	+ if( !fts3SegReaderIsRootOnly(pReader) ){
	115340	+ sqlite3_free(pReader->aNode);
	115341	+ }
	115342	+ }
	115343	+ sqlite3_free(pReader);
113801	115344	}
113802	115345
113803	115346	/*
113804	115347	** Allocate a new SegReader object.
113805	115348	*/
		@@ -113815,77 +115358,35 @@
113815	115358	){
113816	115359	int rc = SQLITE_OK; /* Return code */
113817	115360	Fts3SegReader pReader; / Newly allocated SegReader object */
113818	115361	int nExtra = 0; /* Bytes to allocate segment root node */
113819	115362
	115363	+ assert( iStartLeaf<=iEndLeaf );
113820	115364	if( iStartLeaf==0 ){
113821		- nExtra = nRoot;
	115365	+ nExtra = nRoot + FTS3_NODE_PADDING;
113822	115366	}
113823	115367
113824	115368	pReader = (Fts3SegReader *)sqlite3_malloc(sizeof(Fts3SegReader) + nExtra);
113825	115369	if( !pReader ){
113826	115370	return SQLITE_NOMEM;
113827	115371	}
113828	115372	memset(pReader, 0, sizeof(Fts3SegReader));
113829		- pReader->iStartBlock = iStartLeaf;
113830	115373	pReader->iIdx = iAge;
	115374	+ pReader->iStartBlock = iStartLeaf;
	115375	+ pReader->iLeafEndBlock = iEndLeaf;
113831	115376	pReader->iEndBlock = iEndBlock;
113832	115377
113833	115378	if( nExtra ){
113834	115379	/* The entire segment is stored in the root node. */
113835	115380	pReader->aNode = (char *)&pReader[1];
113836	115381	pReader->nNode = nRoot;
113837	115382	memcpy(pReader->aNode, zRoot, nRoot);
113838		- }else{
113839		- /* If the text of the SQL statement to iterate through a contiguous
113840		- ** set of entries in the %_segments table has not yet been composed,
113841		- ** compose it now.
113842		- */
113843		- if( !p->zSelectLeaves ){
113844		- p->zSelectLeaves = sqlite3_mprintf(
113845		- "SELECT block FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ? "
113846		- "ORDER BY blockid", p->zDb, p->zName
113847		- );
113848		- if( !p->zSelectLeaves ){
113849		- rc = SQLITE_NOMEM;
113850		- goto finished;
113851		- }
113852		- }
113853		-
113854		- /* If there are no free statements in the aLeavesStmt[] array, prepare
113855		- ** a new statement now. Otherwise, reuse a prepared statement from
113856		- ** aLeavesStmt[].
113857		- */
113858		- if( p->nLeavesStmt==0 ){
113859		- if( p->nLeavesTotal==p->nLeavesAlloc ){
113860		- int nNew = p->nLeavesAlloc + 16;
113861		- sqlite3_stmt aNew = (sqlite3_stmt )sqlite3_realloc(
113862		- p->aLeavesStmt, nNewsizeof(sqlite3_stmt )
113863		- );
113864		- if( !aNew ){
113865		- rc = SQLITE_NOMEM;
113866		- goto finished;
113867		- }
113868		- p->nLeavesAlloc = nNew;
113869		- p->aLeavesStmt = aNew;
113870		- }
113871		- rc = sqlite3_prepare_v2(p->db, p->zSelectLeaves, -1, &pReader->pStmt, 0);
113872		- if( rc!=SQLITE_OK ){
113873		- goto finished;
113874		- }
113875		- p->nLeavesTotal++;
113876		- }else{
113877		- pReader->pStmt = p->aLeavesStmt[--p->nLeavesStmt];
113878		- }
113879		-
113880		- /* Bind the start and end leaf blockids to the prepared SQL statement. */
113881		- sqlite3_bind_int64(pReader->pStmt, 1, iStartLeaf);
113882		- sqlite3_bind_int64(pReader->pStmt, 2, iEndLeaf);
113883		- }
113884		- rc = fts3SegReaderNext(pReader);
113885		-
113886		- finished:
	115383	+ memset(&pReader->aNode[nRoot], 0, FTS3_NODE_PADDING);
	115384	+ }else{
	115385	+ pReader->iCurrentBlock = iStartLeaf-1;
	115386	+ }
	115387	+
113887	115388	if( rc==SQLITE_OK ){
113888	115389	*ppReader = pReader;
113889	115390	}else{
113890	115391	sqlite3Fts3SegReaderFree(p, pReader);
113891	115392	}
		@@ -113976,11 +115477,10 @@
113976	115477	}else{
113977	115478	memset(pReader, 0, nByte);
113978	115479	pReader->iIdx = 0x7FFFFFFF;
113979	115480	pReader->ppNextElem = (Fts3HashElem **)&pReader[1];
113980	115481	memcpy(pReader->ppNextElem, aElem, nElemsizeof(Fts3HashElem ));
113981		- fts3SegReaderNext(pReader);
113982	115482	}
113983	115483	}
113984	115484
113985	115485	if( isPrefix ){
113986	115486	sqlite3_free(aElem);
		@@ -114218,11 +115718,11 @@
114218	115718	/*
114219	115719	** Add term zTerm to the SegmentNode. It is guaranteed that zTerm is larger
114220	115720	** (according to memcmp) than the previous term.
114221	115721	*/
114222	115722	static int fts3NodeAddTerm(
114223		- Fts3Table p, / Virtual table handle */
	115723	+ Fts3Table p, / Virtual table handle */
114224	115724	SegmentNode *ppTree, / IN/OUT: SegmentNode handle */
114225	115725	int isCopyTerm, /* True if zTerm/nTerm is transient */
114226	115726	const char zTerm, / Pointer to buffer containing term */
114227	115727	int nTerm /* Size of term in bytes */
114228	115728	){
		@@ -114848,19 +116348,18 @@
114848	116348	** for, then advance each segment iterator until it points to a term of
114849	116349	** equal or greater value than the specified term. This prevents many
114850	116350	** unnecessary merge/sort operations for the case where single segment
114851	116351	** b-tree leaf nodes contain more than one term.
114852	116352	*/
114853		- if( pFilter->zTerm ){
	116353	+ for(i=0; i<nSegment; i++){
114854	116354	int nTerm = pFilter->nTerm;
114855	116355	const char *zTerm = pFilter->zTerm;
114856		- for(i=0; i<nSegment; i++){
114857		- Fts3SegReader *pSeg = apSegment[i];
114858		- while( fts3SegReaderTermCmp(pSeg, zTerm, nTerm)<0 ){
114859		- rc = fts3SegReaderNext(pSeg);
114860		- if( rc!=SQLITE_OK ) goto finished; }
114861		- }
	116356	+ Fts3SegReader *pSeg = apSegment[i];
	116357	+ do {
	116358	+ rc = fts3SegReaderNext(p, pSeg);
	116359	+ if( rc!=SQLITE_OK ) goto finished;
	116360	+ }while( zTerm && fts3SegReaderTermCmp(pSeg, zTerm, nTerm)<0 );
114862	116361	}
114863	116362
114864	116363	fts3SegReaderSort(apSegment, nSegment, nSegment, fts3SegReaderCmp);
114865	116364	while( apSegment[0]->aNode ){
114866	116365	int nTerm = apSegment[0]->nTerm;
		@@ -114965,11 +116464,11 @@
114965	116464	if( pFilter->zTerm && !isPrefix ){
114966	116465	goto finished;
114967	116466	}
114968	116467
114969	116468	for(i=0; i<nMerge; i++){
114970		- rc = fts3SegReaderNext(apSegment[i]);
	116469	+ rc = fts3SegReaderNext(p, apSegment[i]);
114971	116470	if( rc!=SQLITE_OK ) goto finished;
114972	116471	}
114973	116472	fts3SegReaderSort(apSegment, nSegment, nMerge, fts3SegReaderCmp);
114974	116473	}
114975	116474
		@@ -114991,11 +116490,11 @@
114991	116490	*/
114992	116491	static int fts3SegmentMerge(Fts3Table *p, int iLevel){
114993	116492	int i; /* Iterator variable */
114994	116493	int rc; /* Return code */
114995	116494	int iIdx; /* Index of new segment */
114996		- int iNewLevel; /* Level to create new segment at */
	116495	+ int iNewLevel = 0; /* Level to create new segment at */
114997	116496	sqlite3_stmt *pStmt = 0;
114998	116497	SegmentWriter *pWriter = 0;
114999	116498	int nSegment = 0; /* Number of segments being merged */
115000	116499	Fts3SegReader *apSegment = 0; / Array of Segment iterators */
115001	116500	Fts3SegReader pPending = 0; / Iterator for pending-terms */
		@@ -115280,64 +116779,76 @@
115280	116779	sqlite3_step(pStmt);
115281	116780	*pRC = sqlite3_reset(pStmt);
115282	116781	}
115283	116782
115284	116783	/*
115285		-** Update the 0 record of the %_stat table so that it holds a blob
115286		-** which contains the document count followed by the cumulative
115287		-** document sizes for all columns.
	116784	+** Record 0 of the %_stat table contains a blob consisting of N varints,
	116785	+** where N is the number of user defined columns in the fts3 table plus
	116786	+** two. If nCol is the number of user defined columns, then values of the
	116787	+** varints are set as follows:
	116788	+**
	116789	+** Varint 0: Total number of rows in the table.
	116790	+**
	116791	+** Varint 1..nCol: For each column, the total number of tokens stored in
	116792	+** the column for all rows of the table.
	116793	+**
	116794	+** Varint 1+nCol: The total size, in bytes, of all text values in all
	116795	+** columns of all rows of the table.
	116796	+**
115288	116797	*/
115289	116798	static void fts3UpdateDocTotals(
115290		- int pRC, / The result code */
115291		- Fts3Table p, / Table being updated */
115292		- u32 aSzIns, / Size increases */
115293		- u32 aSzDel, / Size decreases */
115294		- int nChng /* Change in the number of documents */
	116799	+ int pRC, / The result code */
	116800	+ Fts3Table p, / Table being updated */
	116801	+ u32 aSzIns, / Size increases */
	116802	+ u32 aSzDel, / Size decreases */
	116803	+ int nChng /* Change in the number of documents */
115295	116804	){
115296	116805	char pBlob; / Storage for BLOB written into %_stat */
115297	116806	int nBlob; /* Size of BLOB written into %_stat */
115298	116807	u32 a; / Array of integers that becomes the BLOB */
115299	116808	sqlite3_stmt pStmt; / Statement for reading and writing */
115300	116809	int i; /* Loop counter */
115301	116810	int rc; /* Result code from subfunctions */
115302	116811
	116812	+ const int nStat = p->nColumn+2;
	116813	+
115303	116814	if( *pRC ) return;
115304		- a = sqlite3_malloc( (sizeof(u32)+10)*(p->nColumn+1) );
	116815	+ a = sqlite3_malloc( (sizeof(u32)+10)*nStat );
115305	116816	if( a==0 ){
115306	116817	*pRC = SQLITE_NOMEM;
115307	116818	return;
115308	116819	}
115309		- pBlob = (char*)&a[p->nColumn+1];
	116820	+ pBlob = (char*)&a[nStat];
115310	116821	rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
115311	116822	if( rc ){
115312	116823	sqlite3_free(a);
115313	116824	*pRC = rc;
115314	116825	return;
115315	116826	}
115316	116827	if( sqlite3_step(pStmt)==SQLITE_ROW ){
115317		- fts3DecodeIntArray(p->nColumn+1, a,
	116828	+ fts3DecodeIntArray(nStat, a,
115318	116829	sqlite3_column_blob(pStmt, 0),
115319	116830	sqlite3_column_bytes(pStmt, 0));
115320	116831	}else{
115321		- memset(a, 0, sizeof(u32)*(p->nColumn+1) );
	116832	+ memset(a, 0, sizeof(u32)*(nStat) );
115322	116833	}
115323	116834	sqlite3_reset(pStmt);
115324	116835	if( nChng<0 && a[0]<(u32)(-nChng) ){
115325	116836	a[0] = 0;
115326	116837	}else{
115327	116838	a[0] += nChng;
115328	116839	}
115329		- for(i=0; i<p->nColumn; i++){
	116840	+ for(i=0; i<p->nColumn+1; i++){
115330	116841	u32 x = a[i+1];
115331	116842	if( x+aSzIns[i] < aSzDel[i] ){
115332	116843	x = 0;
115333	116844	}else{
115334	116845	x = x + aSzIns[i] - aSzDel[i];
115335	116846	}
115336	116847	a[i+1] = x;
115337	116848	}
115338		- fts3EncodeIntArray(p->nColumn+1, a, pBlob, &nBlob);
	116849	+ fts3EncodeIntArray(nStat, a, pBlob, &nBlob);
115339	116850	rc = fts3SqlStmt(p, SQL_REPLACE_DOCTOTAL, &pStmt, 0);
115340	116851	if( rc ){
115341	116852	sqlite3_free(a);
115342	116853	*pRC = rc;
115343	116854	return;
		@@ -115379,13 +116890,163 @@
115379	116890	rc = SQLITE_OK;
115380	116891	#endif
115381	116892	}else{
115382	116893	rc = SQLITE_ERROR;
115383	116894	}
	116895	+
	116896	+ sqlite3Fts3SegmentsClose(p);
	116897	+ return rc;
	116898	+}
	116899	+
	116900	+/*
	116901	+** Return the deferred doclist associated with deferred token pDeferred.
	116902	+** This function assumes that sqlite3Fts3CacheDeferredDoclists() has already
	116903	+** been called to allocate and populate the doclist.
	116904	+*/
	116905	+SQLITE_PRIVATE char sqlite3Fts3DeferredDoclist(Fts3DeferredToken pDeferred, int *pnByte){
	116906	+ if( pDeferred->pList ){
	116907	+ *pnByte = pDeferred->pList->nData;
	116908	+ return pDeferred->pList->aData;
	116909	+ }
	116910	+ *pnByte = 0;
	116911	+ return 0;
	116912	+}
	116913	+
	116914	+/*
	116915	+** Helper fucntion for FreeDeferredDoclists(). This function removes all
	116916	+** references to deferred doclists from within the tree of Fts3Expr
	116917	+** structures headed by
	116918	+*/
	116919	+static void fts3DeferredDoclistClear(Fts3Expr *pExpr){
	116920	+ if( pExpr ){
	116921	+ fts3DeferredDoclistClear(pExpr->pLeft);
	116922	+ fts3DeferredDoclistClear(pExpr->pRight);
	116923	+ if( pExpr->isLoaded ){
	116924	+ sqlite3_free(pExpr->aDoclist);
	116925	+ pExpr->isLoaded = 0;
	116926	+ pExpr->aDoclist = 0;
	116927	+ pExpr->nDoclist = 0;
	116928	+ pExpr->pCurrent = 0;
	116929	+ pExpr->iCurrent = 0;
	116930	+ }
	116931	+ }
	116932	+}
	116933	+
	116934	+/*
	116935	+** Delete all cached deferred doclists. Deferred doclists are cached
	116936	+** (allocated) by the sqlite3Fts3CacheDeferredDoclists() function.
	116937	+*/
	116938	+SQLITE_PRIVATE void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *pCsr){
	116939	+ Fts3DeferredToken *pDef;
	116940	+ for(pDef=pCsr->pDeferred; pDef; pDef=pDef->pNext){
	116941	+ sqlite3_free(pDef->pList);
	116942	+ pDef->pList = 0;
	116943	+ }
	116944	+ if( pCsr->pDeferred ){
	116945	+ fts3DeferredDoclistClear(pCsr->pExpr);
	116946	+ }
	116947	+}
	116948	+
	116949	+/*
	116950	+** Free all entries in the pCsr->pDeffered list. Entries are added to
	116951	+** this list using sqlite3Fts3DeferToken().
	116952	+*/
	116953	+SQLITE_PRIVATE void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *pCsr){
	116954	+ Fts3DeferredToken *pDef;
	116955	+ Fts3DeferredToken *pNext;
	116956	+ for(pDef=pCsr->pDeferred; pDef; pDef=pNext){
	116957	+ pNext = pDef->pNext;
	116958	+ sqlite3_free(pDef->pList);
	116959	+ sqlite3_free(pDef);
	116960	+ }
	116961	+ pCsr->pDeferred = 0;
	116962	+}
	116963	+
	116964	+/*
	116965	+** Generate deferred-doclists for all tokens in the pCsr->pDeferred list
	116966	+** based on the row that pCsr currently points to.
	116967	+**
	116968	+** A deferred-doclist is like any other doclist with position information
	116969	+** included, except that it only contains entries for a single row of the
	116970	+** table, not for all rows.
	116971	+*/
	116972	+SQLITE_PRIVATE int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *pCsr){
	116973	+ int rc = SQLITE_OK; /* Return code */
	116974	+ if( pCsr->pDeferred ){
	116975	+ int i; /* Used to iterate through table columns */
	116976	+ sqlite3_int64 iDocid; /* Docid of the row pCsr points to */
	116977	+ Fts3DeferredToken pDef; / Used to iterate through deferred tokens */
	116978	+
	116979	+ Fts3Table p = (Fts3Table )pCsr->base.pVtab;
	116980	+ sqlite3_tokenizer *pT = p->pTokenizer;
	116981	+ sqlite3_tokenizer_module const *pModule = pT->pModule;
	116982	+
	116983	+ assert( pCsr->isRequireSeek==0 );
	116984	+ iDocid = sqlite3_column_int64(pCsr->pStmt, 0);
	116985	+
	116986	+ for(i=0; i<p->nColumn && rc==SQLITE_OK; i++){
	116987	+ const char zText = (const char )sqlite3_column_text(pCsr->pStmt, i+1);
	116988	+ sqlite3_tokenizer_cursor *pTC = 0;
	116989	+
	116990	+ rc = pModule->xOpen(pT, zText, -1, &pTC);
	116991	+ while( rc==SQLITE_OK ){
	116992	+ char const zToken; / Buffer containing token */
	116993	+ int nToken; /* Number of bytes in token */
	116994	+ int iDum1, iDum2; /* Dummy variables */
	116995	+ int iPos; /* Position of token in zText */
	116996	+
	116997	+ pTC->pTokenizer = pT;
	116998	+ rc = pModule->xNext(pTC, &zToken, &nToken, &iDum1, &iDum2, &iPos);
	116999	+ for(pDef=pCsr->pDeferred; pDef && rc==SQLITE_OK; pDef=pDef->pNext){
	117000	+ Fts3PhraseToken *pPT = pDef->pToken;
	117001	+ if( (pDef->iCol>=p->nColumn \|\| pDef->iCol==i)
	117002	+ && (pPT->n==nToken \|\| (pPT->isPrefix && pPT->n<nToken))
	117003	+ && (0==memcmp(zToken, pPT->z, pPT->n))
	117004	+ ){
	117005	+ fts3PendingListAppend(&pDef->pList, iDocid, i, iPos, &rc);
	117006	+ }
	117007	+ }
	117008	+ }
	117009	+ if( pTC ) pModule->xClose(pTC);
	117010	+ if( rc==SQLITE_DONE ) rc = SQLITE_OK;
	117011	+ }
	117012	+
	117013	+ for(pDef=pCsr->pDeferred; pDef && rc==SQLITE_OK; pDef=pDef->pNext){
	117014	+ if( pDef->pList ){
	117015	+ rc = fts3PendingListAppendVarint(&pDef->pList, 0);
	117016	+ }
	117017	+ }
	117018	+ }
115384	117019
115385	117020	return rc;
115386	117021	}
	117022	+
	117023	+/*
	117024	+** Add an entry for token pToken to the pCsr->pDeferred list.
	117025	+*/
	117026	+SQLITE_PRIVATE int sqlite3Fts3DeferToken(
	117027	+ Fts3Cursor pCsr, / Fts3 table cursor */
	117028	+ Fts3PhraseToken pToken, / Token to defer */
	117029	+ int iCol /* Column that token must appear in (or -1) */
	117030	+){
	117031	+ Fts3DeferredToken *pDeferred;
	117032	+ pDeferred = sqlite3_malloc(sizeof(*pDeferred));
	117033	+ if( !pDeferred ){
	117034	+ return SQLITE_NOMEM;
	117035	+ }
	117036	+ memset(pDeferred, 0, sizeof(*pDeferred));
	117037	+ pDeferred->pToken = pToken;
	117038	+ pDeferred->pNext = pCsr->pDeferred;
	117039	+ pDeferred->iCol = iCol;
	117040	+ pCsr->pDeferred = pDeferred;
	117041	+
	117042	+ assert( pToken->pDeferred==0 );
	117043	+ pToken->pDeferred = pDeferred;
	117044	+
	117045	+ return SQLITE_OK;
	117046	+}
	117047	+
115387	117048
115388	117049	/*
115389	117050	** This function does the work for the xUpdate method of FTS3 virtual
115390	117051	** tables.
115391	117052	*/
		@@ -115401,20 +117062,21 @@
115401	117062	sqlite3_int64 iRemove = 0; /* Rowid removed by UPDATE or DELETE */
115402	117063	u32 aSzIns; / Sizes of inserted documents */
115403	117064	u32 aSzDel; / Sizes of deleted documents */
115404	117065	int nChng = 0; /* Net change in number of documents */
115405	117066
	117067	+ assert( p->pSegments==0 );
115406	117068
115407	117069	/* Allocate space to hold the change in document sizes */
115408		- aSzIns = sqlite3_malloc( sizeof(aSzIns[0])p->nColumn2 );
	117070	+ aSzIns = sqlite3_malloc( sizeof(aSzIns[0])(p->nColumn+1)2 );
115409	117071	if( aSzIns==0 ) return SQLITE_NOMEM;
115410		- aSzDel = &aSzIns[p->nColumn];
115411		- memset(aSzIns, 0, sizeof(aSzIns[0])p->nColumn2);
	117072	+ aSzDel = &aSzIns[p->nColumn+1];
	117073	+ memset(aSzIns, 0, sizeof(aSzIns[0])(p->nColumn+1)2);
115412	117074
115413	117075	/* If this is a DELETE or UPDATE operation, remove the old record. */
115414	117076	if( sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
115415		- int isEmpty;
	117077	+ int isEmpty = 0;
115416	117078	rc = fts3IsEmpty(p, apVal, &isEmpty);
115417	117079	if( rc==SQLITE_OK ){
115418	117080	if( isEmpty ){
115419	117081	/* Deleting this row means the whole table is empty. In this case
115420	117082	** delete the contents of all three tables and throw away any
		@@ -115427,12 +117089,12 @@
115427	117089	rc = fts3PendingTermsDocid(p, iRemove);
115428	117090	fts3DeleteTerms(&rc, p, apVal, aSzDel);
115429	117091	fts3SqlExec(&rc, p, SQL_DELETE_CONTENT, apVal);
115430	117092	if( p->bHasDocsize ){
115431	117093	fts3SqlExec(&rc, p, SQL_DELETE_DOCSIZE, apVal);
115432		- nChng--;
115433	117094	}
	117095	+ nChng--;
115434	117096	}
115435	117097	}
115436	117098	}else if( sqlite3_value_type(apVal[p->nColumn+2])!=SQLITE_NULL ){
115437	117099	sqlite3_free(aSzIns);
115438	117100	return fts3SpecialInsert(p, apVal[p->nColumn+2]);
		@@ -115446,20 +117108,21 @@
115446	117108	}
115447	117109	if( rc==SQLITE_OK ){
115448	117110	rc = fts3InsertTerms(p, apVal, aSzIns);
115449	117111	}
115450	117112	if( p->bHasDocsize ){
115451		- nChng++;
115452	117113	fts3InsertDocsize(&rc, p, aSzIns);
115453	117114	}
	117115	+ nChng++;
115454	117116	}
115455	117117
115456		- if( p->bHasDocsize ){
	117118	+ if( p->bHasStat ){
115457	117119	fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng);
115458	117120	}
115459	117121
115460	117122	sqlite3_free(aSzIns);
	117123	+ sqlite3Fts3SegmentsClose(p);
115461	117124	return rc;
115462	117125	}
115463	117126
115464	117127	/*
115465	117128	** Flush any data in the pending-terms hash table to disk. If successful,
		@@ -115479,10 +117142,11 @@
115479	117142	}else{
115480	117143	sqlite3_exec(p->db, "ROLLBACK TO fts3", 0, 0, 0);
115481	117144	sqlite3_exec(p->db, "RELEASE fts3", 0, 0, 0);
115482	117145	}
115483	117146	}
	117147	+ sqlite3Fts3SegmentsClose(p);
115484	117148	return rc;
115485	117149	}
115486	117150
115487	117151	#endif
115488	117152
		@@ -115509,11 +117173,11 @@
115509	117173	** Used as an fts3ExprIterate() context when loading phrase doclists to
115510	117174	** Fts3Expr.aDoclist[]/nDoclist.
115511	117175	*/
115512	117176	typedef struct LoadDoclistCtx LoadDoclistCtx;
115513	117177	struct LoadDoclistCtx {
115514		- Fts3Table pTab; / FTS3 Table */
	117178	+ Fts3Cursor pCsr; / FTS3 Cursor */
115515	117179	int nPhrase; /* Number of phrases seen so far */
115516	117180	int nToken; /* Number of tokens seen so far */
115517	117181	};
115518	117182
115519	117183	/*
		@@ -115703,11 +117367,11 @@
115703	117367
115704	117368	p->nPhrase++;
115705	117369	p->nToken += pExpr->pPhrase->nToken;
115706	117370
115707	117371	if( pExpr->isLoaded==0 ){
115708		- rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr);
	117372	+ rc = sqlite3Fts3ExprLoadDoclist(p->pCsr, pExpr);
115709	117373	pExpr->isLoaded = 1;
115710	117374	if( rc==SQLITE_OK ){
115711	117375	rc = fts3ExprNearTrim(pExpr);
115712	117376	}
115713	117377	}
		@@ -115746,11 +117410,11 @@
115746	117410	int pnPhrase, / OUT: Number of phrases in query */
115747	117411	int pnToken / OUT: Number of tokens in query */
115748	117412	){
115749	117413	int rc; /* Return Code */
115750	117414	LoadDoclistCtx sCtx = {0,0,0}; /* Context for fts3ExprIterate() */
115751		- sCtx.pTab = (Fts3Table *)pCsr->base.pVtab;
	117415	+ sCtx.pCsr = pCsr;
115752	117416	rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx);
115753	117417	if( rc==SQLITE_OK ){
115754	117418	(void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0);
115755	117419	}
115756	117420	if( pnPhrase ) *pnPhrase = sCtx.nPhrase;
		@@ -116277,24 +117941,51 @@
116277	117941	Fts3Expr pExpr, / Phrase expression node */
116278	117942	int iPhrase, /* Phrase number (numbered from zero) */
116279	117943	void pCtx / Pointer to MatchInfo structure */
116280	117944	){
116281	117945	MatchInfo p = (MatchInfo )pCtx;
116282		- char *pCsr;
	117946	+ Fts3Cursor *pCsr = p->pCursor;
	117947	+ char *pIter;
116283	117948	char *pEnd;
	117949	+ char *pFree = 0;
116284	117950	const int iStart = 2 + (iPhrase * p->nCol * 3) + 1;
116285	117951
116286	117952	assert( pExpr->isLoaded );
	117953	+ assert( pExpr->eType==FTSQUERY_PHRASE );
	117954	+
	117955	+ if( pCsr->pDeferred ){
	117956	+ Fts3Phrase *pPhrase = pExpr->pPhrase;
	117957	+ int ii;
	117958	+ for(ii=0; ii<pPhrase->nToken; ii++){
	117959	+ if( pPhrase->aToken[ii].bFulltext ) break;
	117960	+ }
	117961	+ if( ii<pPhrase->nToken ){
	117962	+ int nFree = 0;
	117963	+ int rc = sqlite3Fts3ExprLoadFtDoclist(pCsr, pExpr, &pFree, &nFree);
	117964	+ if( rc!=SQLITE_OK ) return rc;
	117965	+ pIter = pFree;
	117966	+ pEnd = &pFree[nFree];
	117967	+ }else{
	117968	+ int nDoc = p->aMatchinfo[2 + 3p->nColp->aMatchinfo[0]];
	117969	+ for(ii=0; ii<p->nCol; ii++){
	117970	+ p->aMatchinfo[iStart + ii*3] = nDoc;
	117971	+ p->aMatchinfo[iStart + ii*3 + 1] = nDoc;
	117972	+ }
	117973	+ return SQLITE_OK;
	117974	+ }
	117975	+ }else{
	117976	+ pIter = pExpr->aDoclist;
	117977	+ pEnd = &pExpr->aDoclist[pExpr->nDoclist];
	117978	+ }
116287	117979
116288	117980	/* Fill in the global hit count matrix row for this phrase. */
116289		- pCsr = pExpr->aDoclist;
116290		- pEnd = &pExpr->aDoclist[pExpr->nDoclist];
116291		- while( pCsr<pEnd ){
116292		- while( pCsr++ & 0x80 ); / Skip past docid. */
116293		- fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 1);
	117981	+ while( pIter<pEnd ){
	117982	+ while( pIter++ & 0x80 ); / Skip past docid. */
	117983	+ fts3LoadColumnlistCounts(&pIter, &p->aMatchinfo[iStart], 1);
116294	117984	}
116295	117985
	117986	+ sqlite3_free(pFree);
116296	117987	return SQLITE_OK;
116297	117988	}
116298	117989
116299	117990	/*
116300	117991	** fts3ExprIterate() callback used to collect the "local" matchinfo stats
		@@ -116359,19 +118050,18 @@
116359	118050	if( !sInfo.aMatchinfo ){
116360	118051	return SQLITE_NOMEM;
116361	118052	}
116362	118053	memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo);
116363	118054
116364		-
116365	118055	/* First element of match-info is the number of phrases in the query */
116366	118056	sInfo.aMatchinfo[0] = nPhrase;
116367	118057	sInfo.aMatchinfo[1] = sInfo.nCol;
116368		- (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);
116369	118058	if( pTab->bHasDocsize ){
116370	118059	int ofst = 2 + 3sInfo.aMatchinfo[0]sInfo.aMatchinfo[1];
116371	118060	rc = sqlite3Fts3MatchinfoDocsizeGlobal(pCsr, &sInfo.aMatchinfo[ofst]);
116372	118061	}
	118062	+ (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);
116373	118063	pCsr->aMatchinfo = sInfo.aMatchinfo;
116374	118064	pCsr->isMatchinfoNeeded = 1;
116375	118065	}
116376	118066
116377	118067	sInfo.aMatchinfo = pCsr->aMatchinfo;
		@@ -116477,10 +118167,11 @@
116477	118167	i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res
116478	118168	);
116479	118169	}
116480	118170
116481	118171	snippet_out:
	118172	+ sqlite3Fts3SegmentsClose(pTab);
116482	118173	if( rc!=SQLITE_OK ){
116483	118174	sqlite3_result_error_code(pCtx, rc);
116484	118175	sqlite3_free(res.z);
116485	118176	}else{
116486	118177	sqlite3_result_text(pCtx, res.z, -1, sqlite3_free);
		@@ -116656,10 +118347,11 @@
116656	118347	}
116657	118348
116658	118349	offsets_out:
116659	118350	sqlite3_free(sCtx.aTerm);
116660	118351	assert( rc!=SQLITE_DONE );
	118352	+ sqlite3Fts3SegmentsClose(pTab);
116661	118353	if( rc!=SQLITE_OK ){
116662	118354	sqlite3_result_error_code(pCtx, rc);
116663	118355	sqlite3_free(res.z);
116664	118356	}else{
116665	118357	sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free);
		@@ -116675,10 +118367,11 @@
116675	118367	if( !pCsr->pExpr ){
116676	118368	sqlite3_result_blob(pContext, "", 0, SQLITE_STATIC);
116677	118369	return;
116678	118370	}
116679	118371	rc = fts3GetMatchinfo(pCsr);
	118372	+ sqlite3Fts3SegmentsClose((Fts3Table *)pCsr->base.pVtab );
116680	118373	if( rc!=SQLITE_OK ){
116681	118374	sqlite3_result_error_code(pContext, rc);
116682	118375	}else{
116683	118376	Fts3Table pTab = (Fts3Table)pCsr->base.pVtab;
116684	118377	int n = sizeof(u32)(2+pCsr->aMatchinfo[0]pCsr->aMatchinfo[1]*3);
116685	118378

	--- src/sqlite3.c
	+++ src/sqlite3.c
	@@ -1,12 +1,12 @@
1	/******************************************************************************
2	** This file is an amalgamation of many separate C source files from SQLite
3	** version 3.7.3. By combining all the individual C code files into this
4	** single large file, the entire code can be compiled as a one translation
5	** unit. This allows many compilers to do optimizations that would not be
6	** possible if the files were compiled separately. Performance improvements
7	** of 5% are more are commonly seen when SQLite is compiled as a single
8	** translation unit.
9	**
10	** This file is all you need to compile SQLite. To use SQLite in other
11	** programs, you need this file and the "sqlite3.h" header file that defines
12	** the programming interface to the SQLite library. (If you do not have
	@@ -648,13 +648,13 @@
648	**
649	** See also: [sqlite3_libversion()],
650	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
651	** [sqlite_version()] and [sqlite_source_id()].
652	*/
653	#define SQLITE_VERSION "3.7.3"
654	#define SQLITE_VERSION_NUMBER 3007003
655	#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e"
656
657	/*
658	** CAPI3REF: Run-Time Library Version Numbers
659	** KEYWORDS: sqlite3_version, sqlite3_sourceid
660	**
	@@ -3273,11 +3273,14 @@
3273	** ^If the fourth parameter is negative, the length of the string is
3274	** the number of bytes up to the first zero terminator.
3275	**
3276	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
3277	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
3278	** string after SQLite has finished with it. ^If the fifth argument is



3279	** the special value [SQLITE_STATIC], then SQLite assumes that the
3280	** information is in static, unmanaged space and does not need to be freed.
3281	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
3282	** SQLite makes its own private copy of the data immediately, before
3283	** the sqlite3_bind_*() routine returns.
	@@ -3913,16 +3916,19 @@
3913	** parameters. ^An aggregate SQL function requires an implementation of xStep
3914	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3915	** SQL function or aggregate, pass NULL poiners for all three function
3916	** callbacks.
3917	**
3918	** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3919	** then it is invoked when the function is deleted, either by being
3920	** overloaded or when the database connection closes.
3921	** ^When the destructure callback of the tenth parameter is invoked, it
3922	** is passed a single argument which is a copy of the pointer which was
3923	** the fifth parameter to sqlite3_create_function_v2().



3924	**
3925	** ^It is permitted to register multiple implementations of the same
3926	** functions with the same name but with either differing numbers of
3927	** arguments or differing preferred text encodings. ^SQLite will use
3928	** the implementation that most closely matches the way in which the
	@@ -4381,10 +4387,19 @@
4381	** with the addition that the xDestroy callback is invoked on pArg when
4382	** the collating function is deleted.
4383	** ^Collating functions are deleted when they are overridden by later
4384	** calls to the collation creation functions or when the
4385	** [database connection] is closed using [sqlite3_close()].









4386	**
4387	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
4388	*/
4389	SQLITE_API int sqlite3_create_collation(
4390	sqlite3*,
	@@ -5136,11 +5151,13 @@
5136	** when a new virtual table is be being created or reinitialized.
5137	**
5138	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
5139	** is a pointer to a destructor for the pClientData. ^SQLite will
5140	** invoke the destructor function (if it is not NULL) when SQLite
5141	** no longer needs the pClientData pointer. ^The sqlite3_create_module()


5142	** interface is equivalent to sqlite3_create_module_v2() with a NULL
5143	** destructor.
5144	*/
5145	SQLITE_API int sqlite3_create_module(
5146	sqlite3 db, / SQLite connection to register module with */
	@@ -5319,10 +5336,33 @@
5319	sqlite3_int64 iRow,
5320	int flags,
5321	sqlite3_blob **ppBlob
5322	);
5323























5324	/*
5325	** CAPI3REF: Close A BLOB Handle
5326	**
5327	** ^Closes an open [BLOB handle].
5328	**
	@@ -9890,10 +9930,13 @@
9890	Expr pOn; / The ON clause of a join */
9891	IdList pUsing; / The USING clause of a join */
9892	Bitmask colUsed; /* Bit N (1<<N) set if column N of pTab is used */
9893	char zIndex; / Identifier from "INDEXED BY <zIndex>" clause */
9894	Index pIndex; / Index structure corresponding to zIndex, if any */



9895	} a[1]; /* One entry for each identifier on the list */
9896	};
9897
9898	/*
9899	** Permitted values of the SrcList.a.jointype field
	@@ -9922,10 +9965,11 @@
9922	** case that more than one of these conditions is true.
9923	*/
9924	struct WherePlan {
9925	u32 wsFlags; /* WHERE_* flags that describe the strategy */
9926	u32 nEq; /* Number of == constraints */

9927	union {
9928	Index pIdx; / Index when WHERE_INDEXED is true */
9929	struct WhereTerm pTerm; / WHERE clause term for OR-search */
9930	sqlite3_index_info pVtabIdx; / Virtual table index to use */
9931	} u;
	@@ -10276,10 +10320,15 @@
10276	Table *apVtabLock; / Pointer to virtual tables needing locking */
10277	#endif
10278	int nHeight; /* Expression tree height of current sub-select */
10279	Table pZombieTab; / List of Table objects to delete after code gen */
10280	TriggerPrg pTriggerPrg; / Linked list of coded triggers */





10281	};
10282
10283	#ifdef SQLITE_OMIT_VIRTUALTABLE
10284	#define IN_DECLARE_VTAB 0
10285	#else
	@@ -27311,13 +27360,28 @@
27311	if( flags & (SQLITE_OPEN_WAL\|SQLITE_OPEN_MAIN_JOURNAL) ){
27312	char zDb[MAX_PATHNAME+1]; /* Database file path */
27313	int nDb; /* Number of valid bytes in zDb */
27314	struct stat sStat; /* Output of stat() on database file */
27315
27316	nDb = sqlite3Strlen30(zPath) - ((flags & SQLITE_OPEN_WAL) ? 4 : 8);














27317	memcpy(zDb, zPath, nDb);
27318	zDb[nDb] = '\0';

27319	if( 0==stat(zDb, &sStat) ){
27320	*pMode = sStat.st_mode & 0777;
27321	}else{
27322	rc = SQLITE_IOERR_FSTAT;
27323	}
	@@ -34519,10 +34583,11 @@
34519	# define sqlite3WalSavepointUndo(y,z) 0
34520	# define sqlite3WalFrames(u,v,w,x,y,z) 0
34521	# define sqlite3WalCheckpoint(u,v,w,x) 0
34522	# define sqlite3WalCallback(z) 0
34523	# define sqlite3WalExclusiveMode(y,z) 0

34524	#else
34525
34526	#define WAL_SAVEPOINT_NDATA 4
34527
34528	/* Connection to a write-ahead log (WAL) file.
	@@ -34529,11 +34594,11 @@
34529	** There is one object of this type for each pager.
34530	*/
34531	typedef struct Wal Wal;
34532
34533	/* Open and close a connection to a write-ahead log. */
34534	SQLITE_PRIVATE int sqlite3WalOpen(sqlite3_vfs, sqlite3_file, const char zName, Wal*);
34535	SQLITE_PRIVATE int sqlite3WalClose(Wal pWal, int sync_flags, int, u8 );
34536
34537	/* Used by readers to open (lock) and close (unlock) a snapshot. A
34538	** snapshot is like a read-transaction. It is the state of the database
34539	** at an instant in time. sqlite3WalOpenSnapshot gets a read lock and
	@@ -34585,10 +34650,16 @@
34585
34586	/* Tell the wal layer that an EXCLUSIVE lock has been obtained (or released)
34587	** by the pager layer on the database file.
34588	*/
34589	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op);






34590
34591	#endif /* ifndef SQLITE_OMIT_WAL */
34592	#endif /* _WAL_H_ */
34593
34594	/************ End of wal.h ***********************************************/
	@@ -35496,11 +35567,13 @@
35496	break;
35497	}
35498
35499	return 1;
35500	}

35501

35502	/*
35503	** Return a pointer to a human readable string in a static buffer
35504	** containing the state of the Pager object passed as an argument. This
35505	** is intended to be used within debuggers. For example, as an alternative
35506	** to "print *pPager" in gdb:
	@@ -35620,11 +35693,11 @@
35620	** UNKNOWN_LOCK for an explanation of this.
35621	*/
35622	static int pagerUnlockDb(Pager *pPager, int eLock){
35623	int rc = SQLITE_OK;
35624
35625	assert( !pPager->exclusiveMode );
35626	assert( eLock==NO_LOCK \|\| eLock==SHARED_LOCK );
35627	assert( eLock!=NO_LOCK \|\| pagerUseWal(pPager)==0 );
35628	if( isOpen(pPager->fd) ){
35629	assert( pPager->eLock>=eLock );
35630	rc = sqlite3OsUnlock(pPager->fd, eLock);
	@@ -39094,11 +39167,11 @@
39094	if( rc==SQLITE_OK ){
39095	if( nPage==0 ){
39096	sqlite3BeginBenignMalloc();
39097	if( pagerLockDb(pPager, RESERVED_LOCK)==SQLITE_OK ){
39098	sqlite3OsDelete(pVfs, pPager->zJournal, 0);
39099	pagerUnlockDb(pPager, SHARED_LOCK);
39100	}
39101	sqlite3EndBenignMalloc();
39102	}else{
39103	/* The journal file exists and no other connection has a reserved
39104	** or greater lock on the database file. Now check that there is
	@@ -40920,11 +40993,12 @@
40920	assert( eMode==PAGER_LOCKINGMODE_QUERY
40921	\|\| eMode==PAGER_LOCKINGMODE_NORMAL
40922	\|\| eMode==PAGER_LOCKINGMODE_EXCLUSIVE );
40923	assert( PAGER_LOCKINGMODE_QUERY<0 );
40924	assert( PAGER_LOCKINGMODE_NORMAL>=0 && PAGER_LOCKINGMODE_EXCLUSIVE>=0 );
40925	if( eMode>=0 && !pPager->tempFile ){

40926	pPager->exclusiveMode = (u8)eMode;
40927	}
40928	return (int)pPager->exclusiveMode;
40929	}
40930
	@@ -41107,12 +41181,64 @@
41107	** Return true if the underlying VFS for the given pager supports the
41108	** primitives necessary for write-ahead logging.
41109	*/
41110	SQLITE_PRIVATE int sqlite3PagerWalSupported(Pager *pPager){
41111	const sqlite3_io_methods *pMethods = pPager->fd->pMethods;
41112	return pMethods->iVersion>=2 && pMethods->xShmMap!=0;
41113	}




















































41114
41115	/*
41116	** The caller must be holding a SHARED lock on the database file to call
41117	** this function.
41118	**
	@@ -41143,15 +41269,11 @@
41143	if( !sqlite3PagerWalSupported(pPager) ) return SQLITE_CANTOPEN;
41144
41145	/* Close any rollback journal previously open */
41146	sqlite3OsClose(pPager->jfd);
41147
41148	/* Open the connection to the log file. If this operation fails,
41149	** (e.g. due to malloc() failure), unlock the database file and
41150	** return an error code.
41151	*/
41152	rc = sqlite3WalOpen(pPager->pVfs, pPager->fd, pPager->zWal, &pPager->pWal);
41153	if( rc==SQLITE_OK ){
41154	pPager->journalMode = PAGER_JOURNALMODE_WAL;
41155	pPager->eState = PAGER_OPEN;
41156	}
41157	}else{
	@@ -41186,30 +41308,25 @@
41186	rc = sqlite3OsAccess(
41187	pPager->pVfs, pPager->zWal, SQLITE_ACCESS_EXISTS, &logexists
41188	);
41189	}
41190	if( rc==SQLITE_OK && logexists ){
41191	rc = sqlite3WalOpen(pPager->pVfs, pPager->fd,
41192	pPager->zWal, &pPager->pWal);
41193	}
41194	}
41195
41196	/* Checkpoint and close the log. Because an EXCLUSIVE lock is held on
41197	** the database file, the log and log-summary files will be deleted.
41198	*/
41199	if( rc==SQLITE_OK && pPager->pWal ){
41200	rc = pagerLockDb(pPager, EXCLUSIVE_LOCK);
41201	if( rc==SQLITE_OK ){
41202	rc = sqlite3WalClose(pPager->pWal,
41203	(pPager->noSync ? 0 : pPager->sync_flags),
41204	pPager->pageSize, (u8*)pPager->pTmpSpace
41205	);
41206	pPager->pWal = 0;
41207	}else{
41208	/* If we cannot get an EXCLUSIVE lock, downgrade the PENDING lock
41209	** that we did get back to SHARED. */
41210	pagerUnlockDb(pPager, SQLITE_LOCK_SHARED);
41211	}
41212	}
41213	return rc;
41214	}
41215
	@@ -41661,10 +41778,17 @@
41661	#ifdef SQLITE_DEBUG
41662	u8 lockError; /* True if a locking error has occurred */
41663	#endif
41664	};
41665







41666	/*
41667	** Each page of the wal-index mapping contains a hash-table made up of
41668	** an array of HASHTABLE_NSLOT elements of the following type.
41669	*/
41670	typedef u16 ht_slot;
	@@ -41747,13 +41871,18 @@
41747	pWal->nWiData = iPage+1;
41748	}
41749
41750	/* Request a pointer to the required page from the VFS */
41751	if( pWal->apWiData[iPage]==0 ){
41752	rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ,
41753	pWal->writeLock, (void volatile **)&pWal->apWiData[iPage]
41754	);





41755	}
41756
41757	*ppPage = pWal->apWiData[iPage];
41758	assert( iPage==0 \|\| *ppPage \|\| rc!=SQLITE_OK );
41759	return rc;
	@@ -41831,10 +41960,16 @@
41831	}
41832
41833	aOut[0] = s1;
41834	aOut[1] = s2;
41835	}






41836
41837	/*
41838	** Write the header information in pWal->hdr into the wal-index.
41839	**
41840	** The checksum on pWal->hdr is updated before it is written.
	@@ -41846,11 +41981,11 @@
41846	assert( pWal->writeLock );
41847	pWal->hdr.isInit = 1;
41848	pWal->hdr.iVersion = WALINDEX_MAX_VERSION;
41849	walChecksumBytes(1, (u8*)&pWal->hdr, nCksum, 0, pWal->hdr.aCksum);
41850	memcpy((void )&aHdr[1], (void )&pWal->hdr, sizeof(WalIndexHdr));
41851	sqlite3OsShmBarrier(pWal->pDbFd);
41852	memcpy((void )&aHdr[0], (void )&pWal->hdr, sizeof(WalIndexHdr));
41853	}
41854
41855	/*
41856	** This function encodes a single frame header and writes it to a buffer
	@@ -42418,11 +42553,19 @@
42418
42419	/*
42420	** Close an open wal-index.
42421	*/
42422	static void walIndexClose(Wal *pWal, int isDelete){
42423	sqlite3OsShmUnmap(pWal->pDbFd, isDelete);








42424	}
42425
42426	/*
42427	** Open a connection to the WAL file zWalName. The database file must
42428	** already be opened on connection pDbFd. The buffer that zWalName points
	@@ -42440,10 +42583,11 @@
42440	*/
42441	SQLITE_PRIVATE int sqlite3WalOpen(
42442	sqlite3_vfs pVfs, / vfs module to open wal and wal-index */
42443	sqlite3_file pDbFd, / The open database file */
42444	const char zWalName, / Name of the WAL file */

42445	Wal *ppWal / OUT: Allocated Wal handle */
42446	){
42447	int rc; /* Return Code */
42448	Wal pRet; / Object to allocate and return */
42449	int flags; /* Flags passed to OsOpen() */
	@@ -42473,10 +42617,11 @@
42473	pRet->pVfs = pVfs;
42474	pRet->pWalFd = (sqlite3_file *)&pRet[1];
42475	pRet->pDbFd = pDbFd;
42476	pRet->readLock = -1;
42477	pRet->zWalName = zWalName;

42478
42479	/* Open file handle on the write-ahead log file. */
42480	flags = (SQLITE_OPEN_READWRITE\|SQLITE_OPEN_CREATE\|SQLITE_OPEN_WAL);
42481	rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, flags, &flags);
42482	if( rc==SQLITE_OK && flags&SQLITE_OPEN_READONLY ){
	@@ -42906,11 +43051,13 @@
42906	**
42907	** The EXCLUSIVE lock is not released before returning.
42908	*/
42909	rc = sqlite3OsLock(pWal->pDbFd, SQLITE_LOCK_EXCLUSIVE);
42910	if( rc==SQLITE_OK ){
42911	pWal->exclusiveMode = 1;


42912	rc = sqlite3WalCheckpoint(pWal, sync_flags, nBuf, zBuf);
42913	if( rc==SQLITE_OK ){
42914	isDelete = 1;
42915	}
42916	}
	@@ -42962,11 +43109,11 @@
42962	** Memory barriers are used to prevent the compiler or the hardware from
42963	** reordering the reads and writes.
42964	*/
42965	aHdr = walIndexHdr(pWal);
42966	memcpy(&h1, (void *)&aHdr[0], sizeof(h1));
42967	sqlite3OsShmBarrier(pWal->pDbFd);
42968	memcpy(&h2, (void *)&aHdr[1], sizeof(h2));
42969
42970	if( memcmp(&h1, &h2, sizeof(h1))!=0 ){
42971	return 1; /* Dirty read */
42972	}
	@@ -43163,11 +43310,11 @@
43163	if( !useWal && pInfo->nBackfill==pWal->hdr.mxFrame ){
43164	/* The WAL has been completely backfilled (or it is empty).
43165	** and can be safely ignored.
43166	*/
43167	rc = walLockShared(pWal, WAL_READ_LOCK(0));
43168	sqlite3OsShmBarrier(pWal->pDbFd);
43169	if( rc==SQLITE_OK ){
43170	if( memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr)) ){
43171	/* It is not safe to allow the reader to continue here if frames
43172	** may have been appended to the log before READ_LOCK(0) was obtained.
43173	** When holding READ_LOCK(0), the reader ignores the entire log file,
	@@ -43257,11 +43404,11 @@
43257	** date before proceeding. That would not be possible without somehow
43258	** blocking writers. It only guarantees that a dangerous checkpoint or
43259	** log-wrap (either of which would require an exclusive lock on
43260	** WAL_READ_LOCK(mxI)) has not occurred since the snapshot was valid.
43261	*/
43262	sqlite3OsShmBarrier(pWal->pDbFd);
43263	if( pInfo->aReadMark[mxI]!=mxReadMark
43264	\|\| memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr))
43265	){
43266	walUnlockShared(pWal, WAL_READ_LOCK(mxI));
43267	return WAL_RETRY;
	@@ -43900,17 +44047,18 @@
43900	** WAL is already in exclusive-locking mode - meaning that this
43901	** routine is a no-op. The pager must already hold the exclusive lock
43902	** on the main database file before invoking this operation.
43903	**
43904	** If op is negative, then do a dry-run of the op==1 case but do
43905	** not actually change anything. The pager uses this to see if it
43906	** should acquire the database exclusive lock prior to invoking
43907	** the op==1 case.
43908	*/
43909	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op){
43910	int rc;
43911	assert( pWal->writeLock==0 );

43912
43913	/* pWal->readLock is usually set, but might be -1 if there was a
43914	** prior error while attempting to acquire are read-lock. This cannot
43915	** happen if the connection is actually in exclusive mode (as no xShmLock
43916	** locks are taken in this case). Nor should the pager attempt to
	@@ -43939,10 +44087,19 @@
43939	}else{
43940	rc = pWal->exclusiveMode==0;
43941	}
43942	return rc;
43943	}









43944
43945	#endif /* #ifndef SQLITE_OMIT_WAL */
43946
43947	/************ End of wal.c ***********************************************/
43948	/************ Begin file btmutex.c ***************************************/
	@@ -48119,20 +48276,21 @@
48119	**
48120	** This will release the write lock on the database file. If there
48121	** are no active cursors, it also releases the read lock.
48122	*/
48123	SQLITE_PRIVATE int sqlite3BtreeCommitPhaseTwo(Btree *p){
48124	BtShared *pBt = p->pBt;
48125

48126	sqlite3BtreeEnter(p);
48127	btreeIntegrity(p);
48128
48129	/* If the handle has a write-transaction open, commit the shared-btrees
48130	** transaction and set the shared state to TRANS_READ.
48131	*/
48132	if( p->inTrans==TRANS_WRITE ){
48133	int rc;

48134	assert( pBt->inTransaction==TRANS_WRITE );
48135	assert( pBt->nTransaction>0 );
48136	rc = sqlite3PagerCommitPhaseTwo(pBt->pPager);
48137	if( rc!=SQLITE_OK ){
48138	sqlite3BtreeLeave(p);
	@@ -53047,12 +53205,11 @@
53047	** sqlite3BtreePutData()).
53048	*/
53049	SQLITE_PRIVATE void sqlite3BtreeCacheOverflow(BtCursor *pCur){
53050	assert( cursorHoldsMutex(pCur) );
53051	assert( sqlite3_mutex_held(pCur->pBtree->db->mutex) );
53052	assert(!pCur->isIncrblobHandle);
53053	assert(!pCur->aOverflow);
53054	pCur->isIncrblobHandle = 1;
53055	}
53056	#endif
53057
53058	/*
	@@ -56081,16 +56238,14 @@
56081	pMem->flags = MEM_Int;
56082	pMem->u.i = pOp->p2; /* P2 */
56083	pMem->type = SQLITE_INTEGER;
56084	pMem++;
56085
56086	if( p->explain==1 ){
56087	pMem->flags = MEM_Int;
56088	pMem->u.i = pOp->p3; /* P3 */
56089	pMem->type = SQLITE_INTEGER;
56090	pMem++;
56091	}
56092
56093	if( sqlite3VdbeMemGrow(pMem, 32, 0) ){ /* P4 */
56094	assert( p->db->mallocFailed );
56095	return SQLITE_ERROR;
56096	}
	@@ -56131,11 +56286,11 @@
56131	pMem->flags = MEM_Null; /* Comment */
56132	pMem->type = SQLITE_NULL;
56133	}
56134	}
56135
56136	p->nResColumn = 8 - 5*(p->explain-1);
56137	p->rc = SQLITE_OK;
56138	rc = SQLITE_ROW;
56139	}
56140	return rc;
56141	}
	@@ -59078,10 +59233,12 @@
59078	}
59079	sqlite3Error(p->db, rc, 0);
59080	rc = sqlite3ApiExit(p->db, rc);
59081	}
59082	sqlite3_mutex_leave(p->db->mutex);


59083	}
59084	return rc;
59085	}
59086
59087
	@@ -66071,15 +66228,86 @@
66071	typedef struct Incrblob Incrblob;
66072	struct Incrblob {
66073	int flags; /* Copy of "flags" passed to sqlite3_blob_open() */
66074	int nByte; /* Size of open blob, in bytes */
66075	int iOffset; /* Byte offset of blob in cursor data */

66076	BtCursor pCsr; / Cursor pointing at blob row */
66077	sqlite3_stmt pStmt; / Statement holding cursor open */
66078	sqlite3 db; / The associated database */
66079	};
66080






































































66081	/*
66082	** Open a blob handle.
66083	*/
66084	SQLITE_API int sqlite3_blob_open(
66085	sqlite3* db, /* The database connection */
	@@ -66116,33 +66344,39 @@
66116	/* One of the following two instructions is replaced by an OP_Noop. */
66117	{OP_OpenRead, 0, 0, 0}, /* 3: Open cursor 0 for reading */
66118	{OP_OpenWrite, 0, 0, 0}, /* 4: Open cursor 0 for read/write */
66119
66120	{OP_Variable, 1, 1, 1}, /* 5: Push the rowid to the stack */
66121	{OP_NotExists, 0, 9, 1}, /* 6: Seek the cursor */
66122	{OP_Column, 0, 0, 1}, /* 7 */
66123	{OP_ResultRow, 1, 0, 0}, /* 8 */
66124	{OP_Close, 0, 0, 0}, /* 9 */
66125	{OP_Halt, 0, 0, 0}, /* 10 */

66126	};
66127
66128	Vdbe *v = 0;
66129	int rc = SQLITE_OK;
66130	char *zErr = 0;
66131	Table *pTab;
66132	Parse *pParse;

66133

66134	*ppBlob = 0;

66135	sqlite3_mutex_enter(db->mutex);



66136	pParse = sqlite3StackAllocRaw(db, sizeof(*pParse));
66137	if( pParse==0 ){
66138	rc = SQLITE_NOMEM;
66139	goto blob_open_out;
66140	}
66141	do {
66142	memset(pParse, 0, sizeof(Parse));
66143	pParse->db = db;


66144
66145	sqlite3BtreeEnterAll(db);
66146	pTab = sqlite3LocateTable(pParse, 0, zTable, zDb);
66147	if( pTab && IsVirtual(pTab) ){
66148	pTab = 0;
	@@ -66164,11 +66398,11 @@
66164	sqlite3BtreeLeaveAll(db);
66165	goto blob_open_out;
66166	}
66167
66168	/* Now search pTab for the exact column. */
66169	for(iCol=0; iCol < pTab->nCol; iCol++) {
66170	if( sqlite3StrICmp(pTab->aCol[iCol].zName, zColumn)==0 ){
66171	break;
66172	}
66173	}
66174	if( iCol==pTab->nCol ){
	@@ -66218,15 +66452,18 @@
66218	sqlite3BtreeLeaveAll(db);
66219	goto blob_open_out;
66220	}
66221	}
66222
66223	v = sqlite3VdbeCreate(db);
66224	if( v ){


66225	int iDb = sqlite3SchemaToIndex(db, pTab->pSchema);

66226	sqlite3VdbeAddOpList(v, sizeof(openBlob)/sizeof(VdbeOpList), openBlob);
66227	flags = !!flags; /* flags = (flags ? 1 : 0); */
66228
66229	/* Configure the OP_Transaction */
66230	sqlite3VdbeChangeP1(v, 0, iDb);
66231	sqlite3VdbeChangeP2(v, 0, flags);
66232
	@@ -66265,69 +66502,29 @@
66265	if( !db->mallocFailed ){
66266	sqlite3VdbeMakeReady(v, 1, 1, 1, 0, 0, 0);
66267	}
66268	}
66269



66270	sqlite3BtreeLeaveAll(db);
66271	if( db->mallocFailed ){
66272	goto blob_open_out;
66273	}
66274
66275	sqlite3_bind_int64((sqlite3_stmt *)v, 1, iRow);
66276	rc = sqlite3_step((sqlite3_stmt *)v);
66277	if( rc!=SQLITE_ROW ){
66278	nAttempt++;
66279	rc = sqlite3_finalize((sqlite3_stmt *)v);
66280	sqlite3DbFree(db, zErr);
66281	zErr = sqlite3MPrintf(db, sqlite3_errmsg(db));
66282	v = 0;
66283	}
66284	} while( nAttempt<5 && rc==SQLITE_SCHEMA );
66285
66286	if( rc==SQLITE_ROW ){
66287	/* The row-record has been opened successfully. Check that the
66288	** column in question contains text or a blob. If it contains
66289	** text, it is up to the caller to get the encoding right.
66290	*/
66291	Incrblob *pBlob;
66292	u32 type = v->apCsr[0]->aType[iCol];
66293
66294	if( type<12 ){
66295	sqlite3DbFree(db, zErr);
66296	zErr = sqlite3MPrintf(db, "cannot open value of type %s",
66297	type==0?"null": type==7?"real": "integer"
66298	);
66299	rc = SQLITE_ERROR;
66300	goto blob_open_out;
66301	}
66302	pBlob = (Incrblob *)sqlite3DbMallocZero(db, sizeof(Incrblob));
66303	if( db->mallocFailed ){
66304	sqlite3DbFree(db, pBlob);
66305	goto blob_open_out;
66306	}
66307	pBlob->flags = flags;
66308	pBlob->pCsr = v->apCsr[0]->pCursor;
66309	sqlite3BtreeEnterCursor(pBlob->pCsr);
66310	sqlite3BtreeCacheOverflow(pBlob->pCsr);
66311	sqlite3BtreeLeaveCursor(pBlob->pCsr);
66312	pBlob->pStmt = (sqlite3_stmt *)v;
66313	pBlob->iOffset = v->apCsr[0]->aOffset[iCol];
66314	pBlob->nByte = sqlite3VdbeSerialTypeLen(type);
66315	pBlob->db = db;
66316	ppBlob = (sqlite3_blob )pBlob;
66317	rc = SQLITE_OK;
66318	}else if( rc==SQLITE_OK ){
66319	sqlite3DbFree(db, zErr);
66320	zErr = sqlite3MPrintf(db, "no such rowid: %lld", iRow);
66321	rc = SQLITE_ERROR;
66322	}
66323
66324	blob_open_out:
66325	if( v && (rc!=SQLITE_OK \|\| db->mallocFailed) ){
66326	sqlite3VdbeFinalize(v);



66327	}
66328	sqlite3Error(db, rc, zErr);
66329	sqlite3DbFree(db, zErr);
66330	sqlite3StackFree(db, pParse);
66331	rc = sqlite3ApiExit(db, rc);
66332	sqlite3_mutex_leave(db->mutex);
66333	return rc;
	@@ -66376,11 +66573,11 @@
66376
66377	if( n<0 \|\| iOffset<0 \|\| (iOffset+n)>p->nByte ){
66378	/* Request is out of range. Return a transient error. */
66379	rc = SQLITE_ERROR;
66380	sqlite3Error(db, SQLITE_ERROR, 0);
66381	} else if( v==0 ){
66382	/* If there is no statement handle, then the blob-handle has
66383	** already been invalidated. Return SQLITE_ABORT in this case.
66384	*/
66385	rc = SQLITE_ABORT;
66386	}else{
	@@ -66426,10 +66623,49 @@
66426	*/
66427	SQLITE_API int sqlite3_blob_bytes(sqlite3_blob *pBlob){
66428	Incrblob p = (Incrblob )pBlob;
66429	return p ? p->nByte : 0;
66430	}







































66431
66432	#endif /* #ifndef SQLITE_OMIT_INCRBLOB */
66433
66434	/************ End of vdbeblob.c ******************************************/
66435	/************ Begin file journal.c ***************************************/
	@@ -68757,10 +68993,13 @@
68757	Expr pRight, / Right operand */
68758	const Token pToken / Argument token */
68759	){
68760	Expr *p = sqlite3ExprAlloc(pParse->db, op, pToken, 1);
68761	sqlite3ExprAttachSubtrees(pParse->db, p, pLeft, pRight);



68762	return p;
68763	}
68764
68765	/*
68766	** Join two expressions using an AND operator. If either expression is
	@@ -69869,10 +70108,20 @@
69869	int mem = ++pParse->nMem;
69870	sqlite3VdbeAddOp1(v, OP_If, mem);
69871	testAddr = sqlite3VdbeAddOp2(v, OP_Integer, 1, mem);
69872	assert( testAddr>0 \|\| pParse->db->mallocFailed );
69873	}










69874
69875	switch( pExpr->op ){
69876	case TK_IN: {
69877	char affinity; /* Affinity of the LHS of the IN */
69878	KeyInfo keyInfo; /* Keyinfo for the generated table */
	@@ -84056,10 +84305,31 @@
84056	int (extended_result_codes)(sqlite3,int);
84057	int (limit)(sqlite3,int,int);
84058	sqlite3_stmt (next_stmt)(sqlite3,sqlite3_stmt);
84059	const char (sql)(sqlite3_stmt*);
84060	int (status)(int,int,int*,int);





















84061	};
84062
84063	/*
84064	** The following macros redefine the API routines so that they are
84065	** redirected throught the global sqlite3_api structure.
	@@ -84235,10 +84505,31 @@
84235	#define sqlite3_extended_result_codes sqlite3_api->extended_result_codes
84236	#define sqlite3_limit sqlite3_api->limit
84237	#define sqlite3_next_stmt sqlite3_api->next_stmt
84238	#define sqlite3_sql sqlite3_api->sql
84239	#define sqlite3_status sqlite3_api->status





















84240	#endif /* SQLITE_CORE */
84241
84242	#define SQLITE_EXTENSION_INIT1 const sqlite3_api_routines *sqlite3_api = 0;
84243	#define SQLITE_EXTENSION_INIT2(v) sqlite3_api = v;
84244
	@@ -84552,10 +84843,50 @@
84552	sqlite3_extended_result_codes,
84553	sqlite3_limit,
84554	sqlite3_next_stmt,
84555	sqlite3_sql,
84556	sqlite3_status,








































84557	};
84558
84559	/*
84560	** Attempt to load an SQLite extension library contained in the file
84561	** zFile. The entry point is zProc. zProc may be 0 in which case a
	@@ -86976,17 +87307,17 @@
86976
86977	#ifndef SQLITE_OMIT_EXPLAIN
86978	if( rc==SQLITE_OK && pParse->pVdbe && pParse->explain ){
86979	static const char * const azColName[] = {
86980	"addr", "opcode", "p1", "p2", "p3", "p4", "p5", "comment",
86981	"order", "from", "detail"
86982	};
86983	int iFirst, mx;
86984	if( pParse->explain==2 ){
86985	sqlite3VdbeSetNumCols(pParse->pVdbe, 3);
86986	iFirst = 8;
86987	mx = 11;
86988	}else{
86989	sqlite3VdbeSetNumCols(pParse->pVdbe, 8);
86990	iFirst = 0;
86991	mx = 8;
86992	}
	@@ -87981,10 +88312,92 @@
87981	}
87982	}
87983	return pInfo;
87984	}
87985


















































































87986
87987	/*
87988	** If the inner loop was generated using a non-null pOrderBy argument,
87989	** then the results were placed in a sorter. After the loop is terminated
87990	** we need to run the sorter and output the results. The following
	@@ -88328,26 +88741,10 @@
88328	}
88329	}
88330	generateColumnTypes(pParse, pTabList, pEList);
88331	}
88332
88333	#ifndef SQLITE_OMIT_COMPOUND_SELECT
88334	/*
88335	** Name of the connection operator, used for error messages.
88336	*/
88337	static const char *selectOpName(int id){
88338	char *z;
88339	switch( id ){
88340	case TK_ALL: z = "UNION ALL"; break;
88341	case TK_INTERSECT: z = "INTERSECT"; break;
88342	case TK_EXCEPT: z = "EXCEPT"; break;
88343	default: z = "UNION"; break;
88344	}
88345	return z;
88346	}
88347	#endif /* SQLITE_OMIT_COMPOUND_SELECT */
88348
88349	/*
88350	** Given a an expression list (which is really the list of expressions
88351	** that form the result set of a SELECT statement) compute appropriate
88352	** column names for a table that would hold the expression list.
88353	**
	@@ -88506,10 +88903,11 @@
88506	/* The sqlite3ResultSetOfSelect() is only used n contexts where lookaside
88507	** is disabled */
88508	assert( db->lookaside.bEnabled==0 );
88509	pTab->nRef = 1;
88510	pTab->zName = 0;

88511	selectColumnsFromExprList(pParse, pSelect->pEList, &pTab->nCol, &pTab->aCol);
88512	selectAddColumnTypeAndCollation(pParse, pTab->nCol, pTab->aCol, pSelect);
88513	pTab->iPKey = -1;
88514	if( db->mallocFailed ){
88515	sqlite3DeleteTable(db, pTab);
	@@ -88676,10 +89074,14 @@
88676	Select pPrior; / Another SELECT immediately to our left */
88677	Vdbe v; / Generate code to this VDBE */
88678	SelectDest dest; /* Alternative data destination */
88679	Select pDelete = 0; / Chain of simple selects to delete */
88680	sqlite3 db; / Database connection */




88681
88682	/* Make sure there is no ORDER BY or LIMIT clause on prior SELECTs. Only
88683	** the last (right-most) SELECT in the series may have an ORDER BY or LIMIT.
88684	*/
88685	assert( p && p->pPrior ); /* Calling function guarantees this much */
	@@ -88736,10 +89138,11 @@
88736	case TK_ALL: {
88737	int addr = 0;
88738	assert( !pPrior->pLimit );
88739	pPrior->pLimit = p->pLimit;
88740	pPrior->pOffset = p->pOffset;

88741	rc = sqlite3Select(pParse, pPrior, &dest);
88742	p->pLimit = 0;
88743	p->pOffset = 0;
88744	if( rc ){
88745	goto multi_select_end;
	@@ -88749,10 +89152,11 @@
88749	p->iOffset = pPrior->iOffset;
88750	if( p->iLimit ){
88751	addr = sqlite3VdbeAddOp1(v, OP_IfZero, p->iLimit);
88752	VdbeComment((v, "Jump ahead if LIMIT reached"));
88753	}

88754	rc = sqlite3Select(pParse, p, &dest);
88755	testcase( rc!=SQLITE_OK );
88756	pDelete = p->pPrior;
88757	p->pPrior = pPrior;
88758	if( addr ){
	@@ -88796,10 +89200,11 @@
88796
88797	/* Code the SELECT statements to our left
88798	*/
88799	assert( !pPrior->pOrderBy );
88800	sqlite3SelectDestInit(&uniondest, priorOp, unionTab);

88801	rc = sqlite3Select(pParse, pPrior, &uniondest);
88802	if( rc ){
88803	goto multi_select_end;
88804	}
88805
	@@ -88815,10 +89220,11 @@
88815	pLimit = p->pLimit;
88816	p->pLimit = 0;
88817	pOffset = p->pOffset;
88818	p->pOffset = 0;
88819	uniondest.eDest = op;

88820	rc = sqlite3Select(pParse, p, &uniondest);
88821	testcase( rc!=SQLITE_OK );
88822	/* Query flattening in sqlite3Select() might refill p->pOrderBy.
88823	** Be sure to delete p->pOrderBy, therefore, to avoid a memory leak. */
88824	sqlite3ExprListDelete(db, p->pOrderBy);
	@@ -88880,10 +89286,11 @@
88880	assert( p->pEList );
88881
88882	/* Code the SELECTs to our left into temporary table "tab1".
88883	*/
88884	sqlite3SelectDestInit(&intersectdest, SRT_Union, tab1);

88885	rc = sqlite3Select(pParse, pPrior, &intersectdest);
88886	if( rc ){
88887	goto multi_select_end;
88888	}
88889
	@@ -88896,10 +89303,11 @@
88896	pLimit = p->pLimit;
88897	p->pLimit = 0;
88898	pOffset = p->pOffset;
88899	p->pOffset = 0;
88900	intersectdest.iParm = tab2;

88901	rc = sqlite3Select(pParse, p, &intersectdest);
88902	testcase( rc!=SQLITE_OK );
88903	pDelete = p->pPrior;
88904	p->pPrior = pPrior;
88905	sqlite3ExprDelete(db, p->pLimit);
	@@ -88931,10 +89339,12 @@
88931	sqlite3VdbeAddOp2(v, OP_Close, tab2, 0);
88932	sqlite3VdbeAddOp2(v, OP_Close, tab1, 0);
88933	break;
88934	}
88935	}


88936
88937	/* Compute collating sequences used by
88938	** temporary tables needed to implement the compound select.
88939	** Attach the KeyInfo structure to all temporary tables.
88940	**
	@@ -89275,10 +89685,14 @@
89275	KeyInfo pKeyMerge; / Comparison information for merging rows */
89276	sqlite3 db; / Database connection */
89277	ExprList pOrderBy; / The ORDER BY clause */
89278	int nOrderBy; /* Number of terms in the ORDER BY clause */
89279	int aPermute; / Mapping from ORDER BY terms to result set columns */




89280
89281	assert( p->pOrderBy!=0 );
89282	assert( pKeyDup==0 ); /* "Managed" code needs this. Ticket #3382. */
89283	db = pParse->db;
89284	v = pParse->pVdbe;
	@@ -89428,10 +89842,11 @@
89428	/* Generate a coroutine to evaluate the SELECT statement to the
89429	** left of the compound operator - the "A" select.
89430	*/
89431	VdbeNoopComment((v, "Begin coroutine for left SELECT"));
89432	pPrior->iLimit = regLimitA;

89433	sqlite3Select(pParse, pPrior, &destA);
89434	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofA);
89435	sqlite3VdbeAddOp1(v, OP_Yield, regAddrA);
89436	VdbeNoopComment((v, "End coroutine for left SELECT"));
89437
	@@ -89442,10 +89857,11 @@
89442	VdbeNoopComment((v, "Begin coroutine for right SELECT"));
89443	savedLimit = p->iLimit;
89444	savedOffset = p->iOffset;
89445	p->iLimit = regLimitB;
89446	p->iOffset = 0;

89447	sqlite3Select(pParse, p, &destB);
89448	p->iLimit = savedLimit;
89449	p->iOffset = savedOffset;
89450	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofB);
89451	sqlite3VdbeAddOp1(v, OP_Yield, regAddrB);
	@@ -89572,10 +89988,11 @@
89572	}
89573	p->pPrior = pPrior;
89574
89575	/*** TBD: Insert subroutine calls to close cursors on incomplete
89576	** subqueries **/

89577	return SQLITE_OK;
89578	}
89579	#endif
89580
89581	#if !defined(SQLITE_OMIT_SUBQUERY) \|\| !defined(SQLITE_OMIT_VIEW)
	@@ -90305,10 +90722,11 @@
90305	pTab->nRef = 1;
90306	pTab->zName = sqlite3MPrintf(db, "sqlite_subquery_%p_", (void*)pTab);
90307	while( pSel->pPrior ){ pSel = pSel->pPrior; }
90308	selectColumnsFromExprList(pParse, pSel->pEList, &pTab->nCol, &pTab->aCol);
90309	pTab->iPKey = -1;

90310	pTab->tabFlags \|= TF_Ephemeral;
90311	#endif
90312	}else{
90313	/* An ordinary table or view name in the FROM clause */
90314	assert( pFrom->pTab==0 );
	@@ -90797,10 +91215,15 @@
90797	int rc = 1; /* Value to return from this function */
90798	int addrSortIndex; /* Address of an OP_OpenEphemeral instruction */
90799	AggInfo sAggInfo; /* Information used by aggregate queries */
90800	int iEnd; /* Address of the end of the query */
90801	sqlite3 db; / The database connection */





90802
90803	db = pParse->db;
90804	if( p==0 \|\| db->mallocFailed \|\| pParse->nErr ){
90805	return 1;
90806	}
	@@ -90869,10 +91292,11 @@
90869	}
90870	i = -1;
90871	}else{
90872	sqlite3SelectDestInit(&dest, SRT_EphemTab, pItem->iCursor);
90873	assert( pItem->isPopulated==0 );

90874	sqlite3Select(pParse, pSub, &dest);
90875	pItem->isPopulated = 1;
90876	}
90877	if( /pParse->nErr \|\|/ db->mallocFailed ){
90878	goto select_end;
	@@ -90904,14 +91328,16 @@
90904	pRight = pLoop;
90905	}
90906	mxSelect = db->aLimit[SQLITE_LIMIT_COMPOUND_SELECT];
90907	if( mxSelect && cnt>mxSelect ){
90908	sqlite3ErrorMsg(pParse, "too many terms in compound SELECT");
90909	return 1;
90910	}
90911	}
90912	return multiSelect(pParse, p, pDest);


90913	}
90914	#endif
90915
90916	/* If possible, rewrite the query to use GROUP BY instead of DISTINCT.
90917	** GROUP BY might use an index, DISTINCT never does.
	@@ -90919,11 +91345,10 @@
90919	assert( p->pGroupBy==0 \|\| (p->selFlags & SF_Aggregate)!=0 );
90920	if( (p->selFlags & (SF_Distinct\|SF_Aggregate))==SF_Distinct ){
90921	p->pGroupBy = sqlite3ExprListDup(db, p->pEList, 0);
90922	pGroupBy = p->pGroupBy;
90923	p->selFlags &= ~SF_Distinct;
90924	isDistinct = 0;
90925	}
90926
90927	/* If there is both a GROUP BY and an ORDER BY clause and they are
90928	** identical, then disable the ORDER BY clause since the GROUP BY
90929	** will cause elements to come out in the correct order. This is
	@@ -90966,11 +91391,11 @@
90966	iEnd = sqlite3VdbeMakeLabel(v);
90967	computeLimitRegisters(pParse, p, iEnd);
90968
90969	/* Open a virtual index to use for the distinct set.
90970	*/
90971	if( isDistinct ){
90972	KeyInfo *pKeyInfo;
90973	assert( isAgg \|\| pGroupBy );
90974	distinct = pParse->nTab++;
90975	pKeyInfo = keyInfoFromExprList(pParse, p->pEList);
90976	sqlite3VdbeAddOp4(v, OP_OpenEphemeral, distinct, 0, 0,
	@@ -91125,10 +91550,13 @@
91125	int regBase;
91126	int regRecord;
91127	int nCol;
91128	int nGroupBy;
91129



91130	groupBySort = 1;
91131	nGroupBy = pGroupBy->nExpr;
91132	nCol = nGroupBy + 1;
91133	j = nGroupBy+1;
91134	for(i=0; i<sAggInfo.nColumn; i++){
	@@ -91385,15 +91813,20 @@
91385	sqlite3ExprListDelete(db, pDel);
91386	}
91387	sqlite3VdbeResolveLabel(v, addrEnd);
91388
91389	} /* endif aggregate query */




91390
91391	/* If there is an ORDER BY clause, then we need to sort the results
91392	** and send them to the callback one by one.
91393	*/
91394	if( pOrderBy ){

91395	generateSortTail(pParse, p, v, pEList->nExpr, pDest);
91396	}
91397
91398	/* Jump here to skip this query
91399	*/
	@@ -91406,10 +91839,11 @@
91406
91407	/* Control jumps to here if an error is encountered above, or upon
91408	** successful coding of the SELECT.
91409	*/
91410	select_end:

91411
91412	/* Identify column names if results of the SELECT are to be output.
91413	*/
91414	if( rc==SQLITE_OK && pDest->eDest==SRT_Output ){
91415	generateColumnNames(pParse, pTabList, pEList);
	@@ -94495,11 +94929,11 @@
94495	pParse->pNewTable->nCol = 0;
94496	pParse->pNewTable->aCol = 0;
94497	}
94498	db->pVTab = 0;
94499	}else{
94500	sqlite3Error(db, SQLITE_ERROR, zErr);
94501	sqlite3DbFree(db, zErr);
94502	rc = SQLITE_ERROR;
94503	}
94504	pParse->declareVtab = 0;
94505
	@@ -94957,11 +95391,10 @@
94957	** cost of pursuing that strategy.
94958	*/
94959	struct WhereCost {
94960	WherePlan plan; /* The lookup strategy */
94961	double rCost; /* Overall cost of pursuing this search strategy */
94962	double nRow; /* Estimated number of output rows */
94963	Bitmask used; /* Bitmask of cursors used by this plan */
94964	};
94965
94966	/*
94967	** Bitmasks for the operators that indices are able to exploit. An
	@@ -95000,11 +95433,11 @@
95000	#define WHERE_COLUMN_EQ 0x00010000 /* x=EXPR or x IN (...) or x IS NULL */
95001	#define WHERE_COLUMN_RANGE 0x00020000 /* x<EXPR and/or x>EXPR */
95002	#define WHERE_COLUMN_IN 0x00040000 /* x IN (...) */
95003	#define WHERE_COLUMN_NULL 0x00080000 /* x IS NULL */
95004	#define WHERE_INDEXED 0x000f0000 /* Anything that uses an index */
95005	#define WHERE_NOT_FULLSCAN 0x000f3000 /* Does not do a full table scan */
95006	#define WHERE_IN_ABLE 0x000f1000 /* Able to support an IN operator */
95007	#define WHERE_TOP_LIMIT 0x00100000 /* x<EXPR or x<=EXPR constraint */
95008	#define WHERE_BTM_LIMIT 0x00200000 /* x>EXPR or x>=EXPR constraint */
95009	#define WHERE_IDX_ONLY 0x00800000 /* Use index only - omit table */
95010	#define WHERE_ORDERBY 0x01000000 /* Output will appear in correct order */
	@@ -96346,12 +96779,13 @@
96346	const int iCur = pSrc->iCursor; /* The cursor of the table to be accessed */
96347	const Bitmask maskSrc = getMask(pWC->pMaskSet, iCur); /* Bitmask for pSrc */
96348	WhereTerm * const pWCEnd = &pWC->a[pWC->nTerm]; /* End of pWC->a[] */
96349	WhereTerm pTerm; / A single term of the WHERE clause */
96350
96351	/* No OR-clause optimization allowed if the NOT INDEXED clause is used */
96352	if( pSrc->notIndexed ){

96353	return;
96354	}
96355
96356	/* Search the WHERE clause terms for a usable WO_OR term. */
96357	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
	@@ -96385,11 +96819,11 @@
96385	bestIndex(pParse, &tempWC, pSrc, notReady, notValid, 0, &sTermCost);
96386	}else{
96387	continue;
96388	}
96389	rTotal += sTermCost.rCost;
96390	nRow += sTermCost.nRow;
96391	used \|= sTermCost.used;
96392	if( rTotal>=pCost->rCost ) break;
96393	}
96394
96395	/* If there is an ORDER BY clause, increase the scan cost to account
	@@ -96404,12 +96838,12 @@
96404	** less than the current cost stored in pCost, replace the contents
96405	** of pCost. */
96406	WHERETRACE(("... multi-index OR cost=%.9g nrow=%.9g\n", rTotal, nRow));
96407	if( rTotal<pCost->rCost ){
96408	pCost->rCost = rTotal;
96409	pCost->nRow = nRow;
96410	pCost->used = used;

96411	pCost->plan.wsFlags = flags;
96412	pCost->plan.u.pTerm = pTerm;
96413	}
96414	}
96415	}
	@@ -96489,11 +96923,11 @@
96489	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
96490	if( termCanDriveIndex(pTerm, pSrc, notReady) ){
96491	WHERETRACE(("auto-index reduces cost from %.2f to %.2f\n",
96492	pCost->rCost, costTempIdx));
96493	pCost->rCost = costTempIdx;
96494	pCost->nRow = logN + 1;
96495	pCost->plan.wsFlags = WHERE_TEMP_INDEX;
96496	pCost->used = pTerm->prereqRight;
96497	break;
96498	}
96499	}
	@@ -97562,15 +97996,15 @@
97562
97563	/* If this index is the best we have seen so far, then record this
97564	** index and its cost in the pCost structure.
97565	*/
97566	if( (!pIdx \|\| wsFlags)
97567	&& (cost<pCost->rCost \|\| (cost<=pCost->rCost && nRow<pCost->nRow))
97568	){
97569	pCost->rCost = cost;
97570	pCost->nRow = nRow;
97571	pCost->used = used;

97572	pCost->plan.wsFlags = (wsFlags&wsFlagMask);
97573	pCost->plan.nEq = nEq;
97574	pCost->plan.u.pIdx = pIdx;
97575	}
97576
	@@ -97894,10 +98328,162 @@
97894	}
97895	}
97896	*pzAff = zAff;
97897	return regBase;
97898	}
























































































































































97899
97900	/*
97901	** Generate code for the start of the iLevel-th loop in the WHERE clause
97902	** implementation described by pWInfo.
97903	*/
	@@ -98436,10 +99022,13 @@
98436	/* Loop through table entries that match term pOrTerm. */
98437	pSubWInfo = sqlite3WhereBegin(pParse, pOrTab, pOrTerm->pExpr, 0,
98438	WHERE_OMIT_OPEN \| WHERE_OMIT_CLOSE \|
98439	WHERE_FORCE_TABLE \| WHERE_ONETABLE_ONLY);
98440	if( pSubWInfo ){



98441	if( (wctrlFlags & WHERE_DUPLICATES_OK)==0 ){
98442	int iSet = ((ii==pOrWc->nTerm-1)?-1:ii);
98443	int r;
98444	r = sqlite3ExprCodeGetColumn(pParse, pTabItem->pTab, -1, iCur,
98445	regRowid);
	@@ -98831,10 +99420,11 @@
98831	int nUnconstrained; /* Number tables without INDEXED BY */
98832	Bitmask notIndexed; /* Mask of tables that cannot use an index */
98833
98834	memset(&bestPlan, 0, sizeof(bestPlan));
98835	bestPlan.rCost = SQLITE_BIG_DBL;

98836
98837	/* Loop through the remaining entries in the FROM clause to find the
98838	** next nested loop. The loop tests all FROM clause entries
98839	** either once or twice.
98840	**
	@@ -98895,10 +99485,12 @@
98895	}
98896	mask = (isOptimal ? m : notReady);
98897	pOrderBy = ((i==0 && ppOrderBy )?*ppOrderBy:0);
98898	if( pTabItem->pIndex==0 ) nUnconstrained++;
98899


98900	assert( pTabItem->pTab );
98901	#ifndef SQLITE_OMIT_VIRTUALTABLE
98902	if( IsVirtual(pTabItem->pTab) ){
98903	sqlite3_index_info **pp = &pWInfo->a[j].pIdxInfo;
98904	bestVirtualIndex(pParse, pWC, pTabItem, mask, notReady, pOrderBy,
	@@ -98945,24 +99537,27 @@
98945	&& (bestJ<0 \|\| (notIndexed&m)!=0 /* (2) */
98946	\|\| (sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0)
98947	&& (nUnconstrained==0 \|\| pTabItem->pIndex==0 /* (3) */
98948	\|\| NEVER((sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0))
98949	&& (bestJ<0 \|\| sCost.rCost<bestPlan.rCost /* (4) */
98950	\|\| (sCost.rCost<=bestPlan.rCost && sCost.nRow<bestPlan.nRow))

98951	){
98952	WHERETRACE(("... best so far with cost=%g and nRow=%g\n",
98953	sCost.rCost, sCost.nRow));

98954	bestPlan = sCost;
98955	bestJ = j;
98956	}
98957	if( doNotReorder ) break;
98958	}
98959	}
98960	assert( bestJ>=0 );
98961	assert( notReady & getMask(pMaskSet, pTabList->a[bestJ].iCursor) );
98962	WHERETRACE(("*** Optimizer selects table %d for loop %d\n", bestJ,
98963	pLevel-pWInfo->a));

98964	if( (bestPlan.plan.wsFlags & WHERE_ORDERBY)!=0 ){
98965	*ppOrderBy = 0;
98966	}
98967	andFlags &= bestPlan.plan.wsFlags;
98968	pLevel->plan = bestPlan.plan;
	@@ -98973,11 +99568,13 @@
98973	}else{
98974	pLevel->iIdxCur = -1;
98975	}
98976	notReady &= ~getMask(pMaskSet, pTabList->a[bestJ].iCursor);
98977	pLevel->iFrom = (u8)bestJ;
98978	if( bestPlan.nRow>=(double)1 ) pParse->nQueryLoop *= bestPlan.nRow;


98979
98980	/* Check that if the table scanned by this loop iteration had an
98981	** INDEXED BY clause attached to it, that the named index is being
98982	** used for the scan. If not, then query compilation has failed.
98983	** Return an error.
	@@ -99025,41 +99622,10 @@
99025	notReady = ~(Bitmask)0;
99026	for(i=0, pLevel=pWInfo->a; i<nTabList; i++, pLevel++){
99027	Table pTab; / Table to open */
99028	int iDb; /* Index of database containing table/index */
99029
99030	#ifndef SQLITE_OMIT_EXPLAIN
99031	if( pParse->explain==2 ){
99032	char *zMsg;
99033	struct SrcList_item *pItem = &pTabList->a[pLevel->iFrom];
99034	zMsg = sqlite3MPrintf(db, "TABLE %s", pItem->zName);
99035	if( pItem->zAlias ){
99036	zMsg = sqlite3MAppendf(db, zMsg, "%s AS %s", zMsg, pItem->zAlias);
99037	}
99038	if( (pLevel->plan.wsFlags & WHERE_TEMP_INDEX)!=0 ){
99039	zMsg = sqlite3MAppendf(db, zMsg, "%s WITH AUTOMATIC INDEX", zMsg);
99040	}else if( (pLevel->plan.wsFlags & WHERE_INDEXED)!=0 ){
99041	zMsg = sqlite3MAppendf(db, zMsg, "%s WITH INDEX %s",
99042	zMsg, pLevel->plan.u.pIdx->zName);
99043	}else if( pLevel->plan.wsFlags & WHERE_MULTI_OR ){
99044	zMsg = sqlite3MAppendf(db, zMsg, "%s VIA MULTI-INDEX UNION", zMsg);
99045	}else if( pLevel->plan.wsFlags & (WHERE_ROWID_EQ\|WHERE_ROWID_RANGE) ){
99046	zMsg = sqlite3MAppendf(db, zMsg, "%s USING PRIMARY KEY", zMsg);
99047	}
99048	#ifndef SQLITE_OMIT_VIRTUALTABLE
99049	else if( (pLevel->plan.wsFlags & WHERE_VIRTUALTABLE)!=0 ){
99050	sqlite3_index_info *pVtabIdx = pLevel->plan.u.pVtabIdx;
99051	zMsg = sqlite3MAppendf(db, zMsg, "%s VIRTUAL TABLE INDEX %d:%s", zMsg,
99052	pVtabIdx->idxNum, pVtabIdx->idxStr);
99053	}
99054	#endif
99055	if( pLevel->plan.wsFlags & WHERE_ORDERBY ){
99056	zMsg = sqlite3MAppendf(db, zMsg, "%s ORDER BY", zMsg);
99057	}
99058	sqlite3VdbeAddOp4(v, OP_Explain, i, pLevel->iFrom, 0, zMsg, P4_DYNAMIC);
99059	}
99060	#endif /* SQLITE_OMIT_EXPLAIN */
99061	pTabItem = &pTabList->a[pLevel->iFrom];
99062	pTab = pTabItem->pTab;
99063	pLevel->iTabCur = pTabItem->iCursor;
99064	iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
99065	if( (pTab->tabFlags & TF_Ephemeral)!=0 \|\| pTab->pSelect ){
	@@ -99114,12 +99680,14 @@
99114	** loop below generates code for a single nested loop of the VM
99115	** program.
99116	*/
99117	notReady = ~(Bitmask)0;
99118	for(i=0; i<nTabList; i++){


99119	notReady = codeOneLoopStart(pWInfo, i, wctrlFlags, notReady);
99120	pWInfo->iContinue = pWInfo->a[i].addrCont;
99121	}
99122
99123	#ifdef SQLITE_TEST /* For testing and debugging use only */
99124	/* Record in the query plan information about the current table
99125	** and the index used to access it (if any). If the table itself
	@@ -105517,17 +106085,16 @@
105517	}
105518	}
105519	}
105520
105521	pColl = sqlite3FindCollSeq(db, (u8)enc2, zName, 1);
105522	if( pColl ){
105523	pColl->xCmp = xCompare;
105524	pColl->pUser = pCtx;
105525	pColl->xDel = xDel;
105526	pColl->enc = (u8)(enc2 \| (enc & SQLITE_UTF16_ALIGNED));
105527	pColl->type = collType;
105528	}
105529	sqlite3Error(db, SQLITE_OK, 0);
105530	return SQLITE_OK;
105531	}
105532
105533
	@@ -107491,12 +108058,18 @@
107491	#ifndef SQLITE_AMALGAMATION
107492	/*
107493	** Macros indicating that conditional expressions are always true or
107494	** false.
107495	*/




107496	# define ALWAYS(x) (x)
107497	# define NEVER(X) (x)


107498	/*
107499	** Internal types used by SQLite.
107500	*/
107501	typedef unsigned char u8; /* 1-byte (or larger) unsigned integer */
107502	typedef short int i16; /* 2-byte (or larger) signed integer */
	@@ -107510,12 +108083,16 @@
107510
107511	typedef struct Fts3Table Fts3Table;
107512	typedef struct Fts3Cursor Fts3Cursor;
107513	typedef struct Fts3Expr Fts3Expr;
107514	typedef struct Fts3Phrase Fts3Phrase;
107515	typedef struct Fts3SegReader Fts3SegReader;

107516	typedef struct Fts3SegFilter Fts3SegFilter;



107517
107518	/*
107519	** A connection to a fulltext index is an instance of the following
107520	** structure. The xCreate and xConnect methods create an instance
107521	** of this structure and xDestroy and xDisconnect free that instance.
	@@ -107532,26 +108109,18 @@
107532	sqlite3_tokenizer pTokenizer; / tokenizer for inserts and queries */
107533
107534	/* Precompiled statements used by the implementation. Each of these
107535	** statements is run and reset within a single virtual table API call.
107536	*/
107537	sqlite3_stmt *aStmt[25];
107538
107539	/* Pointer to string containing the SQL:
107540	**
107541	** "SELECT block FROM %_segments WHERE blockid BETWEEN ? AND ?
107542	** ORDER BY blockid"
107543	*/
107544	char *zSelectLeaves;
107545	int nLeavesStmt; /* Valid statements in aLeavesStmt */
107546	int nLeavesTotal; /* Total number of prepared leaves stmts */
107547	int nLeavesAlloc; /* Allocated size of aLeavesStmt */
107548	sqlite3_stmt *aLeavesStmt; / Array of prepared zSelectLeaves stmts */
107549
107550	int nNodeSize; /* Soft limit for node size */
107551	u8 bHasContent; /* True if %_content table exists */
107552	u8 bHasDocsize; /* True if %_docsize table exists */



107553
107554	/* The following hash table is used to buffer pending index updates during
107555	** transactions. Variable nPendingData estimates the memory size of the
107556	** pending data, including hash table overhead, but not malloc overhead.
107557	** When nPendingData exceeds nMaxPendingData, the buffer is flushed
	@@ -107574,17 +108143,24 @@
107574	i16 eSearch; /* Search strategy (see below) */
107575	u8 isEof; /* True if at End Of Results */
107576	u8 isRequireSeek; /* True if must seek pStmt to %_content row */
107577	sqlite3_stmt pStmt; / Prepared statement in use by the cursor */
107578	Fts3Expr pExpr; / Parsed MATCH query string */

107579	sqlite3_int64 iPrevId; /* Previous id read from aDoclist */
107580	char pNextId; / Pointer into the body of aDoclist */
107581	char aDoclist; / List of docids for full-text queries */
107582	int nDoclist; /* Size of buffer at aDoclist */
107583	int isMatchinfoNeeded; /* True when aMatchinfo[] needs filling in */
107584	u32 aMatchinfo; / Information about most recent match */


107585	};




107586
107587	/*
107588	** The Fts3Cursor.eSearch member is always set to one of the following.
107589	** Actualy, Fts3Cursor.eSearch can be greater than or equal to
107590	** FTS3_FULLTEXT_SEARCH. If so, then Fts3Cursor.eSearch - 2 is the index
	@@ -107604,22 +108180,34 @@
107604	#define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */
107605
107606	/*
107607	** A "phrase" is a sequence of one or more tokens that must match in
107608	** sequence. A single token is the base case and the most common case.
107609	** For a sequence of tokens contained in "...", nToken will be the number
107610	** of tokens in the string.






107611	*/









107612	struct Fts3Phrase {

107613	int nToken; /* Number of tokens in the phrase */
107614	int iColumn; /* Index of column this phrase must match */
107615	int isNot; /* Phrase prefixed by unary not (-) operator */
107616	struct PhraseToken {
107617	char z; / Text of the token */
107618	int n; /* Number of bytes in buffer pointed to by z */
107619	int isPrefix; /* True if token ends in with a "" character /
107620	} aToken[1]; /* One entry for each token in the phrase */
107621	};
107622
107623	/*
107624	** A tree of these objects forms the RHS of a MATCH operator.
107625	**
	@@ -107665,15 +108253,10 @@
107665	#define FTSQUERY_AND 3
107666	#define FTSQUERY_OR 4
107667	#define FTSQUERY_PHRASE 5
107668
107669
107670	/* fts3_init.c */
107671	SQLITE_PRIVATE int sqlite3Fts3DeleteVtab(int, sqlite3_vtab *);
107672	SQLITE_PRIVATE int sqlite3Fts3InitVtab(int, sqlite3, void, int, const charconst,
107673	sqlite3_vtab , char );
107674
107675	/* fts3_write.c */
107676	SQLITE_PRIVATE int sqlite3Fts3UpdateMethod(sqlite3_vtab,int,sqlite3_value,sqlite3_int64);
107677	SQLITE_PRIVATE int sqlite3Fts3PendingTermsFlush(Fts3Table *);
107678	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *);
107679	SQLITE_PRIVATE int sqlite3Fts3Optimize(Fts3Table *);
	@@ -107683,15 +108266,24 @@
107683	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table , Fts3SegReader );
107684	SQLITE_PRIVATE int sqlite3Fts3SegReaderIterate(
107685	Fts3Table , Fts3SegReader , int, Fts3SegFilter ,
107686	int ()(Fts3Table , void , char , int, char , int), void
107687	);
107688	SQLITE_PRIVATE int sqlite3Fts3ReadBlock(Fts3Table, sqlite3_int64, char const, int);
107689	SQLITE_PRIVATE int sqlite3Fts3AllSegdirs(Fts3Table, sqlite3_stmt *);
107690	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeLocal(Fts3Cursor, u32);
107691	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeGlobal(Fts3Cursor, u32);
107692	SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *);









107693
107694	/* Flags allowed as part of the 4th argument to SegmentReaderIterate() */
107695	#define FTS3_SEGMENT_REQUIRE_POS 0x00000001
107696	#define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002
107697	#define FTS3_SEGMENT_COLUMN_FILTER 0x00000004
	@@ -107711,19 +108303,21 @@
107711	SQLITE_PRIVATE int sqlite3Fts3GetVarint32(const char , int );
107712	SQLITE_PRIVATE int sqlite3Fts3VarintLen(sqlite3_uint64);
107713	SQLITE_PRIVATE void sqlite3Fts3Dequote(char *);
107714
107715	SQLITE_PRIVATE char sqlite3Fts3FindPositions(Fts3Expr , sqlite3_int64, int);
107716	SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Table , Fts3Expr );

107717	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr , Fts3Expr , int);
107718
107719	/* fts3_tokenizer.c */
107720	SQLITE_PRIVATE const char sqlite3Fts3NextToken(const char , int *);
107721	SQLITE_PRIVATE int sqlite3Fts3InitHashTable(sqlite3 , Fts3Hash , const char *);
107722	SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(Fts3Hash *pHash,
107723	const char , sqlite3_tokenizer , const char , char *
107724	);

107725
107726	/* fts3_snippet.c */
107727	SQLITE_PRIVATE void sqlite3Fts3Offsets(sqlite3_context, Fts3Cursor);
107728	SQLITE_PRIVATE void sqlite3Fts3Snippet(sqlite3_context , Fts3Cursor , const char *,
107729	const char , const char , int, int
	@@ -107875,20 +108469,17 @@
107875	static int fts3DisconnectMethod(sqlite3_vtab *pVtab){
107876	Fts3Table p = (Fts3Table )pVtab;
107877	int i;
107878
107879	assert( p->nPendingData==0 );

107880
107881	/* Free any prepared statements held */
107882	for(i=0; i<SizeofArray(p->aStmt); i++){
107883	sqlite3_finalize(p->aStmt[i]);
107884	}
107885	for(i=0; i<p->nLeavesStmt; i++){
107886	sqlite3_finalize(p->aLeavesStmt[i]);
107887	}
107888	sqlite3_free(p->zSelectLeaves);
107889	sqlite3_free(p->aLeavesStmt);
107890
107891	/* Invoke the tokenizer destructor to free the tokenizer. */
107892	p->pTokenizer->pModule->xDestroy(p->pTokenizer);
107893
107894	sqlite3_free(p);
	@@ -107895,11 +108486,11 @@
107895	return SQLITE_OK;
107896	}
107897
107898	/*
107899	** Construct one or more SQL statements from the format string given
107900	** and then evaluate those statements. The success code is writting
107901	** into *pRc.
107902	**
107903	** If *pRc is initially non-zero then this routine is a no-op.
107904	*/
107905	static void fts3DbExec(
	@@ -107947,37 +108538,42 @@
107947
107948	/*
107949	** Invoke sqlite3_declare_vtab() to declare the schema for the FTS3 table
107950	** passed as the first argument. This is done as part of the xConnect()
107951	** and xCreate() methods.




107952	*/
107953	static int fts3DeclareVtab(Fts3Table *p){
107954	int i; /* Iterator variable */
107955	int rc; /* Return code */
107956	char zSql; / SQL statement passed to declare_vtab() */
107957	char zCols; / List of user defined columns */
107958
107959	/* Create a list of user columns for the virtual table */
107960	zCols = sqlite3_mprintf("%Q, ", p->azColumn[0]);
107961	for(i=1; zCols && i<p->nColumn; i++){
107962	zCols = sqlite3_mprintf("%z%Q, ", zCols, p->azColumn[i]);
107963	}
107964
107965	/* Create the whole "CREATE TABLE" statement to pass to SQLite */
107966	zSql = sqlite3_mprintf(
107967	"CREATE TABLE x(%s %Q HIDDEN, docid HIDDEN)", zCols, p->zName
107968	);
107969
107970	if( !zCols \|\| !zSql ){
107971	rc = SQLITE_NOMEM;
107972	}else{
107973	rc = sqlite3_declare_vtab(p->db, zSql);
107974	}
107975
107976	sqlite3_free(zSql);
107977	sqlite3_free(zCols);
107978	return rc;

107979	}
107980
107981	/*
107982	** Create the backing store tables (%_content, %_segments and %_segdir)
107983	** required by the FTS3 table passed as the only argument. This is done
	@@ -107992,25 +108588,23 @@
107992	int i; /* Iterator variable */
107993	char zContentCols; / Columns of %_content table */
107994	sqlite3 db = p->db; / The database connection */
107995
107996	/* Create a list of user columns for the content table */
107997	if( p->bHasContent ){
107998	zContentCols = sqlite3_mprintf("docid INTEGER PRIMARY KEY");
107999	for(i=0; zContentCols && i<p->nColumn; i++){
108000	char *z = p->azColumn[i];
108001	zContentCols = sqlite3_mprintf("%z, 'c%d%q'", zContentCols, i, z);
108002	}
108003	if( zContentCols==0 ) rc = SQLITE_NOMEM;
108004
108005	/* Create the content table */
108006	fts3DbExec(&rc, db,
108007	"CREATE TABLE %Q.'%q_content'(%s)",
108008	p->zDb, p->zName, zContentCols
108009	);
108010	sqlite3_free(zContentCols);
108011	}
108012	/* Create other tables */
108013	fts3DbExec(&rc, db,
108014	"CREATE TABLE %Q.'%q_segments'(blockid INTEGER PRIMARY KEY, block BLOB);",
108015	p->zDb, p->zName
108016	);
	@@ -108029,10 +108623,12 @@
108029	if( p->bHasDocsize ){
108030	fts3DbExec(&rc, db,
108031	"CREATE TABLE %Q.'%q_docsize'(docid INTEGER PRIMARY KEY, size BLOB);",
108032	p->zDb, p->zName
108033	);


108034	fts3DbExec(&rc, db,
108035	"CREATE TABLE %Q.'%q_stat'(id INTEGER PRIMARY KEY, value BLOB);",
108036	p->zDb, p->zName
108037	);
108038	}
	@@ -108072,10 +108668,70 @@
108072	rc = sqlite3_exec(db, zSql, fts3TableExistsCallback, &res, 0);
108073	sqlite3_free(zSql);
108074	*pResult = (u8)(res & 0xff);
108075	if( rc!=SQLITE_ABORT ) *pRc = rc;
108076	}




























































108077
108078	/*
108079	** This function is the implementation of both the xConnect and xCreate
108080	** methods of the FTS3 virtual table.
108081	**
	@@ -108094,48 +108750,103 @@
108094	const char * const argv, / xCreate/xConnect argument array */
108095	sqlite3_vtab *ppVTab, / Write the resulting vtab structure here */
108096	char *pzErr / Write any error message here */
108097	){
108098	Fts3Hash pHash = (Fts3Hash )pAux;
108099	Fts3Table p; / Pointer to allocated vtab */
108100	int rc; /* Return code */
108101	int i; /* Iterator variable */
108102	int nByte; /* Size of allocation used for p /
108103	int iCol; /* Column index */
108104	int nString = 0; /* Bytes required to hold all column names */
108105	int nCol = 0; /* Number of columns in the FTS table */
108106	char zCsr; / Space for holding column names */
108107	int nDb; /* Bytes required to hold database name */
108108	int nName; /* Bytes required to hold table name */
108109
108110	const char zTokenizer = 0; / Name of tokenizer to use */

108111	sqlite3_tokenizer pTokenizer = 0; / Tokenizer for this table */





108112
108113	nDb = (int)strlen(argv[1]) + 1;
108114	nName = (int)strlen(argv[2]) + 1;
108115	for(i=3; i<argc; i++){
















108116	char const *z = argv[i];
108117	rc = sqlite3Fts3InitTokenizer(pHash, z, &pTokenizer, &zTokenizer, pzErr);
108118	if( rc!=SQLITE_OK ){
108119	return rc;







108120	}
108121	if( z!=zTokenizer ){






















108122	nString += (int)(strlen(z) + 1);

108123	}
108124	}
108125	nCol = argc - 3 - (zTokenizer!=0);
108126	if( zTokenizer==0 ){
108127	rc = sqlite3Fts3InitTokenizer(pHash, 0, &pTokenizer, 0, pzErr);
108128	if( rc!=SQLITE_OK ){
108129	return rc;
108130	}
108131	assert( pTokenizer );
108132	}
108133
108134	if( nCol==0 ){



108135	nCol = 1;
108136	}







108137
108138	/* Allocate and populate the Fts3Table structure. */
108139	nByte = sizeof(Fts3Table) + /* Fts3Table */
108140	nCol * sizeof(char ) + / azColumn */
108141	nName + /* zName */
	@@ -108145,77 +108856,70 @@
108145	if( p==0 ){
108146	rc = SQLITE_NOMEM;
108147	goto fts3_init_out;
108148	}
108149	memset(p, 0, nByte);
108150
108151	p->db = db;
108152	p->nColumn = nCol;
108153	p->nPendingData = 0;
108154	p->azColumn = (char **)&p[1];
108155	p->pTokenizer = pTokenizer;
108156	p->nNodeSize = 1000;
108157	p->nMaxPendingData = FTS3_MAX_PENDING_DATA;
108158	zCsr = (char *)&p->azColumn[nCol];
108159
108160	fts3HashInit(&p->pendingTerms, FTS3_HASH_STRING, 1);
108161
108162	/* Fill in the zName and zDb fields of the vtab structure. */

108163	p->zName = zCsr;
108164	memcpy(zCsr, argv[2], nName);
108165	zCsr += nName;
108166	p->zDb = zCsr;
108167	memcpy(zCsr, argv[1], nDb);
108168	zCsr += nDb;
108169
108170	/* Fill in the azColumn array */
108171	iCol = 0;
108172	for(i=3; i<argc; i++){
108173	if( argv[i]!=zTokenizer ){
108174	char *z;
108175	int n;
108176	z = (char *)sqlite3Fts3NextToken(argv[i], &n);
108177	memcpy(zCsr, z, n);
108178	zCsr[n] = '\0';
108179	sqlite3Fts3Dequote(zCsr);
108180	p->azColumn[iCol++] = zCsr;
108181	zCsr += n+1;
108182	assert( zCsr <= &((char *)p)[nByte] );
108183	}
108184	}
108185	if( iCol==0 ){
108186	assert( nCol==1 );
108187	p->azColumn[0] = "content";
108188	}
108189
108190	/* If this is an xCreate call, create the underlying tables in the
108191	** database. TODO: For xConnect(), it could verify that said tables exist.
108192	*/
108193	if( isCreate ){
108194	p->bHasContent = 1;
108195	p->bHasDocsize = argv[0][3]=='4';
108196	rc = fts3CreateTables(p);
108197	}else{
108198	rc = SQLITE_OK;
108199	fts3TableExists(&rc, db, argv[1], argv[2], "_content", &p->bHasContent);
108200	fts3TableExists(&rc, db, argv[1], argv[2], "_docsize", &p->bHasDocsize);
108201	}
108202	if( rc!=SQLITE_OK ) goto fts3_init_out;
108203
108204	rc = fts3DeclareVtab(p);
108205	if( rc!=SQLITE_OK ) goto fts3_init_out;
108206
108207	*ppVTab = &p->base;
108208
108209	fts3_init_out:
108210	assert( p \|\| (pTokenizer && rc!=SQLITE_OK) );

108211	if( rc!=SQLITE_OK ){
108212	if( p ){
108213	fts3DisconnectMethod((sqlite3_vtab *)p);
108214	}else{
108215	pTokenizer->pModule->xDestroy(pTokenizer);
108216	}


108217	}
108218	return rc;
108219	}
108220
108221	/*
	@@ -108323,14 +109027,16 @@
108323
108324	/*
108325	** Close the cursor. For additional information see the documentation
108326	** on the xClose method of the virtual table interface.
108327	*/
108328	static int fulltextClose(sqlite3_vtab_cursor *pCursor){
108329	Fts3Cursor pCsr = (Fts3Cursor )pCursor;

108330	sqlite3_finalize(pCsr->pStmt);
108331	sqlite3Fts3ExprFree(pCsr->pExpr);

108332	sqlite3_free(pCsr->aDoclist);
108333	sqlite3_free(pCsr->aMatchinfo);
108334	sqlite3_free(pCsr);
108335	return SQLITE_OK;
108336	}
	@@ -108365,134 +109071,186 @@
108365	return SQLITE_OK;
108366	}
108367	}
108368
108369	/*
108370	** Advance the cursor to the next row in the %_content table that
108371	** matches the search criteria. For a MATCH search, this will be
108372	** the next row that matches. For a full-table scan, this will be
108373	** simply the next row in the %_content table. For a docid lookup,
108374	** this routine simply sets the EOF flag.
108375	**
108376	** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned
108377	** even if we reach end-of-file. The fts3EofMethod() will be called
108378	** subsequently to determine whether or not an EOF was hit.





108379	*/
108380	static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){








108381	int rc = SQLITE_OK; /* Return code */
108382	Fts3Cursor pCsr = (Fts3Cursor )pCursor;
108383
108384	if( pCsr->aDoclist==0 ){
108385	if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){
108386	pCsr->isEof = 1;
108387	rc = sqlite3_reset(pCsr->pStmt);
108388	}
108389	}else if( pCsr->pNextId>=&pCsr->aDoclist[pCsr->nDoclist] ){
108390	pCsr->isEof = 1;
108391	}else{
108392	sqlite3_reset(pCsr->pStmt);
108393	fts3GetDeltaVarint(&pCsr->pNextId, &pCsr->iPrevId);
108394	pCsr->isRequireSeek = 1;
108395	pCsr->isMatchinfoNeeded = 1;
108396	}
108397	return rc;
108398	}
108399
108400
108401	/*
108402	** The buffer pointed to by argument zNode (size nNode bytes) contains the
108403	** root node of a b-tree segment. The segment is guaranteed to be at least
108404	** one level high (i.e. the root node is not also a leaf). If successful,
108405	** this function locates the leaf node of the segment that may contain the
108406	** term specified by arguments zTerm and nTerm and writes its block number
108407	** to *piLeaf.
108408	**
108409	** It is possible that the returned leaf node does not contain the specified
108410	** term. However, if the segment does contain said term, it is stored on
108411	** the identified leaf node. Because this function only inspects interior
108412	** segment nodes (and never loads leaf nodes into memory), it is not possible
108413	** to be sure.












































































108414	**
108415	** If an error occurs, an error code other than SQLITE_OK is returned.
108416	*/
108417	static int fts3SelectLeaf(
108418	Fts3Table p, / Virtual table handle */
108419	const char zTerm, / Term to select leaves for */
108420	int nTerm, /* Size of term zTerm in bytes */
108421	const char zNode, / Buffer containing segment interior node */
108422	int nNode, /* Size of buffer at zNode */
108423	sqlite3_int64 piLeaf / Selected leaf node */
108424	){
108425	int rc = SQLITE_OK; /* Return code */
108426	const char zCsr = zNode; / Cursor to iterate through node */
108427	const char zEnd = &zCsr[nNode];/ End of interior node buffer */
108428	char zBuffer = 0; / Buffer to load terms into */
108429	int nAlloc = 0; /* Size of allocated buffer */
108430
108431	while( 1 ){
108432	int isFirstTerm = 1; /* True when processing first term on page */
108433	int iHeight; /* Height of this node in tree */
108434	sqlite3_int64 iChild; /* Block id of child node to descend to */
108435	int nBlock; /* Size of child node in bytes */
108436
108437	zCsr += sqlite3Fts3GetVarint32(zCsr, &iHeight);
108438	zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
108439
108440	while( zCsr<zEnd ){
108441	int cmp; /* memcmp() result */
108442	int nSuffix; /* Size of term suffix */
108443	int nPrefix = 0; /* Size of term prefix */
108444	int nBuffer; /* Total term size */
108445
108446	/* Load the next term on the node into zBuffer */
108447	if( !isFirstTerm ){
108448	zCsr += sqlite3Fts3GetVarint32(zCsr, &nPrefix);
108449	}
108450	isFirstTerm = 0;
108451	zCsr += sqlite3Fts3GetVarint32(zCsr, &nSuffix);
108452	if( nPrefix+nSuffix>nAlloc ){
108453	char *zNew;
108454	nAlloc = (nPrefix+nSuffix) * 2;
108455	zNew = (char *)sqlite3_realloc(zBuffer, nAlloc);
108456	if( !zNew ){
108457	sqlite3_free(zBuffer);
108458	return SQLITE_NOMEM;
108459	}
108460	zBuffer = zNew;
108461	}
108462	memcpy(&zBuffer[nPrefix], zCsr, nSuffix);
108463	nBuffer = nPrefix + nSuffix;
108464	zCsr += nSuffix;
108465
108466	/* Compare the term we are searching for with the term just loaded from
108467	** the interior node. If the specified term is greater than or equal
108468	** to the term from the interior node, then all terms on the sub-tree
108469	** headed by node iChild are smaller than zTerm. No need to search
108470	** iChild.
108471	**
108472	** If the interior node term is larger than the specified term, then
108473	** the tree headed by iChild may contain the specified term.
108474	*/
108475	cmp = memcmp(zTerm, zBuffer, (nBuffer>nTerm ? nTerm : nBuffer));
108476	if( cmp<0 \|\| (cmp==0 && nBuffer>nTerm) ) break;
108477	iChild++;
108478	};
108479
108480	/* If (iHeight==1), the children of this interior node are leaves. The
108481	** specified term may be present on leaf node iChild.
108482	*/
108483	if( iHeight==1 ){
108484	*piLeaf = iChild;
108485	break;
108486	}
108487
108488	/* Descend to interior node iChild. */
108489	rc = sqlite3Fts3ReadBlock(p, iChild, &zCsr, &nBlock);
108490	if( rc!=SQLITE_OK ) break;
108491	zEnd = &zCsr[nBlock];
108492	}
108493	sqlite3_free(zBuffer);
108494	return rc;
108495	}
108496
108497	/*
108498	** This function is used to create delta-encoded serialized lists of FTS3
	@@ -108720,24 +109478,48 @@
108720	*pp2 = p2 + 1;
108721	}
108722
108723	/*
108724	** nToken==1 searches for adjacent positions.




















108725	*/
108726	static int fts3PoslistPhraseMerge(
108727	char *pp, / Output buffer */
108728	int nToken, /* Maximum difference in token positions */
108729	int isSaveLeft, /* Save the left position */
108730	char *pp1, / Left input list */
108731	char *pp2 / Right input list */

108732	){
108733	char p = (pp ? pp : 0);
108734	char p1 = pp1;
108735	char p2 = pp2;
108736
108737	int iCol1 = 0;
108738	int iCol2 = 0;




108739	assert( p1!=0 && p2!=0 );
108740	if( *p1==POS_COLUMN ){
108741	p1++;
108742	p1 += sqlite3Fts3GetVarint32(p1, &iCol1);
108743	}
	@@ -108762,11 +109544,13 @@
108762	assert( p2!=POS_END && p2!=POS_COLUMN );
108763	fts3GetDeltaVarint(&p1, &iPos1); iPos1 -= 2;
108764	fts3GetDeltaVarint(&p2, &iPos2); iPos2 -= 2;
108765
108766	while( 1 ){
108767	if( iPos2>iPos1 && iPos2<=iPos1+nToken ){


108768	sqlite3_int64 iSave;
108769	if( !pp ){
108770	fts3PoslistCopy(0, &p2);
108771	fts3PoslistCopy(0, &p1);
108772	*pp1 = p1;
	@@ -108845,25 +109629,25 @@
108845	){
108846	char p1 = pp1;
108847	char p2 = pp2;
108848
108849	if( !pp ){
108850	if( fts3PoslistPhraseMerge(0, nRight, 0, pp1, pp2) ) return 1;
108851	*pp1 = p1;
108852	*pp2 = p2;
108853	return fts3PoslistPhraseMerge(0, nLeft, 0, pp2, pp1);
108854	}else{
108855	char *pTmp1 = aTmp;
108856	char *pTmp2;
108857	char *aTmp2;
108858	int res = 1;
108859
108860	fts3PoslistPhraseMerge(&pTmp1, nRight, 0, pp1, pp2);
108861	aTmp2 = pTmp2 = pTmp1;
108862	*pp1 = p1;
108863	*pp2 = p2;
108864	fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, pp2, pp1);
108865	if( pTmp1!=aTmp && pTmp2!=aTmp2 ){
108866	fts3PoslistMerge(pp, &aTmp, &aTmp2);
108867	}else if( pTmp1!=aTmp ){
108868	fts3PoslistCopy(pp, &aTmp);
108869	}else if( pTmp2!=aTmp2 ){
	@@ -108905,11 +109689,12 @@
108905	char aBuffer, / Pre-allocated output buffer */
108906	int pnBuffer, / OUT: Bytes written to aBuffer */
108907	char a1, / Buffer containing first doclist */
108908	int n1, /* Size of buffer a1 */
108909	char a2, / Buffer containing second doclist */
108910	int n2 /* Size of buffer a2 */

108911	){
108912	sqlite3_int64 i1 = 0;
108913	sqlite3_int64 i2 = 0;
108914	sqlite3_int64 iPrev = 0;
108915
	@@ -108916,10 +109701,11 @@
108916	char *p = aBuffer;
108917	char *p1 = a1;
108918	char *p2 = a2;
108919	char *pEnd1 = &a1[n1];
108920	char *pEnd2 = &a2[n2];

108921
108922	assert( mergetype==MERGE_OR \|\| mergetype==MERGE_POS_OR
108923	\|\| mergetype==MERGE_AND \|\| mergetype==MERGE_NOT
108924	\|\| mergetype==MERGE_PHRASE \|\| mergetype==MERGE_POS_PHRASE
108925	\|\| mergetype==MERGE_NEAR \|\| mergetype==MERGE_POS_NEAR
	@@ -108959,10 +109745,11 @@
108959	while( p1 && p2 ){
108960	if( i1==i2 ){
108961	fts3PutDeltaVarint(&p, &iPrev, i1);
108962	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108963	fts3GetDeltaVarint2(&p2, pEnd2, &i2);

108964	}else if( i1<i2 ){
108965	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108966	}else{
108967	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
108968	}
	@@ -108989,13 +109776,15 @@
108989	while( p1 && p2 ){
108990	if( i1==i2 ){
108991	char *pSave = p;
108992	sqlite3_int64 iPrevSave = iPrev;
108993	fts3PutDeltaVarint(&p, &iPrev, i1);
108994	if( 0==fts3PoslistPhraseMerge(ppPos, 1, 0, &p1, &p2) ){
108995	p = pSave;
108996	iPrev = iPrevSave;


108997	}
108998	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
108999	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
109000	}else if( i1<i2 ){
109001	fts3PoslistCopy(0, &p1);
	@@ -109044,10 +109833,11 @@
109044	sqlite3_free(aTmp);
109045	break;
109046	}
109047	}
109048

109049	*pnBuffer = (int)(p-aBuffer);
109050	return SQLITE_OK;
109051	}
109052
109053	/*
	@@ -109082,20 +109872,20 @@
109082	for(i=0; i<SizeofArray(pTS->aaOutput); i++){
109083	if( pTS->aaOutput[i] ){
109084	if( !aOut ){
109085	aOut = pTS->aaOutput[i];
109086	nOut = pTS->anOutput[i];
109087	pTS->aaOutput[0] = 0;
109088	}else{
109089	int nNew = nOut + pTS->anOutput[i];
109090	char *aNew = sqlite3_malloc(nNew);
109091	if( !aNew ){
109092	sqlite3_free(aOut);
109093	return SQLITE_NOMEM;
109094	}
109095	fts3DoclistMerge(mergetype, 0, 0,
109096	aNew, &nNew, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut
109097	);
109098	sqlite3_free(pTS->aaOutput[i]);
109099	sqlite3_free(aOut);
109100	pTS->aaOutput[i] = 0;
109101	aOut = aNew;
	@@ -109162,12 +109952,12 @@
109162	if( aMerge!=aDoclist ){
109163	sqlite3_free(aMerge);
109164	}
109165	return SQLITE_NOMEM;
109166	}
109167	fts3DoclistMerge(mergetype, 0, 0,
109168	aNew, &nNew, pTS->aaOutput[iOut], pTS->anOutput[iOut], aMerge, nMerge
109169	);
109170
109171	if( iOut>0 ) sqlite3_free(aMerge);
109172	sqlite3_free(pTS->aaOutput[iOut]);
109173	pTS->aaOutput[iOut] = 0;
	@@ -109180,10 +109970,165 @@
109180	}
109181	}
109182	}
109183	return SQLITE_OK;
109184	}



























































































































































109185
109186	/*
109187	** This function retreives the doclist for the specified term (or term
109188	** prefix) from the database.
109189	**
	@@ -109194,145 +110139,150 @@
109194	** in the database without the found length specifier at the start of on-disk
109195	** doclists.
109196	*/
109197	static int fts3TermSelect(
109198	Fts3Table p, / Virtual table handle */

109199	int iColumn, /* Column to query (or -ve for all columns) */
109200	const char zTerm, / Term to query for */
109201	int nTerm, /* Size of zTerm in bytes */
109202	int isPrefix, /* True for a prefix search */
109203	int isReqPos, /* True to include position lists in output */
109204	int pnOut, / OUT: Size of buffer at ppOut /
109205	char *ppOut / OUT: Malloced result buffer */
109206	){
109207	int i;
109208	TermSelect tsc;
109209	Fts3SegFilter filter; /* Segment term filter configuration */
109210	Fts3SegReader *apSegment; / Array of segments to read data from */
109211	int nSegment = 0; /* Size of apSegment array */
109212	int nAlloc = 16; /* Allocated size of segment array */
109213	int rc; /* Return code */
109214	sqlite3_stmt pStmt = 0; / SQL statement to scan %_segdir table */
109215	int iAge = 0; /* Used to assign ages to segments */
109216
109217	apSegment = (Fts3SegReader *)sqlite3_malloc(sizeof(Fts3SegReader)*nAlloc);
109218	if( !apSegment ) return SQLITE_NOMEM;
109219	rc = sqlite3Fts3SegReaderPending(p, zTerm, nTerm, isPrefix, &apSegment[0]);
109220	if( rc!=SQLITE_OK ) goto finished;
109221	if( apSegment[0] ){
109222	nSegment = 1;
109223	}
109224
109225	/* Loop through the entire %_segdir table. For each segment, create a
109226	** Fts3SegReader to iterate through the subset of the segment leaves
109227	** that may contain a term that matches zTerm/nTerm. For non-prefix
109228	** searches, this is always a single leaf. For prefix searches, this
109229	** may be a contiguous block of leaves.
109230	**
109231	** The code in this loop does not actually load any leaves into memory
109232	** (unless the root node happens to be a leaf). It simply examines the
109233	** b-tree structure to determine which leaves need to be inspected.
109234	*/
109235	rc = sqlite3Fts3AllSegdirs(p, &pStmt);
109236	while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){
109237	Fts3SegReader *pNew = 0;
109238	int nRoot = sqlite3_column_bytes(pStmt, 4);
109239	char const *zRoot = sqlite3_column_blob(pStmt, 4);
109240	if( sqlite3_column_int64(pStmt, 1)==0 ){
109241	/* The entire segment is stored on the root node (which must be a
109242	** leaf). Do not bother inspecting any data in this case, just
109243	** create a Fts3SegReader to scan the single leaf.
109244	*/
109245	rc = sqlite3Fts3SegReaderNew(p, iAge, 0, 0, 0, zRoot, nRoot, &pNew);
109246	}else{
109247	int rc2; /* Return value of sqlite3Fts3ReadBlock() */
109248	sqlite3_int64 i1; /* Blockid of leaf that may contain zTerm */
109249	rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &i1);
109250	if( rc==SQLITE_OK ){
109251	sqlite3_int64 i2 = sqlite3_column_int64(pStmt, 2);
109252	rc = sqlite3Fts3SegReaderNew(p, iAge, i1, i2, 0, 0, 0, &pNew);
109253	}
109254
109255	/* The following call to ReadBlock() serves to reset the SQL statement
109256	** used to retrieve blocks of data from the %_segments table. If it is
109257	** not reset here, then it may remain classified as an active statement
109258	** by SQLite, which may lead to "DROP TABLE" or "DETACH" commands
109259	** failing.
109260	*/
109261	rc2 = sqlite3Fts3ReadBlock(p, 0, 0, 0);
109262	if( rc==SQLITE_OK ){
109263	rc = rc2;
109264	}
109265	}
109266	iAge++;
109267
109268	/* If a new Fts3SegReader was allocated, add it to the apSegment array. */
109269	assert( pNew!=0 \|\| rc!=SQLITE_OK );
109270	if( pNew ){
109271	if( nSegment==nAlloc ){
109272	Fts3SegReader **pArray;
109273	nAlloc += 16;
109274	pArray = (Fts3SegReader **)sqlite3_realloc(
109275	apSegment, nAllocsizeof(Fts3SegReader )
109276	);
109277	if( !pArray ){
109278	sqlite3Fts3SegReaderFree(p, pNew);
109279	rc = SQLITE_NOMEM;
109280	goto finished;
109281	}
109282	apSegment = pArray;
109283	}
109284	apSegment[nSegment++] = pNew;
109285	}
109286	}
109287	if( rc!=SQLITE_DONE ){
109288	assert( rc!=SQLITE_OK );
109289	goto finished;
109290	}
109291
109292	memset(&tsc, 0, sizeof(TermSelect));
109293	tsc.isReqPos = isReqPos;
109294
109295	filter.flags = FTS3_SEGMENT_IGNORE_EMPTY
109296	\| (isPrefix ? FTS3_SEGMENT_PREFIX : 0)
109297	\| (isReqPos ? FTS3_SEGMENT_REQUIRE_POS : 0)
109298	\| (iColumn<p->nColumn ? FTS3_SEGMENT_COLUMN_FILTER : 0);
109299	filter.iCol = iColumn;
109300	filter.zTerm = zTerm;
109301	filter.nTerm = nTerm;
109302
109303	rc = sqlite3Fts3SegReaderIterate(p, apSegment, nSegment, &filter,
109304	fts3TermSelectCb, (void *)&tsc
109305	);
109306	if( rc==SQLITE_OK ){
109307	rc = fts3TermSelectMerge(&tsc);
109308	}
109309
109310	if( rc==SQLITE_OK ){
109311	*ppOut = tsc.aaOutput[0];
109312	*pnOut = tsc.anOutput[0];
109313	}else{

109314	for(i=0; i<SizeofArray(tsc.aaOutput); i++){
109315	sqlite3_free(tsc.aaOutput[i]);
109316	}
109317	}
109318
109319	finished:
109320	sqlite3_reset(pStmt);
109321	for(i=0; i<nSegment; i++){
109322	sqlite3Fts3SegReaderFree(p, apSegment[i]);
109323	}
109324	sqlite3_free(apSegment);
109325	return rc;
109326	}
109327



















































































109328
109329	/*
109330	** Return a DocList corresponding to the phrase *pPhrase.






109331	*/
109332	static int fts3PhraseSelect(
109333	Fts3Table p, / Virtual table handle */
109334	Fts3Phrase pPhrase, / Phrase to return a doclist for */
109335	int isReqPos, /* True if output should contain positions */
109336	char *paOut, / OUT: Pointer to malloc'd result buffer */
109337	int pnOut / OUT: Size of buffer at paOut /
109338	){
	@@ -109340,54 +110290,156 @@
109340	int nOut = 0;
109341	int rc = SQLITE_OK;
109342	int ii;
109343	int iCol = pPhrase->iColumn;
109344	int isTermPos = (pPhrase->nToken>1 \|\| isReqPos);

























109345
109346	for(ii=0; ii<pPhrase->nToken; ii++){
109347	struct PhraseToken *pTok = &pPhrase->aToken[ii];
109348	char z = pTok->z; / Next token of the phrase */
109349	int n = pTok->n; /* Size of z in bytes */
109350	int isPrefix = pTok->isPrefix;/* True if token is a prefix */
109351	char pList; / Pointer to token doclist */
109352	int nList; /* Size of buffer at pList */
109353
109354	rc = fts3TermSelect(p, iCol, z, n, isPrefix, isTermPos, &nList, &pList);












































109355	if( rc!=SQLITE_OK ) break;
109356
109357	if( ii==0 ){
109358	pOut = pList;
109359	nOut = nList;





109360	}else{
109361	/* Merge the new term list and the current output. If this is the
109362	** last term in the phrase, and positions are not required in the
109363	** output of this function, the positions can be dropped as part
109364	** of this merge. Either way, the result of this merge will be
109365	** smaller than nList bytes. The code in fts3DoclistMerge() is written
109366	** so that it is safe to use pList as the output as well as an input
109367	** in this case.


109368	*/
109369	int mergetype = MERGE_POS_PHRASE;
109370	if( ii==pPhrase->nToken-1 && !isReqPos ){
109371	mergetype = MERGE_PHRASE;














109372	}
109373	fts3DoclistMerge(mergetype, 0, 0, pList, &nOut, pOut, nOut, pList, nList);
109374	sqlite3_free(pOut);
109375	pOut = pList;


109376	}
109377	assert( nOut==0 \|\| pOut!=0 );
109378	}
109379
109380	if( rc==SQLITE_OK ){




109381	*paOut = pOut;
109382	*pnOut = nOut;
109383	}else{
109384	sqlite3_free(pOut);
109385	}
109386	return rc;
109387	}
109388








109389	static int fts3NearMerge(
109390	int mergetype, /* MERGE_POS_NEAR or MERGE_NEAR */
109391	int nNear, /* Parameter to NEAR operator */
109392	int nTokenLeft, /* Number of tokens in LHS phrase arg */
109393	char aLeft, / Doclist for LHS (incl. positions) */
	@@ -109396,21 +110448,21 @@
109396	char aRight, / As aLeft */
109397	int nRight, /* As nRight */
109398	char *paOut, / OUT: Results of merge (malloced) */
109399	int pnOut / OUT: Sized of output buffer */
109400	){
109401	char *aOut;
109402	int rc;
109403
109404	assert( mergetype==MERGE_POS_NEAR \|\| MERGE_NEAR );
109405
109406	aOut = sqlite3_malloc(nLeft+nRight+1);
109407	if( aOut==0 ){
109408	rc = SQLITE_NOMEM;
109409	}else{
109410	rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft,
109411	aOut, pnOut, aLeft, nLeft, aRight, nRight
109412	);
109413	if( rc!=SQLITE_OK ){
109414	sqlite3_free(aOut);
109415	aOut = 0;
109416	}
	@@ -109418,21 +110470,36 @@
109418
109419	*paOut = aOut;
109420	return rc;
109421	}
109422










109423	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr pLeft, Fts3Expr pRight, int nNear){
109424	int rc;





109425	if( pLeft->aDoclist==0 \|\| pRight->aDoclist==0 ){
109426	sqlite3_free(pLeft->aDoclist);
109427	sqlite3_free(pRight->aDoclist);
109428	pRight->aDoclist = 0;
109429	pLeft->aDoclist = 0;
109430	rc = SQLITE_OK;
109431	}else{
109432	char *aOut;
109433	int nOut;
109434
109435	rc = fts3NearMerge(MERGE_POS_NEAR, nNear,
109436	pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
109437	pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
109438	&aOut, &nOut
	@@ -109451,19 +110518,158 @@
109451	pLeft->aDoclist = aOut;
109452	pLeft->nDoclist = nOut;
109453	}
109454	return rc;
109455	}

109456
109457	/*
109458	** Evaluate the full-text expression pExpr against fts3 table pTab. Store
109459	** the resulting doclist in paOut and pnOut. This routine mallocs for
109460	** the space needed to store the output. The caller is responsible for








































































































109461	** freeing the space when it has finished.


































109462	*/
109463	static int evalFts3Expr(
109464	Fts3Table p, / Virtual table handle */
109465	Fts3Expr pExpr, / Parsed fts3 expression */
109466	char *paOut, / OUT: Pointer to malloc'd result buffer */
109467	int pnOut, / OUT: Size of buffer at paOut /
109468	int isReqPos /* Require positions in output buffer */
109469	){
	@@ -109472,37 +110678,101 @@
109472	/* Zero the output parameters. */
109473	*paOut = 0;
109474	*pnOut = 0;
109475
109476	if( pExpr ){
109477	assert( pExpr->eType==FTSQUERY_PHRASE
109478	\|\| pExpr->eType==FTSQUERY_NEAR
109479	\|\| isReqPos==0
109480	);


109481	if( pExpr->eType==FTSQUERY_PHRASE ){
109482	rc = fts3PhraseSelect(p, pExpr->pPhrase,
109483	isReqPos \|\| (pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR),
109484	paOut, pnOut
109485	);




























































109486	}else{
109487	char *aLeft;
109488	char *aRight;
109489	int nLeft;
109490	int nRight;
109491
109492	if( 0==(rc = evalFts3Expr(p, pExpr->pRight, &aRight, &nRight, isReqPos))
109493	&& 0==(rc = evalFts3Expr(p, pExpr->pLeft, &aLeft, &nLeft, isReqPos))






109494	){
109495	assert( pExpr->eType==FTSQUERY_NEAR \|\| pExpr->eType==FTSQUERY_OR
109496	\|\| pExpr->eType==FTSQUERY_AND \|\| pExpr->eType==FTSQUERY_NOT
109497	);
109498	switch( pExpr->eType ){
109499	case FTSQUERY_NEAR: {
109500	Fts3Expr *pLeft;
109501	Fts3Expr *pRight;
109502	int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR;
109503
109504	if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
109505	mergetype = MERGE_POS_NEAR;
109506	}
109507	pLeft = pExpr->pLeft;
109508	while( pLeft->eType==FTSQUERY_NEAR ){
	@@ -109527,21 +110797,21 @@
109527	** so that a buffer of zero bytes is never allocated - this can
109528	** cause fts3DoclistMerge() to incorrectly return SQLITE_NOMEM.
109529	*/
109530	char *aBuffer = sqlite3_malloc(nRight+nLeft+1);
109531	rc = fts3DoclistMerge(MERGE_OR, 0, 0, aBuffer, pnOut,
109532	aLeft, nLeft, aRight, nRight
109533	);
109534	*paOut = aBuffer;
109535	sqlite3_free(aLeft);
109536	break;
109537	}
109538
109539	default: {
109540	assert( FTSQUERY_NOT==MERGE_NOT && FTSQUERY_AND==MERGE_AND );
109541	fts3DoclistMerge(pExpr->eType, 0, 0, aLeft, pnOut,
109542	aLeft, nLeft, aRight, nRight
109543	);
109544	*paOut = aLeft;
109545	break;
109546	}
109547	}
	@@ -109548,10 +110818,92 @@
109548	}
109549	sqlite3_free(aRight);
109550	}
109551	}
109552


















































































109553	return rc;
109554	}
109555
109556	/*
109557	** This is the xFilter interface for the virtual table. See
	@@ -109567,15 +110919,10 @@
109567	** If idxNum>=FTS3_FULLTEXT_SEARCH then use the full text index. The
109568	** column on the left-hand side of the MATCH operator is column
109569	** number idxNum-FTS3_FULLTEXT_SEARCH, 0 indexed. argv[0] is the right-hand
109570	** side of the MATCH operator.
109571	*/
109572	/* TODO(shess) Upgrade the cursor initialization and destruction to
109573	** account for fts3FilterMethod() being called multiple times on the
109574	** same cursor. The current solution is very fragile. Apply fix to
109575	** fts3 as appropriate.
109576	*/
109577	static int fts3FilterMethod(
109578	sqlite3_vtab_cursor pCursor, / The cursor used for this query */
109579	int idxNum, /* Strategy index */
109580	const char idxStr, / Unused */
109581	int nVal, /* Number of elements in apVal */
	@@ -109594,35 +110941,19 @@
109594	UNUSED_PARAMETER(nVal);
109595
109596	assert( idxNum>=0 && idxNum<=(FTS3_FULLTEXT_SEARCH+p->nColumn) );
109597	assert( nVal==0 \|\| nVal==1 );
109598	assert( (nVal==0)==(idxNum==FTS3_FULLSCAN_SEARCH) );

109599
109600	/* In case the cursor has been used before, clear it now. */
109601	sqlite3_finalize(pCsr->pStmt);
109602	sqlite3_free(pCsr->aDoclist);
109603	sqlite3Fts3ExprFree(pCsr->pExpr);
109604	memset(&pCursor[1], 0, sizeof(Fts3Cursor)-sizeof(sqlite3_vtab_cursor));
109605
109606	/* Compile a SELECT statement for this cursor. For a full-table-scan, the
109607	** statement loops through all rows of the %_content table. For a
109608	** full-text query or docid lookup, the statement retrieves a single
109609	** row by docid.
109610	*/
109611	zSql = sqlite3_mprintf(azSql[idxNum==FTS3_FULLSCAN_SEARCH], p->zDb, p->zName);
109612	if( !zSql ){
109613	rc = SQLITE_NOMEM;
109614	}else{
109615	rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0);
109616	sqlite3_free(zSql);
109617	}
109618	if( rc!=SQLITE_OK ) return rc;
109619	pCsr->eSearch = (i16)idxNum;
109620
109621	if( idxNum==FTS3_DOCID_SEARCH ){
109622	rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]);
109623	}else if( idxNum!=FTS3_FULLSCAN_SEARCH ){
109624	int iCol = idxNum-FTS3_FULLTEXT_SEARCH;
109625	const char zQuery = (const char )sqlite3_value_text(apVal[0]);
109626
109627	if( zQuery==0 && sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
109628	return SQLITE_NOMEM;
	@@ -109640,14 +110971,33 @@
109640	}
109641
109642	rc = sqlite3Fts3ReadLock(p);
109643	if( rc!=SQLITE_OK ) return rc;
109644
109645	rc = evalFts3Expr(p, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0);


109646	pCsr->pNextId = pCsr->aDoclist;
109647	pCsr->iPrevId = 0;
109648	}

















109649
109650	if( rc!=SQLITE_OK ) return rc;
109651	return fts3NextMethod(pCursor);
109652	}
109653
	@@ -109668,10 +111018,15 @@
109668	static int fts3RowidMethod(sqlite3_vtab_cursor pCursor, sqlite_int64 pRowid){
109669	Fts3Cursor pCsr = (Fts3Cursor ) pCursor;
109670	if( pCsr->aDoclist ){
109671	*pRowid = pCsr->iPrevId;
109672	}else{





109673	*pRowid = sqlite3_column_int64(pCsr->pStmt, 0);
109674	}
109675	return SQLITE_OK;
109676	}
109677
	@@ -109730,11 +111085,13 @@
109730	/*
109731	** Implementation of xSync() method. Flush the contents of the pending-terms
109732	** hash-table to the database.
109733	*/
109734	static int fts3SyncMethod(sqlite3_vtab *pVtab){
109735	return sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab);


109736	}
109737
109738	/*
109739	** Implementation of xBegin() method. This is a no-op.
109740	*/
	@@ -109768,12 +111125,31 @@
109768	** Load the doclist associated with expression pExpr to pExpr->aDoclist.
109769	** The loaded doclist contains positions as well as the document ids.
109770	** This is used by the matchinfo(), snippet() and offsets() auxillary
109771	** functions.
109772	*/
109773	SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Table pTab, Fts3Expr pExpr){
109774	return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);



















109775	}
109776
109777	/*
109778	** After ExprLoadDoclist() (see above) has been called, this function is
109779	** used to iterate/search through the position lists that make up the doclist
	@@ -109835,11 +111211,11 @@
109835	*/
109836	static int fts3FunctionArg(
109837	sqlite3_context pContext, / SQL function call context */
109838	const char zFunc, / Function name */
109839	sqlite3_value pVal, / argv[0] passed to function */
109840	Fts3Cursor *ppCsr / OUT: Store cursor handle here */
109841	){
109842	Fts3Cursor *pRet;
109843	if( sqlite3_value_type(pVal)!=SQLITE_BLOB
109844	\|\| sqlite3_value_bytes(pVal)!=sizeof(Fts3Cursor *)
109845	){
	@@ -109961,17 +111337,11 @@
109961	sqlite3_context pContext, / SQLite function call context */
109962	int nVal, /* Size of argument array */
109963	sqlite3_value *apVal / Array of arguments */
109964	){
109965	Fts3Cursor pCsr; / Cursor handle passed through apVal[0] */
109966
109967	if( nVal!=1 ){
109968	sqlite3_result_error(pContext,
109969	"wrong number of arguments to function matchinfo()", -1);
109970	return;
109971	}
109972
109973	if( SQLITE_OK==fts3FunctionArg(pContext, "matchinfo", apVal[0], &pCsr) ){
109974	sqlite3Fts3Matchinfo(pContext, pCsr);
109975	}
109976	}
109977
	@@ -110030,16 +111400,17 @@
110030
110031	fts3DbExec(&rc, db,
110032	"ALTER TABLE %Q.'%q_content' RENAME TO '%q_content';",
110033	p->zDb, p->zName, zName
110034	);
110035	if( rc==SQLITE_ERROR ) rc = SQLITE_OK;
110036	if( p->bHasDocsize ){
110037	fts3DbExec(&rc, db,
110038	"ALTER TABLE %Q.'%q_docsize' RENAME TO '%q_docsize';",
110039	p->zDb, p->zName, zName
110040	);


110041	fts3DbExec(&rc, db,
110042	"ALTER TABLE %Q.'%q_stat' RENAME TO '%q_stat';",
110043	p->zDb, p->zName, zName
110044	);
110045	}
	@@ -110060,11 +111431,11 @@
110060	/* xConnect */ fts3ConnectMethod,
110061	/* xBestIndex */ fts3BestIndexMethod,
110062	/* xDisconnect */ fts3DisconnectMethod,
110063	/* xDestroy */ fts3DestroyMethod,
110064	/* xOpen */ fts3OpenMethod,
110065	/* xClose */ fulltextClose,
110066	/* xFilter */ fts3FilterMethod,
110067	/* xNext */ fts3NextMethod,
110068	/* xEof */ fts3EofMethod,
110069	/* xColumn */ fts3ColumnMethod,
110070	/* xRowid */ fts3RowidMethod,
	@@ -110087,23 +111458,24 @@
110087	sqlite3Fts3HashClear(pHash);
110088	sqlite3_free(pHash);
110089	}
110090
110091	/*
110092	** The fts3 built-in tokenizers - "simple" and "porter" - are implemented
110093	** in files fts3_tokenizer1.c and fts3_porter.c respectively. The following
110094	** two forward declarations are for functions declared in these files
110095	** used to retrieve the respective implementations.
110096	**
110097	** Calling sqlite3Fts3SimpleTokenizerModule() sets the value pointed
110098	** to by the argument to point to the "simple" tokenizer implementation.
110099	** Function ...PorterTokenizerModule() sets *pModule to point to the
110100	** porter tokenizer/stemmer implementation.
110101	*/
110102	SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule);
110103	SQLITE_PRIVATE void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule);

110104	SQLITE_PRIVATE void sqlite3Fts3IcuTokenizerModule(sqlite3_tokenizer_module const**ppModule);

110105
110106	/*
110107	** Initialise the fts3 extension. If this extension is built as part
110108	** of the sqlite library, then this function is called directly by
110109	** SQLite. If fts3 is built as a dynamically loadable extension, this
	@@ -110155,11 +111527,11 @@
110155	*/
110156	if( SQLITE_OK==rc
110157	&& SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
110158	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))
110159	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
110160	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1))
110161	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
110162	){
110163	rc = sqlite3_create_module_v2(
110164	db, "fts3", &fts3Module, (void *)pHash, hashDestroy
110165	);
	@@ -110297,10 +111669,22 @@
110297	** negative values).
110298	*/
110299	static int fts3isspace(char c){
110300	return c==' ' \|\| c=='\t' \|\| c=='\n' \|\| c=='\r' \|\| c=='\v' \|\| c=='\f';
110301	}












110302
110303	/*
110304	** Extract the next token from buffer z (length n) using the tokenizer
110305	** and other information (column names etc.) in pParse. Create an Fts3Expr
110306	** structure of type FTSQUERY_PHRASE containing a phrase consisting of this
	@@ -110335,15 +111719,14 @@
110335	pCursor->pTokenizer = pTokenizer;
110336	rc = pModule->xNext(pCursor, &zToken, &nToken, &iStart, &iEnd, &iPosition);
110337
110338	if( rc==SQLITE_OK ){
110339	nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase) + nToken;
110340	pRet = (Fts3Expr *)sqlite3_malloc(nByte);
110341	if( !pRet ){
110342	rc = SQLITE_NOMEM;
110343	}else{
110344	memset(pRet, 0, nByte);
110345	pRet->eType = FTSQUERY_PHRASE;
110346	pRet->pPhrase = (Fts3Phrase *)&pRet[1];
110347	pRet->pPhrase->nToken = 1;
110348	pRet->pPhrase->iColumn = iCol;
110349	pRet->pPhrase->aToken[0].n = nToken;
	@@ -110415,20 +111798,21 @@
110415	const char *zToken;
110416	int nToken, iBegin, iEnd, iPos;
110417	rc = pModule->xNext(pCursor, &zToken, &nToken, &iBegin, &iEnd, &iPos);
110418	if( rc==SQLITE_OK ){
110419	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
110420	p = fts3ReallocOrFree(p, nByte+ii*sizeof(struct PhraseToken));
110421	zTemp = fts3ReallocOrFree(zTemp, nTemp + nToken);
110422	if( !p \|\| !zTemp ){
110423	goto no_mem;
110424	}
110425	if( ii==0 ){
110426	memset(p, 0, nByte);
110427	p->pPhrase = (Fts3Phrase *)&p[1];
110428	}
110429	p->pPhrase = (Fts3Phrase *)&p[1];

110430	p->pPhrase->nToken = ii+1;
110431	p->pPhrase->aToken[ii].n = nToken;
110432	memcpy(&zTemp[nTemp], zToken, nToken);
110433	nTemp += nToken;
110434	if( iEnd<nInput && zInput[iEnd]=='*' ){
	@@ -110446,11 +111830,11 @@
110446	if( rc==SQLITE_DONE ){
110447	int jj;
110448	char *zNew = NULL;
110449	int nNew = 0;
110450	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
110451	nByte += (p?(p->pPhrase->nToken-1):0) * sizeof(struct PhraseToken);
110452	p = fts3ReallocOrFree(p, nByte + nTemp);
110453	if( !p ){
110454	goto no_mem;
110455	}
110456	if( zTemp ){
	@@ -110564,15 +111948,14 @@
110564	*/
110565	cNext = zInput[nKey];
110566	if( fts3isspace(cNext)
110567	\|\| cNext=='"' \|\| cNext=='(' \|\| cNext==')' \|\| cNext==0
110568	){
110569	pRet = (Fts3Expr *)sqlite3_malloc(sizeof(Fts3Expr));
110570	if( !pRet ){
110571	return SQLITE_NOMEM;
110572	}
110573	memset(pRet, 0, sizeof(Fts3Expr));
110574	pRet->eType = pKey->eType;
110575	pRet->nNear = nNear;
110576	*ppExpr = pRet;
110577	*pnConsumed = (int)((zInput - z) + nKey);
110578	return SQLITE_OK;
	@@ -110744,17 +112127,16 @@
110744
110745	if( !sqlite3_fts3_enable_parentheses
110746	&& p->eType==FTSQUERY_PHRASE && p->pPhrase->isNot
110747	){
110748	/* Create an implicit NOT operator. */
110749	Fts3Expr *pNot = sqlite3_malloc(sizeof(Fts3Expr));
110750	if( !pNot ){
110751	sqlite3Fts3ExprFree(p);
110752	rc = SQLITE_NOMEM;
110753	goto exprparse_out;
110754	}
110755	memset(pNot, 0, sizeof(Fts3Expr));
110756	pNot->eType = FTSQUERY_NOT;
110757	pNot->pRight = p;
110758	if( pNotBranch ){
110759	pNot->pLeft = pNotBranch;
110760	}
	@@ -110778,17 +112160,16 @@
110778
110779	if( isPhrase && !isRequirePhrase ){
110780	/* Insert an implicit AND operator. */
110781	Fts3Expr *pAnd;
110782	assert( pRet && pPrev );
110783	pAnd = sqlite3_malloc(sizeof(Fts3Expr));
110784	if( !pAnd ){
110785	sqlite3Fts3ExprFree(p);
110786	rc = SQLITE_NOMEM;
110787	goto exprparse_out;
110788	}
110789	memset(pAnd, 0, sizeof(Fts3Expr));
110790	pAnd->eType = FTSQUERY_AND;
110791	insertBinaryOperator(&pRet, pPrev, pAnd);
110792	pPrev = pAnd;
110793	}
110794
	@@ -112234,11 +113615,11 @@
112234	}
112235
112236	sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
112237	}
112238
112239	static int fts3IsIdChar(char c){
112240	static const char isFtsIdChar[] = {
112241	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x */
112242	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 1x */
112243	0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 2x */
112244	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 3x */
	@@ -112272,13 +113653,13 @@
112272	while( *z2 && z2[0]!=']' ) z2++;
112273	if( *z2 ) z2++;
112274	break;
112275
112276	default:
112277	if( fts3IsIdChar(*z1) ){
112278	z2 = &z1[1];
112279	while( fts3IsIdChar(*z2) ) z2++;
112280	}else{
112281	z1++;
112282	}
112283	}
112284	}
	@@ -112287,42 +113668,30 @@
112287	return z1;
112288	}
112289
112290	SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(
112291	Fts3Hash pHash, / Tokenizer hash table */
112292	const char zArg, / Possible tokenizer specification */
112293	sqlite3_tokenizer *ppTok, / OUT: Tokenizer (if applicable) */
112294	const char *pzTokenizer, / OUT: Set to zArg if is tokenizer */
112295	char *pzErr / OUT: Set to malloced error message */
112296	){
112297	int rc;
112298	char z = (char )zArg;
112299	int n;
112300	char *zCopy;
112301	char zEnd; / Pointer to nul-term of zCopy */
112302	sqlite3_tokenizer_module *m;
112303
112304	if( !z ){
112305	zCopy = sqlite3_mprintf("simple");
112306	}else{
112307	if( sqlite3_strnicmp(z, "tokenize", 8) \|\| fts3IsIdChar(z[8])){
112308	return SQLITE_OK;
112309	}
112310	zCopy = sqlite3_mprintf("%s", &z[8]);
112311	*pzTokenizer = zArg;
112312	}
112313	if( !zCopy ){
112314	return SQLITE_NOMEM;
112315	}
112316
112317	zEnd = &zCopy[strlen(zCopy)];
112318
112319	z = (char *)sqlite3Fts3NextToken(zCopy, &n);
112320	z[n] = '\0';
112321	sqlite3Fts3Dequote(z);
112322
112323	m = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash, z, (int)strlen(z)+1);
112324	if( !m ){
112325	*pzErr = sqlite3_mprintf("unknown tokenizer: %s", z);
112326	rc = SQLITE_ERROR;
112327	}else{
112328	char const **aArg = 0;
	@@ -112887,10 +114256,22 @@
112887	*/
112888
112889	#if !defined(SQLITE_CORE) \|\| defined(SQLITE_ENABLE_FTS3)
112890
112891












112892	typedef struct PendingList PendingList;
112893	typedef struct SegmentNode SegmentNode;
112894	typedef struct SegmentWriter SegmentWriter;
112895
112896	/*
	@@ -112905,19 +114286,31 @@
112905	sqlite3_int64 iLastDocid;
112906	sqlite3_int64 iLastCol;
112907	sqlite3_int64 iLastPos;
112908	};
112909











112910	/*
112911	** An instance of this structure is used to iterate through the terms on
112912	** a contiguous set of segment b-tree leaf nodes. Although the details of
112913	** this structure are only manipulated by code in this file, opaque handles
112914	** of type Fts3SegReader* are also used by code in fts3.c to iterate through
112915	** terms when querying the full-text index. See functions:
112916	**
112917	** sqlite3Fts3SegReaderNew()
112918	** sqlite3Fts3SegReaderFree()

112919	** sqlite3Fts3SegReaderIterate()
112920	**
112921	** Methods used to manipulate Fts3SegReader structures:
112922	**
112923	** fts3SegReaderNext()
	@@ -112924,34 +114317,38 @@
112924	** fts3SegReaderFirstDocid()
112925	** fts3SegReaderNextDocid()
112926	*/
112927	struct Fts3SegReader {
112928	int iIdx; /* Index within level, or 0x7FFFFFFF for PT */
112929	sqlite3_int64 iStartBlock;
112930	sqlite3_int64 iEndBlock;
112931	sqlite3_stmt pStmt; / SQL Statement to access leaf nodes */



112932	char aNode; / Pointer to node data (or NULL) */
112933	int nNode; /* Size of buffer at aNode (or 0) */
112934	int nTermAlloc; /* Allocated size of zTerm buffer */
112935	Fts3HashElem **ppNextElem;
112936
112937	/* Variables set by fts3SegReaderNext(). These may be read directly
112938	** by the caller. They are valid from the time SegmentReaderNew() returns
112939	** until SegmentReaderNext() returns something other than SQLITE_OK
112940	** (i.e. SQLITE_DONE).
112941	*/
112942	int nTerm; /* Number of bytes in current term */
112943	char zTerm; / Pointer to current term */

112944	char aDoclist; / Pointer to doclist of current entry */
112945	int nDoclist; /* Size of doclist in current entry */
112946
112947	/* The following variables are used to iterate through the current doclist */
112948	char *pOffsetList;
112949	sqlite3_int64 iDocid;
112950	};
112951
112952	#define fts3SegReaderIsPending(p) ((p)->ppNextElem!=0)

112953
112954	/*
112955	** An instance of this structure is used to create a segment b-tree in the
112956	** database. The internal details of this type are only accessed by the
112957	** following functions:
	@@ -113016,16 +114413,15 @@
113016	#define SQL_SELECT_LEVEL_COUNT 14
113017	#define SQL_SELECT_SEGDIR_COUNT_MAX 15
113018	#define SQL_DELETE_SEGDIR_BY_LEVEL 16
113019	#define SQL_DELETE_SEGMENTS_RANGE 17
113020	#define SQL_CONTENT_INSERT 18
113021	#define SQL_GET_BLOCK 19
113022	#define SQL_DELETE_DOCSIZE 20
113023	#define SQL_REPLACE_DOCSIZE 21
113024	#define SQL_SELECT_DOCSIZE 22
113025	#define SQL_SELECT_DOCTOTAL 23
113026	#define SQL_REPLACE_DOCTOTAL 24
113027
113028	/*
113029	** This function is used to obtain an SQLite prepared statement handle
113030	** for the statement identified by the second argument. If successful,
113031	** *pp is set to the requested statement handle and SQLITE_OK returned.
	@@ -113066,16 +114462,15 @@
113066	/* 15 / "SELECT count(), max(level) FROM %Q.'%q_segdir'",
113067
113068	/* 16 */ "DELETE FROM %Q.'%q_segdir' WHERE level = ?",
113069	/* 17 */ "DELETE FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ?",
113070	/* 18 */ "INSERT INTO %Q.'%q_content' VALUES(%z)",
113071	/* 19 */ "SELECT block FROM %Q.'%q_segments' WHERE blockid = ?",
113072	/* 20 */ "DELETE FROM %Q.'%q_docsize' WHERE docid = ?",
113073	/* 21 */ "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)",
113074	/* 22 */ "SELECT size FROM %Q.'%q_docsize' WHERE docid=?",
113075	/* 23 */ "SELECT value FROM %Q.'%q_stat' WHERE id=0",
113076	/* 24 */ "REPLACE INTO %Q.'%q_stat' VALUES(0,?)",
113077	};
113078	int rc = SQLITE_OK;
113079	sqlite3_stmt *pStmt;
113080
113081	assert( SizeofArray(azSql)==SizeofArray(p->aStmt) );
	@@ -113146,49 +114541,10 @@
113146	}
113147	*pRC = rc;
113148	}
113149
113150
113151	/*
113152	** Read a single block from the %_segments table. If the specified block
113153	** does not exist, return SQLITE_CORRUPT. If some other error (malloc, IO
113154	** etc.) occurs, return the appropriate SQLite error code.
113155	**
113156	** Otherwise, if successful, set *pzBlock to point to a buffer containing
113157	** the block read from the database, and *pnBlock to the size of the read
113158	** block in bytes.
113159	**
113160	** WARNING: The returned buffer is only valid until the next call to
113161	** sqlite3Fts3ReadBlock().
113162	*/
113163	SQLITE_PRIVATE int sqlite3Fts3ReadBlock(
113164	Fts3Table *p,
113165	sqlite3_int64 iBlock,
113166	char const **pzBlock,
113167	int *pnBlock
113168	){
113169	sqlite3_stmt *pStmt;
113170	int rc = fts3SqlStmt(p, SQL_GET_BLOCK, &pStmt, 0);
113171	if( rc!=SQLITE_OK ) return rc;
113172	sqlite3_reset(pStmt);
113173
113174	if( pzBlock ){
113175	sqlite3_bind_int64(pStmt, 1, iBlock);
113176	rc = sqlite3_step(pStmt);
113177	if( rc!=SQLITE_ROW ){
113178	return (rc==SQLITE_DONE ? SQLITE_CORRUPT : rc);
113179	}
113180
113181	*pnBlock = sqlite3_column_bytes(pStmt, 0);
113182	pzBlock = (char )sqlite3_column_blob(pStmt, 0);
113183	if( sqlite3_column_type(pStmt, 0)!=SQLITE_BLOB ){
113184	return SQLITE_CORRUPT;
113185	}
113186	}
113187	return SQLITE_OK;
113188	}
113189
113190	/*
113191	** This function ensures that the caller has obtained a shared-cache
113192	** table-lock on the %_content table. This is required before reading
113193	** data from the fts3 table. If this lock is not acquired first, then
113194	** the caller may end up holding read-locks on the %_segments and %_segdir
	@@ -113353,14 +114709,14 @@
113353	** p->iPrevDocid, and the column is specified by argument iCol.
113354	**
113355	** If successful, SQLITE_OK is returned. Otherwise, an SQLite error code.
113356	*/
113357	static int fts3PendingTermsAdd(
113358	Fts3Table p, / FTS table into which text will be inserted */
113359	const char zText, / Text of document to be inseted */
113360	int iCol, /* Column number into which text is inserted */
113361	u32 pnWord / OUT: Number of tokens inserted */
113362	){
113363	int rc;
113364	int iStart;
113365	int iEnd;
113366	int iPos;
	@@ -113441,10 +114797,13 @@
113441	}
113442	p->iPrevDocid = iDocid;
113443	return SQLITE_OK;
113444	}
113445



113446	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *p){
113447	Fts3HashElem *pElem;
113448	for(pElem=fts3HashFirst(&p->pendingTerms); pElem; pElem=fts3HashNext(pElem)){
113449	sqlite3_free(fts3HashData(pElem));
113450	}
	@@ -113468,10 +114827,11 @@
113468	int rc = fts3PendingTermsAdd(p, zText, i-2, &aSz[i-2]);
113469	if( rc!=SQLITE_OK ){
113470	return rc;
113471	}
113472	}

113473	}
113474	return SQLITE_OK;
113475	}
113476
113477	/*
	@@ -113555,10 +114915,12 @@
113555	fts3SqlExec(&rc, p, SQL_DELETE_ALL_CONTENT, 0);
113556	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGMENTS, 0);
113557	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGDIR, 0);
113558	if( p->bHasDocsize ){
113559	fts3SqlExec(&rc, p, SQL_DELETE_ALL_DOCSIZE, 0);


113560	fts3SqlExec(&rc, p, SQL_DELETE_ALL_STAT, 0);
113561	}
113562	return rc;
113563	}
113564
	@@ -113565,11 +114927,11 @@
113565	/*
113566	** The first element in the apVal[] array is assumed to contain the docid
113567	** (an integer) of a row about to be deleted. Remove all terms from the
113568	** full-text index.
113569	*/
113570	static void fts3DeleteTerms(
113571	int pRC, / Result code */
113572	Fts3Table p, / The FTS table to delete from */
113573	sqlite3_value *apVal, / apVal[] contains the docid to be deleted */
113574	u32 aSz / Sizes of deleted document written here */
113575	){
	@@ -113587,10 +114949,11 @@
113587	if( rc!=SQLITE_OK ){
113588	sqlite3_reset(pSelect);
113589	*pRC = rc;
113590	return;
113591	}

113592	}
113593	}
113594	rc = sqlite3_reset(pSelect);
113595	}else{
113596	sqlite3_reset(pSelect);
	@@ -113648,17 +115011,98 @@
113648	}
113649	}
113650
113651	return rc;
113652	}

















































































113653
113654	/*
113655	** Move the iterator passed as the first argument to the next term in the
113656	** segment. If successful, SQLITE_OK is returned. If there is no next term,
113657	** SQLITE_DONE. Otherwise, an SQLite error code.
113658	*/
113659	static int fts3SegReaderNext(Fts3SegReader *pReader){
113660	char pNext; / Cursor variable */
113661	int nPrefix; /* Number of bytes in term prefix */
113662	int nSuffix; /* Number of bytes in term suffix */
113663
113664	if( !pReader->aDoclist ){
	@@ -113666,11 +115110,12 @@
113666	}else{
113667	pNext = &pReader->aDoclist[pReader->nDoclist];
113668	}
113669
113670	if( !pNext \|\| pNext>=&pReader->aNode[pReader->nNode] ){
113671	int rc;

113672	if( fts3SegReaderIsPending(pReader) ){
113673	Fts3HashElem pElem = (pReader->ppNextElem);
113674	if( pElem==0 ){
113675	pReader->aNode = 0;
113676	}else{
	@@ -113682,26 +115127,40 @@
113682	pReader->ppNextElem++;
113683	assert( pReader->aNode );
113684	}
113685	return SQLITE_OK;
113686	}
113687	if( !pReader->pStmt ){
113688	pReader->aNode = 0;








113689	return SQLITE_OK;
113690	}
113691	rc = sqlite3_step(pReader->pStmt);
113692	if( rc!=SQLITE_ROW ){
113693	pReader->aNode = 0;
113694	return (rc==SQLITE_DONE ? SQLITE_OK : rc);
113695	}
113696	pReader->nNode = sqlite3_column_bytes(pReader->pStmt, 0);
113697	pReader->aNode = (char *)sqlite3_column_blob(pReader->pStmt, 0);
113698	pNext = pReader->aNode;
113699	}
113700



113701	pNext += sqlite3Fts3GetVarint32(pNext, &nPrefix);
113702	pNext += sqlite3Fts3GetVarint32(pNext, &nSuffix);





113703
113704	if( nPrefix+nSuffix>pReader->nTermAlloc ){
113705	int nNew = (nPrefix+nSuffix)*2;
113706	char *zNew = sqlite3_realloc(pReader->zTerm, nNew);
113707	if( !zNew ){
	@@ -113712,13 +115171,22 @@
113712	}
113713	memcpy(&pReader->zTerm[nPrefix], pNext, nSuffix);
113714	pReader->nTerm = nPrefix+nSuffix;
113715	pNext += nSuffix;
113716	pNext += sqlite3Fts3GetVarint32(pNext, &pReader->nDoclist);
113717	assert( pNext<&pReader->aNode[pReader->nNode] );
113718	pReader->aDoclist = pNext;
113719	pReader->pOffsetList = 0;










113720	return SQLITE_OK;
113721	}
113722
113723	/*
113724	** Set the SegReader to point to the first docid in the doclist associated
	@@ -113776,30 +115244,105 @@
113776	sqlite3_int64 iDelta;
113777	pReader->pOffsetList = p + sqlite3Fts3GetVarint(p, &iDelta);
113778	pReader->iDocid += iDelta;
113779	}
113780	}


















































































113781
113782	/*
113783	** Free all allocations associated with the iterator passed as the
113784	** second argument.
113785	*/
113786	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table p, Fts3SegReader pReader){
113787	if( pReader ){
113788	if( pReader->pStmt ){
113789	/* Move the leaf-range SELECT statement to the aLeavesStmt[] array,
113790	** so that it can be reused when required by another query.
113791	*/
113792	assert( p->nLeavesStmt<p->nLeavesTotal );
113793	sqlite3_reset(pReader->pStmt);
113794	p->aLeavesStmt[p->nLeavesStmt++] = pReader->pStmt;
113795	}
113796	if( !fts3SegReaderIsPending(pReader) ){
113797	sqlite3_free(pReader->zTerm);
113798	}
113799	sqlite3_free(pReader);
113800	}
113801	}
113802
113803	/*
113804	** Allocate a new SegReader object.
113805	*/
	@@ -113815,77 +115358,35 @@
113815	){
113816	int rc = SQLITE_OK; /* Return code */
113817	Fts3SegReader pReader; / Newly allocated SegReader object */
113818	int nExtra = 0; /* Bytes to allocate segment root node */
113819

113820	if( iStartLeaf==0 ){
113821	nExtra = nRoot;
113822	}
113823
113824	pReader = (Fts3SegReader *)sqlite3_malloc(sizeof(Fts3SegReader) + nExtra);
113825	if( !pReader ){
113826	return SQLITE_NOMEM;
113827	}
113828	memset(pReader, 0, sizeof(Fts3SegReader));
113829	pReader->iStartBlock = iStartLeaf;
113830	pReader->iIdx = iAge;


113831	pReader->iEndBlock = iEndBlock;
113832
113833	if( nExtra ){
113834	/* The entire segment is stored in the root node. */
113835	pReader->aNode = (char *)&pReader[1];
113836	pReader->nNode = nRoot;
113837	memcpy(pReader->aNode, zRoot, nRoot);
113838	}else{
113839	/* If the text of the SQL statement to iterate through a contiguous
113840	** set of entries in the %_segments table has not yet been composed,
113841	** compose it now.
113842	*/
113843	if( !p->zSelectLeaves ){
113844	p->zSelectLeaves = sqlite3_mprintf(
113845	"SELECT block FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ? "
113846	"ORDER BY blockid", p->zDb, p->zName
113847	);
113848	if( !p->zSelectLeaves ){
113849	rc = SQLITE_NOMEM;
113850	goto finished;
113851	}
113852	}
113853
113854	/* If there are no free statements in the aLeavesStmt[] array, prepare
113855	** a new statement now. Otherwise, reuse a prepared statement from
113856	** aLeavesStmt[].
113857	*/
113858	if( p->nLeavesStmt==0 ){
113859	if( p->nLeavesTotal==p->nLeavesAlloc ){
113860	int nNew = p->nLeavesAlloc + 16;
113861	sqlite3_stmt aNew = (sqlite3_stmt )sqlite3_realloc(
113862	p->aLeavesStmt, nNewsizeof(sqlite3_stmt )
113863	);
113864	if( !aNew ){
113865	rc = SQLITE_NOMEM;
113866	goto finished;
113867	}
113868	p->nLeavesAlloc = nNew;
113869	p->aLeavesStmt = aNew;
113870	}
113871	rc = sqlite3_prepare_v2(p->db, p->zSelectLeaves, -1, &pReader->pStmt, 0);
113872	if( rc!=SQLITE_OK ){
113873	goto finished;
113874	}
113875	p->nLeavesTotal++;
113876	}else{
113877	pReader->pStmt = p->aLeavesStmt[--p->nLeavesStmt];
113878	}
113879
113880	/* Bind the start and end leaf blockids to the prepared SQL statement. */
113881	sqlite3_bind_int64(pReader->pStmt, 1, iStartLeaf);
113882	sqlite3_bind_int64(pReader->pStmt, 2, iEndLeaf);
113883	}
113884	rc = fts3SegReaderNext(pReader);
113885
113886	finished:
113887	if( rc==SQLITE_OK ){
113888	*ppReader = pReader;
113889	}else{
113890	sqlite3Fts3SegReaderFree(p, pReader);
113891	}
	@@ -113976,11 +115477,10 @@
113976	}else{
113977	memset(pReader, 0, nByte);
113978	pReader->iIdx = 0x7FFFFFFF;
113979	pReader->ppNextElem = (Fts3HashElem **)&pReader[1];
113980	memcpy(pReader->ppNextElem, aElem, nElemsizeof(Fts3HashElem ));
113981	fts3SegReaderNext(pReader);
113982	}
113983	}
113984
113985	if( isPrefix ){
113986	sqlite3_free(aElem);
	@@ -114218,11 +115718,11 @@
114218	/*
114219	** Add term zTerm to the SegmentNode. It is guaranteed that zTerm is larger
114220	** (according to memcmp) than the previous term.
114221	*/
114222	static int fts3NodeAddTerm(
114223	Fts3Table p, / Virtual table handle */
114224	SegmentNode *ppTree, / IN/OUT: SegmentNode handle */
114225	int isCopyTerm, /* True if zTerm/nTerm is transient */
114226	const char zTerm, / Pointer to buffer containing term */
114227	int nTerm /* Size of term in bytes */
114228	){
	@@ -114848,19 +116348,18 @@
114848	** for, then advance each segment iterator until it points to a term of
114849	** equal or greater value than the specified term. This prevents many
114850	** unnecessary merge/sort operations for the case where single segment
114851	** b-tree leaf nodes contain more than one term.
114852	*/
114853	if( pFilter->zTerm ){
114854	int nTerm = pFilter->nTerm;
114855	const char *zTerm = pFilter->zTerm;
114856	for(i=0; i<nSegment; i++){
114857	Fts3SegReader *pSeg = apSegment[i];
114858	while( fts3SegReaderTermCmp(pSeg, zTerm, nTerm)<0 ){
114859	rc = fts3SegReaderNext(pSeg);
114860	if( rc!=SQLITE_OK ) goto finished; }
114861	}
114862	}
114863
114864	fts3SegReaderSort(apSegment, nSegment, nSegment, fts3SegReaderCmp);
114865	while( apSegment[0]->aNode ){
114866	int nTerm = apSegment[0]->nTerm;
	@@ -114965,11 +116464,11 @@
114965	if( pFilter->zTerm && !isPrefix ){
114966	goto finished;
114967	}
114968
114969	for(i=0; i<nMerge; i++){
114970	rc = fts3SegReaderNext(apSegment[i]);
114971	if( rc!=SQLITE_OK ) goto finished;
114972	}
114973	fts3SegReaderSort(apSegment, nSegment, nMerge, fts3SegReaderCmp);
114974	}
114975
	@@ -114991,11 +116490,11 @@
114991	*/
114992	static int fts3SegmentMerge(Fts3Table *p, int iLevel){
114993	int i; /* Iterator variable */
114994	int rc; /* Return code */
114995	int iIdx; /* Index of new segment */
114996	int iNewLevel; /* Level to create new segment at */
114997	sqlite3_stmt *pStmt = 0;
114998	SegmentWriter *pWriter = 0;
114999	int nSegment = 0; /* Number of segments being merged */
115000	Fts3SegReader *apSegment = 0; / Array of Segment iterators */
115001	Fts3SegReader pPending = 0; / Iterator for pending-terms */
	@@ -115280,64 +116779,76 @@
115280	sqlite3_step(pStmt);
115281	*pRC = sqlite3_reset(pStmt);
115282	}
115283
115284	/*
115285	** Update the 0 record of the %_stat table so that it holds a blob
115286	** which contains the document count followed by the cumulative
115287	** document sizes for all columns.










115288	*/
115289	static void fts3UpdateDocTotals(
115290	int pRC, / The result code */
115291	Fts3Table p, / Table being updated */
115292	u32 aSzIns, / Size increases */
115293	u32 aSzDel, / Size decreases */
115294	int nChng /* Change in the number of documents */
115295	){
115296	char pBlob; / Storage for BLOB written into %_stat */
115297	int nBlob; /* Size of BLOB written into %_stat */
115298	u32 a; / Array of integers that becomes the BLOB */
115299	sqlite3_stmt pStmt; / Statement for reading and writing */
115300	int i; /* Loop counter */
115301	int rc; /* Result code from subfunctions */
115302


115303	if( *pRC ) return;
115304	a = sqlite3_malloc( (sizeof(u32)+10)*(p->nColumn+1) );
115305	if( a==0 ){
115306	*pRC = SQLITE_NOMEM;
115307	return;
115308	}
115309	pBlob = (char*)&a[p->nColumn+1];
115310	rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
115311	if( rc ){
115312	sqlite3_free(a);
115313	*pRC = rc;
115314	return;
115315	}
115316	if( sqlite3_step(pStmt)==SQLITE_ROW ){
115317	fts3DecodeIntArray(p->nColumn+1, a,
115318	sqlite3_column_blob(pStmt, 0),
115319	sqlite3_column_bytes(pStmt, 0));
115320	}else{
115321	memset(a, 0, sizeof(u32)*(p->nColumn+1) );
115322	}
115323	sqlite3_reset(pStmt);
115324	if( nChng<0 && a[0]<(u32)(-nChng) ){
115325	a[0] = 0;
115326	}else{
115327	a[0] += nChng;
115328	}
115329	for(i=0; i<p->nColumn; i++){
115330	u32 x = a[i+1];
115331	if( x+aSzIns[i] < aSzDel[i] ){
115332	x = 0;
115333	}else{
115334	x = x + aSzIns[i] - aSzDel[i];
115335	}
115336	a[i+1] = x;
115337	}
115338	fts3EncodeIntArray(p->nColumn+1, a, pBlob, &nBlob);
115339	rc = fts3SqlStmt(p, SQL_REPLACE_DOCTOTAL, &pStmt, 0);
115340	if( rc ){
115341	sqlite3_free(a);
115342	*pRC = rc;
115343	return;
	@@ -115379,13 +116890,163 @@
115379	rc = SQLITE_OK;
115380	#endif
115381	}else{
115382	rc = SQLITE_ERROR;
115383	}




























































































































115384
115385	return rc;
115386	}


























115387
115388	/*
115389	** This function does the work for the xUpdate method of FTS3 virtual
115390	** tables.
115391	*/
	@@ -115401,20 +117062,21 @@
115401	sqlite3_int64 iRemove = 0; /* Rowid removed by UPDATE or DELETE */
115402	u32 aSzIns; / Sizes of inserted documents */
115403	u32 aSzDel; / Sizes of deleted documents */
115404	int nChng = 0; /* Net change in number of documents */
115405

115406
115407	/* Allocate space to hold the change in document sizes */
115408	aSzIns = sqlite3_malloc( sizeof(aSzIns[0])p->nColumn2 );
115409	if( aSzIns==0 ) return SQLITE_NOMEM;
115410	aSzDel = &aSzIns[p->nColumn];
115411	memset(aSzIns, 0, sizeof(aSzIns[0])p->nColumn2);
115412
115413	/* If this is a DELETE or UPDATE operation, remove the old record. */
115414	if( sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
115415	int isEmpty;
115416	rc = fts3IsEmpty(p, apVal, &isEmpty);
115417	if( rc==SQLITE_OK ){
115418	if( isEmpty ){
115419	/* Deleting this row means the whole table is empty. In this case
115420	** delete the contents of all three tables and throw away any
	@@ -115427,12 +117089,12 @@
115427	rc = fts3PendingTermsDocid(p, iRemove);
115428	fts3DeleteTerms(&rc, p, apVal, aSzDel);
115429	fts3SqlExec(&rc, p, SQL_DELETE_CONTENT, apVal);
115430	if( p->bHasDocsize ){
115431	fts3SqlExec(&rc, p, SQL_DELETE_DOCSIZE, apVal);
115432	nChng--;
115433	}

115434	}
115435	}
115436	}else if( sqlite3_value_type(apVal[p->nColumn+2])!=SQLITE_NULL ){
115437	sqlite3_free(aSzIns);
115438	return fts3SpecialInsert(p, apVal[p->nColumn+2]);
	@@ -115446,20 +117108,21 @@
115446	}
115447	if( rc==SQLITE_OK ){
115448	rc = fts3InsertTerms(p, apVal, aSzIns);
115449	}
115450	if( p->bHasDocsize ){
115451	nChng++;
115452	fts3InsertDocsize(&rc, p, aSzIns);
115453	}

115454	}
115455
115456	if( p->bHasDocsize ){
115457	fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng);
115458	}
115459
115460	sqlite3_free(aSzIns);

115461	return rc;
115462	}
115463
115464	/*
115465	** Flush any data in the pending-terms hash table to disk. If successful,
	@@ -115479,10 +117142,11 @@
115479	}else{
115480	sqlite3_exec(p->db, "ROLLBACK TO fts3", 0, 0, 0);
115481	sqlite3_exec(p->db, "RELEASE fts3", 0, 0, 0);
115482	}
115483	}

115484	return rc;
115485	}
115486
115487	#endif
115488
	@@ -115509,11 +117173,11 @@
115509	** Used as an fts3ExprIterate() context when loading phrase doclists to
115510	** Fts3Expr.aDoclist[]/nDoclist.
115511	*/
115512	typedef struct LoadDoclistCtx LoadDoclistCtx;
115513	struct LoadDoclistCtx {
115514	Fts3Table pTab; / FTS3 Table */
115515	int nPhrase; /* Number of phrases seen so far */
115516	int nToken; /* Number of tokens seen so far */
115517	};
115518
115519	/*
	@@ -115703,11 +117367,11 @@
115703
115704	p->nPhrase++;
115705	p->nToken += pExpr->pPhrase->nToken;
115706
115707	if( pExpr->isLoaded==0 ){
115708	rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr);
115709	pExpr->isLoaded = 1;
115710	if( rc==SQLITE_OK ){
115711	rc = fts3ExprNearTrim(pExpr);
115712	}
115713	}
	@@ -115746,11 +117410,11 @@
115746	int pnPhrase, / OUT: Number of phrases in query */
115747	int pnToken / OUT: Number of tokens in query */
115748	){
115749	int rc; /* Return Code */
115750	LoadDoclistCtx sCtx = {0,0,0}; /* Context for fts3ExprIterate() */
115751	sCtx.pTab = (Fts3Table *)pCsr->base.pVtab;
115752	rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx);
115753	if( rc==SQLITE_OK ){
115754	(void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0);
115755	}
115756	if( pnPhrase ) *pnPhrase = sCtx.nPhrase;
	@@ -116277,24 +117941,51 @@
116277	Fts3Expr pExpr, / Phrase expression node */
116278	int iPhrase, /* Phrase number (numbered from zero) */
116279	void pCtx / Pointer to MatchInfo structure */
116280	){
116281	MatchInfo p = (MatchInfo )pCtx;
116282	char *pCsr;

116283	char *pEnd;

116284	const int iStart = 2 + (iPhrase * p->nCol * 3) + 1;
116285
116286	assert( pExpr->isLoaded );


























116287
116288	/* Fill in the global hit count matrix row for this phrase. */
116289	pCsr = pExpr->aDoclist;
116290	pEnd = &pExpr->aDoclist[pExpr->nDoclist];
116291	while( pCsr<pEnd ){
116292	while( pCsr++ & 0x80 ); / Skip past docid. */
116293	fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 1);
116294	}
116295

116296	return SQLITE_OK;
116297	}
116298
116299	/*
116300	** fts3ExprIterate() callback used to collect the "local" matchinfo stats
	@@ -116359,19 +118050,18 @@
116359	if( !sInfo.aMatchinfo ){
116360	return SQLITE_NOMEM;
116361	}
116362	memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo);
116363
116364
116365	/* First element of match-info is the number of phrases in the query */
116366	sInfo.aMatchinfo[0] = nPhrase;
116367	sInfo.aMatchinfo[1] = sInfo.nCol;
116368	(void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);
116369	if( pTab->bHasDocsize ){
116370	int ofst = 2 + 3sInfo.aMatchinfo[0]sInfo.aMatchinfo[1];
116371	rc = sqlite3Fts3MatchinfoDocsizeGlobal(pCsr, &sInfo.aMatchinfo[ofst]);
116372	}

116373	pCsr->aMatchinfo = sInfo.aMatchinfo;
116374	pCsr->isMatchinfoNeeded = 1;
116375	}
116376
116377	sInfo.aMatchinfo = pCsr->aMatchinfo;
	@@ -116477,10 +118167,11 @@
116477	i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res
116478	);
116479	}
116480
116481	snippet_out:

116482	if( rc!=SQLITE_OK ){
116483	sqlite3_result_error_code(pCtx, rc);
116484	sqlite3_free(res.z);
116485	}else{
116486	sqlite3_result_text(pCtx, res.z, -1, sqlite3_free);
	@@ -116656,10 +118347,11 @@
116656	}
116657
116658	offsets_out:
116659	sqlite3_free(sCtx.aTerm);
116660	assert( rc!=SQLITE_DONE );

116661	if( rc!=SQLITE_OK ){
116662	sqlite3_result_error_code(pCtx, rc);
116663	sqlite3_free(res.z);
116664	}else{
116665	sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free);
	@@ -116675,10 +118367,11 @@
116675	if( !pCsr->pExpr ){
116676	sqlite3_result_blob(pContext, "", 0, SQLITE_STATIC);
116677	return;
116678	}
116679	rc = fts3GetMatchinfo(pCsr);

116680	if( rc!=SQLITE_OK ){
116681	sqlite3_result_error_code(pContext, rc);
116682	}else{
116683	Fts3Table pTab = (Fts3Table)pCsr->base.pVtab;
116684	int n = sizeof(u32)(2+pCsr->aMatchinfo[0]pCsr->aMatchinfo[1]*3);
116685

	--- src/sqlite3.c
	+++ src/sqlite3.c
	@@ -1,12 +1,12 @@
1	/******************************************************************************
2	** This file is an amalgamation of many separate C source files from SQLite
3	** version 3.7.4. By combining all the individual C code files into this
4	** single large file, the entire code can be compiled as a one translation
5	** unit. This allows many compilers to do optimizations that would not be
6	** possible if the files were compiled separately. Performance improvements
7	** of 5% or more are commonly seen when SQLite is compiled as a single
8	** translation unit.
9	**
10	** This file is all you need to compile SQLite. To use SQLite in other
11	** programs, you need this file and the "sqlite3.h" header file that defines
12	** the programming interface to the SQLite library. (If you do not have
	@@ -648,13 +648,13 @@
648	**
649	** See also: [sqlite3_libversion()],
650	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
651	** [sqlite_version()] and [sqlite_source_id()].
652	*/
653	#define SQLITE_VERSION "3.7.4"
654	#define SQLITE_VERSION_NUMBER 3007004
655	#define SQLITE_SOURCE_ID "2010-11-15 16:29:31 136c2ac24ee1663bc0904bce1a619ecef3d11c1c"
656
657	/*
658	** CAPI3REF: Run-Time Library Version Numbers
659	** KEYWORDS: sqlite3_version, sqlite3_sourceid
660	**
	@@ -3273,11 +3273,14 @@
3273	** ^If the fourth parameter is negative, the length of the string is
3274	** the number of bytes up to the first zero terminator.
3275	**
3276	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
3277	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
3278	** string after SQLite has finished with it. ^The destructor is called
3279	** to dispose of the BLOB or string even if the call to sqlite3_bind_blob(),
3280	** sqlite3_bind_text(), or sqlite3_bind_text16() fails.
3281	** ^If the fifth argument is
3282	** the special value [SQLITE_STATIC], then SQLite assumes that the
3283	** information is in static, unmanaged space and does not need to be freed.
3284	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
3285	** SQLite makes its own private copy of the data immediately, before
3286	** the sqlite3_bind_*() routine returns.
	@@ -3913,16 +3916,19 @@
3916	** parameters. ^An aggregate SQL function requires an implementation of xStep
3917	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3918	** SQL function or aggregate, pass NULL poiners for all three function
3919	** callbacks.
3920	**
3921	** ^(If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3922	** then it is destructor for the application data pointer.
3923	** The destructor is invoked when the function is deleted, either by being
3924	** overloaded or when the database connection closes.)^
3925	** ^The destructor is also invoked if the call to
3926	** sqlite3_create_function_v2() fails.
3927	** ^When the destructor callback of the tenth parameter is invoked, it
3928	** is passed a single argument which is a copy of the application data
3929	** pointer which was the fifth parameter to sqlite3_create_function_v2().
3930	**
3931	** ^It is permitted to register multiple implementations of the same
3932	** functions with the same name but with either differing numbers of
3933	** arguments or differing preferred text encodings. ^SQLite will use
3934	** the implementation that most closely matches the way in which the
	@@ -4381,10 +4387,19 @@
4387	** with the addition that the xDestroy callback is invoked on pArg when
4388	** the collating function is deleted.
4389	** ^Collating functions are deleted when they are overridden by later
4390	** calls to the collation creation functions or when the
4391	** [database connection] is closed using [sqlite3_close()].
4392	**
4393	** ^The xDestroy callback is <u>not</u> called if the
4394	** sqlite3_create_collation_v2() function fails. Applications that invoke
4395	** sqlite3_create_collation_v2() with a non-NULL xDestroy argument should
4396	** check the return code and dispose of the application data pointer
4397	** themselves rather than expecting SQLite to deal with it for them.
4398	** This is different from every other SQLite interface. The inconsistency
4399	** is unfortunate but cannot be changed without breaking backwards
4400	** compatibility.
4401	**
4402	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
4403	*/
4404	SQLITE_API int sqlite3_create_collation(
4405	sqlite3*,
	@@ -5136,11 +5151,13 @@
5151	** when a new virtual table is be being created or reinitialized.
5152	**
5153	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
5154	** is a pointer to a destructor for the pClientData. ^SQLite will
5155	** invoke the destructor function (if it is not NULL) when SQLite
5156	** no longer needs the pClientData pointer. ^The destructor will also
5157	** be invoked if the call to sqlite3_create_module_v2() fails.
5158	** ^The sqlite3_create_module()
5159	** interface is equivalent to sqlite3_create_module_v2() with a NULL
5160	** destructor.
5161	*/
5162	SQLITE_API int sqlite3_create_module(
5163	sqlite3 db, / SQLite connection to register module with */
	@@ -5319,10 +5336,33 @@
5336	sqlite3_int64 iRow,
5337	int flags,
5338	sqlite3_blob **ppBlob
5339	);
5340
5341	/*
5342	** CAPI3REF: Move a BLOB Handle to a New Row
5343	**
5344	** ^This function is used to move an existing blob handle so that it points
5345	** to a different row of the same database table. ^The new row is identified
5346	** by the rowid value passed as the second argument. Only the row can be
5347	** changed. ^The database, table and column on which the blob handle is open
5348	** remain the same. Moving an existing blob handle to a new row can be
5349	** faster than closing the existing handle and opening a new one.
5350	**
5351	** ^(The new row must meet the same criteria as for [sqlite3_blob_open()] -
5352	** it must exist and there must be either a blob or text value stored in
5353	** the nominated column.)^ ^If the new row is not present in the table, or if
5354	** it does not contain a blob or text value, or if another error occurs, an
5355	** SQLite error code is returned and the blob handle is considered aborted.
5356	** ^All subsequent calls to [sqlite3_blob_read()], [sqlite3_blob_write()] or
5357	** [sqlite3_blob_reopen()] on an aborted blob handle immediately return
5358	** SQLITE_ABORT.
5359	**
5360	** ^This function sets the database handle error code and message.
5361	*/
5362	SQLITE_API SQLITE_EXPERIMENTAL int sqlite3_blob_reopen(sqlite3_blob *, sqlite3_int64);
5363
5364	/*
5365	** CAPI3REF: Close A BLOB Handle
5366	**
5367	** ^Closes an open [BLOB handle].
5368	**
	@@ -9890,10 +9930,13 @@
9930	Expr pOn; / The ON clause of a join */
9931	IdList pUsing; / The USING clause of a join */
9932	Bitmask colUsed; /* Bit N (1<<N) set if column N of pTab is used */
9933	char zIndex; / Identifier from "INDEXED BY <zIndex>" clause */
9934	Index pIndex; / Index structure corresponding to zIndex, if any */
9935	#ifndef SQLITE_OMIT_EXPLAIN
9936	int iSelectId; /* If pSelect!=0, the id of the sub-select in EQP */
9937	#endif
9938	} a[1]; /* One entry for each identifier on the list */
9939	};
9940
9941	/*
9942	** Permitted values of the SrcList.a.jointype field
	@@ -9922,10 +9965,11 @@
9965	** case that more than one of these conditions is true.
9966	*/
9967	struct WherePlan {
9968	u32 wsFlags; /* WHERE_* flags that describe the strategy */
9969	u32 nEq; /* Number of == constraints */
9970	double nRow; /* Estimated number of rows (for EQP) */
9971	union {
9972	Index pIdx; / Index when WHERE_INDEXED is true */
9973	struct WhereTerm pTerm; / WHERE clause term for OR-search */
9974	sqlite3_index_info pVtabIdx; / Virtual table index to use */
9975	} u;
	@@ -10276,10 +10320,15 @@
10320	Table *apVtabLock; / Pointer to virtual tables needing locking */
10321	#endif
10322	int nHeight; /* Expression tree height of current sub-select */
10323	Table pZombieTab; / List of Table objects to delete after code gen */
10324	TriggerPrg pTriggerPrg; / Linked list of coded triggers */
10325
10326	#ifndef SQLITE_OMIT_EXPLAIN
10327	int iSelectId;
10328	int iNextSelectId;
10329	#endif
10330	};
10331
10332	#ifdef SQLITE_OMIT_VIRTUALTABLE
10333	#define IN_DECLARE_VTAB 0
10334	#else
	@@ -27311,13 +27360,28 @@
27360	if( flags & (SQLITE_OPEN_WAL\|SQLITE_OPEN_MAIN_JOURNAL) ){
27361	char zDb[MAX_PATHNAME+1]; /* Database file path */
27362	int nDb; /* Number of valid bytes in zDb */
27363	struct stat sStat; /* Output of stat() on database file */
27364
27365	/* zPath is a path to a WAL or journal file. The following block derives
27366	** the path to the associated database file from zPath. This block handles
27367	** the following naming conventions:
27368	**
27369	** "<path to db>-journal"
27370	** "<path to db>-wal"
27371	** "<path to db>-journal-NNNN"
27372	** "<path to db>-wal-NNNN"
27373	**
27374	** where NNNN is a 4 digit decimal number. The NNNN naming schemes are
27375	** used by the test_multiplex.c module.
27376	*/
27377	nDb = sqlite3Strlen30(zPath) - 1;
27378	while( nDb>0 && zPath[nDb]!='l' ) nDb--;
27379	nDb -= ((flags & SQLITE_OPEN_WAL) ? 3 : 7);
27380	memcpy(zDb, zPath, nDb);
27381	zDb[nDb] = '\0';
27382
27383	if( 0==stat(zDb, &sStat) ){
27384	*pMode = sStat.st_mode & 0777;
27385	}else{
27386	rc = SQLITE_IOERR_FSTAT;
27387	}
	@@ -34519,10 +34583,11 @@
34583	# define sqlite3WalSavepointUndo(y,z) 0
34584	# define sqlite3WalFrames(u,v,w,x,y,z) 0
34585	# define sqlite3WalCheckpoint(u,v,w,x) 0
34586	# define sqlite3WalCallback(z) 0
34587	# define sqlite3WalExclusiveMode(y,z) 0
34588	# define sqlite3WalHeapMemory(z) 0
34589	#else
34590
34591	#define WAL_SAVEPOINT_NDATA 4
34592
34593	/* Connection to a write-ahead log (WAL) file.
	@@ -34529,11 +34594,11 @@
34594	** There is one object of this type for each pager.
34595	*/
34596	typedef struct Wal Wal;
34597
34598	/* Open and close a connection to a write-ahead log. */
34599	SQLITE_PRIVATE int sqlite3WalOpen(sqlite3_vfs, sqlite3_file, const char zName, int, Wal*);
34600	SQLITE_PRIVATE int sqlite3WalClose(Wal pWal, int sync_flags, int, u8 );
34601
34602	/* Used by readers to open (lock) and close (unlock) a snapshot. A
34603	** snapshot is like a read-transaction. It is the state of the database
34604	** at an instant in time. sqlite3WalOpenSnapshot gets a read lock and
	@@ -34585,10 +34650,16 @@
34650
34651	/* Tell the wal layer that an EXCLUSIVE lock has been obtained (or released)
34652	** by the pager layer on the database file.
34653	*/
34654	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op);
34655
34656	/* Return true if the argument is non-NULL and the WAL module is using
34657	** heap-memory for the wal-index. Otherwise, if the argument is NULL or the
34658	** WAL module is using shared-memory, return false.
34659	*/
34660	SQLITE_PRIVATE int sqlite3WalHeapMemory(Wal *pWal);
34661
34662	#endif /* ifndef SQLITE_OMIT_WAL */
34663	#endif /* _WAL_H_ */
34664
34665	/************ End of wal.h ***********************************************/
	@@ -35496,11 +35567,13 @@
35567	break;
35568	}
35569
35570	return 1;
35571	}
35572	#endif /* ifndef NDEBUG */
35573
35574	#ifdef SQLITE_DEBUG
35575	/*
35576	** Return a pointer to a human readable string in a static buffer
35577	** containing the state of the Pager object passed as an argument. This
35578	** is intended to be used within debuggers. For example, as an alternative
35579	** to "print *pPager" in gdb:
	@@ -35620,11 +35693,11 @@
35693	** UNKNOWN_LOCK for an explanation of this.
35694	*/
35695	static int pagerUnlockDb(Pager *pPager, int eLock){
35696	int rc = SQLITE_OK;
35697
35698	assert( !pPager->exclusiveMode \|\| pPager->eLock==eLock );
35699	assert( eLock==NO_LOCK \|\| eLock==SHARED_LOCK );
35700	assert( eLock!=NO_LOCK \|\| pagerUseWal(pPager)==0 );
35701	if( isOpen(pPager->fd) ){
35702	assert( pPager->eLock>=eLock );
35703	rc = sqlite3OsUnlock(pPager->fd, eLock);
	@@ -39094,11 +39167,11 @@
39167	if( rc==SQLITE_OK ){
39168	if( nPage==0 ){
39169	sqlite3BeginBenignMalloc();
39170	if( pagerLockDb(pPager, RESERVED_LOCK)==SQLITE_OK ){
39171	sqlite3OsDelete(pVfs, pPager->zJournal, 0);
39172	if( !pPager->exclusiveMode ) pagerUnlockDb(pPager, SHARED_LOCK);
39173	}
39174	sqlite3EndBenignMalloc();
39175	}else{
39176	/* The journal file exists and no other connection has a reserved
39177	** or greater lock on the database file. Now check that there is
	@@ -40920,11 +40993,12 @@
40993	assert( eMode==PAGER_LOCKINGMODE_QUERY
40994	\|\| eMode==PAGER_LOCKINGMODE_NORMAL
40995	\|\| eMode==PAGER_LOCKINGMODE_EXCLUSIVE );
40996	assert( PAGER_LOCKINGMODE_QUERY<0 );
40997	assert( PAGER_LOCKINGMODE_NORMAL>=0 && PAGER_LOCKINGMODE_EXCLUSIVE>=0 );
40998	assert( pPager->exclusiveMode \|\| 0==sqlite3WalHeapMemory(pPager->pWal) );
40999	if( eMode>=0 && !pPager->tempFile && !sqlite3WalHeapMemory(pPager->pWal) ){
41000	pPager->exclusiveMode = (u8)eMode;
41001	}
41002	return (int)pPager->exclusiveMode;
41003	}
41004
	@@ -41107,12 +41181,64 @@
41181	** Return true if the underlying VFS for the given pager supports the
41182	** primitives necessary for write-ahead logging.
41183	*/
41184	SQLITE_PRIVATE int sqlite3PagerWalSupported(Pager *pPager){
41185	const sqlite3_io_methods *pMethods = pPager->fd->pMethods;
41186	return pPager->exclusiveMode \|\| (pMethods->iVersion>=2 && pMethods->xShmMap);
41187	}
41188
41189	/*
41190	** Attempt to take an exclusive lock on the database file. If a PENDING lock
41191	** is obtained instead, immediately release it.
41192	*/
41193	static int pagerExclusiveLock(Pager *pPager){
41194	int rc; /* Return code */
41195
41196	assert( pPager->eLock==SHARED_LOCK \|\| pPager->eLock==EXCLUSIVE_LOCK );
41197	rc = pagerLockDb(pPager, EXCLUSIVE_LOCK);
41198	if( rc!=SQLITE_OK ){
41199	/* If the attempt to grab the pending lock failed, release the
41200	** exclusive lock that may have been obtained instead. */
41201	pagerUnlockDb(pPager, SHARED_LOCK);
41202	}
41203
41204	return rc;
41205	}
41206
41207	/*
41208	** Call sqlite3WalOpen() to open the WAL handle. If the pager is in
41209	** exclusive-locking mode when this function is called, take an EXCLUSIVE
41210	** lock on the database file and use heap-memory to store the wal-index
41211	** in. Otherwise, use the normal shared-memory.
41212	*/
41213	static int pagerOpenWal(Pager *pPager){
41214	int rc = SQLITE_OK;
41215
41216	assert( pPager->pWal==0 && pPager->tempFile==0 );
41217	assert( pPager->eLock==SHARED_LOCK \|\| pPager->eLock==EXCLUSIVE_LOCK \|\| pPager->noReadlock);
41218
41219	/* If the pager is already in exclusive-mode, the WAL module will use
41220	** heap-memory for the wal-index instead of the VFS shared-memory
41221	** implementation. Take the exclusive lock now, before opening the WAL
41222	** file, to make sure this is safe.
41223	*/
41224	if( pPager->exclusiveMode ){
41225	rc = pagerExclusiveLock(pPager);
41226	}
41227
41228	/* Open the connection to the log file. If this operation fails,
41229	** (e.g. due to malloc() failure), return an error code.
41230	*/
41231	if( rc==SQLITE_OK ){
41232	rc = sqlite3WalOpen(pPager->pVfs,
41233	pPager->fd, pPager->zWal, pPager->exclusiveMode, &pPager->pWal
41234	);
41235	}
41236
41237	return rc;
41238	}
41239
41240
41241	/*
41242	** The caller must be holding a SHARED lock on the database file to call
41243	** this function.
41244	**
	@@ -41143,15 +41269,11 @@
41269	if( !sqlite3PagerWalSupported(pPager) ) return SQLITE_CANTOPEN;
41270
41271	/* Close any rollback journal previously open */
41272	sqlite3OsClose(pPager->jfd);
41273
41274	rc = pagerOpenWal(pPager);




41275	if( rc==SQLITE_OK ){
41276	pPager->journalMode = PAGER_JOURNALMODE_WAL;
41277	pPager->eState = PAGER_OPEN;
41278	}
41279	}else{
	@@ -41186,30 +41308,25 @@
41308	rc = sqlite3OsAccess(
41309	pPager->pVfs, pPager->zWal, SQLITE_ACCESS_EXISTS, &logexists
41310	);
41311	}
41312	if( rc==SQLITE_OK && logexists ){
41313	rc = pagerOpenWal(pPager);

41314	}
41315	}
41316
41317	/* Checkpoint and close the log. Because an EXCLUSIVE lock is held on
41318	** the database file, the log and log-summary files will be deleted.
41319	*/
41320	if( rc==SQLITE_OK && pPager->pWal ){
41321	rc = pagerExclusiveLock(pPager);
41322	if( rc==SQLITE_OK ){
41323	rc = sqlite3WalClose(pPager->pWal,
41324	(pPager->noSync ? 0 : pPager->sync_flags),
41325	pPager->pageSize, (u8*)pPager->pTmpSpace
41326	);
41327	pPager->pWal = 0;




41328	}
41329	}
41330	return rc;
41331	}
41332
	@@ -41661,10 +41778,17 @@
41778	#ifdef SQLITE_DEBUG
41779	u8 lockError; /* True if a locking error has occurred */
41780	#endif
41781	};
41782
41783	/*
41784	** Candidate values for Wal.exclusiveMode.
41785	*/
41786	#define WAL_NORMAL_MODE 0
41787	#define WAL_EXCLUSIVE_MODE 1
41788	#define WAL_HEAPMEMORY_MODE 2
41789
41790	/*
41791	** Each page of the wal-index mapping contains a hash-table made up of
41792	** an array of HASHTABLE_NSLOT elements of the following type.
41793	*/
41794	typedef u16 ht_slot;
	@@ -41747,13 +41871,18 @@
41871	pWal->nWiData = iPage+1;
41872	}
41873
41874	/* Request a pointer to the required page from the VFS */
41875	if( pWal->apWiData[iPage]==0 ){
41876	if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){
41877	pWal->apWiData[iPage] = (u32 volatile *)sqlite3MallocZero(WALINDEX_PGSZ);
41878	if( !pWal->apWiData[iPage] ) rc = SQLITE_NOMEM;
41879	}else{
41880	rc = sqlite3OsShmMap(pWal->pDbFd, iPage, WALINDEX_PGSZ,
41881	pWal->writeLock, (void volatile **)&pWal->apWiData[iPage]
41882	);
41883	}
41884	}
41885
41886	*ppPage = pWal->apWiData[iPage];
41887	assert( iPage==0 \|\| *ppPage \|\| rc!=SQLITE_OK );
41888	return rc;
	@@ -41831,10 +41960,16 @@
41960	}
41961
41962	aOut[0] = s1;
41963	aOut[1] = s2;
41964	}
41965
41966	static void walShmBarrier(Wal *pWal){
41967	if( pWal->exclusiveMode!=WAL_HEAPMEMORY_MODE ){
41968	sqlite3OsShmBarrier(pWal->pDbFd);
41969	}
41970	}
41971
41972	/*
41973	** Write the header information in pWal->hdr into the wal-index.
41974	**
41975	** The checksum on pWal->hdr is updated before it is written.
	@@ -41846,11 +41981,11 @@
41981	assert( pWal->writeLock );
41982	pWal->hdr.isInit = 1;
41983	pWal->hdr.iVersion = WALINDEX_MAX_VERSION;
41984	walChecksumBytes(1, (u8*)&pWal->hdr, nCksum, 0, pWal->hdr.aCksum);
41985	memcpy((void )&aHdr[1], (void )&pWal->hdr, sizeof(WalIndexHdr));
41986	walShmBarrier(pWal);
41987	memcpy((void )&aHdr[0], (void )&pWal->hdr, sizeof(WalIndexHdr));
41988	}
41989
41990	/*
41991	** This function encodes a single frame header and writes it to a buffer
	@@ -42418,11 +42553,19 @@
42553
42554	/*
42555	** Close an open wal-index.
42556	*/
42557	static void walIndexClose(Wal *pWal, int isDelete){
42558	if( pWal->exclusiveMode==WAL_HEAPMEMORY_MODE ){
42559	int i;
42560	for(i=0; i<pWal->nWiData; i++){
42561	sqlite3_free((void *)pWal->apWiData[i]);
42562	pWal->apWiData[i] = 0;
42563	}
42564	}else{
42565	sqlite3OsShmUnmap(pWal->pDbFd, isDelete);
42566	}
42567	}
42568
42569	/*
42570	** Open a connection to the WAL file zWalName. The database file must
42571	** already be opened on connection pDbFd. The buffer that zWalName points
	@@ -42440,10 +42583,11 @@
42583	*/
42584	SQLITE_PRIVATE int sqlite3WalOpen(
42585	sqlite3_vfs pVfs, / vfs module to open wal and wal-index */
42586	sqlite3_file pDbFd, / The open database file */
42587	const char zWalName, / Name of the WAL file */
42588	int bNoShm, /* True to run in heap-memory mode */
42589	Wal *ppWal / OUT: Allocated Wal handle */
42590	){
42591	int rc; /* Return Code */
42592	Wal pRet; / Object to allocate and return */
42593	int flags; /* Flags passed to OsOpen() */
	@@ -42473,10 +42617,11 @@
42617	pRet->pVfs = pVfs;
42618	pRet->pWalFd = (sqlite3_file *)&pRet[1];
42619	pRet->pDbFd = pDbFd;
42620	pRet->readLock = -1;
42621	pRet->zWalName = zWalName;
42622	pRet->exclusiveMode = (bNoShm ? WAL_HEAPMEMORY_MODE: WAL_NORMAL_MODE);
42623
42624	/* Open file handle on the write-ahead log file. */
42625	flags = (SQLITE_OPEN_READWRITE\|SQLITE_OPEN_CREATE\|SQLITE_OPEN_WAL);
42626	rc = sqlite3OsOpen(pVfs, zWalName, pRet->pWalFd, flags, &flags);
42627	if( rc==SQLITE_OK && flags&SQLITE_OPEN_READONLY ){
	@@ -42906,11 +43051,13 @@
43051	**
43052	** The EXCLUSIVE lock is not released before returning.
43053	*/
43054	rc = sqlite3OsLock(pWal->pDbFd, SQLITE_LOCK_EXCLUSIVE);
43055	if( rc==SQLITE_OK ){
43056	if( pWal->exclusiveMode==WAL_NORMAL_MODE ){
43057	pWal->exclusiveMode = WAL_EXCLUSIVE_MODE;
43058	}
43059	rc = sqlite3WalCheckpoint(pWal, sync_flags, nBuf, zBuf);
43060	if( rc==SQLITE_OK ){
43061	isDelete = 1;
43062	}
43063	}
	@@ -42962,11 +43109,11 @@
43109	** Memory barriers are used to prevent the compiler or the hardware from
43110	** reordering the reads and writes.
43111	*/
43112	aHdr = walIndexHdr(pWal);
43113	memcpy(&h1, (void *)&aHdr[0], sizeof(h1));
43114	walShmBarrier(pWal);
43115	memcpy(&h2, (void *)&aHdr[1], sizeof(h2));
43116
43117	if( memcmp(&h1, &h2, sizeof(h1))!=0 ){
43118	return 1; /* Dirty read */
43119	}
	@@ -43163,11 +43310,11 @@
43310	if( !useWal && pInfo->nBackfill==pWal->hdr.mxFrame ){
43311	/* The WAL has been completely backfilled (or it is empty).
43312	** and can be safely ignored.
43313	*/
43314	rc = walLockShared(pWal, WAL_READ_LOCK(0));
43315	walShmBarrier(pWal);
43316	if( rc==SQLITE_OK ){
43317	if( memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr)) ){
43318	/* It is not safe to allow the reader to continue here if frames
43319	** may have been appended to the log before READ_LOCK(0) was obtained.
43320	** When holding READ_LOCK(0), the reader ignores the entire log file,
	@@ -43257,11 +43404,11 @@
43404	** date before proceeding. That would not be possible without somehow
43405	** blocking writers. It only guarantees that a dangerous checkpoint or
43406	** log-wrap (either of which would require an exclusive lock on
43407	** WAL_READ_LOCK(mxI)) has not occurred since the snapshot was valid.
43408	*/
43409	walShmBarrier(pWal);
43410	if( pInfo->aReadMark[mxI]!=mxReadMark
43411	\|\| memcmp((void *)walIndexHdr(pWal), &pWal->hdr, sizeof(WalIndexHdr))
43412	){
43413	walUnlockShared(pWal, WAL_READ_LOCK(mxI));
43414	return WAL_RETRY;
	@@ -43900,17 +44047,18 @@
44047	** WAL is already in exclusive-locking mode - meaning that this
44048	** routine is a no-op. The pager must already hold the exclusive lock
44049	** on the main database file before invoking this operation.
44050	**
44051	** If op is negative, then do a dry-run of the op==1 case but do
44052	** not actually change anything. The pager uses this to see if it
44053	** should acquire the database exclusive lock prior to invoking
44054	** the op==1 case.
44055	*/
44056	SQLITE_PRIVATE int sqlite3WalExclusiveMode(Wal *pWal, int op){
44057	int rc;
44058	assert( pWal->writeLock==0 );
44059	assert( pWal->exclusiveMode!=WAL_HEAPMEMORY_MODE \|\| op==-1 );
44060
44061	/* pWal->readLock is usually set, but might be -1 if there was a
44062	** prior error while attempting to acquire are read-lock. This cannot
44063	** happen if the connection is actually in exclusive mode (as no xShmLock
44064	** locks are taken in this case). Nor should the pager attempt to
	@@ -43939,10 +44087,19 @@
44087	}else{
44088	rc = pWal->exclusiveMode==0;
44089	}
44090	return rc;
44091	}
44092
44093	/*
44094	** Return true if the argument is non-NULL and the WAL module is using
44095	** heap-memory for the wal-index. Otherwise, if the argument is NULL or the
44096	** WAL module is using shared-memory, return false.
44097	*/
44098	SQLITE_PRIVATE int sqlite3WalHeapMemory(Wal *pWal){
44099	return (pWal && pWal->exclusiveMode==WAL_HEAPMEMORY_MODE );
44100	}
44101
44102	#endif /* #ifndef SQLITE_OMIT_WAL */
44103
44104	/************ End of wal.c ***********************************************/
44105	/************ Begin file btmutex.c ***************************************/
	@@ -48119,20 +48276,21 @@
48276	**
48277	** This will release the write lock on the database file. If there
48278	** are no active cursors, it also releases the read lock.
48279	*/
48280	SQLITE_PRIVATE int sqlite3BtreeCommitPhaseTwo(Btree *p){

48281
48282	if( p->inTrans==TRANS_NONE ) return SQLITE_OK;
48283	sqlite3BtreeEnter(p);
48284	btreeIntegrity(p);
48285
48286	/* If the handle has a write-transaction open, commit the shared-btrees
48287	** transaction and set the shared state to TRANS_READ.
48288	*/
48289	if( p->inTrans==TRANS_WRITE ){
48290	int rc;
48291	BtShared *pBt = p->pBt;
48292	assert( pBt->inTransaction==TRANS_WRITE );
48293	assert( pBt->nTransaction>0 );
48294	rc = sqlite3PagerCommitPhaseTwo(pBt->pPager);
48295	if( rc!=SQLITE_OK ){
48296	sqlite3BtreeLeave(p);
	@@ -53047,12 +53205,11 @@
53205	** sqlite3BtreePutData()).
53206	*/
53207	SQLITE_PRIVATE void sqlite3BtreeCacheOverflow(BtCursor *pCur){
53208	assert( cursorHoldsMutex(pCur) );
53209	assert( sqlite3_mutex_held(pCur->pBtree->db->mutex) );
53210	invalidateOverflowCache(pCur);

53211	pCur->isIncrblobHandle = 1;
53212	}
53213	#endif
53214
53215	/*
	@@ -56081,16 +56238,14 @@
56238	pMem->flags = MEM_Int;
56239	pMem->u.i = pOp->p2; /* P2 */
56240	pMem->type = SQLITE_INTEGER;
56241	pMem++;
56242
56243	pMem->flags = MEM_Int;
56244	pMem->u.i = pOp->p3; /* P3 */
56245	pMem->type = SQLITE_INTEGER;
56246	pMem++;


56247
56248	if( sqlite3VdbeMemGrow(pMem, 32, 0) ){ /* P4 */
56249	assert( p->db->mallocFailed );
56250	return SQLITE_ERROR;
56251	}
	@@ -56131,11 +56286,11 @@
56286	pMem->flags = MEM_Null; /* Comment */
56287	pMem->type = SQLITE_NULL;
56288	}
56289	}
56290
56291	p->nResColumn = 8 - 4*(p->explain-1);
56292	p->rc = SQLITE_OK;
56293	rc = SQLITE_ROW;
56294	}
56295	return rc;
56296	}
	@@ -59078,10 +59233,12 @@
59233	}
59234	sqlite3Error(p->db, rc, 0);
59235	rc = sqlite3ApiExit(p->db, rc);
59236	}
59237	sqlite3_mutex_leave(p->db->mutex);
59238	}else if( xDel!=SQLITE_STATIC && xDel!=SQLITE_TRANSIENT ){
59239	xDel((void*)zData);
59240	}
59241	return rc;
59242	}
59243
59244
	@@ -66071,15 +66228,86 @@
66228	typedef struct Incrblob Incrblob;
66229	struct Incrblob {
66230	int flags; /* Copy of "flags" passed to sqlite3_blob_open() */
66231	int nByte; /* Size of open blob, in bytes */
66232	int iOffset; /* Byte offset of blob in cursor data */
66233	int iCol; /* Table column this handle is open on */
66234	BtCursor pCsr; / Cursor pointing at blob row */
66235	sqlite3_stmt pStmt; / Statement holding cursor open */
66236	sqlite3 db; / The associated database */
66237	};
66238
66239
66240	/*
66241	** This function is used by both blob_open() and blob_reopen(). It seeks
66242	** the b-tree cursor associated with blob handle p to point to row iRow.
66243	** If successful, SQLITE_OK is returned and subsequent calls to
66244	** sqlite3_blob_read() or sqlite3_blob_write() access the specified row.
66245	**
66246	** If an error occurs, or if the specified row does not exist or does not
66247	** contain a value of type TEXT or BLOB in the column nominated when the
66248	** blob handle was opened, then an error code is returned and *pzErr may
66249	** be set to point to a buffer containing an error message. It is the
66250	** responsibility of the caller to free the error message buffer using
66251	** sqlite3DbFree().
66252	**
66253	** If an error does occur, then the b-tree cursor is closed. All subsequent
66254	** calls to sqlite3_blob_read(), blob_write() or blob_reopen() will
66255	** immediately return SQLITE_ABORT.
66256	*/
66257	static int blobSeekToRow(Incrblob p, sqlite3_int64 iRow, char *pzErr){
66258	int rc; /* Error code */
66259	char zErr = 0; / Error message */
66260	Vdbe v = (Vdbe )p->pStmt;
66261
66262	/* Set the value of the SQL statements only variable to integer iRow.
66263	** This is done directly instead of using sqlite3_bind_int64() to avoid
66264	** triggering asserts related to mutexes.
66265	*/
66266	assert( v->aVar[0].flags&MEM_Int );
66267	v->aVar[0].u.i = iRow;
66268
66269	rc = sqlite3_step(p->pStmt);
66270	if( rc==SQLITE_ROW ){
66271	u32 type = v->apCsr[0]->aType[p->iCol];
66272	if( type<12 ){
66273	zErr = sqlite3MPrintf(p->db, "cannot open value of type %s",
66274	type==0?"null": type==7?"real": "integer"
66275	);
66276	rc = SQLITE_ERROR;
66277	sqlite3_finalize(p->pStmt);
66278	p->pStmt = 0;
66279	}else{
66280	p->iOffset = v->apCsr[0]->aOffset[p->iCol];
66281	p->nByte = sqlite3VdbeSerialTypeLen(type);
66282	p->pCsr = v->apCsr[0]->pCursor;
66283	sqlite3BtreeEnterCursor(p->pCsr);
66284	sqlite3BtreeCacheOverflow(p->pCsr);
66285	sqlite3BtreeLeaveCursor(p->pCsr);
66286	}
66287	}
66288
66289	if( rc==SQLITE_ROW ){
66290	rc = SQLITE_OK;
66291	}else if( p->pStmt ){
66292	rc = sqlite3_finalize(p->pStmt);
66293	p->pStmt = 0;
66294	if( rc==SQLITE_OK ){
66295	zErr = sqlite3MPrintf(p->db, "no such rowid: %lld", iRow);
66296	rc = SQLITE_ERROR;
66297	}else{
66298	zErr = sqlite3MPrintf(p->db, "%s", sqlite3_errmsg(p->db));
66299	}
66300	}
66301
66302	assert( rc!=SQLITE_OK \|\| zErr==0 );
66303	assert( rc!=SQLITE_ROW && rc!=SQLITE_DONE );
66304
66305	*pzErr = zErr;
66306	return rc;
66307	}
66308
66309	/*
66310	** Open a blob handle.
66311	*/
66312	SQLITE_API int sqlite3_blob_open(
66313	sqlite3* db, /* The database connection */
	@@ -66116,33 +66344,39 @@
66344	/* One of the following two instructions is replaced by an OP_Noop. */
66345	{OP_OpenRead, 0, 0, 0}, /* 3: Open cursor 0 for reading */
66346	{OP_OpenWrite, 0, 0, 0}, /* 4: Open cursor 0 for read/write */
66347
66348	{OP_Variable, 1, 1, 1}, /* 5: Push the rowid to the stack */
66349	{OP_NotExists, 0, 10, 1}, /* 6: Seek the cursor */
66350	{OP_Column, 0, 0, 1}, /* 7 */
66351	{OP_ResultRow, 1, 0, 0}, /* 8 */
66352	{OP_Goto, 0, 5, 0}, /* 9 */
66353	{OP_Close, 0, 0, 0}, /* 10 */
66354	{OP_Halt, 0, 0, 0}, /* 11 */
66355	};
66356

66357	int rc = SQLITE_OK;
66358	char *zErr = 0;
66359	Table *pTab;
66360	Parse *pParse = 0;
66361	Incrblob *pBlob = 0;
66362
66363	flags = !!flags; /* flags = (flags ? 1 : 0); */
66364	*ppBlob = 0;
66365
66366	sqlite3_mutex_enter(db->mutex);
66367
66368	pBlob = (Incrblob *)sqlite3DbMallocZero(db, sizeof(Incrblob));
66369	if( !pBlob ) goto blob_open_out;
66370	pParse = sqlite3StackAllocRaw(db, sizeof(*pParse));
66371	if( !pParse ) goto blob_open_out;
66372


66373	do {
66374	memset(pParse, 0, sizeof(Parse));
66375	pParse->db = db;
66376	sqlite3DbFree(db, zErr);
66377	zErr = 0;
66378
66379	sqlite3BtreeEnterAll(db);
66380	pTab = sqlite3LocateTable(pParse, 0, zTable, zDb);
66381	if( pTab && IsVirtual(pTab) ){
66382	pTab = 0;
	@@ -66164,11 +66398,11 @@
66398	sqlite3BtreeLeaveAll(db);
66399	goto blob_open_out;
66400	}
66401
66402	/* Now search pTab for the exact column. */
66403	for(iCol=0; iCol<pTab->nCol; iCol++) {
66404	if( sqlite3StrICmp(pTab->aCol[iCol].zName, zColumn)==0 ){
66405	break;
66406	}
66407	}
66408	if( iCol==pTab->nCol ){
	@@ -66218,15 +66452,18 @@
66452	sqlite3BtreeLeaveAll(db);
66453	goto blob_open_out;
66454	}
66455	}
66456
66457	pBlob->pStmt = (sqlite3_stmt *)sqlite3VdbeCreate(db);
66458	assert( pBlob->pStmt \|\| db->mallocFailed );
66459	if( pBlob->pStmt ){
66460	Vdbe v = (Vdbe )pBlob->pStmt;
66461	int iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
66462
66463	sqlite3VdbeAddOpList(v, sizeof(openBlob)/sizeof(VdbeOpList), openBlob);
66464
66465
66466	/* Configure the OP_Transaction */
66467	sqlite3VdbeChangeP1(v, 0, iDb);
66468	sqlite3VdbeChangeP2(v, 0, flags);
66469
	@@ -66265,69 +66502,29 @@
66502	if( !db->mallocFailed ){
66503	sqlite3VdbeMakeReady(v, 1, 1, 1, 0, 0, 0);
66504	}
66505	}
66506
66507	pBlob->flags = flags;
66508	pBlob->iCol = iCol;
66509	pBlob->db = db;
66510	sqlite3BtreeLeaveAll(db);
66511	if( db->mallocFailed ){
66512	goto blob_open_out;
66513	}
66514	sqlite3_bind_int64(pBlob->pStmt, 1, iRow);
66515	rc = blobSeekToRow(pBlob, iRow, &zErr);
66516	} while( (++nAttempt)<5 && rc==SQLITE_SCHEMA );














































66517
66518	blob_open_out:
66519	if( rc==SQLITE_OK && db->mallocFailed==0 ){
66520	ppBlob = (sqlite3_blob )pBlob;
66521	}else{
66522	if( pBlob && pBlob->pStmt ) sqlite3VdbeFinalize((Vdbe *)pBlob->pStmt);
66523	sqlite3DbFree(db, pBlob);
66524	}
66525	sqlite3Error(db, rc, (zErr ? "%s" : 0), zErr);
66526	sqlite3DbFree(db, zErr);
66527	sqlite3StackFree(db, pParse);
66528	rc = sqlite3ApiExit(db, rc);
66529	sqlite3_mutex_leave(db->mutex);
66530	return rc;
	@@ -66376,11 +66573,11 @@
66573
66574	if( n<0 \|\| iOffset<0 \|\| (iOffset+n)>p->nByte ){
66575	/* Request is out of range. Return a transient error. */
66576	rc = SQLITE_ERROR;
66577	sqlite3Error(db, SQLITE_ERROR, 0);
66578	}else if( v==0 ){
66579	/* If there is no statement handle, then the blob-handle has
66580	** already been invalidated. Return SQLITE_ABORT in this case.
66581	*/
66582	rc = SQLITE_ABORT;
66583	}else{
	@@ -66426,10 +66623,49 @@
66623	*/
66624	SQLITE_API int sqlite3_blob_bytes(sqlite3_blob *pBlob){
66625	Incrblob p = (Incrblob )pBlob;
66626	return p ? p->nByte : 0;
66627	}
66628
66629	/*
66630	** Move an existing blob handle to point to a different row of the same
66631	** database table.
66632	**
66633	** If an error occurs, or if the specified row does not exist or does not
66634	** contain a blob or text value, then an error code is returned and the
66635	** database handle error code and message set. If this happens, then all
66636	** subsequent calls to sqlite3_blob_xxx() functions (except blob_close())
66637	** immediately return SQLITE_ABORT.
66638	*/
66639	SQLITE_API int sqlite3_blob_reopen(sqlite3_blob *pBlob, sqlite3_int64 iRow){
66640	int rc;
66641	Incrblob p = (Incrblob )pBlob;
66642	sqlite3 *db;
66643
66644	if( p==0 ) return SQLITE_MISUSE_BKPT;
66645	db = p->db;
66646	sqlite3_mutex_enter(db->mutex);
66647
66648	if( p->pStmt==0 ){
66649	/* If there is no statement handle, then the blob-handle has
66650	** already been invalidated. Return SQLITE_ABORT in this case.
66651	*/
66652	rc = SQLITE_ABORT;
66653	}else{
66654	char *zErr;
66655	rc = blobSeekToRow(p, iRow, &zErr);
66656	if( rc!=SQLITE_OK ){
66657	sqlite3Error(db, rc, (zErr ? "%s" : 0), zErr);
66658	sqlite3DbFree(db, zErr);
66659	}
66660	assert( rc!=SQLITE_SCHEMA );
66661	}
66662
66663	rc = sqlite3ApiExit(db, rc);
66664	sqlite3_mutex_leave(db->mutex);
66665	return rc;
66666	}
66667
66668	#endif /* #ifndef SQLITE_OMIT_INCRBLOB */
66669
66670	/************ End of vdbeblob.c ******************************************/
66671	/************ Begin file journal.c ***************************************/
	@@ -68757,10 +68993,13 @@
68993	Expr pRight, / Right operand */
68994	const Token pToken / Argument token */
68995	){
68996	Expr *p = sqlite3ExprAlloc(pParse->db, op, pToken, 1);
68997	sqlite3ExprAttachSubtrees(pParse->db, p, pLeft, pRight);
68998	if( p ) {
68999	sqlite3ExprCheckHeight(pParse, p->nHeight);
69000	}
69001	return p;
69002	}
69003
69004	/*
69005	** Join two expressions using an AND operator. If either expression is
	@@ -69869,10 +70108,20 @@
70108	int mem = ++pParse->nMem;
70109	sqlite3VdbeAddOp1(v, OP_If, mem);
70110	testAddr = sqlite3VdbeAddOp2(v, OP_Integer, 1, mem);
70111	assert( testAddr>0 \|\| pParse->db->mallocFailed );
70112	}
70113
70114	#ifndef SQLITE_OMIT_EXPLAIN
70115	if( pParse->explain==2 ){
70116	char *zMsg = sqlite3MPrintf(
70117	pParse->db, "EXECUTE %s%s SUBQUERY %d", testAddr?"":"CORRELATED ",
70118	pExpr->op==TK_IN?"LIST":"SCALAR", pParse->iNextSelectId
70119	);
70120	sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
70121	}
70122	#endif
70123
70124	switch( pExpr->op ){
70125	case TK_IN: {
70126	char affinity; /* Affinity of the LHS of the IN */
70127	KeyInfo keyInfo; /* Keyinfo for the generated table */
	@@ -84056,10 +84305,31 @@
84305	int (extended_result_codes)(sqlite3,int);
84306	int (limit)(sqlite3,int,int);
84307	sqlite3_stmt (next_stmt)(sqlite3,sqlite3_stmt);
84308	const char (sql)(sqlite3_stmt*);
84309	int (status)(int,int,int*,int);
84310	int (backup_finish)(sqlite3_backup);
84311	sqlite3_backup (backup_init)(sqlite3,const char,sqlite3,const char);
84312	int (backup_pagecount)(sqlite3_backup);
84313	int (backup_remaining)(sqlite3_backup);
84314	int (backup_step)(sqlite3_backup,int);
84315	const char (compileoption_get)(int);
84316	int (compileoption_used)(const char);
84317	int (create_function_v2)(sqlite3,const char,int,int,void,void (xFunc)(sqlite3_context,int,sqlite3_value*),void (xStep)(sqlite3_context,int,sqlite3_value),void (xFinal)(sqlite3_context),void(xDestroy)(void*));
84318	int (db_config)(sqlite3,int,...);
84319	sqlite3_mutex (db_mutex)(sqlite3*);
84320	int (db_status)(sqlite3,int,int,int,int);
84321	int (extended_errcode)(sqlite3);
84322	void (log)(int,const char,...);
84323	sqlite3_int64 (*soft_heap_limit64)(sqlite3_int64);
84324	const char (sourceid)(void);
84325	int (stmt_status)(sqlite3_stmt,int,int);
84326	int (strnicmp)(const char,const char*,int);
84327	int (unlock_notify)(sqlite3,void()(void,int),void);
84328	int (wal_autocheckpoint)(sqlite3,int);
84329	int (wal_checkpoint)(sqlite3,const char*);
84330	void (wal_hook)(sqlite3,int()(void,sqlite3,const char,int),void);
84331	};
84332
84333	/*
84334	** The following macros redefine the API routines so that they are
84335	** redirected throught the global sqlite3_api structure.
	@@ -84235,10 +84505,31 @@
84505	#define sqlite3_extended_result_codes sqlite3_api->extended_result_codes
84506	#define sqlite3_limit sqlite3_api->limit
84507	#define sqlite3_next_stmt sqlite3_api->next_stmt
84508	#define sqlite3_sql sqlite3_api->sql
84509	#define sqlite3_status sqlite3_api->status
84510	#define sqlite3_backup_finish sqlite3_api->backup_finish
84511	#define sqlite3_backup_init sqlite3_api->backup_init
84512	#define sqlite3_backup_pagecount sqlite3_api->backup_pagecount
84513	#define sqlite3_backup_remaining sqlite3_api->backup_remaining
84514	#define sqlite3_backup_step sqlite3_api->backup_step
84515	#define sqlite3_compileoption_get sqlite3_api->compileoption_get
84516	#define sqlite3_compileoption_used sqlite3_api->compileoption_used
84517	#define sqlite3_create_function_v2 sqlite3_api->create_function_v2
84518	#define sqlite3_db_config sqlite3_api->db_config
84519	#define sqlite3_db_mutex sqlite3_api->db_mutex
84520	#define sqlite3_db_status sqlite3_api->db_status
84521	#define sqlite3_extended_errcode sqlite3_api->extended_errcode
84522	#define sqlite3_log sqlite3_api->log
84523	#define sqlite3_soft_heap_limit64 sqlite3_api->soft_heap_limit64
84524	#define sqlite3_sourceid sqlite3_api->sourceid
84525	#define sqlite3_stmt_status sqlite3_api->stmt_status
84526	#define sqlite3_strnicmp sqlite3_api->strnicmp
84527	#define sqlite3_unlock_notify sqlite3_api->unlock_notify
84528	#define sqlite3_wal_autocheckpoint sqlite3_api->wal_autocheckpoint
84529	#define sqlite3_wal_checkpoint sqlite3_api->wal_checkpoint
84530	#define sqlite3_wal_hook sqlite3_api->wal_hook
84531	#endif /* SQLITE_CORE */
84532
84533	#define SQLITE_EXTENSION_INIT1 const sqlite3_api_routines *sqlite3_api = 0;
84534	#define SQLITE_EXTENSION_INIT2(v) sqlite3_api = v;
84535
	@@ -84552,10 +84843,50 @@
84843	sqlite3_extended_result_codes,
84844	sqlite3_limit,
84845	sqlite3_next_stmt,
84846	sqlite3_sql,
84847	sqlite3_status,
84848
84849	/*
84850	** Added for 3.7.4
84851	*/
84852	sqlite3_backup_finish,
84853	sqlite3_backup_init,
84854	sqlite3_backup_pagecount,
84855	sqlite3_backup_remaining,
84856	sqlite3_backup_step,
84857	#ifndef SQLITE_OMIT_COMPILEOPTION_DIAGS
84858	sqlite3_compileoption_get,
84859	sqlite3_compileoption_used,
84860	#else
84861	0,
84862	0,
84863	#endif
84864	sqlite3_create_function_v2,
84865	sqlite3_db_config,
84866	sqlite3_db_mutex,
84867	sqlite3_db_status,
84868	sqlite3_extended_errcode,
84869	sqlite3_log,
84870	sqlite3_soft_heap_limit64,
84871	sqlite3_sourceid,
84872	sqlite3_stmt_status,
84873	sqlite3_strnicmp,
84874	#ifdef SQLITE_ENABLE_UNLOCK_NOTIFY
84875	sqlite3_unlock_notify,
84876	#else
84877	0,
84878	#endif
84879	#ifndef SQLITE_OMIT_WAL
84880	sqlite3_wal_autocheckpoint,
84881	sqlite3_wal_checkpoint,
84882	sqlite3_wal_hook,
84883	#else
84884	0,
84885	0,
84886	0,
84887	#endif
84888	};
84889
84890	/*
84891	** Attempt to load an SQLite extension library contained in the file
84892	** zFile. The entry point is zProc. zProc may be 0 in which case a
	@@ -86976,17 +87307,17 @@
87307
87308	#ifndef SQLITE_OMIT_EXPLAIN
87309	if( rc==SQLITE_OK && pParse->pVdbe && pParse->explain ){
87310	static const char * const azColName[] = {
87311	"addr", "opcode", "p1", "p2", "p3", "p4", "p5", "comment",
87312	"selectid", "order", "from", "detail"
87313	};
87314	int iFirst, mx;
87315	if( pParse->explain==2 ){
87316	sqlite3VdbeSetNumCols(pParse->pVdbe, 4);
87317	iFirst = 8;
87318	mx = 12;
87319	}else{
87320	sqlite3VdbeSetNumCols(pParse->pVdbe, 8);
87321	iFirst = 0;
87322	mx = 8;
87323	}
	@@ -87981,10 +88312,92 @@
88312	}
88313	}
88314	return pInfo;
88315	}
88316
88317	#ifndef SQLITE_OMIT_COMPOUND_SELECT
88318	/*
88319	** Name of the connection operator, used for error messages.
88320	*/
88321	static const char *selectOpName(int id){
88322	char *z;
88323	switch( id ){
88324	case TK_ALL: z = "UNION ALL"; break;
88325	case TK_INTERSECT: z = "INTERSECT"; break;
88326	case TK_EXCEPT: z = "EXCEPT"; break;
88327	default: z = "UNION"; break;
88328	}
88329	return z;
88330	}
88331	#endif /* SQLITE_OMIT_COMPOUND_SELECT */
88332
88333	#ifndef SQLITE_OMIT_EXPLAIN
88334	/*
88335	** Unless an "EXPLAIN QUERY PLAN" command is being processed, this function
88336	** is a no-op. Otherwise, it adds a single row of output to the EQP result,
88337	** where the caption is of the form:
88338	**
88339	** "USE TEMP B-TREE FOR xxx"
88340	**
88341	** where xxx is one of "DISTINCT", "ORDER BY" or "GROUP BY". Exactly which
88342	** is determined by the zUsage argument.
88343	*/
88344	static void explainTempTable(Parse pParse, const char zUsage){
88345	if( pParse->explain==2 ){
88346	Vdbe *v = pParse->pVdbe;
88347	char *zMsg = sqlite3MPrintf(pParse->db, "USE TEMP B-TREE FOR %s", zUsage);
88348	sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
88349	}
88350	}
88351
88352	/*
88353	** Unless an "EXPLAIN QUERY PLAN" command is being processed, this function
88354	** is a no-op. Otherwise, it adds a single row of output to the EQP result,
88355	** where the caption is of one of the two forms:
88356	**
88357	** "COMPOSITE SUBQUERIES iSub1 and iSub2 (op)"
88358	** "COMPOSITE SUBQUERIES iSub1 and iSub2 USING TEMP B-TREE (op)"
88359	**
88360	** where iSub1 and iSub2 are the integers passed as the corresponding
88361	** function parameters, and op is the text representation of the parameter
88362	** of the same name. The parameter "op" must be one of TK_UNION, TK_EXCEPT,
88363	** TK_INTERSECT or TK_ALL. The first form is used if argument bUseTmp is
88364	** false, or the second form if it is true.
88365	*/
88366	static void explainComposite(
88367	Parse pParse, / Parse context */
88368	int op, /* One of TK_UNION, TK_EXCEPT etc. */
88369	int iSub1, /* Subquery id 1 */
88370	int iSub2, /* Subquery id 2 */
88371	int bUseTmp /* True if a temp table was used */
88372	){
88373	assert( op==TK_UNION \|\| op==TK_EXCEPT \|\| op==TK_INTERSECT \|\| op==TK_ALL );
88374	if( pParse->explain==2 ){
88375	Vdbe *v = pParse->pVdbe;
88376	char *zMsg = sqlite3MPrintf(
88377	pParse->db, "COMPOUND SUBQUERIES %d AND %d %s(%s)", iSub1, iSub2,
88378	bUseTmp?"USING TEMP B-TREE ":"", selectOpName(op)
88379	);
88380	sqlite3VdbeAddOp4(v, OP_Explain, pParse->iSelectId, 0, 0, zMsg, P4_DYNAMIC);
88381	}
88382	}
88383
88384	/*
88385	** Assign expression b to lvalue a. A second, no-op, version of this macro
88386	** is provided when SQLITE_OMIT_EXPLAIN is defined. This allows the code
88387	** in sqlite3Select() to assign values to structure member variables that
88388	** only exist if SQLITE_OMIT_EXPLAIN is not defined without polluting the
88389	** code with #ifndef directives.
88390	*/
88391	# define explainSetInteger(a, b) a = b
88392
88393	#else
88394	/* No-op versions of the explainXXX() functions and macros. */
88395	# define explainTempTable(y,z)
88396	# define explainComposite(v,w,x,y,z)
88397	# define explainSetInteger(y,z)
88398	#endif
88399
88400	/*
88401	** If the inner loop was generated using a non-null pOrderBy argument,
88402	** then the results were placed in a sorter. After the loop is terminated
88403	** we need to run the sorter and output the results. The following
	@@ -88328,26 +88741,10 @@
88741	}
88742	}
88743	generateColumnTypes(pParse, pTabList, pEList);
88744	}
88745
















88746	/*
88747	** Given a an expression list (which is really the list of expressions
88748	** that form the result set of a SELECT statement) compute appropriate
88749	** column names for a table that would hold the expression list.
88750	**
	@@ -88506,10 +88903,11 @@
88903	/* The sqlite3ResultSetOfSelect() is only used n contexts where lookaside
88904	** is disabled */
88905	assert( db->lookaside.bEnabled==0 );
88906	pTab->nRef = 1;
88907	pTab->zName = 0;
88908	pTab->nRowEst = 1000000;
88909	selectColumnsFromExprList(pParse, pSelect->pEList, &pTab->nCol, &pTab->aCol);
88910	selectAddColumnTypeAndCollation(pParse, pTab->nCol, pTab->aCol, pSelect);
88911	pTab->iPKey = -1;
88912	if( db->mallocFailed ){
88913	sqlite3DeleteTable(db, pTab);
	@@ -88676,10 +89074,14 @@
89074	Select pPrior; / Another SELECT immediately to our left */
89075	Vdbe v; / Generate code to this VDBE */
89076	SelectDest dest; /* Alternative data destination */
89077	Select pDelete = 0; / Chain of simple selects to delete */
89078	sqlite3 db; / Database connection */
89079	#ifndef SQLITE_OMIT_EXPLAIN
89080	int iSub1; /* EQP id of left-hand query */
89081	int iSub2; /* EQP id of right-hand query */
89082	#endif
89083
89084	/* Make sure there is no ORDER BY or LIMIT clause on prior SELECTs. Only
89085	** the last (right-most) SELECT in the series may have an ORDER BY or LIMIT.
89086	*/
89087	assert( p && p->pPrior ); /* Calling function guarantees this much */
	@@ -88736,10 +89138,11 @@
89138	case TK_ALL: {
89139	int addr = 0;
89140	assert( !pPrior->pLimit );
89141	pPrior->pLimit = p->pLimit;
89142	pPrior->pOffset = p->pOffset;
89143	explainSetInteger(iSub1, pParse->iNextSelectId);
89144	rc = sqlite3Select(pParse, pPrior, &dest);
89145	p->pLimit = 0;
89146	p->pOffset = 0;
89147	if( rc ){
89148	goto multi_select_end;
	@@ -88749,10 +89152,11 @@
89152	p->iOffset = pPrior->iOffset;
89153	if( p->iLimit ){
89154	addr = sqlite3VdbeAddOp1(v, OP_IfZero, p->iLimit);
89155	VdbeComment((v, "Jump ahead if LIMIT reached"));
89156	}
89157	explainSetInteger(iSub2, pParse->iNextSelectId);
89158	rc = sqlite3Select(pParse, p, &dest);
89159	testcase( rc!=SQLITE_OK );
89160	pDelete = p->pPrior;
89161	p->pPrior = pPrior;
89162	if( addr ){
	@@ -88796,10 +89200,11 @@
89200
89201	/* Code the SELECT statements to our left
89202	*/
89203	assert( !pPrior->pOrderBy );
89204	sqlite3SelectDestInit(&uniondest, priorOp, unionTab);
89205	explainSetInteger(iSub1, pParse->iNextSelectId);
89206	rc = sqlite3Select(pParse, pPrior, &uniondest);
89207	if( rc ){
89208	goto multi_select_end;
89209	}
89210
	@@ -88815,10 +89220,11 @@
89220	pLimit = p->pLimit;
89221	p->pLimit = 0;
89222	pOffset = p->pOffset;
89223	p->pOffset = 0;
89224	uniondest.eDest = op;
89225	explainSetInteger(iSub2, pParse->iNextSelectId);
89226	rc = sqlite3Select(pParse, p, &uniondest);
89227	testcase( rc!=SQLITE_OK );
89228	/* Query flattening in sqlite3Select() might refill p->pOrderBy.
89229	** Be sure to delete p->pOrderBy, therefore, to avoid a memory leak. */
89230	sqlite3ExprListDelete(db, p->pOrderBy);
	@@ -88880,10 +89286,11 @@
89286	assert( p->pEList );
89287
89288	/* Code the SELECTs to our left into temporary table "tab1".
89289	*/
89290	sqlite3SelectDestInit(&intersectdest, SRT_Union, tab1);
89291	explainSetInteger(iSub1, pParse->iNextSelectId);
89292	rc = sqlite3Select(pParse, pPrior, &intersectdest);
89293	if( rc ){
89294	goto multi_select_end;
89295	}
89296
	@@ -88896,10 +89303,11 @@
89303	pLimit = p->pLimit;
89304	p->pLimit = 0;
89305	pOffset = p->pOffset;
89306	p->pOffset = 0;
89307	intersectdest.iParm = tab2;
89308	explainSetInteger(iSub2, pParse->iNextSelectId);
89309	rc = sqlite3Select(pParse, p, &intersectdest);
89310	testcase( rc!=SQLITE_OK );
89311	pDelete = p->pPrior;
89312	p->pPrior = pPrior;
89313	sqlite3ExprDelete(db, p->pLimit);
	@@ -88931,10 +89339,12 @@
89339	sqlite3VdbeAddOp2(v, OP_Close, tab2, 0);
89340	sqlite3VdbeAddOp2(v, OP_Close, tab1, 0);
89341	break;
89342	}
89343	}
89344
89345	explainComposite(pParse, p->op, iSub1, iSub2, p->op!=TK_ALL);
89346
89347	/* Compute collating sequences used by
89348	** temporary tables needed to implement the compound select.
89349	** Attach the KeyInfo structure to all temporary tables.
89350	**
	@@ -89275,10 +89685,14 @@
89685	KeyInfo pKeyMerge; / Comparison information for merging rows */
89686	sqlite3 db; / Database connection */
89687	ExprList pOrderBy; / The ORDER BY clause */
89688	int nOrderBy; /* Number of terms in the ORDER BY clause */
89689	int aPermute; / Mapping from ORDER BY terms to result set columns */
89690	#ifndef SQLITE_OMIT_EXPLAIN
89691	int iSub1; /* EQP id of left-hand query */
89692	int iSub2; /* EQP id of right-hand query */
89693	#endif
89694
89695	assert( p->pOrderBy!=0 );
89696	assert( pKeyDup==0 ); /* "Managed" code needs this. Ticket #3382. */
89697	db = pParse->db;
89698	v = pParse->pVdbe;
	@@ -89428,10 +89842,11 @@
89842	/* Generate a coroutine to evaluate the SELECT statement to the
89843	** left of the compound operator - the "A" select.
89844	*/
89845	VdbeNoopComment((v, "Begin coroutine for left SELECT"));
89846	pPrior->iLimit = regLimitA;
89847	explainSetInteger(iSub1, pParse->iNextSelectId);
89848	sqlite3Select(pParse, pPrior, &destA);
89849	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofA);
89850	sqlite3VdbeAddOp1(v, OP_Yield, regAddrA);
89851	VdbeNoopComment((v, "End coroutine for left SELECT"));
89852
	@@ -89442,10 +89857,11 @@
89857	VdbeNoopComment((v, "Begin coroutine for right SELECT"));
89858	savedLimit = p->iLimit;
89859	savedOffset = p->iOffset;
89860	p->iLimit = regLimitB;
89861	p->iOffset = 0;
89862	explainSetInteger(iSub2, pParse->iNextSelectId);
89863	sqlite3Select(pParse, p, &destB);
89864	p->iLimit = savedLimit;
89865	p->iOffset = savedOffset;
89866	sqlite3VdbeAddOp2(v, OP_Integer, 1, regEofB);
89867	sqlite3VdbeAddOp1(v, OP_Yield, regAddrB);
	@@ -89572,10 +89988,11 @@
89988	}
89989	p->pPrior = pPrior;
89990
89991	/*** TBD: Insert subroutine calls to close cursors on incomplete
89992	** subqueries **/
89993	explainComposite(pParse, p->op, iSub1, iSub2, 0);
89994	return SQLITE_OK;
89995	}
89996	#endif
89997
89998	#if !defined(SQLITE_OMIT_SUBQUERY) \|\| !defined(SQLITE_OMIT_VIEW)
	@@ -90305,10 +90722,11 @@
90722	pTab->nRef = 1;
90723	pTab->zName = sqlite3MPrintf(db, "sqlite_subquery_%p_", (void*)pTab);
90724	while( pSel->pPrior ){ pSel = pSel->pPrior; }
90725	selectColumnsFromExprList(pParse, pSel->pEList, &pTab->nCol, &pTab->aCol);
90726	pTab->iPKey = -1;
90727	pTab->nRowEst = 1000000;
90728	pTab->tabFlags \|= TF_Ephemeral;
90729	#endif
90730	}else{
90731	/* An ordinary table or view name in the FROM clause */
90732	assert( pFrom->pTab==0 );
	@@ -90797,10 +91215,15 @@
91215	int rc = 1; /* Value to return from this function */
91216	int addrSortIndex; /* Address of an OP_OpenEphemeral instruction */
91217	AggInfo sAggInfo; /* Information used by aggregate queries */
91218	int iEnd; /* Address of the end of the query */
91219	sqlite3 db; / The database connection */
91220
91221	#ifndef SQLITE_OMIT_EXPLAIN
91222	int iRestoreSelectId = pParse->iSelectId;
91223	pParse->iSelectId = pParse->iNextSelectId++;
91224	#endif
91225
91226	db = pParse->db;
91227	if( p==0 \|\| db->mallocFailed \|\| pParse->nErr ){
91228	return 1;
91229	}
	@@ -90869,10 +91292,11 @@
91292	}
91293	i = -1;
91294	}else{
91295	sqlite3SelectDestInit(&dest, SRT_EphemTab, pItem->iCursor);
91296	assert( pItem->isPopulated==0 );
91297	explainSetInteger(pItem->iSelectId, pParse->iNextSelectId);
91298	sqlite3Select(pParse, pSub, &dest);
91299	pItem->isPopulated = 1;
91300	}
91301	if( /pParse->nErr \|\|/ db->mallocFailed ){
91302	goto select_end;
	@@ -90904,14 +91328,16 @@
91328	pRight = pLoop;
91329	}
91330	mxSelect = db->aLimit[SQLITE_LIMIT_COMPOUND_SELECT];
91331	if( mxSelect && cnt>mxSelect ){
91332	sqlite3ErrorMsg(pParse, "too many terms in compound SELECT");
91333	goto select_end;
91334	}
91335	}
91336	rc = multiSelect(pParse, p, pDest);
91337	explainSetInteger(pParse->iSelectId, iRestoreSelectId);
91338	return rc;
91339	}
91340	#endif
91341
91342	/* If possible, rewrite the query to use GROUP BY instead of DISTINCT.
91343	** GROUP BY might use an index, DISTINCT never does.
	@@ -90919,11 +91345,10 @@
91345	assert( p->pGroupBy==0 \|\| (p->selFlags & SF_Aggregate)!=0 );
91346	if( (p->selFlags & (SF_Distinct\|SF_Aggregate))==SF_Distinct ){
91347	p->pGroupBy = sqlite3ExprListDup(db, p->pEList, 0);
91348	pGroupBy = p->pGroupBy;
91349	p->selFlags &= ~SF_Distinct;

91350	}
91351
91352	/* If there is both a GROUP BY and an ORDER BY clause and they are
91353	** identical, then disable the ORDER BY clause since the GROUP BY
91354	** will cause elements to come out in the correct order. This is
	@@ -90966,11 +91391,11 @@
91391	iEnd = sqlite3VdbeMakeLabel(v);
91392	computeLimitRegisters(pParse, p, iEnd);
91393
91394	/* Open a virtual index to use for the distinct set.
91395	*/
91396	if( p->selFlags & SF_Distinct ){
91397	KeyInfo *pKeyInfo;
91398	assert( isAgg \|\| pGroupBy );
91399	distinct = pParse->nTab++;
91400	pKeyInfo = keyInfoFromExprList(pParse, p->pEList);
91401	sqlite3VdbeAddOp4(v, OP_OpenEphemeral, distinct, 0, 0,
	@@ -91125,10 +91550,13 @@
91550	int regBase;
91551	int regRecord;
91552	int nCol;
91553	int nGroupBy;
91554
91555	explainTempTable(pParse,
91556	isDistinct && !(p->selFlags&SF_Distinct)?"DISTINCT":"GROUP BY");
91557
91558	groupBySort = 1;
91559	nGroupBy = pGroupBy->nExpr;
91560	nCol = nGroupBy + 1;
91561	j = nGroupBy+1;
91562	for(i=0; i<sAggInfo.nColumn; i++){
	@@ -91385,15 +91813,20 @@
91813	sqlite3ExprListDelete(db, pDel);
91814	}
91815	sqlite3VdbeResolveLabel(v, addrEnd);
91816
91817	} /* endif aggregate query */
91818
91819	if( distinct>=0 ){
91820	explainTempTable(pParse, "DISTINCT");
91821	}
91822
91823	/* If there is an ORDER BY clause, then we need to sort the results
91824	** and send them to the callback one by one.
91825	*/
91826	if( pOrderBy ){
91827	explainTempTable(pParse, "ORDER BY");
91828	generateSortTail(pParse, p, v, pEList->nExpr, pDest);
91829	}
91830
91831	/* Jump here to skip this query
91832	*/
	@@ -91406,10 +91839,11 @@
91839
91840	/* Control jumps to here if an error is encountered above, or upon
91841	** successful coding of the SELECT.
91842	*/
91843	select_end:
91844	explainSetInteger(pParse->iSelectId, iRestoreSelectId);
91845
91846	/* Identify column names if results of the SELECT are to be output.
91847	*/
91848	if( rc==SQLITE_OK && pDest->eDest==SRT_Output ){
91849	generateColumnNames(pParse, pTabList, pEList);
	@@ -94495,11 +94929,11 @@
94929	pParse->pNewTable->nCol = 0;
94930	pParse->pNewTable->aCol = 0;
94931	}
94932	db->pVTab = 0;
94933	}else{
94934	sqlite3Error(db, SQLITE_ERROR, (zErr ? "%s" : 0), zErr);
94935	sqlite3DbFree(db, zErr);
94936	rc = SQLITE_ERROR;
94937	}
94938	pParse->declareVtab = 0;
94939
	@@ -94957,11 +95391,10 @@
95391	** cost of pursuing that strategy.
95392	*/
95393	struct WhereCost {
95394	WherePlan plan; /* The lookup strategy */
95395	double rCost; /* Overall cost of pursuing this search strategy */

95396	Bitmask used; /* Bitmask of cursors used by this plan */
95397	};
95398
95399	/*
95400	** Bitmasks for the operators that indices are able to exploit. An
	@@ -95000,11 +95433,11 @@
95433	#define WHERE_COLUMN_EQ 0x00010000 /* x=EXPR or x IN (...) or x IS NULL */
95434	#define WHERE_COLUMN_RANGE 0x00020000 /* x<EXPR and/or x>EXPR */
95435	#define WHERE_COLUMN_IN 0x00040000 /* x IN (...) */
95436	#define WHERE_COLUMN_NULL 0x00080000 /* x IS NULL */
95437	#define WHERE_INDEXED 0x000f0000 /* Anything that uses an index */
95438	#define WHERE_NOT_FULLSCAN 0x100f3000 /* Does not do a full table scan */
95439	#define WHERE_IN_ABLE 0x000f1000 /* Able to support an IN operator */
95440	#define WHERE_TOP_LIMIT 0x00100000 /* x<EXPR or x<=EXPR constraint */
95441	#define WHERE_BTM_LIMIT 0x00200000 /* x>EXPR or x>=EXPR constraint */
95442	#define WHERE_IDX_ONLY 0x00800000 /* Use index only - omit table */
95443	#define WHERE_ORDERBY 0x01000000 /* Output will appear in correct order */
	@@ -96346,12 +96779,13 @@
96779	const int iCur = pSrc->iCursor; /* The cursor of the table to be accessed */
96780	const Bitmask maskSrc = getMask(pWC->pMaskSet, iCur); /* Bitmask for pSrc */
96781	WhereTerm * const pWCEnd = &pWC->a[pWC->nTerm]; /* End of pWC->a[] */
96782	WhereTerm pTerm; / A single term of the WHERE clause */
96783
96784	/* No OR-clause optimization allowed if the INDEXED BY or NOT INDEXED clauses
96785	** are used */
96786	if( pSrc->notIndexed \|\| pSrc->pIndex!=0 ){
96787	return;
96788	}
96789
96790	/* Search the WHERE clause terms for a usable WO_OR term. */
96791	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
	@@ -96385,11 +96819,11 @@
96819	bestIndex(pParse, &tempWC, pSrc, notReady, notValid, 0, &sTermCost);
96820	}else{
96821	continue;
96822	}
96823	rTotal += sTermCost.rCost;
96824	nRow += sTermCost.plan.nRow;
96825	used \|= sTermCost.used;
96826	if( rTotal>=pCost->rCost ) break;
96827	}
96828
96829	/* If there is an ORDER BY clause, increase the scan cost to account
	@@ -96404,12 +96838,12 @@
96838	** less than the current cost stored in pCost, replace the contents
96839	** of pCost. */
96840	WHERETRACE(("... multi-index OR cost=%.9g nrow=%.9g\n", rTotal, nRow));
96841	if( rTotal<pCost->rCost ){
96842	pCost->rCost = rTotal;

96843	pCost->used = used;
96844	pCost->plan.nRow = nRow;
96845	pCost->plan.wsFlags = flags;
96846	pCost->plan.u.pTerm = pTerm;
96847	}
96848	}
96849	}
	@@ -96489,11 +96923,11 @@
96923	for(pTerm=pWC->a; pTerm<pWCEnd; pTerm++){
96924	if( termCanDriveIndex(pTerm, pSrc, notReady) ){
96925	WHERETRACE(("auto-index reduces cost from %.2f to %.2f\n",
96926	pCost->rCost, costTempIdx));
96927	pCost->rCost = costTempIdx;
96928	pCost->plan.nRow = logN + 1;
96929	pCost->plan.wsFlags = WHERE_TEMP_INDEX;
96930	pCost->used = pTerm->prereqRight;
96931	break;
96932	}
96933	}
	@@ -97562,15 +97996,15 @@
97996
97997	/* If this index is the best we have seen so far, then record this
97998	** index and its cost in the pCost structure.
97999	*/
98000	if( (!pIdx \|\| wsFlags)
98001	&& (cost<pCost->rCost \|\| (cost<=pCost->rCost && nRow<pCost->plan.nRow))
98002	){
98003	pCost->rCost = cost;

98004	pCost->used = used;
98005	pCost->plan.nRow = nRow;
98006	pCost->plan.wsFlags = (wsFlags&wsFlagMask);
98007	pCost->plan.nEq = nEq;
98008	pCost->plan.u.pIdx = pIdx;
98009	}
98010
	@@ -97894,10 +98328,162 @@
98328	}
98329	}
98330	*pzAff = zAff;
98331	return regBase;
98332	}
98333
98334	#ifndef SQLITE_OMIT_EXPLAIN
98335	/*
98336	** This routine is a helper for explainIndexRange() below
98337	**
98338	** pStr holds the text of an expression that we are building up one term
98339	** at a time. This routine adds a new term to the end of the expression.
98340	** Terms are separated by AND so add the "AND" text for second and subsequent
98341	** terms only.
98342	*/
98343	static void explainAppendTerm(
98344	StrAccum pStr, / The text expression being built */
98345	int iTerm, /* Index of this term. First is zero */
98346	const char zColumn, / Name of the column */
98347	const char zOp / Name of the operator */
98348	){
98349	if( iTerm ) sqlite3StrAccumAppend(pStr, " AND ", 5);
98350	sqlite3StrAccumAppend(pStr, zColumn, -1);
98351	sqlite3StrAccumAppend(pStr, zOp, 1);
98352	sqlite3StrAccumAppend(pStr, "?", 1);
98353	}
98354
98355	/*
98356	** Argument pLevel describes a strategy for scanning table pTab. This
98357	** function returns a pointer to a string buffer containing a description
98358	** of the subset of table rows scanned by the strategy in the form of an
98359	** SQL expression. Or, if all rows are scanned, NULL is returned.
98360	**
98361	** For example, if the query:
98362	**
98363	** SELECT * FROM t1 WHERE a=1 AND b>2;
98364	**
98365	** is run and there is an index on (a, b), then this function returns a
98366	** string similar to:
98367	**
98368	** "a=? AND b>?"
98369	**
98370	** The returned pointer points to memory obtained from sqlite3DbMalloc().
98371	** It is the responsibility of the caller to free the buffer when it is
98372	** no longer required.
98373	*/
98374	static char explainIndexRange(sqlite3 db, WhereLevel pLevel, Table pTab){
98375	WherePlan *pPlan = &pLevel->plan;
98376	Index *pIndex = pPlan->u.pIdx;
98377	int nEq = pPlan->nEq;
98378	int i, j;
98379	Column *aCol = pTab->aCol;
98380	int *aiColumn = pIndex->aiColumn;
98381	StrAccum txt;
98382
98383	if( nEq==0 && (pPlan->wsFlags & (WHERE_BTM_LIMIT\|WHERE_TOP_LIMIT))==0 ){
98384	return 0;
98385	}
98386	sqlite3StrAccumInit(&txt, 0, 0, SQLITE_MAX_LENGTH);
98387	txt.db = db;
98388	sqlite3StrAccumAppend(&txt, " (", 2);
98389	for(i=0; i<nEq; i++){
98390	explainAppendTerm(&txt, i, aCol[aiColumn[i]].zName, "=");
98391	}
98392
98393	j = i;
98394	if( pPlan->wsFlags&WHERE_BTM_LIMIT ){
98395	explainAppendTerm(&txt, i++, aCol[aiColumn[j]].zName, ">");
98396	}
98397	if( pPlan->wsFlags&WHERE_TOP_LIMIT ){
98398	explainAppendTerm(&txt, i, aCol[aiColumn[j]].zName, "<");
98399	}
98400	sqlite3StrAccumAppend(&txt, ")", 1);
98401	return sqlite3StrAccumFinish(&txt);
98402	}
98403
98404	/*
98405	** This function is a no-op unless currently processing an EXPLAIN QUERY PLAN
98406	** command. If the query being compiled is an EXPLAIN QUERY PLAN, a single
98407	** record is added to the output to describe the table scan strategy in
98408	** pLevel.
98409	*/
98410	static void explainOneScan(
98411	Parse pParse, / Parse context */
98412	SrcList pTabList, / Table list this loop refers to */
98413	WhereLevel pLevel, / Scan to write OP_Explain opcode for */
98414	int iLevel, /* Value for "level" column of output */
98415	int iFrom, /* Value for "from" column of output */
98416	u16 wctrlFlags /* Flags passed to sqlite3WhereBegin() */
98417	){
98418	if( pParse->explain==2 ){
98419	u32 flags = pLevel->plan.wsFlags;
98420	struct SrcList_item *pItem = &pTabList->a[pLevel->iFrom];
98421	Vdbe v = pParse->pVdbe; / VM being constructed */
98422	sqlite3 db = pParse->db; / Database handle */
98423	char zMsg; / Text to add to EQP output */
98424	sqlite3_int64 nRow; /* Expected number of rows visited by scan */
98425	int iId = pParse->iSelectId; /* Select id (left-most output column) */
98426	int isSearch; /* True for a SEARCH. False for SCAN. */
98427
98428	if( (flags&WHERE_MULTI_OR) \|\| (wctrlFlags&WHERE_ONETABLE_ONLY) ) return;
98429
98430	isSearch = (pLevel->plan.nEq>0 \|\| flags&(WHERE_BTM_LIMIT\|WHERE_TOP_LIMIT));
98431
98432	zMsg = sqlite3MPrintf(db, "%s", isSearch?"SEARCH":"SCAN");
98433	if( pItem->pSelect ){
98434	zMsg = sqlite3MAppendf(db, zMsg, "%s SUBQUERY %d", zMsg,pItem->iSelectId);
98435	}else{
98436	zMsg = sqlite3MAppendf(db, zMsg, "%s TABLE %s", zMsg, pItem->zName);
98437	}
98438
98439	if( pItem->zAlias ){
98440	zMsg = sqlite3MAppendf(db, zMsg, "%s AS %s", zMsg, pItem->zAlias);
98441	}
98442	if( (flags & WHERE_INDEXED)!=0 ){
98443	char *zWhere = explainIndexRange(db, pLevel, pItem->pTab);
98444	zMsg = sqlite3MAppendf(db, zMsg, "%s USING %s%sINDEX%s%s%s", zMsg,
98445	((flags & WHERE_TEMP_INDEX)?"AUTOMATIC ":""),
98446	((flags & WHERE_IDX_ONLY)?"COVERING ":""),
98447	((flags & WHERE_TEMP_INDEX)?"":" "),
98448	((flags & WHERE_TEMP_INDEX)?"": pLevel->plan.u.pIdx->zName),
98449	zWhere
98450	);
98451	sqlite3DbFree(db, zWhere);
98452	}else if( flags & (WHERE_ROWID_EQ\|WHERE_ROWID_RANGE) ){
98453	zMsg = sqlite3MAppendf(db, zMsg, "%s USING INTEGER PRIMARY KEY", zMsg);
98454
98455	if( flags&WHERE_ROWID_EQ ){
98456	zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid=?)", zMsg);
98457	}else if( flags&WHERE_BTM_LIMIT && flags&WHERE_TOP_LIMIT ){
98458	zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid>? AND rowid<?)", zMsg);
98459	}else if( flags&WHERE_BTM_LIMIT ){
98460	zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid>?)", zMsg);
98461	}else if( flags&WHERE_TOP_LIMIT ){
98462	zMsg = sqlite3MAppendf(db, zMsg, "%s (rowid<?)", zMsg);
98463	}
98464	}
98465	#ifndef SQLITE_OMIT_VIRTUALTABLE
98466	else if( (flags & WHERE_VIRTUALTABLE)!=0 ){
98467	sqlite3_index_info *pVtabIdx = pLevel->plan.u.pVtabIdx;
98468	zMsg = sqlite3MAppendf(db, zMsg, "%s VIRTUAL TABLE INDEX %d:%s", zMsg,
98469	pVtabIdx->idxNum, pVtabIdx->idxStr);
98470	}
98471	#endif
98472	if( wctrlFlags&(WHERE_ORDERBY_MIN\|WHERE_ORDERBY_MAX) ){
98473	nRow = 1;
98474	}else{
98475	nRow = (sqlite3_int64)pLevel->plan.nRow;
98476	}
98477	zMsg = sqlite3MAppendf(db, zMsg, "%s (~%lld rows)", zMsg, nRow);
98478	sqlite3VdbeAddOp4(v, OP_Explain, iId, iLevel, iFrom, zMsg, P4_DYNAMIC);
98479	}
98480	}
98481	#else
98482	# define explainOneScan(u,v,w,x,y,z)
98483	#endif /* SQLITE_OMIT_EXPLAIN */
98484
98485
98486	/*
98487	** Generate code for the start of the iLevel-th loop in the WHERE clause
98488	** implementation described by pWInfo.
98489	*/
	@@ -98436,10 +99022,13 @@
99022	/* Loop through table entries that match term pOrTerm. */
99023	pSubWInfo = sqlite3WhereBegin(pParse, pOrTab, pOrTerm->pExpr, 0,
99024	WHERE_OMIT_OPEN \| WHERE_OMIT_CLOSE \|
99025	WHERE_FORCE_TABLE \| WHERE_ONETABLE_ONLY);
99026	if( pSubWInfo ){
99027	explainOneScan(
99028	pParse, pOrTab, &pSubWInfo->a[0], iLevel, pLevel->iFrom, 0
99029	);
99030	if( (wctrlFlags & WHERE_DUPLICATES_OK)==0 ){
99031	int iSet = ((ii==pOrWc->nTerm-1)?-1:ii);
99032	int r;
99033	r = sqlite3ExprCodeGetColumn(pParse, pTabItem->pTab, -1, iCur,
99034	regRowid);
	@@ -98831,10 +99420,11 @@
99420	int nUnconstrained; /* Number tables without INDEXED BY */
99421	Bitmask notIndexed; /* Mask of tables that cannot use an index */
99422
99423	memset(&bestPlan, 0, sizeof(bestPlan));
99424	bestPlan.rCost = SQLITE_BIG_DBL;
99425	WHERETRACE(("* Begin search for loop %d *\n", i));
99426
99427	/* Loop through the remaining entries in the FROM clause to find the
99428	** next nested loop. The loop tests all FROM clause entries
99429	** either once or twice.
99430	**
	@@ -98895,10 +99485,12 @@
99485	}
99486	mask = (isOptimal ? m : notReady);
99487	pOrderBy = ((i==0 && ppOrderBy )?*ppOrderBy:0);
99488	if( pTabItem->pIndex==0 ) nUnconstrained++;
99489
99490	WHERETRACE(("=== trying table %d with isOptimal=%d ===\n",
99491	j, isOptimal));
99492	assert( pTabItem->pTab );
99493	#ifndef SQLITE_OMIT_VIRTUALTABLE
99494	if( IsVirtual(pTabItem->pTab) ){
99495	sqlite3_index_info **pp = &pWInfo->a[j].pIdxInfo;
99496	bestVirtualIndex(pParse, pWC, pTabItem, mask, notReady, pOrderBy,
	@@ -98945,24 +99537,27 @@
99537	&& (bestJ<0 \|\| (notIndexed&m)!=0 /* (2) */
99538	\|\| (sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0)
99539	&& (nUnconstrained==0 \|\| pTabItem->pIndex==0 /* (3) */
99540	\|\| NEVER((sCost.plan.wsFlags & WHERE_NOT_FULLSCAN)!=0))
99541	&& (bestJ<0 \|\| sCost.rCost<bestPlan.rCost /* (4) */
99542	\|\| (sCost.rCost<=bestPlan.rCost
99543	&& sCost.plan.nRow<bestPlan.plan.nRow))
99544	){
99545	WHERETRACE(("=== table %d is best so far"
99546	" with cost=%g and nRow=%g\n",
99547	j, sCost.rCost, sCost.plan.nRow));
99548	bestPlan = sCost;
99549	bestJ = j;
99550	}
99551	if( doNotReorder ) break;
99552	}
99553	}
99554	assert( bestJ>=0 );
99555	assert( notReady & getMask(pMaskSet, pTabList->a[bestJ].iCursor) );
99556	WHERETRACE(("*** Optimizer selects table %d for loop %d"
99557	" with cost=%g and nRow=%g\n",
99558	bestJ, pLevel-pWInfo->a, bestPlan.rCost, bestPlan.plan.nRow));
99559	if( (bestPlan.plan.wsFlags & WHERE_ORDERBY)!=0 ){
99560	*ppOrderBy = 0;
99561	}
99562	andFlags &= bestPlan.plan.wsFlags;
99563	pLevel->plan = bestPlan.plan;
	@@ -98973,11 +99568,13 @@
99568	}else{
99569	pLevel->iIdxCur = -1;
99570	}
99571	notReady &= ~getMask(pMaskSet, pTabList->a[bestJ].iCursor);
99572	pLevel->iFrom = (u8)bestJ;
99573	if( bestPlan.plan.nRow>=(double)1 ){
99574	pParse->nQueryLoop *= bestPlan.plan.nRow;
99575	}
99576
99577	/* Check that if the table scanned by this loop iteration had an
99578	** INDEXED BY clause attached to it, that the named index is being
99579	** used for the scan. If not, then query compilation has failed.
99580	** Return an error.
	@@ -99025,41 +99622,10 @@
99622	notReady = ~(Bitmask)0;
99623	for(i=0, pLevel=pWInfo->a; i<nTabList; i++, pLevel++){
99624	Table pTab; / Table to open */
99625	int iDb; /* Index of database containing table/index */
99626































99627	pTabItem = &pTabList->a[pLevel->iFrom];
99628	pTab = pTabItem->pTab;
99629	pLevel->iTabCur = pTabItem->iCursor;
99630	iDb = sqlite3SchemaToIndex(db, pTab->pSchema);
99631	if( (pTab->tabFlags & TF_Ephemeral)!=0 \|\| pTab->pSelect ){
	@@ -99114,12 +99680,14 @@
99680	** loop below generates code for a single nested loop of the VM
99681	** program.
99682	*/
99683	notReady = ~(Bitmask)0;
99684	for(i=0; i<nTabList; i++){
99685	WhereLevel *pLevel = &pWInfo->a[i];
99686	explainOneScan(pParse, pTabList, pLevel, i, pLevel->iFrom, wctrlFlags);
99687	notReady = codeOneLoopStart(pWInfo, i, wctrlFlags, notReady);
99688	pWInfo->iContinue = pLevel->addrCont;
99689	}
99690
99691	#ifdef SQLITE_TEST /* For testing and debugging use only */
99692	/* Record in the query plan information about the current table
99693	** and the index used to access it (if any). If the table itself
	@@ -105517,17 +106085,16 @@
106085	}
106086	}
106087	}
106088
106089	pColl = sqlite3FindCollSeq(db, (u8)enc2, zName, 1);
106090	if( pColl==0 ) return SQLITE_NOMEM;
106091	pColl->xCmp = xCompare;
106092	pColl->pUser = pCtx;
106093	pColl->xDel = xDel;
106094	pColl->enc = (u8)(enc2 \| (enc & SQLITE_UTF16_ALIGNED));
106095	pColl->type = collType;

106096	sqlite3Error(db, SQLITE_OK, 0);
106097	return SQLITE_OK;
106098	}
106099
106100
	@@ -107491,12 +108058,18 @@
108058	#ifndef SQLITE_AMALGAMATION
108059	/*
108060	** Macros indicating that conditional expressions are always true or
108061	** false.
108062	*/
108063	#ifdef SQLITE_COVERAGE_TEST
108064	# define ALWAYS(x) (1)
108065	# define NEVER(X) (0)
108066	#else
108067	# define ALWAYS(x) (x)
108068	# define NEVER(X) (x)
108069	#endif
108070
108071	/*
108072	** Internal types used by SQLite.
108073	*/
108074	typedef unsigned char u8; /* 1-byte (or larger) unsigned integer */
108075	typedef short int i16; /* 2-byte (or larger) signed integer */
	@@ -107510,12 +108083,16 @@
108083
108084	typedef struct Fts3Table Fts3Table;
108085	typedef struct Fts3Cursor Fts3Cursor;
108086	typedef struct Fts3Expr Fts3Expr;
108087	typedef struct Fts3Phrase Fts3Phrase;
108088	typedef struct Fts3PhraseToken Fts3PhraseToken;
108089
108090	typedef struct Fts3SegFilter Fts3SegFilter;
108091	typedef struct Fts3DeferredToken Fts3DeferredToken;
108092	typedef struct Fts3SegReader Fts3SegReader;
108093	typedef struct Fts3SegReaderArray Fts3SegReaderArray;
108094
108095	/*
108096	** A connection to a fulltext index is an instance of the following
108097	** structure. The xCreate and xConnect methods create an instance
108098	** of this structure and xDestroy and xDisconnect free that instance.
	@@ -107532,26 +108109,18 @@
108109	sqlite3_tokenizer pTokenizer; / tokenizer for inserts and queries */
108110
108111	/* Precompiled statements used by the implementation. Each of these
108112	** statements is run and reset within a single virtual table API call.
108113	*/
108114	sqlite3_stmt *aStmt[24];











108115
108116	int nNodeSize; /* Soft limit for node size */
108117	u8 bHasStat; /* True if %_stat table exists */
108118	u8 bHasDocsize; /* True if %_docsize table exists */
108119	int nPgsz; /* Page size for host database */
108120	char zSegmentsTbl; / Name of %_segments table */
108121	sqlite3_blob pSegments; / Blob handle open on %_segments table */
108122
108123	/* The following hash table is used to buffer pending index updates during
108124	** transactions. Variable nPendingData estimates the memory size of the
108125	** pending data, including hash table overhead, but not malloc overhead.
108126	** When nPendingData exceeds nMaxPendingData, the buffer is flushed
	@@ -107574,17 +108143,24 @@
108143	i16 eSearch; /* Search strategy (see below) */
108144	u8 isEof; /* True if at End Of Results */
108145	u8 isRequireSeek; /* True if must seek pStmt to %_content row */
108146	sqlite3_stmt pStmt; / Prepared statement in use by the cursor */
108147	Fts3Expr pExpr; / Parsed MATCH query string */
108148	Fts3DeferredToken pDeferred; / Deferred search tokens, if any */
108149	sqlite3_int64 iPrevId; /* Previous id read from aDoclist */
108150	char pNextId; / Pointer into the body of aDoclist */
108151	char aDoclist; / List of docids for full-text queries */
108152	int nDoclist; /* Size of buffer at aDoclist */
108153	int isMatchinfoNeeded; /* True when aMatchinfo[] needs filling in */
108154	u32 aMatchinfo; / Information about most recent match */
108155	int eEvalmode; /* An FTS3_EVAL_XX constant */
108156	int nRowAvg; /* Average size of database rows, in pages */
108157	};
108158
108159	#define FTS3_EVAL_FILTER 0
108160	#define FTS3_EVAL_NEXT 1
108161	#define FTS3_EVAL_MATCHINFO 2
108162
108163	/*
108164	** The Fts3Cursor.eSearch member is always set to one of the following.
108165	** Actualy, Fts3Cursor.eSearch can be greater than or equal to
108166	** FTS3_FULLTEXT_SEARCH. If so, then Fts3Cursor.eSearch - 2 is the index
	@@ -107604,22 +108180,34 @@
108180	#define FTS3_FULLTEXT_SEARCH 2 /* Full-text index search */
108181
108182	/*
108183	** A "phrase" is a sequence of one or more tokens that must match in
108184	** sequence. A single token is the base case and the most common case.
108185	** For a sequence of tokens contained in double-quotes (i.e. "one two three")
108186	** nToken will be the number of tokens in the string.
108187	**
108188	** The nDocMatch and nMatch variables contain data that may be used by the
108189	** matchinfo() function. They are populated when the full-text index is
108190	** queried for hits on the phrase. If one or more tokens in the phrase
108191	** are deferred, the nDocMatch and nMatch variables are populated based
108192	** on the assumption that the
108193	*/
108194	struct Fts3PhraseToken {
108195	char z; / Text of the token */
108196	int n; /* Number of bytes in buffer z */
108197	int isPrefix; /* True if token ends with a "" character /
108198	int bFulltext; /* True if full-text index was used */
108199	Fts3SegReaderArray pArray; / Segment-reader for this token */
108200	Fts3DeferredToken pDeferred; / Deferred token object for this token */
108201	};
108202
108203	struct Fts3Phrase {
108204	/* Variables populated by fts3_expr.c when parsing a MATCH expression */
108205	int nToken; /* Number of tokens in the phrase */
108206	int iColumn; /* Index of column this phrase must match */
108207	int isNot; /* Phrase prefixed by unary not (-) operator */
108208	Fts3PhraseToken aToken[1]; /* One entry for each token in the phrase */




108209	};
108210
108211	/*
108212	** A tree of these objects forms the RHS of a MATCH operator.
108213	**
	@@ -107665,15 +108253,10 @@
108253	#define FTSQUERY_AND 3
108254	#define FTSQUERY_OR 4
108255	#define FTSQUERY_PHRASE 5
108256
108257





108258	/* fts3_write.c */
108259	SQLITE_PRIVATE int sqlite3Fts3UpdateMethod(sqlite3_vtab,int,sqlite3_value,sqlite3_int64);
108260	SQLITE_PRIVATE int sqlite3Fts3PendingTermsFlush(Fts3Table *);
108261	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *);
108262	SQLITE_PRIVATE int sqlite3Fts3Optimize(Fts3Table *);
	@@ -107683,15 +108266,24 @@
108266	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table , Fts3SegReader );
108267	SQLITE_PRIVATE int sqlite3Fts3SegReaderIterate(
108268	Fts3Table , Fts3SegReader , int, Fts3SegFilter ,
108269	int ()(Fts3Table , void , char , int, char , int), void
108270	);
108271	SQLITE_PRIVATE int sqlite3Fts3SegReaderCost(Fts3Cursor , Fts3SegReader , int *);
108272	SQLITE_PRIVATE int sqlite3Fts3AllSegdirs(Fts3Table, sqlite3_stmt *);
108273	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeLocal(Fts3Cursor, u32);
108274	SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeGlobal(Fts3Cursor, u32);
108275	SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *);
108276	SQLITE_PRIVATE int sqlite3Fts3ReadBlock(Fts3Table, sqlite3_int64, char , int);
108277
108278	SQLITE_PRIVATE void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *);
108279	SQLITE_PRIVATE int sqlite3Fts3DeferToken(Fts3Cursor , Fts3PhraseToken , int);
108280	SQLITE_PRIVATE int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *);
108281	SQLITE_PRIVATE void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *);
108282	SQLITE_PRIVATE char sqlite3Fts3DeferredDoclist(Fts3DeferredToken , int *);
108283
108284	SQLITE_PRIVATE void sqlite3Fts3SegmentsClose(Fts3Table *);
108285
108286	/* Flags allowed as part of the 4th argument to SegmentReaderIterate() */
108287	#define FTS3_SEGMENT_REQUIRE_POS 0x00000001
108288	#define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002
108289	#define FTS3_SEGMENT_COLUMN_FILTER 0x00000004
	@@ -107711,19 +108303,21 @@
108303	SQLITE_PRIVATE int sqlite3Fts3GetVarint32(const char , int );
108304	SQLITE_PRIVATE int sqlite3Fts3VarintLen(sqlite3_uint64);
108305	SQLITE_PRIVATE void sqlite3Fts3Dequote(char *);
108306
108307	SQLITE_PRIVATE char sqlite3Fts3FindPositions(Fts3Expr , sqlite3_int64, int);
108308	SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Cursor , Fts3Expr );
108309	SQLITE_PRIVATE int sqlite3Fts3ExprLoadFtDoclist(Fts3Cursor , Fts3Expr , char *, int );
108310	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr , Fts3Expr , int);
108311
108312	/* fts3_tokenizer.c */
108313	SQLITE_PRIVATE const char sqlite3Fts3NextToken(const char , int *);
108314	SQLITE_PRIVATE int sqlite3Fts3InitHashTable(sqlite3 , Fts3Hash , const char *);
108315	SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(Fts3Hash pHash, const char ,
108316	sqlite3_tokenizer , char
108317	);
108318	SQLITE_PRIVATE int sqlite3Fts3IsIdChar(char);
108319
108320	/* fts3_snippet.c */
108321	SQLITE_PRIVATE void sqlite3Fts3Offsets(sqlite3_context, Fts3Cursor);
108322	SQLITE_PRIVATE void sqlite3Fts3Snippet(sqlite3_context , Fts3Cursor , const char *,
108323	const char , const char , int, int
	@@ -107875,20 +108469,17 @@
108469	static int fts3DisconnectMethod(sqlite3_vtab *pVtab){
108470	Fts3Table p = (Fts3Table )pVtab;
108471	int i;
108472
108473	assert( p->nPendingData==0 );
108474	assert( p->pSegments==0 );
108475
108476	/* Free any prepared statements held */
108477	for(i=0; i<SizeofArray(p->aStmt); i++){
108478	sqlite3_finalize(p->aStmt[i]);
108479	}
108480	sqlite3_free(p->zSegmentsTbl);




108481
108482	/* Invoke the tokenizer destructor to free the tokenizer. */
108483	p->pTokenizer->pModule->xDestroy(p->pTokenizer);
108484
108485	sqlite3_free(p);
	@@ -107895,11 +108486,11 @@
108486	return SQLITE_OK;
108487	}
108488
108489	/*
108490	** Construct one or more SQL statements from the format string given
108491	** and then evaluate those statements. The success code is written
108492	** into *pRc.
108493	**
108494	** If *pRc is initially non-zero then this routine is a no-op.
108495	*/
108496	static void fts3DbExec(
	@@ -107947,37 +108538,42 @@
108538
108539	/*
108540	** Invoke sqlite3_declare_vtab() to declare the schema for the FTS3 table
108541	** passed as the first argument. This is done as part of the xConnect()
108542	** and xCreate() methods.
108543	**
108544	** If *pRc is non-zero when this function is called, it is a no-op.
108545	** Otherwise, if an error occurs, an SQLite error code is stored in *pRc
108546	** before returning.
108547	*/
108548	static void fts3DeclareVtab(int pRc, Fts3Table p){
108549	if( *pRc==SQLITE_OK ){
108550	int i; /* Iterator variable */
108551	int rc; /* Return code */
108552	char zSql; / SQL statement passed to declare_vtab() */
108553	char zCols; / List of user defined columns */
108554
108555	/* Create a list of user columns for the virtual table */
108556	zCols = sqlite3_mprintf("%Q, ", p->azColumn[0]);
108557	for(i=1; zCols && i<p->nColumn; i++){
108558	zCols = sqlite3_mprintf("%z%Q, ", zCols, p->azColumn[i]);
108559	}
108560
108561	/* Create the whole "CREATE TABLE" statement to pass to SQLite */
108562	zSql = sqlite3_mprintf(
108563	"CREATE TABLE x(%s %Q HIDDEN, docid HIDDEN)", zCols, p->zName
108564	);
108565	if( !zCols \|\| !zSql ){
108566	rc = SQLITE_NOMEM;
108567	}else{
108568	rc = sqlite3_declare_vtab(p->db, zSql);
108569	}
108570
108571	sqlite3_free(zSql);
108572	sqlite3_free(zCols);
108573	*pRc = rc;
108574	}
108575	}
108576
108577	/*
108578	** Create the backing store tables (%_content, %_segments and %_segdir)
108579	** required by the FTS3 table passed as the only argument. This is done
	@@ -107992,25 +108588,23 @@
108588	int i; /* Iterator variable */
108589	char zContentCols; / Columns of %_content table */
108590	sqlite3 db = p->db; / The database connection */
108591
108592	/* Create a list of user columns for the content table */
108593	zContentCols = sqlite3_mprintf("docid INTEGER PRIMARY KEY");
108594	for(i=0; zContentCols && i<p->nColumn; i++){
108595	char *z = p->azColumn[i];
108596	zContentCols = sqlite3_mprintf("%z, 'c%d%q'", zContentCols, i, z);
108597	}
108598	if( zContentCols==0 ) rc = SQLITE_NOMEM;
108599
108600	/* Create the content table */
108601	fts3DbExec(&rc, db,
108602	"CREATE TABLE %Q.'%q_content'(%s)",
108603	p->zDb, p->zName, zContentCols
108604	);
108605	sqlite3_free(zContentCols);


108606	/* Create other tables */
108607	fts3DbExec(&rc, db,
108608	"CREATE TABLE %Q.'%q_segments'(blockid INTEGER PRIMARY KEY, block BLOB);",
108609	p->zDb, p->zName
108610	);
	@@ -108029,10 +108623,12 @@
108623	if( p->bHasDocsize ){
108624	fts3DbExec(&rc, db,
108625	"CREATE TABLE %Q.'%q_docsize'(docid INTEGER PRIMARY KEY, size BLOB);",
108626	p->zDb, p->zName
108627	);
108628	}
108629	if( p->bHasStat ){
108630	fts3DbExec(&rc, db,
108631	"CREATE TABLE %Q.'%q_stat'(id INTEGER PRIMARY KEY, value BLOB);",
108632	p->zDb, p->zName
108633	);
108634	}
	@@ -108072,10 +108668,70 @@
108668	rc = sqlite3_exec(db, zSql, fts3TableExistsCallback, &res, 0);
108669	sqlite3_free(zSql);
108670	*pResult = (u8)(res & 0xff);
108671	if( rc!=SQLITE_ABORT ) *pRc = rc;
108672	}
108673
108674	/*
108675	** Store the current database page-size in bytes in p->nPgsz.
108676	**
108677	** If *pRc is non-zero when this function is called, it is a no-op.
108678	** Otherwise, if an error occurs, an SQLite error code is stored in *pRc
108679	** before returning.
108680	*/
108681	static void fts3DatabasePageSize(int pRc, Fts3Table p){
108682	if( *pRc==SQLITE_OK ){
108683	int rc; /* Return code */
108684	char zSql; / SQL text "PRAGMA %Q.page_size" */
108685	sqlite3_stmt pStmt; / Compiled "PRAGMA %Q.page_size" statement */
108686
108687	zSql = sqlite3_mprintf("PRAGMA %Q.page_size", p->zDb);
108688	if( !zSql ){
108689	rc = SQLITE_NOMEM;
108690	}else{
108691	rc = sqlite3_prepare(p->db, zSql, -1, &pStmt, 0);
108692	if( rc==SQLITE_OK ){
108693	sqlite3_step(pStmt);
108694	p->nPgsz = sqlite3_column_int(pStmt, 0);
108695	rc = sqlite3_finalize(pStmt);
108696	}
108697	}
108698	assert( p->nPgsz>0 \|\| rc!=SQLITE_OK );
108699	sqlite3_free(zSql);
108700	*pRc = rc;
108701	}
108702	}
108703
108704	/*
108705	** "Special" FTS4 arguments are column specifications of the following form:
108706	**
108707	** <key> = <value>
108708	**
108709	** There may not be whitespace surrounding the "=" character. The <value>
108710	** term may be quoted, but the <key> may not.
108711	*/
108712	static int fts3IsSpecialColumn(
108713	const char *z,
108714	int *pnKey,
108715	char **pzValue
108716	){
108717	char *zValue;
108718	const char *zCsr = z;
108719
108720	while( *zCsr!='=' ){
108721	if( *zCsr=='\0' ) return 0;
108722	zCsr++;
108723	}
108724
108725	*pnKey = zCsr-z;
108726	zValue = sqlite3_mprintf("%s", &zCsr[1]);
108727	if( zValue ){
108728	sqlite3Fts3Dequote(zValue);
108729	}
108730	*pzValue = zValue;
108731	return 1;
108732	}
108733
108734	/*
108735	** This function is the implementation of both the xConnect and xCreate
108736	** methods of the FTS3 virtual table.
108737	**
	@@ -108094,48 +108750,103 @@
108750	const char * const argv, / xCreate/xConnect argument array */
108751	sqlite3_vtab *ppVTab, / Write the resulting vtab structure here */
108752	char *pzErr / Write any error message here */
108753	){
108754	Fts3Hash pHash = (Fts3Hash )pAux;
108755	Fts3Table p = 0; / Pointer to allocated vtab */
108756	int rc = SQLITE_OK; /* Return code */
108757	int i; /* Iterator variable */
108758	int nByte; /* Size of allocation used for p /
108759	int iCol; /* Column index */
108760	int nString = 0; /* Bytes required to hold all column names */
108761	int nCol = 0; /* Number of columns in the FTS table */
108762	char zCsr; / Space for holding column names */
108763	int nDb; /* Bytes required to hold database name */
108764	int nName; /* Bytes required to hold table name */
108765	int isFts4 = (argv[0][3]=='4'); /* True for FTS4, false for FTS3 */
108766	int bNoDocsize = 0; /* True to omit %_docsize table */
108767	const char *aCol; / Array of column names */
108768	sqlite3_tokenizer pTokenizer = 0; / Tokenizer for this table */
108769
108770	assert( strlen(argv[0])==4 );
108771	assert( (sqlite3_strnicmp(argv[0], "fts4", 4)==0 && isFts4)
108772	\|\| (sqlite3_strnicmp(argv[0], "fts3", 4)==0 && !isFts4)
108773	);
108774
108775	nDb = (int)strlen(argv[1]) + 1;
108776	nName = (int)strlen(argv[2]) + 1;
108777
108778	aCol = (const char *)sqlite3_malloc(sizeof(const char ) * (argc-2) );
108779	if( !aCol ) return SQLITE_NOMEM;
108780	memset(aCol, 0, sizeof(const char ) (argc-2));
108781
108782	/* Loop through all of the arguments passed by the user to the FTS3/4
108783	** module (i.e. all the column names and special arguments). This loop
108784	** does the following:
108785	**
108786	** + Figures out the number of columns the FTSX table will have, and
108787	** the number of bytes of space that must be allocated to store copies
108788	** of the column names.
108789	**
108790	** + If there is a tokenizer specification included in the arguments,
108791	** initializes the tokenizer pTokenizer.
108792	*/
108793	for(i=3; rc==SQLITE_OK && i<argc; i++){
108794	char const *z = argv[i];
108795	int nKey;
108796	char *zVal;
108797
108798	/* Check if this is a tokenizer specification */
108799	if( !pTokenizer
108800	&& strlen(z)>8
108801	&& 0==sqlite3_strnicmp(z, "tokenize", 8)
108802	&& 0==sqlite3Fts3IsIdChar(z[8])
108803	){
108804	rc = sqlite3Fts3InitTokenizer(pHash, &z[9], &pTokenizer, pzErr);
108805	}
108806
108807	/* Check if it is an FTS4 special argument. */
108808	else if( isFts4 && fts3IsSpecialColumn(z, &nKey, &zVal) ){
108809	if( !zVal ){
108810	rc = SQLITE_NOMEM;
108811	goto fts3_init_out;
108812	}
108813	if( nKey==9 && 0==sqlite3_strnicmp(z, "matchinfo", 9) ){
108814	if( strlen(zVal)==4 && 0==sqlite3_strnicmp(zVal, "fts3", 4) ){
108815	bNoDocsize = 1;
108816	}else{
108817	*pzErr = sqlite3_mprintf("unrecognized matchinfo: %s", zVal);
108818	rc = SQLITE_ERROR;
108819	}
108820	}else{
108821	*pzErr = sqlite3_mprintf("unrecognized parameter: %s", z);
108822	rc = SQLITE_ERROR;
108823	}
108824	sqlite3_free(zVal);
108825	}
108826
108827	/* Otherwise, the argument is a column name. */
108828	else {
108829	nString += (int)(strlen(z) + 1);
108830	aCol[nCol++] = z;
108831	}
108832	}
108833	if( rc!=SQLITE_OK ) goto fts3_init_out;







108834
108835	if( nCol==0 ){
108836	assert( nString==0 );
108837	aCol[0] = "content";
108838	nString = 8;
108839	nCol = 1;
108840	}
108841
108842	if( pTokenizer==0 ){
108843	rc = sqlite3Fts3InitTokenizer(pHash, "simple", &pTokenizer, pzErr);
108844	if( rc!=SQLITE_OK ) goto fts3_init_out;
108845	}
108846	assert( pTokenizer );
108847
108848
108849	/* Allocate and populate the Fts3Table structure. */
108850	nByte = sizeof(Fts3Table) + /* Fts3Table */
108851	nCol * sizeof(char ) + / azColumn */
108852	nName + /* zName */
	@@ -108145,77 +108856,70 @@
108856	if( p==0 ){
108857	rc = SQLITE_NOMEM;
108858	goto fts3_init_out;
108859	}
108860	memset(p, 0, nByte);

108861	p->db = db;
108862	p->nColumn = nCol;
108863	p->nPendingData = 0;
108864	p->azColumn = (char **)&p[1];
108865	p->pTokenizer = pTokenizer;
108866	p->nNodeSize = 1000;
108867	p->nMaxPendingData = FTS3_MAX_PENDING_DATA;
108868	p->bHasDocsize = (isFts4 && bNoDocsize==0);
108869	p->bHasStat = isFts4;
108870	fts3HashInit(&p->pendingTerms, FTS3_HASH_STRING, 1);
108871
108872	/* Fill in the zName and zDb fields of the vtab structure. */
108873	zCsr = (char *)&p->azColumn[nCol];
108874	p->zName = zCsr;
108875	memcpy(zCsr, argv[2], nName);
108876	zCsr += nName;
108877	p->zDb = zCsr;
108878	memcpy(zCsr, argv[1], nDb);
108879	zCsr += nDb;
108880
108881	/* Fill in the azColumn array */
108882	for(iCol=0; iCol<nCol; iCol++){
108883	char *z;
108884	int n;
108885	z = (char *)sqlite3Fts3NextToken(aCol[iCol], &n);
108886	memcpy(zCsr, z, n);
108887	zCsr[n] = '\0';
108888	sqlite3Fts3Dequote(zCsr);
108889	p->azColumn[iCol] = zCsr;
108890	zCsr += n+1;
108891	assert( zCsr <= &((char *)p)[nByte] );







108892	}
108893
108894	/* If this is an xCreate call, create the underlying tables in the
108895	** database. TODO: For xConnect(), it could verify that said tables exist.
108896	*/
108897	if( isCreate ){


108898	rc = fts3CreateTables(p);
108899	}
108900
108901	/* Figure out the page-size for the database. This is required in order to
108902	** estimate the cost of loading large doclists from the database (see
108903	** function sqlite3Fts3SegReaderCost() for details).
108904	*/
108905	fts3DatabasePageSize(&rc, p);
108906
108907	/* Declare the table schema to SQLite. */
108908	fts3DeclareVtab(&rc, p);

108909
108910	fts3_init_out:
108911
108912	sqlite3_free(aCol);
108913	if( rc!=SQLITE_OK ){
108914	if( p ){
108915	fts3DisconnectMethod((sqlite3_vtab *)p);
108916	}else if( pTokenizer ){
108917	pTokenizer->pModule->xDestroy(pTokenizer);
108918	}
108919	}else{
108920	*ppVTab = &p->base;
108921	}
108922	return rc;
108923	}
108924
108925	/*
	@@ -108323,14 +109027,16 @@
109027
109028	/*
109029	** Close the cursor. For additional information see the documentation
109030	** on the xClose method of the virtual table interface.
109031	*/
109032	static int fts3CloseMethod(sqlite3_vtab_cursor *pCursor){
109033	Fts3Cursor pCsr = (Fts3Cursor )pCursor;
109034	assert( ((Fts3Table *)pCsr->base.pVtab)->pSegments==0 );
109035	sqlite3_finalize(pCsr->pStmt);
109036	sqlite3Fts3ExprFree(pCsr->pExpr);
109037	sqlite3Fts3FreeDeferredTokens(pCsr);
109038	sqlite3_free(pCsr->aDoclist);
109039	sqlite3_free(pCsr->aMatchinfo);
109040	sqlite3_free(pCsr);
109041	return SQLITE_OK;
109042	}
	@@ -108365,134 +109071,186 @@
109071	return SQLITE_OK;
109072	}
109073	}
109074
109075	/*
109076	** This function is used to process a single interior node when searching
109077	** a b-tree for a term or term prefix. The node data is passed to this
109078	** function via the zNode/nNode parameters. The term to search for is
109079	** passed in zTerm/nTerm.

109080	**
109081	** If piFirst is not NULL, then this function sets *piFirst to the blockid
109082	** of the child node that heads the sub-tree that may contain the term.
109083	**
109084	** If piLast is not NULL, then *piLast is set to the right-most child node
109085	** that heads a sub-tree that may contain a term for which zTerm/nTerm is
109086	** a prefix.
109087	**
109088	** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK.
109089	*/
109090	static int fts3ScanInteriorNode(
109091	Fts3Table p, / Virtual table handle */
109092	const char zTerm, / Term to select leaves for */
109093	int nTerm, /* Size of term zTerm in bytes */
109094	const char zNode, / Buffer containing segment interior node */
109095	int nNode, /* Size of buffer at zNode */
109096	sqlite3_int64 piFirst, / OUT: Selected child node */
109097	sqlite3_int64 piLast / OUT: Selected child node */
109098	){
109099	int rc = SQLITE_OK; /* Return code */
109100	const char zCsr = zNode; / Cursor to iterate through node */
109101	const char zEnd = &zCsr[nNode];/ End of interior node buffer */
109102	char zBuffer = 0; / Buffer to load terms into */
109103	int nAlloc = 0; /* Size of allocated buffer */
109104	int isFirstTerm = 1; /* True when processing first term on page */
109105	sqlite3_int64 iChild; /* Block id of child node to descend to */
109106
109107	/* Skip over the 'height' varint that occurs at the start of every
109108	** interior node. Then load the blockid of the left-child of the b-tree
109109	** node into variable iChild.
109110	**
109111	** Even if the data structure on disk is corrupted, this (reading two
109112	** varints from the buffer) does not risk an overread. If zNode is a
109113	** root node, then the buffer comes from a SELECT statement. SQLite does
109114	** not make this guarantee explicitly, but in practice there are always
109115	** either more than 20 bytes of allocated space following the nNode bytes of
109116	** contents, or two zero bytes. Or, if the node is read from the %_segments
109117	** table, then there are always 20 bytes of zeroed padding following the
109118	** nNode bytes of content (see sqlite3Fts3ReadBlock() for details).
109119	*/
109120	zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
109121	zCsr += sqlite3Fts3GetVarint(zCsr, &iChild);
109122	if( zCsr>=zEnd ){
109123	return SQLITE_CORRUPT;
109124	}
109125
109126	while( zCsr<zEnd && (piFirst \|\| piLast) ){
109127	int cmp; /* memcmp() result */
109128	int nSuffix; /* Size of term suffix */
109129	int nPrefix = 0; /* Size of term prefix */
109130	int nBuffer; /* Total term size */
109131
109132	/* Load the next term on the node into zBuffer. Use realloc() to expand
109133	** the size of zBuffer if required. */
109134	if( !isFirstTerm ){
109135	zCsr += sqlite3Fts3GetVarint32(zCsr, &nPrefix);
109136	}
109137	isFirstTerm = 0;
109138	zCsr += sqlite3Fts3GetVarint32(zCsr, &nSuffix);
109139
109140	if( nPrefix<0 \|\| nSuffix<0 \|\| &zCsr[nSuffix]>zEnd ){
109141	rc = SQLITE_CORRUPT;
109142	goto finish_scan;
109143	}
109144	if( nPrefix+nSuffix>nAlloc ){
109145	char *zNew;
109146	nAlloc = (nPrefix+nSuffix) * 2;
109147	zNew = (char *)sqlite3_realloc(zBuffer, nAlloc);
109148	if( !zNew ){
109149	rc = SQLITE_NOMEM;
109150	goto finish_scan;
109151	}
109152	zBuffer = zNew;
109153	}
109154	memcpy(&zBuffer[nPrefix], zCsr, nSuffix);
109155	nBuffer = nPrefix + nSuffix;
109156	zCsr += nSuffix;
109157
109158	/* Compare the term we are searching for with the term just loaded from
109159	** the interior node. If the specified term is greater than or equal
109160	** to the term from the interior node, then all terms on the sub-tree
109161	** headed by node iChild are smaller than zTerm. No need to search
109162	** iChild.
109163	**
109164	** If the interior node term is larger than the specified term, then
109165	** the tree headed by iChild may contain the specified term.
109166	*/
109167	cmp = memcmp(zTerm, zBuffer, (nBuffer>nTerm ? nTerm : nBuffer));
109168	if( piFirst && (cmp<0 \|\| (cmp==0 && nBuffer>nTerm)) ){
109169	*piFirst = iChild;
109170	piFirst = 0;
109171	}
109172
109173	if( piLast && cmp<0 ){
109174	*piLast = iChild;
109175	piLast = 0;
109176	}
109177
109178	iChild++;
109179	};
109180
109181	if( piFirst ) *piFirst = iChild;
109182	if( piLast ) *piLast = iChild;
109183
109184	finish_scan:
109185	sqlite3_free(zBuffer);
109186	return rc;
109187	}
109188
109189
109190	/*
109191	** The buffer pointed to by argument zNode (size nNode bytes) contains an
109192	** interior node of a b-tree segment. The zTerm buffer (size nTerm bytes)
109193	** contains a term. This function searches the sub-tree headed by the zNode
109194	** node for the range of leaf nodes that may contain the specified term
109195	** or terms for which the specified term is a prefix.
109196	**
109197	** If piLeaf is not NULL, then *piLeaf is set to the blockid of the
109198	** left-most leaf node in the tree that may contain the specified term.
109199	** If piLeaf2 is not NULL, then *piLeaf2 is set to the blockid of the
109200	** right-most leaf node that may contain a term for which the specified
109201	** term is a prefix.
109202	**
109203	** It is possible that the range of returned leaf nodes does not contain
109204	** the specified term or any terms for which it is a prefix. However, if the
109205	** segment does contain any such terms, they are stored within the identified
109206	** range. Because this function only inspects interior segment nodes (and
109207	** never loads leaf nodes into memory), it is not possible to be sure.
109208	**
109209	** If an error occurs, an error code other than SQLITE_OK is returned.
109210	*/
109211	static int fts3SelectLeaf(
109212	Fts3Table p, / Virtual table handle */
109213	const char zTerm, / Term to select leaves for */
109214	int nTerm, /* Size of term zTerm in bytes */
109215	const char zNode, / Buffer containing segment interior node */
109216	int nNode, /* Size of buffer at zNode */
109217	sqlite3_int64 piLeaf, / Selected leaf node */
109218	sqlite3_int64 piLeaf2 / Selected leaf node */
109219	){
109220	int rc; /* Return code */
109221	int iHeight; /* Height of this node in tree */
109222
109223	assert( piLeaf \|\| piLeaf2 );
109224
109225	sqlite3Fts3GetVarint32(zNode, &iHeight);
109226	rc = fts3ScanInteriorNode(p, zTerm, nTerm, zNode, nNode, piLeaf, piLeaf2);
109227	assert( !piLeaf2 \|\| !piLeaf \|\| rc!=SQLITE_OK \|\| (piLeaf<=piLeaf2) );
109228
109229	if( rc==SQLITE_OK && iHeight>1 ){
109230	char zBlob = 0; / Blob read from %_segments table */
109231	int nBlob; /* Size of zBlob in bytes */
109232
109233	if( piLeaf && piLeaf2 && (piLeaf!=piLeaf2) ){
109234	rc = sqlite3Fts3ReadBlock(p, *piLeaf, &zBlob, &nBlob);
109235	if( rc==SQLITE_OK ){
109236	rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, 0);
109237	}
109238	sqlite3_free(zBlob);
109239	piLeaf = 0;
109240	zBlob = 0;
109241	}
109242
109243	if( rc==SQLITE_OK ){
109244	rc = sqlite3Fts3ReadBlock(p, piLeaf ? piLeaf : piLeaf2, &zBlob, &nBlob);
109245	}
109246	if( rc==SQLITE_OK ){
109247	rc = fts3SelectLeaf(p, zTerm, nTerm, zBlob, nBlob, piLeaf, piLeaf2);
109248	}
109249	sqlite3_free(zBlob);
109250	}
109251




































109252	return rc;
109253	}
109254
109255	/*
109256	** This function is used to create delta-encoded serialized lists of FTS3
	@@ -108720,24 +109478,48 @@
109478	*pp2 = p2 + 1;
109479	}
109480
109481	/*
109482	** nToken==1 searches for adjacent positions.
109483	**
109484	** This function is used to merge two position lists into one. When it is
109485	** called, pp1 and pp2 must both point to position lists. A position-list is
109486	** the part of a doclist that follows each document id. For example, if a row
109487	** contains:
109488	**
109489	** 'a b c'\|'x y z'\|'a b b a'
109490	**
109491	** Then the position list for this row for token 'b' would consist of:
109492	**
109493	** 0x02 0x01 0x02 0x03 0x03 0x00
109494	**
109495	** When this function returns, both pp1 and pp2 are left pointing to the
109496	** byte following the 0x00 terminator of their respective position lists.
109497	**
109498	** If isSaveLeft is 0, an entry is added to the output position list for
109499	** each position in *pp2 for which there exists one or more positions in
109500	** pp1 so that (pos(pp2)>pos(pp1) && pos(pp2)-pos(*pp1)<=nToken). i.e.
109501	** when the pp1 token appears before the pp2 token, but not more than nToken
109502	** slots before it.
109503	*/
109504	static int fts3PoslistPhraseMerge(
109505	char *pp, / IN/OUT: Preallocated output buffer */
109506	int nToken, /* Maximum difference in token positions */
109507	int isSaveLeft, /* Save the left position */
109508	int isExact, /* If pp1 is exactly nTokens before pp2 */
109509	char *pp1, / IN/OUT: Left input list */
109510	char *pp2 / IN/OUT: Right input list */
109511	){
109512	char p = (pp ? pp : 0);
109513	char p1 = pp1;
109514	char p2 = pp2;

109515	int iCol1 = 0;
109516	int iCol2 = 0;
109517
109518	/* Never set both isSaveLeft and isExact for the same invocation. */
109519	assert( isSaveLeft==0 \|\| isExact==0 );
109520
109521	assert( p1!=0 && p2!=0 );
109522	if( *p1==POS_COLUMN ){
109523	p1++;
109524	p1 += sqlite3Fts3GetVarint32(p1, &iCol1);
109525	}
	@@ -108762,11 +109544,13 @@
109544	assert( p2!=POS_END && p2!=POS_COLUMN );
109545	fts3GetDeltaVarint(&p1, &iPos1); iPos1 -= 2;
109546	fts3GetDeltaVarint(&p2, &iPos2); iPos2 -= 2;
109547
109548	while( 1 ){
109549	if( iPos2==iPos1+nToken
109550	\|\| (isExact==0 && iPos2>iPos1 && iPos2<=iPos1+nToken)
109551	){
109552	sqlite3_int64 iSave;
109553	if( !pp ){
109554	fts3PoslistCopy(0, &p2);
109555	fts3PoslistCopy(0, &p1);
109556	*pp1 = p1;
	@@ -108845,25 +109629,25 @@
109629	){
109630	char p1 = pp1;
109631	char p2 = pp2;
109632
109633	if( !pp ){
109634	if( fts3PoslistPhraseMerge(0, nRight, 0, 0, pp1, pp2) ) return 1;
109635	*pp1 = p1;
109636	*pp2 = p2;
109637	return fts3PoslistPhraseMerge(0, nLeft, 0, 0, pp2, pp1);
109638	}else{
109639	char *pTmp1 = aTmp;
109640	char *pTmp2;
109641	char *aTmp2;
109642	int res = 1;
109643
109644	fts3PoslistPhraseMerge(&pTmp1, nRight, 0, 0, pp1, pp2);
109645	aTmp2 = pTmp2 = pTmp1;
109646	*pp1 = p1;
109647	*pp2 = p2;
109648	fts3PoslistPhraseMerge(&pTmp2, nLeft, 1, 0, pp2, pp1);
109649	if( pTmp1!=aTmp && pTmp2!=aTmp2 ){
109650	fts3PoslistMerge(pp, &aTmp, &aTmp2);
109651	}else if( pTmp1!=aTmp ){
109652	fts3PoslistCopy(pp, &aTmp);
109653	}else if( pTmp2!=aTmp2 ){
	@@ -108905,11 +109689,12 @@
109689	char aBuffer, / Pre-allocated output buffer */
109690	int pnBuffer, / OUT: Bytes written to aBuffer */
109691	char a1, / Buffer containing first doclist */
109692	int n1, /* Size of buffer a1 */
109693	char a2, / Buffer containing second doclist */
109694	int n2, /* Size of buffer a2 */
109695	int pnDoc / OUT: Number of docids in output */
109696	){
109697	sqlite3_int64 i1 = 0;
109698	sqlite3_int64 i2 = 0;
109699	sqlite3_int64 iPrev = 0;
109700
	@@ -108916,10 +109701,11 @@
109701	char *p = aBuffer;
109702	char *p1 = a1;
109703	char *p2 = a2;
109704	char *pEnd1 = &a1[n1];
109705	char *pEnd2 = &a2[n2];
109706	int nDoc = 0;
109707
109708	assert( mergetype==MERGE_OR \|\| mergetype==MERGE_POS_OR
109709	\|\| mergetype==MERGE_AND \|\| mergetype==MERGE_NOT
109710	\|\| mergetype==MERGE_PHRASE \|\| mergetype==MERGE_POS_PHRASE
109711	\|\| mergetype==MERGE_NEAR \|\| mergetype==MERGE_POS_NEAR
	@@ -108959,10 +109745,11 @@
109745	while( p1 && p2 ){
109746	if( i1==i2 ){
109747	fts3PutDeltaVarint(&p, &iPrev, i1);
109748	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
109749	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
109750	nDoc++;
109751	}else if( i1<i2 ){
109752	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
109753	}else{
109754	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
109755	}
	@@ -108989,13 +109776,15 @@
109776	while( p1 && p2 ){
109777	if( i1==i2 ){
109778	char *pSave = p;
109779	sqlite3_int64 iPrevSave = iPrev;
109780	fts3PutDeltaVarint(&p, &iPrev, i1);
109781	if( 0==fts3PoslistPhraseMerge(ppPos, nParam1, 0, 1, &p1, &p2) ){
109782	p = pSave;
109783	iPrev = iPrevSave;
109784	}else{
109785	nDoc++;
109786	}
109787	fts3GetDeltaVarint2(&p1, pEnd1, &i1);
109788	fts3GetDeltaVarint2(&p2, pEnd2, &i2);
109789	}else if( i1<i2 ){
109790	fts3PoslistCopy(0, &p1);
	@@ -109044,10 +109833,11 @@
109833	sqlite3_free(aTmp);
109834	break;
109835	}
109836	}
109837
109838	if( pnDoc ) *pnDoc = nDoc;
109839	*pnBuffer = (int)(p-aBuffer);
109840	return SQLITE_OK;
109841	}
109842
109843	/*
	@@ -109082,20 +109872,20 @@
109872	for(i=0; i<SizeofArray(pTS->aaOutput); i++){
109873	if( pTS->aaOutput[i] ){
109874	if( !aOut ){
109875	aOut = pTS->aaOutput[i];
109876	nOut = pTS->anOutput[i];
109877	pTS->aaOutput[i] = 0;
109878	}else{
109879	int nNew = nOut + pTS->anOutput[i];
109880	char *aNew = sqlite3_malloc(nNew);
109881	if( !aNew ){
109882	sqlite3_free(aOut);
109883	return SQLITE_NOMEM;
109884	}
109885	fts3DoclistMerge(mergetype, 0, 0,
109886	aNew, &nNew, pTS->aaOutput[i], pTS->anOutput[i], aOut, nOut, 0
109887	);
109888	sqlite3_free(pTS->aaOutput[i]);
109889	sqlite3_free(aOut);
109890	pTS->aaOutput[i] = 0;
109891	aOut = aNew;
	@@ -109162,12 +109952,12 @@
109952	if( aMerge!=aDoclist ){
109953	sqlite3_free(aMerge);
109954	}
109955	return SQLITE_NOMEM;
109956	}
109957	fts3DoclistMerge(mergetype, 0, 0, aNew, &nNew,
109958	pTS->aaOutput[iOut], pTS->anOutput[iOut], aMerge, nMerge, 0
109959	);
109960
109961	if( iOut>0 ) sqlite3_free(aMerge);
109962	sqlite3_free(pTS->aaOutput[iOut]);
109963	pTS->aaOutput[iOut] = 0;
	@@ -109180,10 +109970,165 @@
109970	}
109971	}
109972	}
109973	return SQLITE_OK;
109974	}
109975
109976	static int fts3DeferredTermSelect(
109977	Fts3DeferredToken pToken, / Phrase token */
109978	int isTermPos, /* True to include positions */
109979	int pnOut, / OUT: Size of list */
109980	char *ppOut / OUT: Body of list */
109981	){
109982	char *aSource;
109983	int nSource;
109984
109985	aSource = sqlite3Fts3DeferredDoclist(pToken, &nSource);
109986	if( !aSource ){
109987	*pnOut = 0;
109988	*ppOut = 0;
109989	}else if( isTermPos ){
109990	*ppOut = sqlite3_malloc(nSource);
109991	if( !*ppOut ) return SQLITE_NOMEM;
109992	memcpy(*ppOut, aSource, nSource);
109993	*pnOut = nSource;
109994	}else{
109995	sqlite3_int64 docid;
109996	*pnOut = sqlite3Fts3GetVarint(aSource, &docid);
109997	ppOut = sqlite3_malloc(pnOut);
109998	if( !*ppOut ) return SQLITE_NOMEM;
109999	sqlite3Fts3PutVarint(*ppOut, docid);
110000	}
110001
110002	return SQLITE_OK;
110003	}
110004
110005	/*
110006	** An Fts3SegReaderArray is used to store an array of Fts3SegReader objects.
110007	** Elements are added to the array using fts3SegReaderArrayAdd().
110008	*/
110009	struct Fts3SegReaderArray {
110010	int nSegment; /* Number of valid entries in apSegment[] */
110011	int nAlloc; /* Allocated size of apSegment[] */
110012	int nCost; /* The cost of executing SegReaderIterate() */
110013	Fts3SegReader apSegment[1]; / Array of seg-reader objects */
110014	};
110015
110016
110017	/*
110018	** Free an Fts3SegReaderArray object. Also free all seg-readers in the
110019	** array (using sqlite3Fts3SegReaderFree()).
110020	*/
110021	static void fts3SegReaderArrayFree(Fts3SegReaderArray *pArray){
110022	if( pArray ){
110023	int i;
110024	for(i=0; i<pArray->nSegment; i++){
110025	sqlite3Fts3SegReaderFree(0, pArray->apSegment[i]);
110026	}
110027	sqlite3_free(pArray);
110028	}
110029	}
110030
110031	static int fts3SegReaderArrayAdd(
110032	Fts3SegReaderArray **ppArray,
110033	Fts3SegReader *pNew
110034	){
110035	Fts3SegReaderArray pArray = ppArray;
110036
110037	if( !pArray \|\| pArray->nAlloc==pArray->nSegment ){
110038	int nNew = (pArray ? pArray->nAlloc+16 : 16);
110039	pArray = (Fts3SegReaderArray *)sqlite3_realloc(pArray,
110040	sizeof(Fts3SegReaderArray) + (nNew-1) * sizeof(Fts3SegReader*)
110041	);
110042	if( !pArray ){
110043	sqlite3Fts3SegReaderFree(0, pNew);
110044	return SQLITE_NOMEM;
110045	}
110046	if( nNew==16 ){
110047	pArray->nSegment = 0;
110048	pArray->nCost = 0;
110049	}
110050	pArray->nAlloc = nNew;
110051	*ppArray = pArray;
110052	}
110053
110054	pArray->apSegment[pArray->nSegment++] = pNew;
110055	return SQLITE_OK;
110056	}
110057
110058	static int fts3TermSegReaderArray(
110059	Fts3Cursor pCsr, / Virtual table cursor handle */
110060	const char zTerm, / Term to query for */
110061	int nTerm, /* Size of zTerm in bytes */
110062	int isPrefix, /* True for a prefix search */
110063	Fts3SegReaderArray *ppArray / OUT: Allocated seg-reader array */
110064	){
110065	Fts3Table p = (Fts3Table )pCsr->base.pVtab;
110066	int rc; /* Return code */
110067	Fts3SegReaderArray pArray = 0; / Array object to build */
110068	Fts3SegReader pReader = 0; / Seg-reader to add to pArray */
110069	sqlite3_stmt pStmt = 0; / SQL statement to scan %_segdir table */
110070	int iAge = 0; /* Used to assign ages to segments */
110071
110072	/* Allocate a seg-reader to scan the pending terms, if any. */
110073	rc = sqlite3Fts3SegReaderPending(p, zTerm, nTerm, isPrefix, &pReader);
110074	if( rc==SQLITE_OK && pReader ) {
110075	rc = fts3SegReaderArrayAdd(&pArray, pReader);
110076	}
110077
110078	/* Loop through the entire %_segdir table. For each segment, create a
110079	** Fts3SegReader to iterate through the subset of the segment leaves
110080	** that may contain a term that matches zTerm/nTerm. For non-prefix
110081	** searches, this is always a single leaf. For prefix searches, this
110082	** may be a contiguous block of leaves.
110083	*/
110084	if( rc==SQLITE_OK ){
110085	rc = sqlite3Fts3AllSegdirs(p, &pStmt);
110086	}
110087	while( rc==SQLITE_OK && SQLITE_ROW==(rc = sqlite3_step(pStmt)) ){
110088	Fts3SegReader *pNew = 0;
110089	int nRoot = sqlite3_column_bytes(pStmt, 4);
110090	char const *zRoot = sqlite3_column_blob(pStmt, 4);
110091	if( sqlite3_column_int64(pStmt, 1)==0 ){
110092	/* The entire segment is stored on the root node (which must be a
110093	** leaf). Do not bother inspecting any data in this case, just
110094	** create a Fts3SegReader to scan the single leaf.
110095	*/
110096	rc = sqlite3Fts3SegReaderNew(p, iAge, 0, 0, 0, zRoot, nRoot, &pNew);
110097	}else{
110098	sqlite3_int64 i1; /* First leaf that may contain zTerm */
110099	sqlite3_int64 i2; /* Final leaf that may contain zTerm */
110100	rc = fts3SelectLeaf(p, zTerm, nTerm, zRoot, nRoot, &i1, (isPrefix?&i2:0));
110101	if( isPrefix==0 ) i2 = i1;
110102	if( rc==SQLITE_OK ){
110103	rc = sqlite3Fts3SegReaderNew(p, iAge, i1, i2, 0, 0, 0, &pNew);
110104	}
110105	}
110106	assert( (pNew==0)==(rc!=SQLITE_OK) );
110107
110108	/* If a new Fts3SegReader was allocated, add it to the array. */
110109	if( rc==SQLITE_OK ){
110110	rc = fts3SegReaderArrayAdd(&pArray, pNew);
110111	}
110112	if( rc==SQLITE_OK ){
110113	rc = sqlite3Fts3SegReaderCost(pCsr, pNew, &pArray->nCost);
110114	}
110115	iAge++;
110116	}
110117
110118	if( rc==SQLITE_DONE ){
110119	rc = sqlite3_reset(pStmt);
110120	}else{
110121	sqlite3_reset(pStmt);
110122	}
110123	if( rc!=SQLITE_OK ){
110124	fts3SegReaderArrayFree(pArray);
110125	pArray = 0;
110126	}
110127	*ppArray = pArray;
110128	return rc;
110129	}
110130
110131	/*
110132	** This function retreives the doclist for the specified term (or term
110133	** prefix) from the database.
110134	**
	@@ -109194,145 +110139,150 @@
110139	** in the database without the found length specifier at the start of on-disk
110140	** doclists.
110141	*/
110142	static int fts3TermSelect(
110143	Fts3Table p, / Virtual table handle */
110144	Fts3PhraseToken pTok, / Token to query for */
110145	int iColumn, /* Column to query (or -ve for all columns) */



110146	int isReqPos, /* True to include position lists in output */
110147	int pnOut, / OUT: Size of buffer at ppOut /
110148	char *ppOut / OUT: Malloced result buffer */
110149	){






110150	int rc; /* Return code */
110151	Fts3SegReaderArray pArray; / Seg-reader array for this term */
110152	TermSelect tsc; /* Context object for fts3TermSelectCb() */
110153	Fts3SegFilter filter; /* Segment term filter configuration */
110154
110155	pArray = pTok->pArray;









































































110156	memset(&tsc, 0, sizeof(TermSelect));
110157	tsc.isReqPos = isReqPos;
110158
110159	filter.flags = FTS3_SEGMENT_IGNORE_EMPTY
110160	\| (pTok->isPrefix ? FTS3_SEGMENT_PREFIX : 0)
110161	\| (isReqPos ? FTS3_SEGMENT_REQUIRE_POS : 0)
110162	\| (iColumn<p->nColumn ? FTS3_SEGMENT_COLUMN_FILTER : 0);
110163	filter.iCol = iColumn;
110164	filter.zTerm = pTok->z;
110165	filter.nTerm = pTok->n;
110166
110167	rc = sqlite3Fts3SegReaderIterate(p, pArray->apSegment, pArray->nSegment,
110168	&filter, fts3TermSelectCb, (void *)&tsc
110169	);
110170	if( rc==SQLITE_OK ){
110171	rc = fts3TermSelectMerge(&tsc);
110172	}
110173
110174	if( rc==SQLITE_OK ){
110175	*ppOut = tsc.aaOutput[0];
110176	*pnOut = tsc.anOutput[0];
110177	}else{
110178	int i;
110179	for(i=0; i<SizeofArray(tsc.aaOutput); i++){
110180	sqlite3_free(tsc.aaOutput[i]);
110181	}
110182	}
110183
110184	fts3SegReaderArrayFree(pArray);
110185	pTok->pArray = 0;




110186	return rc;
110187	}
110188
110189	/*
110190	** This function counts the total number of docids in the doclist stored
110191	** in buffer aList[], size nList bytes.
110192	**
110193	** If the isPoslist argument is true, then it is assumed that the doclist
110194	** contains a position-list following each docid. Otherwise, it is assumed
110195	** that the doclist is simply a list of docids stored as delta encoded
110196	** varints.
110197	*/
110198	static int fts3DoclistCountDocids(int isPoslist, char *aList, int nList){
110199	int nDoc = 0; /* Return value */
110200	if( aList ){
110201	char aEnd = &aList[nList]; / Pointer to one byte after EOF */
110202	char p = aList; / Cursor */
110203	if( !isPoslist ){
110204	/* The number of docids in the list is the same as the number of
110205	** varints. In FTS3 a varint consists of a single byte with the 0x80
110206	** bit cleared and zero or more bytes with the 0x80 bit set. So to
110207	** count the varints in the buffer, just count the number of bytes
110208	** with the 0x80 bit clear. */
110209	while( p<aEnd ) nDoc += (((*p++)&0x80)==0);
110210	}else{
110211	while( p<aEnd ){
110212	nDoc++;
110213	while( (p++)&0x80 ); / Skip docid varint */
110214	fts3PoslistCopy(0, &p); /* Skip over position list */
110215	}
110216	}
110217	}
110218
110219	return nDoc;
110220	}
110221
110222	/*
110223	** Call sqlite3Fts3DeferToken() for each token in the expression pExpr.
110224	*/
110225	static int fts3DeferExpression(Fts3Cursor pCsr, Fts3Expr pExpr){
110226	int rc = SQLITE_OK;
110227	if( pExpr ){
110228	rc = fts3DeferExpression(pCsr, pExpr->pLeft);
110229	if( rc==SQLITE_OK ){
110230	rc = fts3DeferExpression(pCsr, pExpr->pRight);
110231	}
110232	if( pExpr->eType==FTSQUERY_PHRASE ){
110233	int iCol = pExpr->pPhrase->iColumn;
110234	int i;
110235	for(i=0; rc==SQLITE_OK && i<pExpr->pPhrase->nToken; i++){
110236	Fts3PhraseToken *pToken = &pExpr->pPhrase->aToken[i];
110237	if( pToken->pDeferred==0 ){
110238	rc = sqlite3Fts3DeferToken(pCsr, pToken, iCol);
110239	}
110240	}
110241	}
110242	}
110243	return rc;
110244	}
110245
110246	/*
110247	** This function removes the position information from a doclist. When
110248	** called, buffer aList (size *pnList bytes) contains a doclist that includes
110249	** position information. This function removes the position information so
110250	** that aList contains only docids, and adjusts *pnList to reflect the new
110251	** (possibly reduced) size of the doclist.
110252	*/
110253	static void fts3DoclistStripPositions(
110254	char aList, / IN/OUT: Buffer containing doclist */
110255	int pnList / IN/OUT: Size of doclist in bytes */
110256	){
110257	if( aList ){
110258	char aEnd = &aList[pnList]; /* Pointer to one byte after EOF */
110259	char p = aList; / Input cursor */
110260	char pOut = aList; / Output cursor */
110261
110262	while( p<aEnd ){
110263	sqlite3_int64 delta;
110264	p += sqlite3Fts3GetVarint(p, &delta);
110265	fts3PoslistCopy(0, &p);
110266	pOut += sqlite3Fts3PutVarint(pOut, delta);
110267	}
110268
110269	*pnList = (pOut - aList);
110270	}
110271	}
110272
110273	/*
110274	** Return a DocList corresponding to the phrase *pPhrase.
110275	**
110276	** If this function returns SQLITE_OK, but *pnOut is set to a negative value,
110277	** then no tokens in the phrase were looked up in the full-text index. This
110278	** is only possible when this function is called from within xFilter(). The
110279	** caller should assume that all documents match the phrase. The actual
110280	** filtering will take place in xNext().
110281	*/
110282	static int fts3PhraseSelect(
110283	Fts3Cursor pCsr, / Virtual table cursor handle */
110284	Fts3Phrase pPhrase, / Phrase to return a doclist for */
110285	int isReqPos, /* True if output should contain positions */
110286	char *paOut, / OUT: Pointer to malloc'd result buffer */
110287	int pnOut / OUT: Size of buffer at paOut /
110288	){
	@@ -109340,54 +110290,156 @@
110290	int nOut = 0;
110291	int rc = SQLITE_OK;
110292	int ii;
110293	int iCol = pPhrase->iColumn;
110294	int isTermPos = (pPhrase->nToken>1 \|\| isReqPos);
110295	Fts3Table p = (Fts3Table )pCsr->base.pVtab;
110296	int isFirst = 1;
110297
110298	int iPrevTok = 0;
110299	int nDoc = 0;
110300
110301	/* If this is an xFilter() evaluation, create a segment-reader for each
110302	** phrase token. Or, if this is an xNext() or snippet/offsets/matchinfo
110303	** evaluation, only create segment-readers if there are no Fts3DeferredToken
110304	** objects attached to the phrase-tokens.
110305	*/
110306	for(ii=0; ii<pPhrase->nToken; ii++){
110307	Fts3PhraseToken *pTok = &pPhrase->aToken[ii];
110308	if( pTok->pArray==0 ){
110309	if( (pCsr->eEvalmode==FTS3_EVAL_FILTER)
110310	\|\| (pCsr->eEvalmode==FTS3_EVAL_NEXT && pCsr->pDeferred==0)
110311	\|\| (pCsr->eEvalmode==FTS3_EVAL_MATCHINFO && pTok->bFulltext)
110312	){
110313	rc = fts3TermSegReaderArray(
110314	pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pArray
110315	);
110316	if( rc!=SQLITE_OK ) return rc;
110317	}
110318	}
110319	}
110320
110321	for(ii=0; ii<pPhrase->nToken; ii++){
110322	Fts3PhraseToken pTok; / Token to find doclist for */
110323	int iTok; /* The token being queried this iteration */


110324	char pList; / Pointer to token doclist */
110325	int nList; /* Size of buffer at pList */
110326
110327	/* Select a token to process. If this is an xFilter() call, then tokens
110328	** are processed in order from least to most costly. Otherwise, tokens
110329	** are processed in the order in which they occur in the phrase.
110330	*/
110331	if( pCsr->eEvalmode==FTS3_EVAL_MATCHINFO ){
110332	assert( isReqPos );
110333	iTok = ii;
110334	pTok = &pPhrase->aToken[iTok];
110335	if( pTok->bFulltext==0 ) continue;
110336	}else if( pCsr->eEvalmode==FTS3_EVAL_NEXT \|\| isReqPos ){
110337	iTok = ii;
110338	pTok = &pPhrase->aToken[iTok];
110339	}else{
110340	int nMinCost = 0x7FFFFFFF;
110341	int jj;
110342
110343	/* Find the remaining token with the lowest cost. */
110344	for(jj=0; jj<pPhrase->nToken; jj++){
110345	Fts3SegReaderArray *pArray = pPhrase->aToken[jj].pArray;
110346	if( pArray && pArray->nCost<nMinCost ){
110347	iTok = jj;
110348	nMinCost = pArray->nCost;
110349	}
110350	}
110351	pTok = &pPhrase->aToken[iTok];
110352
110353	/* This branch is taken if it is determined that loading the doclist
110354	** for the next token would require more IO than loading all documents
110355	** currently identified by doclist pOut/nOut. No further doclists will
110356	** be loaded from the full-text index for this phrase.
110357	*/
110358	if( nMinCost>nDoc && ii>0 ){
110359	rc = fts3DeferExpression(pCsr, pCsr->pExpr);
110360	break;
110361	}
110362	}
110363
110364	if( pCsr->eEvalmode==FTS3_EVAL_NEXT && pTok->pDeferred ){
110365	rc = fts3DeferredTermSelect(pTok->pDeferred, isTermPos, &nList, &pList);
110366	}else{
110367	assert( pTok->pArray );
110368	rc = fts3TermSelect(p, pTok, iCol, isTermPos, &nList, &pList);
110369	pTok->bFulltext = 1;
110370	}
110371	assert( rc!=SQLITE_OK \|\| pCsr->eEvalmode \|\| pTok->pArray==0 );
110372	if( rc!=SQLITE_OK ) break;
110373
110374	if( isFirst ){
110375	pOut = pList;
110376	nOut = nList;
110377	if( pCsr->eEvalmode==FTS3_EVAL_FILTER && pPhrase->nToken>1 ){
110378	nDoc = fts3DoclistCountDocids(1, pOut, nOut);
110379	}
110380	isFirst = 0;
110381	iPrevTok = iTok;
110382	}else{
110383	/* Merge the new term list and the current output. */
110384	char aLeft, aRight;
110385	int nLeft, nRight;
110386	int nDist;
110387	int mt;
110388
110389	/* If this is the final token of the phrase, and positions were not
110390	** requested by the caller, use MERGE_PHRASE instead of POS_PHRASE.
110391	** This drops the position information from the output list.
110392	*/
110393	mt = MERGE_POS_PHRASE;
110394	if( ii==pPhrase->nToken-1 && !isReqPos ) mt = MERGE_PHRASE;
110395
110396	assert( iPrevTok!=iTok );
110397	if( iPrevTok<iTok ){
110398	aLeft = pOut;
110399	nLeft = nOut;
110400	aRight = pList;
110401	nRight = nList;
110402	nDist = iTok-iPrevTok;
110403	iPrevTok = iTok;
110404	}else{
110405	aRight = pOut;
110406	nRight = nOut;
110407	aLeft = pList;
110408	nLeft = nList;
110409	nDist = iPrevTok-iTok;
110410	}
110411	pOut = aRight;
110412	fts3DoclistMerge(
110413	mt, nDist, 0, pOut, &nOut, aLeft, nLeft, aRight, nRight, &nDoc
110414	);
110415	sqlite3_free(aLeft);
110416	}
110417	assert( nOut==0 \|\| pOut!=0 );
110418	}
110419
110420	if( rc==SQLITE_OK ){
110421	if( ii!=pPhrase->nToken ){
110422	assert( pCsr->eEvalmode==FTS3_EVAL_FILTER && isReqPos==0 );
110423	fts3DoclistStripPositions(pOut, &nOut);
110424	}
110425	*paOut = pOut;
110426	*pnOut = nOut;
110427	}else{
110428	sqlite3_free(pOut);
110429	}
110430	return rc;
110431	}
110432
110433	/*
110434	** This function merges two doclists according to the requirements of a
110435	** NEAR operator.
110436	**
110437	** Both input doclists must include position information. The output doclist
110438	** includes position information if the first argument to this function
110439	** is MERGE_POS_NEAR, or does not if it is MERGE_NEAR.
110440	*/
110441	static int fts3NearMerge(
110442	int mergetype, /* MERGE_POS_NEAR or MERGE_NEAR */
110443	int nNear, /* Parameter to NEAR operator */
110444	int nTokenLeft, /* Number of tokens in LHS phrase arg */
110445	char aLeft, / Doclist for LHS (incl. positions) */
	@@ -109396,21 +110448,21 @@
110448	char aRight, / As aLeft */
110449	int nRight, /* As nRight */
110450	char *paOut, / OUT: Results of merge (malloced) */
110451	int pnOut / OUT: Sized of output buffer */
110452	){
110453	char aOut; / Buffer to write output doclist to */
110454	int rc; /* Return code */
110455
110456	assert( mergetype==MERGE_POS_NEAR \|\| MERGE_NEAR );
110457
110458	aOut = sqlite3_malloc(nLeft+nRight+1);
110459	if( aOut==0 ){
110460	rc = SQLITE_NOMEM;
110461	}else{
110462	rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft,
110463	aOut, pnOut, aLeft, nLeft, aRight, nRight, 0
110464	);
110465	if( rc!=SQLITE_OK ){
110466	sqlite3_free(aOut);
110467	aOut = 0;
110468	}
	@@ -109418,21 +110470,36 @@
110470
110471	*paOut = aOut;
110472	return rc;
110473	}
110474
110475	/*
110476	** This function is used as part of the processing for the snippet() and
110477	** offsets() functions.
110478	**
110479	** Both pLeft and pRight are expression nodes of type FTSQUERY_PHRASE. Both
110480	** have their respective doclists (including position information) loaded
110481	** in Fts3Expr.aDoclist/nDoclist. This function removes all entries from
110482	** each doclist that are not within nNear tokens of a corresponding entry
110483	** in the other doclist.
110484	*/
110485	SQLITE_PRIVATE int sqlite3Fts3ExprNearTrim(Fts3Expr pLeft, Fts3Expr pRight, int nNear){
110486	int rc; /* Return code */
110487
110488	assert( pLeft->eType==FTSQUERY_PHRASE );
110489	assert( pRight->eType==FTSQUERY_PHRASE );
110490	assert( pLeft->isLoaded && pRight->isLoaded );
110491
110492	if( pLeft->aDoclist==0 \|\| pRight->aDoclist==0 ){
110493	sqlite3_free(pLeft->aDoclist);
110494	sqlite3_free(pRight->aDoclist);
110495	pRight->aDoclist = 0;
110496	pLeft->aDoclist = 0;
110497	rc = SQLITE_OK;
110498	}else{
110499	char aOut; / Buffer in which to assemble new doclist */
110500	int nOut; /* Size of buffer aOut in bytes */
110501
110502	rc = fts3NearMerge(MERGE_POS_NEAR, nNear,
110503	pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
110504	pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
110505	&aOut, &nOut
	@@ -109451,19 +110518,158 @@
110518	pLeft->aDoclist = aOut;
110519	pLeft->nDoclist = nOut;
110520	}
110521	return rc;
110522	}
110523
110524
110525	/*
110526	** Allocate an Fts3SegReaderArray for each token in the expression pExpr.
110527	** The allocated objects are stored in the Fts3PhraseToken.pArray member
110528	** variables of each token structure.
110529	*/
110530	static int fts3ExprAllocateSegReaders(
110531	Fts3Cursor pCsr, / FTS3 table */
110532	Fts3Expr pExpr, / Expression to create seg-readers for */
110533	int pnExpr / OUT: Number of AND'd expressions */
110534	){
110535	int rc = SQLITE_OK; /* Return code */
110536
110537	assert( pCsr->eEvalmode==FTS3_EVAL_FILTER );
110538	if( pnExpr && pExpr->eType!=FTSQUERY_AND ){
110539	(*pnExpr)++;
110540	pnExpr = 0;
110541	}
110542
110543	if( pExpr->eType==FTSQUERY_PHRASE ){
110544	Fts3Phrase *pPhrase = pExpr->pPhrase;
110545	int ii;
110546
110547	for(ii=0; rc==SQLITE_OK && ii<pPhrase->nToken; ii++){
110548	Fts3PhraseToken *pTok = &pPhrase->aToken[ii];
110549	if( pTok->pArray==0 ){
110550	rc = fts3TermSegReaderArray(
110551	pCsr, pTok->z, pTok->n, pTok->isPrefix, &pTok->pArray
110552	);
110553	}
110554	}
110555	}else{
110556	rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pLeft, pnExpr);
110557	if( rc==SQLITE_OK ){
110558	rc = fts3ExprAllocateSegReaders(pCsr, pExpr->pRight, pnExpr);
110559	}
110560	}
110561	return rc;
110562	}
110563
110564	/*
110565	** Free the Fts3SegReaderArray objects associated with each token in the
110566	** expression pExpr. In other words, this function frees the resources
110567	** allocated by fts3ExprAllocateSegReaders().
110568	*/
110569	static void fts3ExprFreeSegReaders(Fts3Expr *pExpr){
110570	if( pExpr ){
110571	Fts3Phrase *pPhrase = pExpr->pPhrase;
110572	if( pPhrase ){
110573	int kk;
110574	for(kk=0; kk<pPhrase->nToken; kk++){
110575	fts3SegReaderArrayFree(pPhrase->aToken[kk].pArray);
110576	pPhrase->aToken[kk].pArray = 0;
110577	}
110578	}
110579	fts3ExprFreeSegReaders(pExpr->pLeft);
110580	fts3ExprFreeSegReaders(pExpr->pRight);
110581	}
110582	}
110583
110584	/*
110585	** Return the sum of the costs of all tokens in the expression pExpr. This
110586	** function must be called after Fts3SegReaderArrays have been allocated
110587	** for all tokens using fts3ExprAllocateSegReaders().
110588	*/
110589	int fts3ExprCost(Fts3Expr *pExpr){
110590	int nCost; /* Return value */
110591	if( pExpr->eType==FTSQUERY_PHRASE ){
110592	Fts3Phrase *pPhrase = pExpr->pPhrase;
110593	int ii;
110594	nCost = 0;
110595	for(ii=0; ii<pPhrase->nToken; ii++){
110596	nCost += pPhrase->aToken[ii].pArray->nCost;
110597	}
110598	}else{
110599	nCost = fts3ExprCost(pExpr->pLeft) + fts3ExprCost(pExpr->pRight);
110600	}
110601	return nCost;
110602	}
110603
110604	/*
110605	** The following is a helper function (and type) for fts3EvalExpr(). It
110606	** must be called after Fts3SegReaders have been allocated for every token
110607	** in the expression. See the context it is called from in fts3EvalExpr()
110608	** for further explanation.
110609	*/
110610	typedef struct ExprAndCost ExprAndCost;
110611	struct ExprAndCost {
110612	Fts3Expr *pExpr;
110613	int nCost;
110614	};
110615	static void fts3ExprAssignCosts(
110616	Fts3Expr pExpr, / Expression to create seg-readers for */
110617	ExprAndCost *ppExprCost / OUT: Write to ppExprCost /
110618	){
110619	if( pExpr->eType==FTSQUERY_AND ){
110620	fts3ExprAssignCosts(pExpr->pLeft, ppExprCost);
110621	fts3ExprAssignCosts(pExpr->pRight, ppExprCost);
110622	}else{
110623	(*ppExprCost)->pExpr = pExpr;
110624	(*ppExprCost)->nCost = fts3ExprCost(pExpr);;
110625	(*ppExprCost)++;
110626	}
110627	}
110628
110629	/*
110630	** Evaluate the full-text expression pExpr against FTS3 table pTab. Store
110631	** the resulting doclist in paOut and pnOut. This routine mallocs for
110632	** the space needed to store the output. The caller is responsible for
110633	** freeing the space when it has finished.
110634	**
110635	** This function is called in two distinct contexts:
110636	**
110637	** * From within the virtual table xFilter() method. In this case, the
110638	** output doclist contains entries for all rows in the table, based on
110639	** data read from the full-text index.
110640	**
110641	** In this case, if the query expression contains one or more tokens that
110642	** are very common, then the returned doclist may contain a superset of
110643	** the documents that actually match the expression.
110644	**
110645	** * From within the virtual table xNext() method. This call is only made
110646	** if the call from within xFilter() found that there were very common
110647	** tokens in the query expression and did return a superset of the
110648	** matching documents. In this case the returned doclist contains only
110649	** entries that correspond to the current row of the table. Instead of
110650	** reading the data for each token from the full-text index, the data is
110651	** already available in-memory in the Fts3PhraseToken.pDeferred structures.
110652	** See fts3EvalDeferred() for how it gets there.
110653	**
110654	** In the first case above, Fts3Cursor.doDeferred==0. In the second (if it is
110655	** required) Fts3Cursor.doDeferred==1.
110656	**
110657	** If the SQLite invokes the snippet(), offsets() or matchinfo() function
110658	** as part of a SELECT on an FTS3 table, this function is called on each
110659	** individual phrase expression in the query. If there were very common tokens
110660	** found in the xFilter() call, then this function is called once for phrase
110661	** for each row visited, and the returned doclist contains entries for the
110662	** current row only. Otherwise, if there were no very common tokens, then this
110663	** function is called once only for each phrase in the query and the returned
110664	** doclist contains entries for all rows of the table.
110665	**
110666	** Fts3Cursor.doDeferred==1 when this function is called on phrases as a
110667	** result of a snippet(), offsets() or matchinfo() invocation.
110668	*/
110669	static int fts3EvalExpr(
110670	Fts3Cursor p, / Virtual table cursor handle */
110671	Fts3Expr pExpr, / Parsed fts3 expression */
110672	char *paOut, / OUT: Pointer to malloc'd result buffer */
110673	int pnOut, / OUT: Size of buffer at paOut /
110674	int isReqPos /* Require positions in output buffer */
110675	){
	@@ -109472,37 +110678,101 @@
110678	/* Zero the output parameters. */
110679	*paOut = 0;
110680	*pnOut = 0;
110681
110682	if( pExpr ){
110683	assert( pExpr->eType==FTSQUERY_NEAR \|\| pExpr->eType==FTSQUERY_OR
110684	\|\| pExpr->eType==FTSQUERY_AND \|\| pExpr->eType==FTSQUERY_NOT
110685	\|\| pExpr->eType==FTSQUERY_PHRASE
110686	);
110687	assert( pExpr->eType==FTSQUERY_PHRASE \|\| isReqPos==0 );
110688
110689	if( pExpr->eType==FTSQUERY_PHRASE ){
110690	rc = fts3PhraseSelect(p, pExpr->pPhrase,
110691	isReqPos \|\| (pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR),
110692	paOut, pnOut
110693	);
110694	fts3ExprFreeSegReaders(pExpr);
110695	}else if( p->eEvalmode==FTS3_EVAL_FILTER && pExpr->eType==FTSQUERY_AND ){
110696	ExprAndCost aExpr = 0; / Array of AND'd expressions and costs */
110697	int nExpr = 0; /* Size of aExpr[] */
110698	char aRet = 0; / Doclist to return to caller */
110699	int nRet = 0; /* Length of aRet[] in bytes */
110700	int nDoc = 0x7FFFFFFF;
110701
110702	assert( !isReqPos );
110703
110704	rc = fts3ExprAllocateSegReaders(p, pExpr, &nExpr);
110705	if( rc==SQLITE_OK ){
110706	assert( nExpr>1 );
110707	aExpr = sqlite3_malloc(sizeof(ExprAndCost) * nExpr);
110708	if( !aExpr ) rc = SQLITE_NOMEM;
110709	}
110710	if( rc==SQLITE_OK ){
110711	int ii; /* Used to iterate through expressions */
110712
110713	fts3ExprAssignCosts(pExpr, &aExpr);
110714	aExpr -= nExpr;
110715	for(ii=0; ii<nExpr; ii++){
110716	char *aNew;
110717	int nNew;
110718	int jj;
110719	ExprAndCost *pBest = 0;
110720
110721	for(jj=0; jj<nExpr; jj++){
110722	ExprAndCost *pCand = &aExpr[jj];
110723	if( pCand->pExpr && (pBest==0 \|\| pCand->nCost<pBest->nCost) ){
110724	pBest = pCand;
110725	}
110726	}
110727
110728	if( pBest->nCost>nDoc ){
110729	rc = fts3DeferExpression(p, p->pExpr);
110730	break;
110731	}else{
110732	rc = fts3EvalExpr(p, pBest->pExpr, &aNew, &nNew, 0);
110733	if( rc!=SQLITE_OK ) break;
110734	pBest->pExpr = 0;
110735	if( ii==0 ){
110736	aRet = aNew;
110737	nRet = nNew;
110738	nDoc = fts3DoclistCountDocids(0, aRet, nRet);
110739	}else{
110740	fts3DoclistMerge(
110741	MERGE_AND, 0, 0, aRet, &nRet, aRet, nRet, aNew, nNew, &nDoc
110742	);
110743	sqlite3_free(aNew);
110744	}
110745	}
110746	}
110747	}
110748
110749	*paOut = aRet;
110750	*pnOut = nRet;
110751	sqlite3_free(aExpr);
110752	fts3ExprFreeSegReaders(pExpr);
110753
110754	}else{
110755	char *aLeft;
110756	char *aRight;
110757	int nLeft;
110758	int nRight;
110759
110760	assert( pExpr->eType==FTSQUERY_NEAR
110761	\|\| pExpr->eType==FTSQUERY_OR
110762	\|\| pExpr->eType==FTSQUERY_NOT
110763	\|\| (pExpr->eType==FTSQUERY_AND && p->eEvalmode==FTS3_EVAL_NEXT)
110764	);
110765
110766	if( 0==(rc = fts3EvalExpr(p, pExpr->pRight, &aRight, &nRight, isReqPos))
110767	&& 0==(rc = fts3EvalExpr(p, pExpr->pLeft, &aLeft, &nLeft, isReqPos))
110768	){



110769	switch( pExpr->eType ){
110770	case FTSQUERY_NEAR: {
110771	Fts3Expr *pLeft;
110772	Fts3Expr *pRight;
110773	int mergetype = MERGE_NEAR;

110774	if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
110775	mergetype = MERGE_POS_NEAR;
110776	}
110777	pLeft = pExpr->pLeft;
110778	while( pLeft->eType==FTSQUERY_NEAR ){
	@@ -109527,21 +110797,21 @@
110797	** so that a buffer of zero bytes is never allocated - this can
110798	** cause fts3DoclistMerge() to incorrectly return SQLITE_NOMEM.
110799	*/
110800	char *aBuffer = sqlite3_malloc(nRight+nLeft+1);
110801	rc = fts3DoclistMerge(MERGE_OR, 0, 0, aBuffer, pnOut,
110802	aLeft, nLeft, aRight, nRight, 0
110803	);
110804	*paOut = aBuffer;
110805	sqlite3_free(aLeft);
110806	break;
110807	}
110808
110809	default: {
110810	assert( FTSQUERY_NOT==MERGE_NOT && FTSQUERY_AND==MERGE_AND );
110811	fts3DoclistMerge(pExpr->eType, 0, 0, aLeft, pnOut,
110812	aLeft, nLeft, aRight, nRight, 0
110813	);
110814	*paOut = aLeft;
110815	break;
110816	}
110817	}
	@@ -109548,10 +110818,92 @@
110818	}
110819	sqlite3_free(aRight);
110820	}
110821	}
110822
110823	return rc;
110824	}
110825
110826	/*
110827	** This function is called from within xNext() for each row visited by
110828	** an FTS3 query. If evaluating the FTS3 query expression within xFilter()
110829	** was able to determine the exact set of matching rows, this function sets
110830	** *pbRes to true and returns SQLITE_IO immediately.
110831	**
110832	** Otherwise, if evaluating the query expression within xFilter() returned a
110833	** superset of the matching documents instead of an exact set (this happens
110834	** when the query includes very common tokens and it is deemed too expensive to
110835	** load their doclists from disk), this function tests if the current row
110836	** really does match the FTS3 query.
110837	**
110838	** If an error occurs, an SQLite error code is returned. Otherwise, SQLITE_OK
110839	** is returned and *pbRes is set to true if the current row matches the
110840	** FTS3 query (and should be included in the results returned to SQLite), or
110841	** false otherwise.
110842	*/
110843	static int fts3EvalDeferred(
110844	Fts3Cursor pCsr, / FTS3 cursor pointing at row to test */
110845	int pbRes / OUT: Set to true if row is a match */
110846	){
110847	int rc = SQLITE_OK;
110848	if( pCsr->pDeferred==0 ){
110849	*pbRes = 1;
110850	}else{
110851	rc = fts3CursorSeek(0, pCsr);
110852	if( rc==SQLITE_OK ){
110853	sqlite3Fts3FreeDeferredDoclists(pCsr);
110854	rc = sqlite3Fts3CacheDeferredDoclists(pCsr);
110855	}
110856	if( rc==SQLITE_OK ){
110857	char *a = 0;
110858	int n = 0;
110859	rc = fts3EvalExpr(pCsr, pCsr->pExpr, &a, &n, 0);
110860	assert( n>=0 );
110861	*pbRes = (n>0);
110862	sqlite3_free(a);
110863	}
110864	}
110865	return rc;
110866	}
110867
110868	/*
110869	** Advance the cursor to the next row in the %_content table that
110870	** matches the search criteria. For a MATCH search, this will be
110871	** the next row that matches. For a full-table scan, this will be
110872	** simply the next row in the %_content table. For a docid lookup,
110873	** this routine simply sets the EOF flag.
110874	**
110875	** Return SQLITE_OK if nothing goes wrong. SQLITE_OK is returned
110876	** even if we reach end-of-file. The fts3EofMethod() will be called
110877	** subsequently to determine whether or not an EOF was hit.
110878	*/
110879	static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){
110880	int res;
110881	int rc = SQLITE_OK; /* Return code */
110882	Fts3Cursor pCsr = (Fts3Cursor )pCursor;
110883
110884	pCsr->eEvalmode = FTS3_EVAL_NEXT;
110885	do {
110886	if( pCsr->aDoclist==0 ){
110887	if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){
110888	pCsr->isEof = 1;
110889	rc = sqlite3_reset(pCsr->pStmt);
110890	break;
110891	}
110892	pCsr->iPrevId = sqlite3_column_int64(pCsr->pStmt, 0);
110893	}else{
110894	if( pCsr->pNextId>=&pCsr->aDoclist[pCsr->nDoclist] ){
110895	pCsr->isEof = 1;
110896	break;
110897	}
110898	sqlite3_reset(pCsr->pStmt);
110899	fts3GetDeltaVarint(&pCsr->pNextId, &pCsr->iPrevId);
110900	pCsr->isRequireSeek = 1;
110901	pCsr->isMatchinfoNeeded = 1;
110902	}
110903	}while( SQLITE_OK==(rc = fts3EvalDeferred(pCsr, &res)) && res==0 );
110904
110905	return rc;
110906	}
110907
110908	/*
110909	** This is the xFilter interface for the virtual table. See
	@@ -109567,15 +110919,10 @@
110919	** If idxNum>=FTS3_FULLTEXT_SEARCH then use the full text index. The
110920	** column on the left-hand side of the MATCH operator is column
110921	** number idxNum-FTS3_FULLTEXT_SEARCH, 0 indexed. argv[0] is the right-hand
110922	** side of the MATCH operator.
110923	*/





110924	static int fts3FilterMethod(
110925	sqlite3_vtab_cursor pCursor, / The cursor used for this query */
110926	int idxNum, /* Strategy index */
110927	const char idxStr, / Unused */
110928	int nVal, /* Number of elements in apVal */
	@@ -109594,35 +110941,19 @@
110941	UNUSED_PARAMETER(nVal);
110942
110943	assert( idxNum>=0 && idxNum<=(FTS3_FULLTEXT_SEARCH+p->nColumn) );
110944	assert( nVal==0 \|\| nVal==1 );
110945	assert( (nVal==0)==(idxNum==FTS3_FULLSCAN_SEARCH) );
110946	assert( p->pSegments==0 );
110947
110948	/* In case the cursor has been used before, clear it now. */
110949	sqlite3_finalize(pCsr->pStmt);
110950	sqlite3_free(pCsr->aDoclist);
110951	sqlite3Fts3ExprFree(pCsr->pExpr);
110952	memset(&pCursor[1], 0, sizeof(Fts3Cursor)-sizeof(sqlite3_vtab_cursor));
110953
110954	if( idxNum!=FTS3_DOCID_SEARCH && idxNum!=FTS3_FULLSCAN_SEARCH ){

















110955	int iCol = idxNum-FTS3_FULLTEXT_SEARCH;
110956	const char zQuery = (const char )sqlite3_value_text(apVal[0]);
110957
110958	if( zQuery==0 && sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
110959	return SQLITE_NOMEM;
	@@ -109640,14 +110971,33 @@
110971	}
110972
110973	rc = sqlite3Fts3ReadLock(p);
110974	if( rc!=SQLITE_OK ) return rc;
110975
110976	rc = fts3EvalExpr(pCsr, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0);
110977	sqlite3Fts3SegmentsClose(p);
110978	if( rc!=SQLITE_OK ) return rc;
110979	pCsr->pNextId = pCsr->aDoclist;
110980	pCsr->iPrevId = 0;
110981	}
110982
110983	/* Compile a SELECT statement for this cursor. For a full-table-scan, the
110984	** statement loops through all rows of the %_content table. For a
110985	** full-text query or docid lookup, the statement retrieves a single
110986	** row by docid.
110987	*/
110988	zSql = sqlite3_mprintf(azSql[idxNum==FTS3_FULLSCAN_SEARCH], p->zDb, p->zName);
110989	if( !zSql ){
110990	rc = SQLITE_NOMEM;
110991	}else{
110992	rc = sqlite3_prepare_v2(p->db, zSql, -1, &pCsr->pStmt, 0);
110993	sqlite3_free(zSql);
110994	}
110995	if( rc==SQLITE_OK && idxNum==FTS3_DOCID_SEARCH ){
110996	rc = sqlite3_bind_value(pCsr->pStmt, 1, apVal[0]);
110997	}
110998	pCsr->eSearch = (i16)idxNum;
110999
111000	if( rc!=SQLITE_OK ) return rc;
111001	return fts3NextMethod(pCursor);
111002	}
111003
	@@ -109668,10 +111018,15 @@
111018	static int fts3RowidMethod(sqlite3_vtab_cursor pCursor, sqlite_int64 pRowid){
111019	Fts3Cursor pCsr = (Fts3Cursor ) pCursor;
111020	if( pCsr->aDoclist ){
111021	*pRowid = pCsr->iPrevId;
111022	}else{
111023	/* This branch runs if the query is implemented using a full-table scan
111024	** (not using the full-text index). In this case grab the rowid from the
111025	** SELECT statement.
111026	*/
111027	assert( pCsr->isRequireSeek==0 );
111028	*pRowid = sqlite3_column_int64(pCsr->pStmt, 0);
111029	}
111030	return SQLITE_OK;
111031	}
111032
	@@ -109730,11 +111085,13 @@
111085	/*
111086	** Implementation of xSync() method. Flush the contents of the pending-terms
111087	** hash-table to the database.
111088	*/
111089	static int fts3SyncMethod(sqlite3_vtab *pVtab){
111090	int rc = sqlite3Fts3PendingTermsFlush((Fts3Table *)pVtab);
111091	sqlite3Fts3SegmentsClose((Fts3Table *)pVtab);
111092	return rc;
111093	}
111094
111095	/*
111096	** Implementation of xBegin() method. This is a no-op.
111097	*/
	@@ -109768,12 +111125,31 @@
111125	** Load the doclist associated with expression pExpr to pExpr->aDoclist.
111126	** The loaded doclist contains positions as well as the document ids.
111127	** This is used by the matchinfo(), snippet() and offsets() auxillary
111128	** functions.
111129	*/
111130	SQLITE_PRIVATE int sqlite3Fts3ExprLoadDoclist(Fts3Cursor pCsr, Fts3Expr pExpr){
111131	int rc;
111132	assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase );
111133	assert( pCsr->eEvalmode==FTS3_EVAL_NEXT );
111134	rc = fts3EvalExpr(pCsr, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
111135	return rc;
111136	}
111137
111138	SQLITE_PRIVATE int sqlite3Fts3ExprLoadFtDoclist(
111139	Fts3Cursor *pCsr,
111140	Fts3Expr *pExpr,
111141	char **paDoclist,
111142	int *pnDoclist
111143	){
111144	int rc;
111145	assert( pCsr->eEvalmode==FTS3_EVAL_NEXT );
111146	assert( pExpr->eType==FTSQUERY_PHRASE && pExpr->pPhrase );
111147	pCsr->eEvalmode = FTS3_EVAL_MATCHINFO;
111148	rc = fts3EvalExpr(pCsr, pExpr, paDoclist, pnDoclist, 1);
111149	pCsr->eEvalmode = FTS3_EVAL_NEXT;
111150	return rc;
111151	}
111152
111153	/*
111154	** After ExprLoadDoclist() (see above) has been called, this function is
111155	** used to iterate/search through the position lists that make up the doclist
	@@ -109835,11 +111211,11 @@
111211	*/
111212	static int fts3FunctionArg(
111213	sqlite3_context pContext, / SQL function call context */
111214	const char zFunc, / Function name */
111215	sqlite3_value pVal, / argv[0] passed to function */
111216	Fts3Cursor *ppCsr / OUT: Store cursor handle here */
111217	){
111218	Fts3Cursor *pRet;
111219	if( sqlite3_value_type(pVal)!=SQLITE_BLOB
111220	\|\| sqlite3_value_bytes(pVal)!=sizeof(Fts3Cursor *)
111221	){
	@@ -109961,17 +111337,11 @@
111337	sqlite3_context pContext, / SQLite function call context */
111338	int nVal, /* Size of argument array */
111339	sqlite3_value *apVal / Array of arguments */
111340	){
111341	Fts3Cursor pCsr; / Cursor handle passed through apVal[0] */
111342	assert( nVal==1 );






111343	if( SQLITE_OK==fts3FunctionArg(pContext, "matchinfo", apVal[0], &pCsr) ){
111344	sqlite3Fts3Matchinfo(pContext, pCsr);
111345	}
111346	}
111347
	@@ -110030,16 +111400,17 @@
111400
111401	fts3DbExec(&rc, db,
111402	"ALTER TABLE %Q.'%q_content' RENAME TO '%q_content';",
111403	p->zDb, p->zName, zName
111404	);

111405	if( p->bHasDocsize ){
111406	fts3DbExec(&rc, db,
111407	"ALTER TABLE %Q.'%q_docsize' RENAME TO '%q_docsize';",
111408	p->zDb, p->zName, zName
111409	);
111410	}
111411	if( p->bHasStat ){
111412	fts3DbExec(&rc, db,
111413	"ALTER TABLE %Q.'%q_stat' RENAME TO '%q_stat';",
111414	p->zDb, p->zName, zName
111415	);
111416	}
	@@ -110060,11 +111431,11 @@
111431	/* xConnect */ fts3ConnectMethod,
111432	/* xBestIndex */ fts3BestIndexMethod,
111433	/* xDisconnect */ fts3DisconnectMethod,
111434	/* xDestroy */ fts3DestroyMethod,
111435	/* xOpen */ fts3OpenMethod,
111436	/* xClose */ fts3CloseMethod,
111437	/* xFilter */ fts3FilterMethod,
111438	/* xNext */ fts3NextMethod,
111439	/* xEof */ fts3EofMethod,
111440	/* xColumn */ fts3ColumnMethod,
111441	/* xRowid */ fts3RowidMethod,
	@@ -110087,23 +111458,24 @@
111458	sqlite3Fts3HashClear(pHash);
111459	sqlite3_free(pHash);
111460	}
111461
111462	/*
111463	** The fts3 built-in tokenizers - "simple", "porter" and "icu"- are
111464	** implemented in files fts3_tokenizer1.c, fts3_porter.c and fts3_icu.c
111465	** respectively. The following three forward declarations are for functions
111466	** declared in these files used to retrieve the respective implementations.
111467	**
111468	** Calling sqlite3Fts3SimpleTokenizerModule() sets the value pointed
111469	** to by the argument to point to the "simple" tokenizer implementation.
111470	** And so on.

111471	*/
111472	SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule);
111473	SQLITE_PRIVATE void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule);
111474	#ifdef SQLITE_ENABLE_ICU
111475	SQLITE_PRIVATE void sqlite3Fts3IcuTokenizerModule(sqlite3_tokenizer_module const**ppModule);
111476	#endif
111477
111478	/*
111479	** Initialise the fts3 extension. If this extension is built as part
111480	** of the sqlite library, then this function is called directly by
111481	** SQLite. If fts3 is built as a dynamically loadable extension, this
	@@ -110155,11 +111527,11 @@
111527	*/
111528	if( SQLITE_OK==rc
111529	&& SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
111530	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))
111531	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
111532	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", 1))
111533	&& SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
111534	){
111535	rc = sqlite3_create_module_v2(
111536	db, "fts3", &fts3Module, (void *)pHash, hashDestroy
111537	);
	@@ -110297,10 +111669,22 @@
111669	** negative values).
111670	*/
111671	static int fts3isspace(char c){
111672	return c==' ' \|\| c=='\t' \|\| c=='\n' \|\| c=='\r' \|\| c=='\v' \|\| c=='\f';
111673	}
111674
111675	/*
111676	** Allocate nByte bytes of memory using sqlite3_malloc(). If successful,
111677	** zero the memory before returning a pointer to it. If unsuccessful,
111678	** return NULL.
111679	*/
111680	static void *fts3MallocZero(int nByte){
111681	void *pRet = sqlite3_malloc(nByte);
111682	if( pRet ) memset(pRet, 0, nByte);
111683	return pRet;
111684	}
111685
111686
111687	/*
111688	** Extract the next token from buffer z (length n) using the tokenizer
111689	** and other information (column names etc.) in pParse. Create an Fts3Expr
111690	** structure of type FTSQUERY_PHRASE containing a phrase consisting of this
	@@ -110335,15 +111719,14 @@
111719	pCursor->pTokenizer = pTokenizer;
111720	rc = pModule->xNext(pCursor, &zToken, &nToken, &iStart, &iEnd, &iPosition);
111721
111722	if( rc==SQLITE_OK ){
111723	nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase) + nToken;
111724	pRet = (Fts3Expr *)fts3MallocZero(nByte);
111725	if( !pRet ){
111726	rc = SQLITE_NOMEM;
111727	}else{

111728	pRet->eType = FTSQUERY_PHRASE;
111729	pRet->pPhrase = (Fts3Phrase *)&pRet[1];
111730	pRet->pPhrase->nToken = 1;
111731	pRet->pPhrase->iColumn = iCol;
111732	pRet->pPhrase->aToken[0].n = nToken;
	@@ -110415,20 +111798,21 @@
111798	const char *zToken;
111799	int nToken, iBegin, iEnd, iPos;
111800	rc = pModule->xNext(pCursor, &zToken, &nToken, &iBegin, &iEnd, &iPos);
111801	if( rc==SQLITE_OK ){
111802	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
111803	p = fts3ReallocOrFree(p, nByte+ii*sizeof(Fts3PhraseToken));
111804	zTemp = fts3ReallocOrFree(zTemp, nTemp + nToken);
111805	if( !p \|\| !zTemp ){
111806	goto no_mem;
111807	}
111808	if( ii==0 ){
111809	memset(p, 0, nByte);
111810	p->pPhrase = (Fts3Phrase *)&p[1];
111811	}
111812	p->pPhrase = (Fts3Phrase *)&p[1];
111813	memset(&p->pPhrase->aToken[ii], 0, sizeof(Fts3PhraseToken));
111814	p->pPhrase->nToken = ii+1;
111815	p->pPhrase->aToken[ii].n = nToken;
111816	memcpy(&zTemp[nTemp], zToken, nToken);
111817	nTemp += nToken;
111818	if( iEnd<nInput && zInput[iEnd]=='*' ){
	@@ -110446,11 +111830,11 @@
111830	if( rc==SQLITE_DONE ){
111831	int jj;
111832	char *zNew = NULL;
111833	int nNew = 0;
111834	int nByte = sizeof(Fts3Expr) + sizeof(Fts3Phrase);
111835	nByte += (p?(p->pPhrase->nToken-1):0) * sizeof(Fts3PhraseToken);
111836	p = fts3ReallocOrFree(p, nByte + nTemp);
111837	if( !p ){
111838	goto no_mem;
111839	}
111840	if( zTemp ){
	@@ -110564,15 +111948,14 @@
111948	*/
111949	cNext = zInput[nKey];
111950	if( fts3isspace(cNext)
111951	\|\| cNext=='"' \|\| cNext=='(' \|\| cNext==')' \|\| cNext==0
111952	){
111953	pRet = (Fts3Expr *)fts3MallocZero(sizeof(Fts3Expr));
111954	if( !pRet ){
111955	return SQLITE_NOMEM;
111956	}

111957	pRet->eType = pKey->eType;
111958	pRet->nNear = nNear;
111959	*ppExpr = pRet;
111960	*pnConsumed = (int)((zInput - z) + nKey);
111961	return SQLITE_OK;
	@@ -110744,17 +112127,16 @@
112127
112128	if( !sqlite3_fts3_enable_parentheses
112129	&& p->eType==FTSQUERY_PHRASE && p->pPhrase->isNot
112130	){
112131	/* Create an implicit NOT operator. */
112132	Fts3Expr *pNot = fts3MallocZero(sizeof(Fts3Expr));
112133	if( !pNot ){
112134	sqlite3Fts3ExprFree(p);
112135	rc = SQLITE_NOMEM;
112136	goto exprparse_out;
112137	}

112138	pNot->eType = FTSQUERY_NOT;
112139	pNot->pRight = p;
112140	if( pNotBranch ){
112141	pNot->pLeft = pNotBranch;
112142	}
	@@ -110778,17 +112160,16 @@
112160
112161	if( isPhrase && !isRequirePhrase ){
112162	/* Insert an implicit AND operator. */
112163	Fts3Expr *pAnd;
112164	assert( pRet && pPrev );
112165	pAnd = fts3MallocZero(sizeof(Fts3Expr));
112166	if( !pAnd ){
112167	sqlite3Fts3ExprFree(p);
112168	rc = SQLITE_NOMEM;
112169	goto exprparse_out;
112170	}

112171	pAnd->eType = FTSQUERY_AND;
112172	insertBinaryOperator(&pRet, pPrev, pAnd);
112173	pPrev = pAnd;
112174	}
112175
	@@ -112234,11 +113615,11 @@
113615	}
113616
113617	sqlite3_result_blob(context, (void *)&pPtr, sizeof(pPtr), SQLITE_TRANSIENT);
113618	}
113619
113620	SQLITE_PRIVATE int sqlite3Fts3IsIdChar(char c){
113621	static const char isFtsIdChar[] = {
113622	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x */
113623	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 1x */
113624	0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 2x */
113625	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 3x */
	@@ -112272,13 +113653,13 @@
113653	while( *z2 && z2[0]!=']' ) z2++;
113654	if( *z2 ) z2++;
113655	break;
113656
113657	default:
113658	if( sqlite3Fts3IsIdChar(*z1) ){
113659	z2 = &z1[1];
113660	while( sqlite3Fts3IsIdChar(*z2) ) z2++;
113661	}else{
113662	z1++;
113663	}
113664	}
113665	}
	@@ -112287,42 +113668,30 @@
113668	return z1;
113669	}
113670
113671	SQLITE_PRIVATE int sqlite3Fts3InitTokenizer(
113672	Fts3Hash pHash, / Tokenizer hash table */
113673	const char zArg, / Tokenizer name */
113674	sqlite3_tokenizer *ppTok, / OUT: Tokenizer (if applicable) */

113675	char *pzErr / OUT: Set to malloced error message */
113676	){
113677	int rc;
113678	char z = (char )zArg;
113679	int n;
113680	char *zCopy;
113681	char zEnd; / Pointer to nul-term of zCopy */
113682	sqlite3_tokenizer_module *m;
113683
113684	zCopy = sqlite3_mprintf("%s", zArg);
113685	if( !zCopy ) return SQLITE_NOMEM;











113686	zEnd = &zCopy[strlen(zCopy)];
113687
113688	z = (char *)sqlite3Fts3NextToken(zCopy, &n);
113689	z[n] = '\0';
113690	sqlite3Fts3Dequote(z);
113691
113692	m = (sqlite3_tokenizer_module *)sqlite3Fts3HashFind(pHash,z,(int)strlen(z)+1);
113693	if( !m ){
113694	*pzErr = sqlite3_mprintf("unknown tokenizer: %s", z);
113695	rc = SQLITE_ERROR;
113696	}else{
113697	char const **aArg = 0;
	@@ -112887,10 +114256,22 @@
114256	*/
114257
114258	#if !defined(SQLITE_CORE) \|\| defined(SQLITE_ENABLE_FTS3)
114259
114260
114261	/*
114262	** When full-text index nodes are loaded from disk, the buffer that they
114263	** are loaded into has the following number of bytes of padding at the end
114264	** of it. i.e. if a full-text index node is 900 bytes in size, then a buffer
114265	** of 920 bytes is allocated for it.
114266	**
114267	** This means that if we have a pointer into a buffer containing node data,
114268	** it is always safe to read up to two varints from it without risking an
114269	** overread, even if the node data is corrupted.
114270	*/
114271	#define FTS3_NODE_PADDING (FTS3_VARINT_MAX*2)
114272
114273	typedef struct PendingList PendingList;
114274	typedef struct SegmentNode SegmentNode;
114275	typedef struct SegmentWriter SegmentWriter;
114276
114277	/*
	@@ -112905,19 +114286,31 @@
114286	sqlite3_int64 iLastDocid;
114287	sqlite3_int64 iLastCol;
114288	sqlite3_int64 iLastPos;
114289	};
114290
114291
114292	/*
114293	** Each cursor has a (possibly empty) linked list of the following objects.
114294	*/
114295	struct Fts3DeferredToken {
114296	Fts3PhraseToken pToken; / Pointer to corresponding expr token */
114297	int iCol; /* Column token must occur in */
114298	Fts3DeferredToken pNext; / Next in list of deferred tokens */
114299	PendingList pList; / Doclist is assembled here */
114300	};
114301
114302	/*
114303	** An instance of this structure is used to iterate through the terms on
114304	** a contiguous set of segment b-tree leaf nodes. Although the details of
114305	** this structure are only manipulated by code in this file, opaque handles
114306	** of type Fts3SegReader* are also used by code in fts3.c to iterate through
114307	** terms when querying the full-text index. See functions:
114308	**
114309	** sqlite3Fts3SegReaderNew()
114310	** sqlite3Fts3SegReaderFree()
114311	** sqlite3Fts3SegReaderCost()
114312	** sqlite3Fts3SegReaderIterate()
114313	**
114314	** Methods used to manipulate Fts3SegReader structures:
114315	**
114316	** fts3SegReaderNext()
	@@ -112924,34 +114317,38 @@
114317	** fts3SegReaderFirstDocid()
114318	** fts3SegReaderNextDocid()
114319	*/
114320	struct Fts3SegReader {
114321	int iIdx; /* Index within level, or 0x7FFFFFFF for PT */
114322
114323	sqlite3_int64 iStartBlock; /* Rowid of first leaf block to traverse */
114324	sqlite3_int64 iLeafEndBlock; /* Rowid of final leaf block to traverse */
114325	sqlite3_int64 iEndBlock; /* Rowid of final block in segment (or 0) */
114326	sqlite3_int64 iCurrentBlock; /* Current leaf block (or 0) */
114327
114328	char aNode; / Pointer to node data (or NULL) */
114329	int nNode; /* Size of buffer at aNode (or 0) */

114330	Fts3HashElem **ppNextElem;
114331
114332	/* Variables set by fts3SegReaderNext(). These may be read directly
114333	** by the caller. They are valid from the time SegmentReaderNew() returns
114334	** until SegmentReaderNext() returns something other than SQLITE_OK
114335	** (i.e. SQLITE_DONE).
114336	*/
114337	int nTerm; /* Number of bytes in current term */
114338	char zTerm; / Pointer to current term */
114339	int nTermAlloc; /* Allocated size of zTerm buffer */
114340	char aDoclist; / Pointer to doclist of current entry */
114341	int nDoclist; /* Size of doclist in current entry */
114342
114343	/* The following variables are used to iterate through the current doclist */
114344	char *pOffsetList;
114345	sqlite3_int64 iDocid;
114346	};
114347
114348	#define fts3SegReaderIsPending(p) ((p)->ppNextElem!=0)
114349	#define fts3SegReaderIsRootOnly(p) ((p)->aNode==(char *)&(p)[1])
114350
114351	/*
114352	** An instance of this structure is used to create a segment b-tree in the
114353	** database. The internal details of this type are only accessed by the
114354	** following functions:
	@@ -113016,16 +114413,15 @@
114413	#define SQL_SELECT_LEVEL_COUNT 14
114414	#define SQL_SELECT_SEGDIR_COUNT_MAX 15
114415	#define SQL_DELETE_SEGDIR_BY_LEVEL 16
114416	#define SQL_DELETE_SEGMENTS_RANGE 17
114417	#define SQL_CONTENT_INSERT 18
114418	#define SQL_DELETE_DOCSIZE 19
114419	#define SQL_REPLACE_DOCSIZE 20
114420	#define SQL_SELECT_DOCSIZE 21
114421	#define SQL_SELECT_DOCTOTAL 22
114422	#define SQL_REPLACE_DOCTOTAL 23

114423
114424	/*
114425	** This function is used to obtain an SQLite prepared statement handle
114426	** for the statement identified by the second argument. If successful,
114427	** *pp is set to the requested statement handle and SQLITE_OK returned.
	@@ -113066,16 +114462,15 @@
114462	/* 15 / "SELECT count(), max(level) FROM %Q.'%q_segdir'",
114463
114464	/* 16 */ "DELETE FROM %Q.'%q_segdir' WHERE level = ?",
114465	/* 17 */ "DELETE FROM %Q.'%q_segments' WHERE blockid BETWEEN ? AND ?",
114466	/* 18 */ "INSERT INTO %Q.'%q_content' VALUES(%z)",
114467	/* 19 */ "DELETE FROM %Q.'%q_docsize' WHERE docid = ?",
114468	/* 20 */ "REPLACE INTO %Q.'%q_docsize' VALUES(?,?)",
114469	/* 21 */ "SELECT size FROM %Q.'%q_docsize' WHERE docid=?",
114470	/* 22 */ "SELECT value FROM %Q.'%q_stat' WHERE id=0",
114471	/* 23 */ "REPLACE INTO %Q.'%q_stat' VALUES(0,?)",

114472	};
114473	int rc = SQLITE_OK;
114474	sqlite3_stmt *pStmt;
114475
114476	assert( SizeofArray(azSql)==SizeofArray(p->aStmt) );
	@@ -113146,49 +114541,10 @@
114541	}
114542	*pRC = rc;
114543	}
114544
114545







































114546	/*
114547	** This function ensures that the caller has obtained a shared-cache
114548	** table-lock on the %_content table. This is required before reading
114549	** data from the fts3 table. If this lock is not acquired first, then
114550	** the caller may end up holding read-locks on the %_segments and %_segdir
	@@ -113353,14 +114709,14 @@
114709	** p->iPrevDocid, and the column is specified by argument iCol.
114710	**
114711	** If successful, SQLITE_OK is returned. Otherwise, an SQLite error code.
114712	*/
114713	static int fts3PendingTermsAdd(
114714	Fts3Table p, / Table into which text will be inserted */
114715	const char zText, / Text of document to be inserted */
114716	int iCol, /* Column into which text is being inserted */
114717	u32 pnWord / OUT: Number of tokens inserted */
114718	){
114719	int rc;
114720	int iStart;
114721	int iEnd;
114722	int iPos;
	@@ -113441,10 +114797,13 @@
114797	}
114798	p->iPrevDocid = iDocid;
114799	return SQLITE_OK;
114800	}
114801
114802	/*
114803	** Discard the contents of the pending-terms hash table.
114804	*/
114805	SQLITE_PRIVATE void sqlite3Fts3PendingTermsClear(Fts3Table *p){
114806	Fts3HashElem *pElem;
114807	for(pElem=fts3HashFirst(&p->pendingTerms); pElem; pElem=fts3HashNext(pElem)){
114808	sqlite3_free(fts3HashData(pElem));
114809	}
	@@ -113468,10 +114827,11 @@
114827	int rc = fts3PendingTermsAdd(p, zText, i-2, &aSz[i-2]);
114828	if( rc!=SQLITE_OK ){
114829	return rc;
114830	}
114831	}
114832	aSz[p->nColumn] += sqlite3_value_bytes(apVal[i]);
114833	}
114834	return SQLITE_OK;
114835	}
114836
114837	/*
	@@ -113555,10 +114915,12 @@
114915	fts3SqlExec(&rc, p, SQL_DELETE_ALL_CONTENT, 0);
114916	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGMENTS, 0);
114917	fts3SqlExec(&rc, p, SQL_DELETE_ALL_SEGDIR, 0);
114918	if( p->bHasDocsize ){
114919	fts3SqlExec(&rc, p, SQL_DELETE_ALL_DOCSIZE, 0);
114920	}
114921	if( p->bHasStat ){
114922	fts3SqlExec(&rc, p, SQL_DELETE_ALL_STAT, 0);
114923	}
114924	return rc;
114925	}
114926
	@@ -113565,11 +114927,11 @@
114927	/*
114928	** The first element in the apVal[] array is assumed to contain the docid
114929	** (an integer) of a row about to be deleted. Remove all terms from the
114930	** full-text index.
114931	*/
114932	static void fts3DeleteTerms(
114933	int pRC, / Result code */
114934	Fts3Table p, / The FTS table to delete from */
114935	sqlite3_value *apVal, / apVal[] contains the docid to be deleted */
114936	u32 aSz / Sizes of deleted document written here */
114937	){
	@@ -113587,10 +114949,11 @@
114949	if( rc!=SQLITE_OK ){
114950	sqlite3_reset(pSelect);
114951	*pRC = rc;
114952	return;
114953	}
114954	aSz[p->nColumn] += sqlite3_column_bytes(pSelect, i);
114955	}
114956	}
114957	rc = sqlite3_reset(pSelect);
114958	}else{
114959	sqlite3_reset(pSelect);
	@@ -113648,17 +115011,98 @@
115011	}
115012	}
115013
115014	return rc;
115015	}
115016
115017	/*
115018	** The %_segments table is declared as follows:
115019	**
115020	** CREATE TABLE %_segments(blockid INTEGER PRIMARY KEY, block BLOB)
115021	**
115022	** This function reads data from a single row of the %_segments table. The
115023	** specific row is identified by the iBlockid parameter. If paBlob is not
115024	** NULL, then a buffer is allocated using sqlite3_malloc() and populated
115025	** with the contents of the blob stored in the "block" column of the
115026	** identified table row is. Whether or not paBlob is NULL, *pnBlob is set
115027	** to the size of the blob in bytes before returning.
115028	**
115029	** If an error occurs, or the table does not contain the specified row,
115030	** an SQLite error code is returned. Otherwise, SQLITE_OK is returned. If
115031	** paBlob is non-NULL, then it is the responsibility of the caller to
115032	** eventually free the returned buffer.
115033	**
115034	** This function may leave an open sqlite3_blob* handle in the
115035	** Fts3Table.pSegments variable. This handle is reused by subsequent calls
115036	** to this function. The handle may be closed by calling the
115037	** sqlite3Fts3SegmentsClose() function. Reusing a blob handle is a handy
115038	** performance improvement, but the blob handle should always be closed
115039	** before control is returned to the user (to prevent a lock being held
115040	** on the database file for longer than necessary). Thus, any virtual table
115041	** method (xFilter etc.) that may directly or indirectly call this function
115042	** must call sqlite3Fts3SegmentsClose() before returning.
115043	*/
115044	SQLITE_PRIVATE int sqlite3Fts3ReadBlock(
115045	Fts3Table p, / FTS3 table handle */
115046	sqlite3_int64 iBlockid, /* Access the row with blockid=$iBlockid */
115047	char *paBlob, / OUT: Blob data in malloc'd buffer */
115048	int pnBlob / OUT: Size of blob data */
115049	){
115050	int rc; /* Return code */
115051
115052	/* pnBlob must be non-NULL. paBlob may be NULL or non-NULL. */
115053	assert( pnBlob);
115054
115055	if( p->pSegments ){
115056	rc = sqlite3_blob_reopen(p->pSegments, iBlockid);
115057	}else{
115058	if( 0==p->zSegmentsTbl ){
115059	p->zSegmentsTbl = sqlite3_mprintf("%s_segments", p->zName);
115060	if( 0==p->zSegmentsTbl ) return SQLITE_NOMEM;
115061	}
115062	rc = sqlite3_blob_open(
115063	p->db, p->zDb, p->zSegmentsTbl, "block", iBlockid, 0, &p->pSegments
115064	);
115065	}
115066
115067	if( rc==SQLITE_OK ){
115068	int nByte = sqlite3_blob_bytes(p->pSegments);
115069	if( paBlob ){
115070	char *aByte = sqlite3_malloc(nByte + FTS3_NODE_PADDING);
115071	if( !aByte ){
115072	rc = SQLITE_NOMEM;
115073	}else{
115074	rc = sqlite3_blob_read(p->pSegments, aByte, nByte, 0);
115075	memset(&aByte[nByte], 0, FTS3_NODE_PADDING);
115076	if( rc!=SQLITE_OK ){
115077	sqlite3_free(aByte);
115078	aByte = 0;
115079	}
115080	}
115081	*paBlob = aByte;
115082	}
115083	*pnBlob = nByte;
115084	}
115085
115086	return rc;
115087	}
115088
115089	/*
115090	** Close the blob handle at p->pSegments, if it is open. See comments above
115091	** the sqlite3Fts3ReadBlock() function for details.
115092	*/
115093	SQLITE_PRIVATE void sqlite3Fts3SegmentsClose(Fts3Table *p){
115094	sqlite3_blob_close(p->pSegments);
115095	p->pSegments = 0;
115096	}
115097
115098	/*
115099	** Move the iterator passed as the first argument to the next term in the
115100	** segment. If successful, SQLITE_OK is returned. If there is no next term,
115101	** SQLITE_DONE. Otherwise, an SQLite error code.
115102	*/
115103	static int fts3SegReaderNext(Fts3Table p, Fts3SegReader pReader){
115104	char pNext; / Cursor variable */
115105	int nPrefix; /* Number of bytes in term prefix */
115106	int nSuffix; /* Number of bytes in term suffix */
115107
115108	if( !pReader->aDoclist ){
	@@ -113666,11 +115110,12 @@
115110	}else{
115111	pNext = &pReader->aDoclist[pReader->nDoclist];
115112	}
115113
115114	if( !pNext \|\| pNext>=&pReader->aNode[pReader->nNode] ){
115115	int rc; /* Return code from Fts3ReadBlock() */
115116
115117	if( fts3SegReaderIsPending(pReader) ){
115118	Fts3HashElem pElem = (pReader->ppNextElem);
115119	if( pElem==0 ){
115120	pReader->aNode = 0;
115121	}else{
	@@ -113682,26 +115127,40 @@
115127	pReader->ppNextElem++;
115128	assert( pReader->aNode );
115129	}
115130	return SQLITE_OK;
115131	}
115132
115133	if( !fts3SegReaderIsRootOnly(pReader) ){
115134	sqlite3_free(pReader->aNode);
115135	}
115136	pReader->aNode = 0;
115137
115138	/* If iCurrentBlock>=iLeafEndBlock, this is an EOF condition. All leaf
115139	** blocks have already been traversed. */
115140	assert( pReader->iCurrentBlock<=pReader->iLeafEndBlock );
115141	if( pReader->iCurrentBlock>=pReader->iLeafEndBlock ){
115142	return SQLITE_OK;
115143	}
115144
115145	rc = sqlite3Fts3ReadBlock(
115146	p, ++pReader->iCurrentBlock, &pReader->aNode, &pReader->nNode
115147	);
115148	if( rc!=SQLITE_OK ) return rc;


115149	pNext = pReader->aNode;
115150	}
115151
115152	/* Because of the FTS3_NODE_PADDING bytes of padding, the following is
115153	** safe (no risk of overread) even if the node data is corrupted.
115154	*/
115155	pNext += sqlite3Fts3GetVarint32(pNext, &nPrefix);
115156	pNext += sqlite3Fts3GetVarint32(pNext, &nSuffix);
115157	if( nPrefix<0 \|\| nSuffix<=0
115158	\|\| &pNext[nSuffix]>&pReader->aNode[pReader->nNode]
115159	){
115160	return SQLITE_CORRUPT;
115161	}
115162
115163	if( nPrefix+nSuffix>pReader->nTermAlloc ){
115164	int nNew = (nPrefix+nSuffix)*2;
115165	char *zNew = sqlite3_realloc(pReader->zTerm, nNew);
115166	if( !zNew ){
	@@ -113712,13 +115171,22 @@
115171	}
115172	memcpy(&pReader->zTerm[nPrefix], pNext, nSuffix);
115173	pReader->nTerm = nPrefix+nSuffix;
115174	pNext += nSuffix;
115175	pNext += sqlite3Fts3GetVarint32(pNext, &pReader->nDoclist);

115176	pReader->aDoclist = pNext;
115177	pReader->pOffsetList = 0;
115178
115179	/* Check that the doclist does not appear to extend past the end of the
115180	** b-tree node. And that the final byte of the doclist is 0x00. If either
115181	** of these statements is untrue, then the data structure is corrupt.
115182	*/
115183	if( &pReader->aDoclist[pReader->nDoclist]>&pReader->aNode[pReader->nNode]
115184	\|\| pReader->aDoclist[pReader->nDoclist-1]
115185	){
115186	return SQLITE_CORRUPT;
115187	}
115188	return SQLITE_OK;
115189	}
115190
115191	/*
115192	** Set the SegReader to point to the first docid in the doclist associated
	@@ -113776,30 +115244,105 @@
115244	sqlite3_int64 iDelta;
115245	pReader->pOffsetList = p + sqlite3Fts3GetVarint(p, &iDelta);
115246	pReader->iDocid += iDelta;
115247	}
115248	}
115249
115250	/*
115251	** This function is called to estimate the amount of data that will be
115252	** loaded from the disk If SegReaderIterate() is called on this seg-reader,
115253	** in units of average document size.
115254	**
115255	** This can be used as follows: If the caller has a small doclist that
115256	** contains references to N documents, and is considering merging it with
115257	** a large doclist (size X "average documents"), it may opt not to load
115258	** the large doclist if X>N.
115259	*/
115260	SQLITE_PRIVATE int sqlite3Fts3SegReaderCost(
115261	Fts3Cursor pCsr, / FTS3 cursor handle */
115262	Fts3SegReader pReader, / Segment-reader handle */
115263	int pnCost / IN/OUT: Number of bytes read */
115264	){
115265	Fts3Table p = (Fts3Table)pCsr->base.pVtab;
115266	int rc = SQLITE_OK; /* Return code */
115267	int nCost = 0; /* Cost in bytes to return */
115268	int pgsz = p->nPgsz; /* Database page size */
115269
115270	/* If this seg-reader is reading the pending-terms table, or if all data
115271	** for the segment is stored on the root page of the b-tree, then the cost
115272	** is zero. In this case all required data is already in main memory.
115273	*/
115274	if( p->bHasStat
115275	&& !fts3SegReaderIsPending(pReader)
115276	&& !fts3SegReaderIsRootOnly(pReader)
115277	){
115278	int nBlob = 0;
115279	sqlite3_int64 iBlock;
115280
115281	if( pCsr->nRowAvg==0 ){
115282	/* The average document size, which is required to calculate the cost
115283	** of each doclist, has not yet been determined. Read the required
115284	** data from the %_stat table to calculate it.
115285	**
115286	** Entry 0 of the %_stat table is a blob containing (nCol+1) FTS3
115287	** varints, where nCol is the number of columns in the FTS3 table.
115288	** The first varint is the number of documents currently stored in
115289	** the table. The following nCol varints contain the total amount of
115290	** data stored in all rows of each column of the table, from left
115291	** to right.
115292	*/
115293	sqlite3_stmt *pStmt;
115294	rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
115295	if( rc ) return rc;
115296	if( sqlite3_step(pStmt)==SQLITE_ROW ){
115297	sqlite3_int64 nDoc = 0;
115298	sqlite3_int64 nByte = 0;
115299	const char *a = sqlite3_column_blob(pStmt, 0);
115300	if( a ){
115301	const char *pEnd = &a[sqlite3_column_bytes(pStmt, 0)];
115302	a += sqlite3Fts3GetVarint(a, &nDoc);
115303	while( a<pEnd ){
115304	a += sqlite3Fts3GetVarint(a, &nByte);
115305	}
115306	}
115307
115308	pCsr->nRowAvg = (((nByte / nDoc) + pgsz - 1) / pgsz);
115309	}
115310	rc = sqlite3_reset(pStmt);
115311	if( rc!=SQLITE_OK \|\| pCsr->nRowAvg==0 ) return rc;
115312	}
115313
115314	/* Assume that a blob flows over onto overflow pages if it is larger
115315	** than (pgsz-35) bytes in size (the file-format documentation
115316	** confirms this).
115317	*/
115318	for(iBlock=pReader->iStartBlock; iBlock<=pReader->iLeafEndBlock; iBlock++){
115319	rc = sqlite3Fts3ReadBlock(p, iBlock, 0, &nBlob);
115320	if( rc!=SQLITE_OK ) break;
115321	if( (nBlob+35)>pgsz ){
115322	int nOvfl = (nBlob + 34)/pgsz;
115323	nCost += ((nOvfl + pCsr->nRowAvg - 1)/pCsr->nRowAvg);
115324	}
115325	}
115326	}
115327
115328	*pnCost += nCost;
115329	return rc;
115330	}
115331
115332	/*
115333	** Free all allocations associated with the iterator passed as the
115334	** second argument.
115335	*/
115336	SQLITE_PRIVATE void sqlite3Fts3SegReaderFree(Fts3Table p, Fts3SegReader pReader){
115337	if( pReader && !fts3SegReaderIsPending(pReader) ){
115338	sqlite3_free(pReader->zTerm);
115339	if( !fts3SegReaderIsRootOnly(pReader) ){
115340	sqlite3_free(pReader->aNode);
115341	}
115342	}
115343	sqlite3_free(pReader);







115344	}
115345
115346	/*
115347	** Allocate a new SegReader object.
115348	*/
	@@ -113815,77 +115358,35 @@
115358	){
115359	int rc = SQLITE_OK; /* Return code */
115360	Fts3SegReader pReader; / Newly allocated SegReader object */
115361	int nExtra = 0; /* Bytes to allocate segment root node */
115362
115363	assert( iStartLeaf<=iEndLeaf );
115364	if( iStartLeaf==0 ){
115365	nExtra = nRoot + FTS3_NODE_PADDING;
115366	}
115367
115368	pReader = (Fts3SegReader *)sqlite3_malloc(sizeof(Fts3SegReader) + nExtra);
115369	if( !pReader ){
115370	return SQLITE_NOMEM;
115371	}
115372	memset(pReader, 0, sizeof(Fts3SegReader));

115373	pReader->iIdx = iAge;
115374	pReader->iStartBlock = iStartLeaf;
115375	pReader->iLeafEndBlock = iEndLeaf;
115376	pReader->iEndBlock = iEndBlock;
115377
115378	if( nExtra ){
115379	/* The entire segment is stored in the root node. */
115380	pReader->aNode = (char *)&pReader[1];
115381	pReader->nNode = nRoot;
115382	memcpy(pReader->aNode, zRoot, nRoot);
115383	memset(&pReader->aNode[nRoot], 0, FTS3_NODE_PADDING);
115384	}else{
115385	pReader->iCurrentBlock = iStartLeaf-1;
115386	}
115387












































115388	if( rc==SQLITE_OK ){
115389	*ppReader = pReader;
115390	}else{
115391	sqlite3Fts3SegReaderFree(p, pReader);
115392	}
	@@ -113976,11 +115477,10 @@
115477	}else{
115478	memset(pReader, 0, nByte);
115479	pReader->iIdx = 0x7FFFFFFF;
115480	pReader->ppNextElem = (Fts3HashElem **)&pReader[1];
115481	memcpy(pReader->ppNextElem, aElem, nElemsizeof(Fts3HashElem ));

115482	}
115483	}
115484
115485	if( isPrefix ){
115486	sqlite3_free(aElem);
	@@ -114218,11 +115718,11 @@
115718	/*
115719	** Add term zTerm to the SegmentNode. It is guaranteed that zTerm is larger
115720	** (according to memcmp) than the previous term.
115721	*/
115722	static int fts3NodeAddTerm(
115723	Fts3Table p, / Virtual table handle */
115724	SegmentNode *ppTree, / IN/OUT: SegmentNode handle */
115725	int isCopyTerm, /* True if zTerm/nTerm is transient */
115726	const char zTerm, / Pointer to buffer containing term */
115727	int nTerm /* Size of term in bytes */
115728	){
	@@ -114848,19 +116348,18 @@
116348	** for, then advance each segment iterator until it points to a term of
116349	** equal or greater value than the specified term. This prevents many
116350	** unnecessary merge/sort operations for the case where single segment
116351	** b-tree leaf nodes contain more than one term.
116352	*/
116353	for(i=0; i<nSegment; i++){
116354	int nTerm = pFilter->nTerm;
116355	const char *zTerm = pFilter->zTerm;
116356	Fts3SegReader *pSeg = apSegment[i];
116357	do {
116358	rc = fts3SegReaderNext(p, pSeg);
116359	if( rc!=SQLITE_OK ) goto finished;
116360	}while( zTerm && fts3SegReaderTermCmp(pSeg, zTerm, nTerm)<0 );

116361	}
116362
116363	fts3SegReaderSort(apSegment, nSegment, nSegment, fts3SegReaderCmp);
116364	while( apSegment[0]->aNode ){
116365	int nTerm = apSegment[0]->nTerm;
	@@ -114965,11 +116464,11 @@
116464	if( pFilter->zTerm && !isPrefix ){
116465	goto finished;
116466	}
116467
116468	for(i=0; i<nMerge; i++){
116469	rc = fts3SegReaderNext(p, apSegment[i]);
116470	if( rc!=SQLITE_OK ) goto finished;
116471	}
116472	fts3SegReaderSort(apSegment, nSegment, nMerge, fts3SegReaderCmp);
116473	}
116474
	@@ -114991,11 +116490,11 @@
116490	*/
116491	static int fts3SegmentMerge(Fts3Table *p, int iLevel){
116492	int i; /* Iterator variable */
116493	int rc; /* Return code */
116494	int iIdx; /* Index of new segment */
116495	int iNewLevel = 0; /* Level to create new segment at */
116496	sqlite3_stmt *pStmt = 0;
116497	SegmentWriter *pWriter = 0;
116498	int nSegment = 0; /* Number of segments being merged */
116499	Fts3SegReader *apSegment = 0; / Array of Segment iterators */
116500	Fts3SegReader pPending = 0; / Iterator for pending-terms */
	@@ -115280,64 +116779,76 @@
116779	sqlite3_step(pStmt);
116780	*pRC = sqlite3_reset(pStmt);
116781	}
116782
116783	/*
116784	** Record 0 of the %_stat table contains a blob consisting of N varints,
116785	** where N is the number of user defined columns in the fts3 table plus
116786	** two. If nCol is the number of user defined columns, then values of the
116787	** varints are set as follows:
116788	**
116789	** Varint 0: Total number of rows in the table.
116790	**
116791	** Varint 1..nCol: For each column, the total number of tokens stored in
116792	** the column for all rows of the table.
116793	**
116794	** Varint 1+nCol: The total size, in bytes, of all text values in all
116795	** columns of all rows of the table.
116796	**
116797	*/
116798	static void fts3UpdateDocTotals(
116799	int pRC, / The result code */
116800	Fts3Table p, / Table being updated */
116801	u32 aSzIns, / Size increases */
116802	u32 aSzDel, / Size decreases */
116803	int nChng /* Change in the number of documents */
116804	){
116805	char pBlob; / Storage for BLOB written into %_stat */
116806	int nBlob; /* Size of BLOB written into %_stat */
116807	u32 a; / Array of integers that becomes the BLOB */
116808	sqlite3_stmt pStmt; / Statement for reading and writing */
116809	int i; /* Loop counter */
116810	int rc; /* Result code from subfunctions */
116811
116812	const int nStat = p->nColumn+2;
116813
116814	if( *pRC ) return;
116815	a = sqlite3_malloc( (sizeof(u32)+10)*nStat );
116816	if( a==0 ){
116817	*pRC = SQLITE_NOMEM;
116818	return;
116819	}
116820	pBlob = (char*)&a[nStat];
116821	rc = fts3SqlStmt(p, SQL_SELECT_DOCTOTAL, &pStmt, 0);
116822	if( rc ){
116823	sqlite3_free(a);
116824	*pRC = rc;
116825	return;
116826	}
116827	if( sqlite3_step(pStmt)==SQLITE_ROW ){
116828	fts3DecodeIntArray(nStat, a,
116829	sqlite3_column_blob(pStmt, 0),
116830	sqlite3_column_bytes(pStmt, 0));
116831	}else{
116832	memset(a, 0, sizeof(u32)*(nStat) );
116833	}
116834	sqlite3_reset(pStmt);
116835	if( nChng<0 && a[0]<(u32)(-nChng) ){
116836	a[0] = 0;
116837	}else{
116838	a[0] += nChng;
116839	}
116840	for(i=0; i<p->nColumn+1; i++){
116841	u32 x = a[i+1];
116842	if( x+aSzIns[i] < aSzDel[i] ){
116843	x = 0;
116844	}else{
116845	x = x + aSzIns[i] - aSzDel[i];
116846	}
116847	a[i+1] = x;
116848	}
116849	fts3EncodeIntArray(nStat, a, pBlob, &nBlob);
116850	rc = fts3SqlStmt(p, SQL_REPLACE_DOCTOTAL, &pStmt, 0);
116851	if( rc ){
116852	sqlite3_free(a);
116853	*pRC = rc;
116854	return;
	@@ -115379,13 +116890,163 @@
116890	rc = SQLITE_OK;
116891	#endif
116892	}else{
116893	rc = SQLITE_ERROR;
116894	}
116895
116896	sqlite3Fts3SegmentsClose(p);
116897	return rc;
116898	}
116899
116900	/*
116901	** Return the deferred doclist associated with deferred token pDeferred.
116902	** This function assumes that sqlite3Fts3CacheDeferredDoclists() has already
116903	** been called to allocate and populate the doclist.
116904	*/
116905	SQLITE_PRIVATE char sqlite3Fts3DeferredDoclist(Fts3DeferredToken pDeferred, int *pnByte){
116906	if( pDeferred->pList ){
116907	*pnByte = pDeferred->pList->nData;
116908	return pDeferred->pList->aData;
116909	}
116910	*pnByte = 0;
116911	return 0;
116912	}
116913
116914	/*
116915	** Helper fucntion for FreeDeferredDoclists(). This function removes all
116916	** references to deferred doclists from within the tree of Fts3Expr
116917	** structures headed by
116918	*/
116919	static void fts3DeferredDoclistClear(Fts3Expr *pExpr){
116920	if( pExpr ){
116921	fts3DeferredDoclistClear(pExpr->pLeft);
116922	fts3DeferredDoclistClear(pExpr->pRight);
116923	if( pExpr->isLoaded ){
116924	sqlite3_free(pExpr->aDoclist);
116925	pExpr->isLoaded = 0;
116926	pExpr->aDoclist = 0;
116927	pExpr->nDoclist = 0;
116928	pExpr->pCurrent = 0;
116929	pExpr->iCurrent = 0;
116930	}
116931	}
116932	}
116933
116934	/*
116935	** Delete all cached deferred doclists. Deferred doclists are cached
116936	** (allocated) by the sqlite3Fts3CacheDeferredDoclists() function.
116937	*/
116938	SQLITE_PRIVATE void sqlite3Fts3FreeDeferredDoclists(Fts3Cursor *pCsr){
116939	Fts3DeferredToken *pDef;
116940	for(pDef=pCsr->pDeferred; pDef; pDef=pDef->pNext){
116941	sqlite3_free(pDef->pList);
116942	pDef->pList = 0;
116943	}
116944	if( pCsr->pDeferred ){
116945	fts3DeferredDoclistClear(pCsr->pExpr);
116946	}
116947	}
116948
116949	/*
116950	** Free all entries in the pCsr->pDeffered list. Entries are added to
116951	** this list using sqlite3Fts3DeferToken().
116952	*/
116953	SQLITE_PRIVATE void sqlite3Fts3FreeDeferredTokens(Fts3Cursor *pCsr){
116954	Fts3DeferredToken *pDef;
116955	Fts3DeferredToken *pNext;
116956	for(pDef=pCsr->pDeferred; pDef; pDef=pNext){
116957	pNext = pDef->pNext;
116958	sqlite3_free(pDef->pList);
116959	sqlite3_free(pDef);
116960	}
116961	pCsr->pDeferred = 0;
116962	}
116963
116964	/*
116965	** Generate deferred-doclists for all tokens in the pCsr->pDeferred list
116966	** based on the row that pCsr currently points to.
116967	**
116968	** A deferred-doclist is like any other doclist with position information
116969	** included, except that it only contains entries for a single row of the
116970	** table, not for all rows.
116971	*/
116972	SQLITE_PRIVATE int sqlite3Fts3CacheDeferredDoclists(Fts3Cursor *pCsr){
116973	int rc = SQLITE_OK; /* Return code */
116974	if( pCsr->pDeferred ){
116975	int i; /* Used to iterate through table columns */
116976	sqlite3_int64 iDocid; /* Docid of the row pCsr points to */
116977	Fts3DeferredToken pDef; / Used to iterate through deferred tokens */
116978
116979	Fts3Table p = (Fts3Table )pCsr->base.pVtab;
116980	sqlite3_tokenizer *pT = p->pTokenizer;
116981	sqlite3_tokenizer_module const *pModule = pT->pModule;
116982
116983	assert( pCsr->isRequireSeek==0 );
116984	iDocid = sqlite3_column_int64(pCsr->pStmt, 0);
116985
116986	for(i=0; i<p->nColumn && rc==SQLITE_OK; i++){
116987	const char zText = (const char )sqlite3_column_text(pCsr->pStmt, i+1);
116988	sqlite3_tokenizer_cursor *pTC = 0;
116989
116990	rc = pModule->xOpen(pT, zText, -1, &pTC);
116991	while( rc==SQLITE_OK ){
116992	char const zToken; / Buffer containing token */
116993	int nToken; /* Number of bytes in token */
116994	int iDum1, iDum2; /* Dummy variables */
116995	int iPos; /* Position of token in zText */
116996
116997	pTC->pTokenizer = pT;
116998	rc = pModule->xNext(pTC, &zToken, &nToken, &iDum1, &iDum2, &iPos);
116999	for(pDef=pCsr->pDeferred; pDef && rc==SQLITE_OK; pDef=pDef->pNext){
117000	Fts3PhraseToken *pPT = pDef->pToken;
117001	if( (pDef->iCol>=p->nColumn \|\| pDef->iCol==i)
117002	&& (pPT->n==nToken \|\| (pPT->isPrefix && pPT->n<nToken))
117003	&& (0==memcmp(zToken, pPT->z, pPT->n))
117004	){
117005	fts3PendingListAppend(&pDef->pList, iDocid, i, iPos, &rc);
117006	}
117007	}
117008	}
117009	if( pTC ) pModule->xClose(pTC);
117010	if( rc==SQLITE_DONE ) rc = SQLITE_OK;
117011	}
117012
117013	for(pDef=pCsr->pDeferred; pDef && rc==SQLITE_OK; pDef=pDef->pNext){
117014	if( pDef->pList ){
117015	rc = fts3PendingListAppendVarint(&pDef->pList, 0);
117016	}
117017	}
117018	}
117019
117020	return rc;
117021	}
117022
117023	/*
117024	** Add an entry for token pToken to the pCsr->pDeferred list.
117025	*/
117026	SQLITE_PRIVATE int sqlite3Fts3DeferToken(
117027	Fts3Cursor pCsr, / Fts3 table cursor */
117028	Fts3PhraseToken pToken, / Token to defer */
117029	int iCol /* Column that token must appear in (or -1) */
117030	){
117031	Fts3DeferredToken *pDeferred;
117032	pDeferred = sqlite3_malloc(sizeof(*pDeferred));
117033	if( !pDeferred ){
117034	return SQLITE_NOMEM;
117035	}
117036	memset(pDeferred, 0, sizeof(*pDeferred));
117037	pDeferred->pToken = pToken;
117038	pDeferred->pNext = pCsr->pDeferred;
117039	pDeferred->iCol = iCol;
117040	pCsr->pDeferred = pDeferred;
117041
117042	assert( pToken->pDeferred==0 );
117043	pToken->pDeferred = pDeferred;
117044
117045	return SQLITE_OK;
117046	}
117047
117048
117049	/*
117050	** This function does the work for the xUpdate method of FTS3 virtual
117051	** tables.
117052	*/
	@@ -115401,20 +117062,21 @@
117062	sqlite3_int64 iRemove = 0; /* Rowid removed by UPDATE or DELETE */
117063	u32 aSzIns; / Sizes of inserted documents */
117064	u32 aSzDel; / Sizes of deleted documents */
117065	int nChng = 0; /* Net change in number of documents */
117066
117067	assert( p->pSegments==0 );
117068
117069	/* Allocate space to hold the change in document sizes */
117070	aSzIns = sqlite3_malloc( sizeof(aSzIns[0])(p->nColumn+1)2 );
117071	if( aSzIns==0 ) return SQLITE_NOMEM;
117072	aSzDel = &aSzIns[p->nColumn+1];
117073	memset(aSzIns, 0, sizeof(aSzIns[0])(p->nColumn+1)2);
117074
117075	/* If this is a DELETE or UPDATE operation, remove the old record. */
117076	if( sqlite3_value_type(apVal[0])!=SQLITE_NULL ){
117077	int isEmpty = 0;
117078	rc = fts3IsEmpty(p, apVal, &isEmpty);
117079	if( rc==SQLITE_OK ){
117080	if( isEmpty ){
117081	/* Deleting this row means the whole table is empty. In this case
117082	** delete the contents of all three tables and throw away any
	@@ -115427,12 +117089,12 @@
117089	rc = fts3PendingTermsDocid(p, iRemove);
117090	fts3DeleteTerms(&rc, p, apVal, aSzDel);
117091	fts3SqlExec(&rc, p, SQL_DELETE_CONTENT, apVal);
117092	if( p->bHasDocsize ){
117093	fts3SqlExec(&rc, p, SQL_DELETE_DOCSIZE, apVal);

117094	}
117095	nChng--;
117096	}
117097	}
117098	}else if( sqlite3_value_type(apVal[p->nColumn+2])!=SQLITE_NULL ){
117099	sqlite3_free(aSzIns);
117100	return fts3SpecialInsert(p, apVal[p->nColumn+2]);
	@@ -115446,20 +117108,21 @@
117108	}
117109	if( rc==SQLITE_OK ){
117110	rc = fts3InsertTerms(p, apVal, aSzIns);
117111	}
117112	if( p->bHasDocsize ){

117113	fts3InsertDocsize(&rc, p, aSzIns);
117114	}
117115	nChng++;
117116	}
117117
117118	if( p->bHasStat ){
117119	fts3UpdateDocTotals(&rc, p, aSzIns, aSzDel, nChng);
117120	}
117121
117122	sqlite3_free(aSzIns);
117123	sqlite3Fts3SegmentsClose(p);
117124	return rc;
117125	}
117126
117127	/*
117128	** Flush any data in the pending-terms hash table to disk. If successful,
	@@ -115479,10 +117142,11 @@
117142	}else{
117143	sqlite3_exec(p->db, "ROLLBACK TO fts3", 0, 0, 0);
117144	sqlite3_exec(p->db, "RELEASE fts3", 0, 0, 0);
117145	}
117146	}
117147	sqlite3Fts3SegmentsClose(p);
117148	return rc;
117149	}
117150
117151	#endif
117152
	@@ -115509,11 +117173,11 @@
117173	** Used as an fts3ExprIterate() context when loading phrase doclists to
117174	** Fts3Expr.aDoclist[]/nDoclist.
117175	*/
117176	typedef struct LoadDoclistCtx LoadDoclistCtx;
117177	struct LoadDoclistCtx {
117178	Fts3Cursor pCsr; / FTS3 Cursor */
117179	int nPhrase; /* Number of phrases seen so far */
117180	int nToken; /* Number of tokens seen so far */
117181	};
117182
117183	/*
	@@ -115703,11 +117367,11 @@
117367
117368	p->nPhrase++;
117369	p->nToken += pExpr->pPhrase->nToken;
117370
117371	if( pExpr->isLoaded==0 ){
117372	rc = sqlite3Fts3ExprLoadDoclist(p->pCsr, pExpr);
117373	pExpr->isLoaded = 1;
117374	if( rc==SQLITE_OK ){
117375	rc = fts3ExprNearTrim(pExpr);
117376	}
117377	}
	@@ -115746,11 +117410,11 @@
117410	int pnPhrase, / OUT: Number of phrases in query */
117411	int pnToken / OUT: Number of tokens in query */
117412	){
117413	int rc; /* Return Code */
117414	LoadDoclistCtx sCtx = {0,0,0}; /* Context for fts3ExprIterate() */
117415	sCtx.pCsr = pCsr;
117416	rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx);
117417	if( rc==SQLITE_OK ){
117418	(void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0);
117419	}
117420	if( pnPhrase ) *pnPhrase = sCtx.nPhrase;
	@@ -116277,24 +117941,51 @@
117941	Fts3Expr pExpr, / Phrase expression node */
117942	int iPhrase, /* Phrase number (numbered from zero) */
117943	void pCtx / Pointer to MatchInfo structure */
117944	){
117945	MatchInfo p = (MatchInfo )pCtx;
117946	Fts3Cursor *pCsr = p->pCursor;
117947	char *pIter;
117948	char *pEnd;
117949	char *pFree = 0;
117950	const int iStart = 2 + (iPhrase * p->nCol * 3) + 1;
117951
117952	assert( pExpr->isLoaded );
117953	assert( pExpr->eType==FTSQUERY_PHRASE );
117954
117955	if( pCsr->pDeferred ){
117956	Fts3Phrase *pPhrase = pExpr->pPhrase;
117957	int ii;
117958	for(ii=0; ii<pPhrase->nToken; ii++){
117959	if( pPhrase->aToken[ii].bFulltext ) break;
117960	}
117961	if( ii<pPhrase->nToken ){
117962	int nFree = 0;
117963	int rc = sqlite3Fts3ExprLoadFtDoclist(pCsr, pExpr, &pFree, &nFree);
117964	if( rc!=SQLITE_OK ) return rc;
117965	pIter = pFree;
117966	pEnd = &pFree[nFree];
117967	}else{
117968	int nDoc = p->aMatchinfo[2 + 3p->nColp->aMatchinfo[0]];
117969	for(ii=0; ii<p->nCol; ii++){
117970	p->aMatchinfo[iStart + ii*3] = nDoc;
117971	p->aMatchinfo[iStart + ii*3 + 1] = nDoc;
117972	}
117973	return SQLITE_OK;
117974	}
117975	}else{
117976	pIter = pExpr->aDoclist;
117977	pEnd = &pExpr->aDoclist[pExpr->nDoclist];
117978	}
117979
117980	/* Fill in the global hit count matrix row for this phrase. */
117981	while( pIter<pEnd ){
117982	while( pIter++ & 0x80 ); / Skip past docid. */
117983	fts3LoadColumnlistCounts(&pIter, &p->aMatchinfo[iStart], 1);


117984	}
117985
117986	sqlite3_free(pFree);
117987	return SQLITE_OK;
117988	}
117989
117990	/*
117991	** fts3ExprIterate() callback used to collect the "local" matchinfo stats
	@@ -116359,19 +118050,18 @@
118050	if( !sInfo.aMatchinfo ){
118051	return SQLITE_NOMEM;
118052	}
118053	memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo);
118054

118055	/* First element of match-info is the number of phrases in the query */
118056	sInfo.aMatchinfo[0] = nPhrase;
118057	sInfo.aMatchinfo[1] = sInfo.nCol;

118058	if( pTab->bHasDocsize ){
118059	int ofst = 2 + 3sInfo.aMatchinfo[0]sInfo.aMatchinfo[1];
118060	rc = sqlite3Fts3MatchinfoDocsizeGlobal(pCsr, &sInfo.aMatchinfo[ofst]);
118061	}
118062	(void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);
118063	pCsr->aMatchinfo = sInfo.aMatchinfo;
118064	pCsr->isMatchinfoNeeded = 1;
118065	}
118066
118067	sInfo.aMatchinfo = pCsr->aMatchinfo;
	@@ -116477,10 +118167,11 @@
118167	i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res
118168	);
118169	}
118170
118171	snippet_out:
118172	sqlite3Fts3SegmentsClose(pTab);
118173	if( rc!=SQLITE_OK ){
118174	sqlite3_result_error_code(pCtx, rc);
118175	sqlite3_free(res.z);
118176	}else{
118177	sqlite3_result_text(pCtx, res.z, -1, sqlite3_free);
	@@ -116656,10 +118347,11 @@
118347	}
118348
118349	offsets_out:
118350	sqlite3_free(sCtx.aTerm);
118351	assert( rc!=SQLITE_DONE );
118352	sqlite3Fts3SegmentsClose(pTab);
118353	if( rc!=SQLITE_OK ){
118354	sqlite3_result_error_code(pCtx, rc);
118355	sqlite3_free(res.z);
118356	}else{
118357	sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free);
	@@ -116675,10 +118367,11 @@
118367	if( !pCsr->pExpr ){
118368	sqlite3_result_blob(pContext, "", 0, SQLITE_STATIC);
118369	return;
118370	}
118371	rc = fts3GetMatchinfo(pCsr);
118372	sqlite3Fts3SegmentsClose((Fts3Table *)pCsr->base.pVtab );
118373	if( rc!=SQLITE_OK ){
118374	sqlite3_result_error_code(pContext, rc);
118375	}else{
118376	Fts3Table pTab = (Fts3Table)pCsr->base.pVtab;
118377	int n = sizeof(u32)(2+pCsr->aMatchinfo[0]pCsr->aMatchinfo[1]*3);
118378

M src/sqlite3.h

+51 -11

		--- src/sqlite3.h
		+++ src/sqlite3.h
		@@ -105,13 +105,13 @@
105	105	**
106	106	** See also: [sqlite3_libversion()],
107	107	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
108	108	** [sqlite_version()] and [sqlite_source_id()].
109	109	*/
110		-#define SQLITE_VERSION "3.7.3"
111		-#define SQLITE_VERSION_NUMBER 3007003
112		-#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e"
	110	+#define SQLITE_VERSION "3.7.4"
	111	+#define SQLITE_VERSION_NUMBER 3007004
	112	+#define SQLITE_SOURCE_ID "2010-11-15 16:29:31 136c2ac24ee1663bc0904bce1a619ecef3d11c1c"
113	113
114	114	/*
115	115	** CAPI3REF: Run-Time Library Version Numbers
116	116	** KEYWORDS: sqlite3_version, sqlite3_sourceid
117	117	**
		@@ -2730,11 +2730,14 @@
2730	2730	** ^If the fourth parameter is negative, the length of the string is
2731	2731	** the number of bytes up to the first zero terminator.
2732	2732	**
2733	2733	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
2734	2734	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
2735		-** string after SQLite has finished with it. ^If the fifth argument is
	2735	+** string after SQLite has finished with it. ^The destructor is called
	2736	+** to dispose of the BLOB or string even if the call to sqlite3_bind_blob(),
	2737	+** sqlite3_bind_text(), or sqlite3_bind_text16() fails.
	2738	+** ^If the fifth argument is
2736	2739	** the special value [SQLITE_STATIC], then SQLite assumes that the
2737	2740	** information is in static, unmanaged space and does not need to be freed.
2738	2741	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
2739	2742	** SQLite makes its own private copy of the data immediately, before
2740	2743	** the sqlite3_bind_*() routine returns.
		@@ -3370,16 +3373,19 @@
3370	3373	** parameters. ^An aggregate SQL function requires an implementation of xStep
3371	3374	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3372	3375	** SQL function or aggregate, pass NULL poiners for all three function
3373	3376	** callbacks.
3374	3377	**
3375		-** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3376		-** then it is invoked when the function is deleted, either by being
3377		-** overloaded or when the database connection closes.
3378		-** ^When the destructure callback of the tenth parameter is invoked, it
3379		-** is passed a single argument which is a copy of the pointer which was
3380		-** the fifth parameter to sqlite3_create_function_v2().
	3378	+** ^(If the tenth parameter to sqlite3_create_function_v2() is not NULL,
	3379	+** then it is destructor for the application data pointer.
	3380	+** The destructor is invoked when the function is deleted, either by being
	3381	+** overloaded or when the database connection closes.)^
	3382	+** ^The destructor is also invoked if the call to
	3383	+** sqlite3_create_function_v2() fails.
	3384	+** ^When the destructor callback of the tenth parameter is invoked, it
	3385	+** is passed a single argument which is a copy of the application data
	3386	+** pointer which was the fifth parameter to sqlite3_create_function_v2().
3381	3387	**
3382	3388	** ^It is permitted to register multiple implementations of the same
3383	3389	** functions with the same name but with either differing numbers of
3384	3390	** arguments or differing preferred text encodings. ^SQLite will use
3385	3391	** the implementation that most closely matches the way in which the
		@@ -3838,10 +3844,19 @@
3838	3844	** with the addition that the xDestroy callback is invoked on pArg when
3839	3845	** the collating function is deleted.
3840	3846	** ^Collating functions are deleted when they are overridden by later
3841	3847	** calls to the collation creation functions or when the
3842	3848	** [database connection] is closed using [sqlite3_close()].
	3849	+**
	3850	+** ^The xDestroy callback is <u>not</u> called if the
	3851	+** sqlite3_create_collation_v2() function fails. Applications that invoke
	3852	+** sqlite3_create_collation_v2() with a non-NULL xDestroy argument should
	3853	+** check the return code and dispose of the application data pointer
	3854	+** themselves rather than expecting SQLite to deal with it for them.
	3855	+** This is different from every other SQLite interface. The inconsistency
	3856	+** is unfortunate but cannot be changed without breaking backwards
	3857	+** compatibility.
3843	3858	**
3844	3859	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
3845	3860	*/
3846	3861	SQLITE_API int sqlite3_create_collation(
3847	3862	sqlite3*,
		@@ -4593,11 +4608,13 @@
4593	4608	** when a new virtual table is be being created or reinitialized.
4594	4609	**
4595	4610	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
4596	4611	** is a pointer to a destructor for the pClientData. ^SQLite will
4597	4612	** invoke the destructor function (if it is not NULL) when SQLite
4598		-** no longer needs the pClientData pointer. ^The sqlite3_create_module()
	4613	+** no longer needs the pClientData pointer. ^The destructor will also
	4614	+** be invoked if the call to sqlite3_create_module_v2() fails.
	4615	+** ^The sqlite3_create_module()
4599	4616	** interface is equivalent to sqlite3_create_module_v2() with a NULL
4600	4617	** destructor.
4601	4618	*/
4602	4619	SQLITE_API int sqlite3_create_module(
4603	4620	sqlite3 db, / SQLite connection to register module with */
		@@ -4776,10 +4793,33 @@
4776	4793	sqlite3_int64 iRow,
4777	4794	int flags,
4778	4795	sqlite3_blob **ppBlob
4779	4796	);
4780	4797
	4798	+/*
	4799	+** CAPI3REF: Move a BLOB Handle to a New Row
	4800	+**
	4801	+** ^This function is used to move an existing blob handle so that it points
	4802	+** to a different row of the same database table. ^The new row is identified
	4803	+** by the rowid value passed as the second argument. Only the row can be
	4804	+** changed. ^The database, table and column on which the blob handle is open
	4805	+** remain the same. Moving an existing blob handle to a new row can be
	4806	+** faster than closing the existing handle and opening a new one.
	4807	+**
	4808	+** ^(The new row must meet the same criteria as for [sqlite3_blob_open()] -
	4809	+** it must exist and there must be either a blob or text value stored in
	4810	+** the nominated column.)^ ^If the new row is not present in the table, or if
	4811	+** it does not contain a blob or text value, or if another error occurs, an
	4812	+** SQLite error code is returned and the blob handle is considered aborted.
	4813	+** ^All subsequent calls to [sqlite3_blob_read()], [sqlite3_blob_write()] or
	4814	+** [sqlite3_blob_reopen()] on an aborted blob handle immediately return
	4815	+** SQLITE_ABORT.
	4816	+**
	4817	+** ^This function sets the database handle error code and message.
	4818	+*/
	4819	+SQLITE_API SQLITE_EXPERIMENTAL int sqlite3_blob_reopen(sqlite3_blob *, sqlite3_int64);
	4820	+
4781	4821	/*
4782	4822	** CAPI3REF: Close A BLOB Handle
4783	4823	**
4784	4824	** ^Closes an open [BLOB handle].
4785	4825	**
4786	4826

	--- src/sqlite3.h
	+++ src/sqlite3.h
	@@ -105,13 +105,13 @@
105	**
106	** See also: [sqlite3_libversion()],
107	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
108	** [sqlite_version()] and [sqlite_source_id()].
109	*/
110	#define SQLITE_VERSION "3.7.3"
111	#define SQLITE_VERSION_NUMBER 3007003
112	#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e"
113
114	/*
115	** CAPI3REF: Run-Time Library Version Numbers
116	** KEYWORDS: sqlite3_version, sqlite3_sourceid
117	**
	@@ -2730,11 +2730,14 @@
2730	** ^If the fourth parameter is negative, the length of the string is
2731	** the number of bytes up to the first zero terminator.
2732	**
2733	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
2734	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
2735	** string after SQLite has finished with it. ^If the fifth argument is



2736	** the special value [SQLITE_STATIC], then SQLite assumes that the
2737	** information is in static, unmanaged space and does not need to be freed.
2738	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
2739	** SQLite makes its own private copy of the data immediately, before
2740	** the sqlite3_bind_*() routine returns.
	@@ -3370,16 +3373,19 @@
3370	** parameters. ^An aggregate SQL function requires an implementation of xStep
3371	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3372	** SQL function or aggregate, pass NULL poiners for all three function
3373	** callbacks.
3374	**
3375	** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3376	** then it is invoked when the function is deleted, either by being
3377	** overloaded or when the database connection closes.
3378	** ^When the destructure callback of the tenth parameter is invoked, it
3379	** is passed a single argument which is a copy of the pointer which was
3380	** the fifth parameter to sqlite3_create_function_v2().



3381	**
3382	** ^It is permitted to register multiple implementations of the same
3383	** functions with the same name but with either differing numbers of
3384	** arguments or differing preferred text encodings. ^SQLite will use
3385	** the implementation that most closely matches the way in which the
	@@ -3838,10 +3844,19 @@
3838	** with the addition that the xDestroy callback is invoked on pArg when
3839	** the collating function is deleted.
3840	** ^Collating functions are deleted when they are overridden by later
3841	** calls to the collation creation functions or when the
3842	** [database connection] is closed using [sqlite3_close()].









3843	**
3844	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
3845	*/
3846	SQLITE_API int sqlite3_create_collation(
3847	sqlite3*,
	@@ -4593,11 +4608,13 @@
4593	** when a new virtual table is be being created or reinitialized.
4594	**
4595	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
4596	** is a pointer to a destructor for the pClientData. ^SQLite will
4597	** invoke the destructor function (if it is not NULL) when SQLite
4598	** no longer needs the pClientData pointer. ^The sqlite3_create_module()


4599	** interface is equivalent to sqlite3_create_module_v2() with a NULL
4600	** destructor.
4601	*/
4602	SQLITE_API int sqlite3_create_module(
4603	sqlite3 db, / SQLite connection to register module with */
	@@ -4776,10 +4793,33 @@
4776	sqlite3_int64 iRow,
4777	int flags,
4778	sqlite3_blob **ppBlob
4779	);
4780























4781	/*
4782	** CAPI3REF: Close A BLOB Handle
4783	**
4784	** ^Closes an open [BLOB handle].
4785	**
4786

	--- src/sqlite3.h
	+++ src/sqlite3.h
	@@ -105,13 +105,13 @@
105	**
106	** See also: [sqlite3_libversion()],
107	** [sqlite3_libversion_number()], [sqlite3_sourceid()],
108	** [sqlite_version()] and [sqlite_source_id()].
109	*/
110	#define SQLITE_VERSION "3.7.4"
111	#define SQLITE_VERSION_NUMBER 3007004
112	#define SQLITE_SOURCE_ID "2010-11-15 16:29:31 136c2ac24ee1663bc0904bce1a619ecef3d11c1c"
113
114	/*
115	** CAPI3REF: Run-Time Library Version Numbers
116	** KEYWORDS: sqlite3_version, sqlite3_sourceid
117	**
	@@ -2730,11 +2730,14 @@
2730	** ^If the fourth parameter is negative, the length of the string is
2731	** the number of bytes up to the first zero terminator.
2732	**
2733	** ^The fifth argument to sqlite3_bind_blob(), sqlite3_bind_text(), and
2734	** sqlite3_bind_text16() is a destructor used to dispose of the BLOB or
2735	** string after SQLite has finished with it. ^The destructor is called
2736	** to dispose of the BLOB or string even if the call to sqlite3_bind_blob(),
2737	** sqlite3_bind_text(), or sqlite3_bind_text16() fails.
2738	** ^If the fifth argument is
2739	** the special value [SQLITE_STATIC], then SQLite assumes that the
2740	** information is in static, unmanaged space and does not need to be freed.
2741	** ^If the fifth argument has the value [SQLITE_TRANSIENT], then
2742	** SQLite makes its own private copy of the data immediately, before
2743	** the sqlite3_bind_*() routine returns.
	@@ -3370,16 +3373,19 @@
3373	** parameters. ^An aggregate SQL function requires an implementation of xStep
3374	** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing
3375	** SQL function or aggregate, pass NULL poiners for all three function
3376	** callbacks.
3377	**
3378	** ^(If the tenth parameter to sqlite3_create_function_v2() is not NULL,
3379	** then it is destructor for the application data pointer.
3380	** The destructor is invoked when the function is deleted, either by being
3381	** overloaded or when the database connection closes.)^
3382	** ^The destructor is also invoked if the call to
3383	** sqlite3_create_function_v2() fails.
3384	** ^When the destructor callback of the tenth parameter is invoked, it
3385	** is passed a single argument which is a copy of the application data
3386	** pointer which was the fifth parameter to sqlite3_create_function_v2().
3387	**
3388	** ^It is permitted to register multiple implementations of the same
3389	** functions with the same name but with either differing numbers of
3390	** arguments or differing preferred text encodings. ^SQLite will use
3391	** the implementation that most closely matches the way in which the
	@@ -3838,10 +3844,19 @@
3844	** with the addition that the xDestroy callback is invoked on pArg when
3845	** the collating function is deleted.
3846	** ^Collating functions are deleted when they are overridden by later
3847	** calls to the collation creation functions or when the
3848	** [database connection] is closed using [sqlite3_close()].
3849	**
3850	** ^The xDestroy callback is <u>not</u> called if the
3851	** sqlite3_create_collation_v2() function fails. Applications that invoke
3852	** sqlite3_create_collation_v2() with a non-NULL xDestroy argument should
3853	** check the return code and dispose of the application data pointer
3854	** themselves rather than expecting SQLite to deal with it for them.
3855	** This is different from every other SQLite interface. The inconsistency
3856	** is unfortunate but cannot be changed without breaking backwards
3857	** compatibility.
3858	**
3859	** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()].
3860	*/
3861	SQLITE_API int sqlite3_create_collation(
3862	sqlite3*,
	@@ -4593,11 +4608,13 @@
4608	** when a new virtual table is be being created or reinitialized.
4609	**
4610	** ^The sqlite3_create_module_v2() interface has a fifth parameter which
4611	** is a pointer to a destructor for the pClientData. ^SQLite will
4612	** invoke the destructor function (if it is not NULL) when SQLite
4613	** no longer needs the pClientData pointer. ^The destructor will also
4614	** be invoked if the call to sqlite3_create_module_v2() fails.
4615	** ^The sqlite3_create_module()
4616	** interface is equivalent to sqlite3_create_module_v2() with a NULL
4617	** destructor.
4618	*/
4619	SQLITE_API int sqlite3_create_module(
4620	sqlite3 db, / SQLite connection to register module with */
	@@ -4776,10 +4793,33 @@
4793	sqlite3_int64 iRow,
4794	int flags,
4795	sqlite3_blob **ppBlob
4796	);
4797
4798	/*
4799	** CAPI3REF: Move a BLOB Handle to a New Row
4800	**
4801	** ^This function is used to move an existing blob handle so that it points
4802	** to a different row of the same database table. ^The new row is identified
4803	** by the rowid value passed as the second argument. Only the row can be
4804	** changed. ^The database, table and column on which the blob handle is open
4805	** remain the same. Moving an existing blob handle to a new row can be
4806	** faster than closing the existing handle and opening a new one.
4807	**
4808	** ^(The new row must meet the same criteria as for [sqlite3_blob_open()] -
4809	** it must exist and there must be either a blob or text value stored in
4810	** the nominated column.)^ ^If the new row is not present in the table, or if
4811	** it does not contain a blob or text value, or if another error occurs, an
4812	** SQLite error code is returned and the blob handle is considered aborted.
4813	** ^All subsequent calls to [sqlite3_blob_read()], [sqlite3_blob_write()] or
4814	** [sqlite3_blob_reopen()] on an aborted blob handle immediately return
4815	** SQLITE_ABORT.
4816	**
4817	** ^This function sets the database handle error code and message.
4818	*/
4819	SQLITE_API SQLITE_EXPERIMENTAL int sqlite3_blob_reopen(sqlite3_blob *, sqlite3_int64);
4820
4821	/*
4822	** CAPI3REF: Close A BLOB Handle
4823	**
4824	** ^Closes an open [BLOB handle].
4825	**
4826

Fossil SCM

Keyboard Shortcuts