Fossil Forum

Problem

When files are named on the command line, fossil commit still checks every file in the checkout, so a one-file commit gets slower as the checkout grows.

Measurements

One-file commits (fossil commit --nosync -m x FILE) with repo-cksum off, trunk 76742655f0, Windows 11, NTFS, three runs each:

I use Fossil as the store of an automatic file-history tool that learns from the file system which files changed and commits only those, 100 to 200 times per hour. For this use the cost of a commit should follow the size of the change, not the size of the checkout.

Cause

commit_cmd() calls unsaved_changes() before select_commit_files(), and unsaved_changes() runs vfile_check_signature() over the whole checkout. Only the selected files are stored (is_selected(id)), and the other files keep their vfile.mrid in the new manifest, so checking them does not change the result.

Suggestion

Would you consider checking only the named files when files are named? The patch below does this in two places:

• commit_cmd() calls select_commit_files() before unsaved_changes() (the block is moved, not changed);
• vfile_check_signature() adds AND is_selected(id) to its query. is_selected() is true for every file when no files are named, so status, changes and a plain commit behave as before.

No new option or help text. The only difference I know of is that a partial commit no longer refreshes vfile.chnged and vfile.mtime of the files it does not commit; status and changes check them anyway.

If you would rather not change the default, the same could be offered as an option.

--- src/checkin.c
+++ src/checkin.c
@@ -2749,6 +2749,29 @@
     qsort((void*)sCiInfo.azTag, nTag, sizeof(sCiInfo.azTag[0]), tagCmp);
   }

+  /* There are two ways this command may be executed. If there are
+  ** no arguments following the word "commit", then all modified files
+  ** in the checked-out directory are committed. If one or more arguments
+  ** follows "commit", then only those files are committed.
+  **
+  ** After the following function call has returned, the Global.aCommitFile[]
+  ** array is allocated to contain the "id" field from the vfile table
+  ** for each file to be committed. Or, if aCommitFile is NULL, all files
+  ** should be committed.
+  */
+  if( select_commit_files() ){
+    if( !noPrompt ){
+      prompt_user("continue (y/N)? ", &ans);
+      cReply = blob_str(&ans)[0];
+      blob_reset(&ans);
+    }else{
+      cReply = 'N';
+    }
+    if( cReply!='y' && cReply!='Y' ){
+      fossil_exit(1);
+    }
+  }
+
   hasChanges = unsaved_changes(useHash ? CKSIG_HASH : 0);
   if( hasChanges==0 && onlyIfChanges ){
     /* "fossil commit --if-changes" is a no-op if there are no changes. */
@@ -2812,28 +2835,6 @@
     }
   }

-  /* There are two ways this command may be executed. If there are
-  ** no arguments following the word "commit", then all modified files
-  ** in the checked-out directory are committed. If one or more arguments
-  ** follows "commit", then only those files are committed.
-  **
-  ** After the following function call has returned, the Global.aCommitFile[]
-  ** array is allocated to contain the "id" field from the vfile table
-  ** for each file to be committed. Or, if aCommitFile is NULL, all files
-  ** should be committed.
-  */
-  if( select_commit_files() ){
-    if( !noPrompt ){
-      prompt_user("continue (y/N)? ", &ans);
-      cReply = blob_str(&ans)[0];
-      blob_reset(&ans);
-    }else{
-      cReply = 'N';
-    }
-    if( cReply!='y' && cReply!='Y' ){
-      fossil_exit(1);
-    }
-  }
   isAMerge = db_exists("SELECT 1 FROM vmerge WHERE id=0 OR id<-2");
   if( g.aCommitFile && isAMerge ){
     fossil_fatal("cannot do a partial commit of a merge");
--- src/vfile.c
+++ src/vfile.c
@@ -181,7 +181,8 @@
                  "       vfile.mrid, deleted, chnged, uuid, size, mtime,"
                  "      CASE WHEN isexe THEN %d WHEN islink THEN %d ELSE %d END"
                  "  FROM vfile LEFT JOIN blob ON vfile.mrid=blob.rid"
-                 " WHERE vid=%d ", g.zLocalRoot, PERM_EXE, PERM_LNK, PERM_REG,
+                 " WHERE vid=%d AND is_selected(id)",
+                 g.zLocalRoot, PERM_EXE, PERM_LNK, PERM_REG,
                  vid);
   while( db_step(&q)==SQLITE_ROW ){
     int id, rid, isDeleted;

Delta manifests in this use

With the patch, most of the remaining cost is building the baseline manifest (7.8 MB of F-cards for 58,000 files). With --delta (and forbid-delta-manifests off) Fossil does not build it, and the one-file commit drops from 510 ms to 335 ms (last column above). The tool would commit with --delta and make a baseline now and then. I understand why delta manifests are off by default and have read delta-manifests.md; for commits that name a few files in a large checkout they are still a clear win, so I hope --delta stays available.

Testing

I ran the same steps in two checkouts, one committing with trunk and one with trunk + patch, and compared the files of the new check-in (files_of_checkin('tip')), the output of fossil changes, the messages and the exit codes after each step. They were identical for:

1. a named edited file, with another file edited and a third file only touched but not named;
2. a renamed file (fossil mv), naming the new name;
3. a removed file (fossil rm);
4. an added file;
5. a named file that did not change ("none of the selected files have changed");
6. a name Fossil does not know ("fossil knows nothing about");
7. a named directory;
8. a commit without names (the unnamed edits of the earlier steps are committed);
9. a commit without names when nothing changed ("nothing has changed").

I tested on Windows only.