Shawn O. Pearce  committed ac053c0

Allow frontends to bidirectionally communicate with fast-import

The existing checkpoint command is very useful to force fast-import
to dump the branches out to disk so that standard Git tools can
access them and the objects they refer to. However there was not a
way to know when fast-import had finished executing the checkpoint
and it was safe to read those refs.

The progress command can be used to make fast-import output any
message of the frontend's choosing to standard out. The frontend
can scan for these messages using select() or poll() to monitor a
pipe connected to the standard output of fast-import.

Signed-off-by: Shawn O. Pearce <>

  • Participants
  • Parent commits 1fdb649

Comments (0)

Files changed (3)

File Documentation/git-fast-import.txt

 	This command is optional and is not needed to perform
 	an import.
+	Causes fast-import to echo the entire line to its own
+	standard output.  This command is optional and is not needed
+	to perform an import.
 Create or update a branch with a new commit, recording one logical
 The `LF` after the command is optional (it used to be required).
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream.  The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+	'progress' SP <any> LF
+	LF?
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`.  The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+	frontend | git-fast-import | sed 's/^progress //'
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
 Tips and Tricks
 The following tips and tricks have been collected from various
 You only need to expend the effort once, and everyone using your
 project will benefit from the smaller repository.
+Include Some Progress Messages
+Every once in a while have your frontend emit a `progress` message
+to fast-import.  The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
 Packfile Optimization

File fast-import.c

         | new_tag
         | reset_branch
         | checkpoint
+        | progress
   new_blob ::= 'blob' lf
   checkpoint ::= 'checkpoint' lf
+  progress ::= 'progress' sp not_lf* lf
+    lf?;
      # note: the first idnum in a stream should be 1 and subsequent
      # idnums should not have gaps between values as this will cause
      # the stream parser to reserve space for the gapped values.  An
+static void cmd_progress(void)
+	fwrite(command_buf.buf, 1, command_buf.len - 1, stdout);
+	fputc('\n', stdout);
+	fflush(stdout);
+	skip_optional_lf();
 static void import_marks(const char *input_file)
 	char line[512];
 		else if (!strcmp("checkpoint", command_buf.buf))
+		else if (!prefixcmp(command_buf.buf, "progress "))
+			cmd_progress();
 			die("Unsupported command: %s", command_buf.buf);

File t/

 	 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
 	 git diff expect actual'
+cat >input <<INPUT_END
+commit refs/heads/O4
+data <<COMMIT
+commit refs/heads/O4
+data <<COMMIT
+progress Two commits down, 2 to go!
+commit refs/heads/O4
+data <<COMMIT
+progress Three commits down, 1 to go!
+commit refs/heads/O4
+data <<COMMIT
+progress I'm done!
+test_expect_success \
+	'O: progress outputs as requested by input' \
+	'git-fast-import <input >actual &&
+	 grep "progress " <input >expect &&
+	 git diff expect actual'