Commits

Shawn O. Pearce  committed ac053c0

Allow frontends to bidirectionally communicate with fast-import

The existing checkpoint command is very useful to force fast-import
to dump the branches out to disk so that standard Git tools can
access them and the objects they refer to. However there was not a
way to know when fast-import had finished executing the checkpoint
and it was safe to read those refs.

The progress command can be used to make fast-import output any
message of the frontend's choosing to standard out. The frontend
can scan for these messages using select() or poll() to monitor a
pipe connected to the standard output of fast-import.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

  • Participants
  • Parent commits 1fdb649

Comments (0)

Files changed (3)

File Documentation/git-fast-import.txt

 	This command is optional and is not needed to perform
 	an import.
 
+`progress`::
+	Causes fast-import to echo the entire line to its own
+	standard output.  This command is optional and is not needed
+	to perform an import.
+
 `commit`
 ~~~~~~~~
 Create or update a branch with a new commit, recording one logical
 
 The `LF` after the command is optional (it used to be required).
 
+`progress`
+~~~~~~~~~~
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream.  The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+
+....
+	'progress' SP <any> LF
+	LF?
+....
+
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`.  The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+
+====
+	frontend | git-fast-import | sed 's/^progress //'
+====
+
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
+
 Tips and Tricks
 ---------------
 The following tips and tricks have been collected from various
 You only need to expend the effort once, and everyone using your
 project will benefit from the smaller repository.
 
+Include Some Progress Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Every once in a while have your frontend emit a `progress` message
+to fast-import.  The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
+
 
 Packfile Optimization
 ---------------------

File fast-import.c

         | new_tag
         | reset_branch
         | checkpoint
+        | progress
         ;
 
   new_blob ::= 'blob' lf
   checkpoint ::= 'checkpoint' lf
     lf?;
 
+  progress ::= 'progress' sp not_lf* lf
+    lf?;
+
      # note: the first idnum in a stream should be 1 and subsequent
      # idnums should not have gaps between values as this will cause
      # the stream parser to reserve space for the gapped values.  An
 	skip_optional_lf();
 }
 
+static void cmd_progress(void)
+{
+	fwrite(command_buf.buf, 1, command_buf.len - 1, stdout);
+	fputc('\n', stdout);
+	fflush(stdout);
+	skip_optional_lf();
+}
+
 static void import_marks(const char *input_file)
 {
 	char line[512];
 			cmd_reset_branch();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			cmd_checkpoint();
+		else if (!prefixcmp(command_buf.buf, "progress "))
+			cmd_progress();
 		else
 			die("Unsupported command: %s", command_buf.buf);
 	}

File t/t9300-fast-import.sh

 	 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
 	 git diff expect actual'
 
+cat >input <<INPUT_END
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zstring
+COMMIT
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zof
+COMMIT
+progress Two commits down, 2 to go!
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zempty
+COMMIT
+progress Three commits down, 1 to go!
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zcommits
+COMMIT
+progress I'm done!
+INPUT_END
+test_expect_success \
+	'O: progress outputs as requested by input' \
+	'git-fast-import <input >actual &&
+	 grep "progress " <input >expect &&
+	 git diff expect actual'
+
 test_done