ls-files.c: add --only-object-name option

From: ZheNing Hu <[email protected]>

`git ls-files --stage` default output format is:

[<tag> ]<mode> <object> <stage> <file>

sometime we want to find a path's corresponding objectname,
we will parse the output and extract objectname from it
again and again.

So introduce a new option `--only-object-name` which can only
output objectname when giving `--stage` or `--resolve-undo`.

Signed-off-by: ZheNing Hu <[email protected]>

Submitted-As: https://lore.kernel.org/git/[email protected]

bundle URIs: design doc and initial git fetch --bundle-uri implementa…

…tion

This is the first of series towards building the bundle URI feature as
discussed in previous RFCs, specifically pulled directly out of [5]:

[1]
https://lore.kernel.org/git/[email protected]/
[2]
https://lore.kernel.org/git/[email protected]/
[3]
https://lore.kernel.org/git/[email protected]
[4]
https://lore.kernel.org/git/[email protected]/
[5]
https://lore.kernel.org/git/[email protected]

The first patch details the long-term design and goals of the bundle URI
feature, including complicated features such as the bundle-uri protocol v2
verb and bundle lists with heuristics.

However, then intention is to start small with the simplest features that
allow user benefit as soon as possible. In that direction, the rest of this
series creates the ability to run 'git fetch --bundle-uri=' to skip fetching
from any remote and instead download the file at the given . Currently, that
data is expected to be a bundle, which Git will then unbundle and modify the
refs to be in the 'refs/bundle/' namespace.

Currently, the can be a literal filename, a file:// URI, or an http[s]://
URI. Tests are added for both of these cases.

As outlined in [5], the next steps after this are:

 1. Add 'git clone --bundle-uri=' to run this 'git fetch --bundle-uri=' step
    before doing a fetch negotiation with the origin remote.
 2. Allow parsing a bundle list as a config file at the given URI. The
    key-value format is unified with the protocol v2 verb (coming in (3)).
 3. Implement the protocol v2 verb, re-using the bundle list logic from (2).
    Use this to auto-discover bundle URIs during 'git clone' (behind a
    config option).
 4. Implement the 'timestamp' heuristic, allowing incremental 'git fetch'
    commands to download a bundle list from a configured URI, and only
    download bundles that are new based on the timestamp values.

As mentioned in the design document, this is not all that is possible. For
instance, Ævar's suggestion to download only the bundle headers can be used
as a second heuristic (and as an augmentation of the timestamp heuristic).

Thanks, -Stolee

Derrick Stolee (6):
  docs: document bundle URI standard
  remote-curl: add 'get' capability
  bundle-uri: create basic file-copy logic
  fetch: add --bundle-uri option
  bundle-uri: add support for http(s):// and file://
  fetch: add 'refs/bundle/' to log.excludeDecoration

 Documentation/fetch-options.txt        |   6 +
 Documentation/git-fetch.txt            |   1 +
 Documentation/gitremote-helpers.txt    |   9 +
 Documentation/technical/bundle-uri.txt | 475 +++++++++++++++++++++++++
 Makefile                               |   1 +
 builtin/fetch.c                        |  10 +
 bundle-uri.c                           | 166 +++++++++
 bundle-uri.h                           |  14 +
 remote-curl.c                          |  33 ++
 t/t5557-http-get.sh                    |  37 ++
 t/t5558-fetch-bundle-uri.sh            |  77 ++++
 transport-helper.c                     |   5 +-
 12 files changed, 833 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/technical/bundle-uri.txt
 create mode 100644 bundle-uri.c
 create mode 100644 bundle-uri.h
 create mode 100755 t/t5557-http-get.sh
 create mode 100755 t/t5558-fetch-bundle-uri.sh

base-commit: 89c6e45

Submitted-As: https://lore.kernel.org/git/[email protected]

range-diff: show submodule changes irrespective of diff.submodule

From: Philippe Blain <[email protected]>

After generating diffs for each range to be compared using a 'git log'
invocation, range-diff.c::read_patches looks for the "diff --git" header
in those diffs to recognize the beginning of a new change.

In a project with submodules, and with 'diff.submodule=log' set in the
config, this header is missing for the diff of a changed submodule, so
any submodule changes are quietly ignored in the range-diff.

When 'diff.submodule=diff' is set in the config, the "diff --git" header
is also missing for the submodule itself, but is shown for submodule
content changes, which can easily confuse 'git range-diff' and lead to
errors such as:

    error: git apply: bad git-diff - inconsistent old filename on line 1
    error: could not parse git header 'diff --git path/to/submodule/and/some/file/within
    '
    error: could not parse log for '@{u}..@{1}'

Force the submodule diff format to its default ("short") when invoking
'git log' to generate the patches for each range, such that submodule
changes are always detected.

Add a test, including an invocation with '--creation-factor=100' to
force the second commit in the range not to be considered a complete
rewrite, in order to verify we do indeed get the "short" format.

Signed-off-by: Philippe Blain <[email protected]>

Submitted-As: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]

remote: create fetch.credentialsInUrl config

From: Derrick Stolee <[email protected]>

Users sometimes provide a "username:password" combination in their
plaintext URLs. Since Git stores these URLs in plaintext in the
.git/config file, this is a very insecure way of storing these
credentials. Credential managers are a more secure way of storing this
information.

System administrators might want to prevent this kind of use by users on
their machines.

Create a new "fetch.credentialsInUrl" config option and teach Git to
warn or die when seeing a URL with this kind of information. The warning
anonymizes the sensitive information of the URL to be clear about the
issue.

This change currently defaults the behavior to "allow" which does
nothing with these URLs. We can consider changing this behavior to
"warn" by default if we wish. At that time, we may want to add some
advice about setting fetch.credentialsInUrl=ignore for users who still
want to follow this pattern (and not receive the warning).

An earlier version of this change injected the logic into
url_normalize() in urlmatch.c. While most code paths that parse URLs
eventually normalize the URL, that normalization does not happen early
enough in the stack to avoid attempting connections to the URL first. By
inserting a check into the remote validation, we identify the issue
before making a connection. In the old code path, this was revealed by
testing the new t5601-clone.sh test under --stress, resulting in an
instance where the return code was 13 (SIGPIPE) instead of 128 from the
die().

However, we can reuse the parsing information from url_normalize() in
order to benefit from its well-worn parsing logic. We can use the struct
url_info that is created in that method to replace the password with
"<redacted>" in our error messages. This comes with a slight downside
that the normalized URL might look slightly different from the input URL
(for instance, the normalized version adds a closing slash). This should
not hinder users figuring out what the problem is and being able to fix
the issue.

As an attempt to ensure the parsing logic did not catch any
unintentional cases, I modified this change locally to to use the "die"
option by default. Running the test suite succeeds except for the
explicit username:password URLs used in t5550-http-fetch-dumb.sh and
t5541-http-push-smart.sh. This means that all other tested URLs did not
trigger this logic.

The tests show that the proper error messages appear (or do not
appear), but also count the number of error messages. When only warning,
each process validates the remote URL and outputs a warning. This
happens twice for clone, three times for fetch, and once for push.

Co-authored-by: Junio C Hamano <[email protected]>
Signed-off-by: Junio C Hamano <[email protected]>
Signed-off-by: Derrick Stolee <[email protected]>

Submitted-As: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]

read-cache.c: reduce unnecessary cache entry name copying

From: ZheNing Hu <[email protected]>

In function create_from_disk, we have already copy the cache
entries name from disk or previous cache entry, we can reduce
unnecessary copy before that.

Signed-off-by: ZheNing Hu <[email protected]>

Submitted-As: https://lore.kernel.org/git/[email protected]

Die preserve ggg

This [2] short series is a follow up to GitGitGadget "Update the die()
preserve-merges messages to help some users (PR gitgitgadget#1155)" [1].

Since v1: Additional patch to translate the user facing die message. Bring
the --abort preclusion to the start of the if && condition for clarity.
Clarify that the pull.rebase config is complementary to this particular
'die'. Updates to the commit messages.

v0: The first patch is a tidy up of the --preserve option to highlight that
it is now Deleted, rather than Deprecated.

In response to Avar's comments that the former error message merely
'tantilised without telling' the user what to do, it became obvious that the
underling problem was that the user was unable to git rebase --abort which
was also fatal, when a preserve-rebase was in progress.

Thus the main update is to allow the rebase --abort command, even when a
--preserve is in progress, to proceed. The --abort code was unchanged by the
removal of the preserve option, as the resetting and clean up of internal
state is common to the other rebase options.

The user facing fatal message now simply advises to abort, or downgrade to a
version that has preserve-merges to complete the rebase.

The final patch highlights that some IDEs still allow the setting of the
preserve-merges option as a pull config setup.

Philip Oakly

[1] GitLore ref [email protected]
https://lore.kernel.org/git/[email protected]/

[2]
https://lore.kernel.org/git/[email protected]/t/#u

Philip Oakley (4):
  rebase.c: state preserve-merges has been removed
  rebase: help users when dying with `preserve-merges`
  rebase: note `preserve` merges may be a pull config option
  rebase: translate a die(preserve-merges) message

 builtin/rebase.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

base-commit: c4f0e30

Submitted-As: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]

rebase: update branches in multi-part topic

This is a feature I've wanted for quite a while. When working on the sparse
index topic, I created a long RFC that actually broke into three topics for
full review upstream. These topics were sequential, so any feedback on an
earlier one required updates to the later ones. I would work on the full
feature and use interactive rebase to update the full list of commits.
However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a
rebase.updateRefs config option) that adds 'git update-ref' commands into
the TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as
they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic
pick 96cb2e35af1 bundle-uri: add support for http(s):// and file://
pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration
exec git update-ref refs/heads/bundle-redo/fetch HEAD 6c5840ed77e1bc41c1fe6fb7c894ceede1b8d730

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth
exec git update-ref refs/heads/bundle-redo/clone HEAD 9e4a6fe9b68a8455b427c9ac8cdbff30c96653b4

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers
pick 3029c3aca15 bundle-uri: create base key-value pair parsing
pick a8b2de79ce8 bundle-uri: create "key=value" line parsing
pick 92625a47673 bundle-uri: unit test "key=value" parsing
pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists
pick 9d6809a8d53 bundle-uri: parse bundle list in config format
pick 287a732b54c bundle-uri: fetch a list of bundles
exec git update-ref refs/heads/bundle-redo/list HEAD 287a732b54c4d95e7f410b3b36ef90d8a19cd346

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton
pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri"
pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config
pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting
pick caf4599a81d bundle-uri: allow relative URLs in bundle lists
pick df255000b7e bundle-uri: download bundles from an advertised list
pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests

exec git update-ref refs/heads/bundle-redo/advertise HEAD c9578391976ab9899c4e4f9b5fa2827650097305

The first two patches are helpers that are needed, but the full logic of the
--update-refs option is introduced in patch 3. The config option is
available in patch 4.

Thanks, -Stolee

Derrick Stolee (4):
  log-tree: create for_each_decoration()
  branch: add branch_checked_out() helper
  rebase: add --update-refs option
  rebase: add rebase.updateRefs config option

 Documentation/config/rebase.txt |   3 +
 Documentation/git-rebase.txt    |  11 ++++
 branch.c                        |  24 ++++---
 branch.h                        |   8 +++
 builtin/rebase.c                |  10 +++
 log-tree.c                      | 111 ++++++++++++++++++++++----------
 log-tree.h                      |   4 ++
 sequencer.c                     |  99 ++++++++++++++++++++++++++++
 sequencer.h                     |   1 +
 t/t3404-rebase-interactive.sh   |  34 ++++++++++
 10 files changed, 262 insertions(+), 43 deletions(-)

base-commit: 2668e36

Submitted-As: https://lore.kernel.org/git/[email protected]

bitmap-format.txt: fix some formatting issues and include checksum info

There are some issues in the bitmap-format html page. For example, some
nested lists are shown as top-level lists (e.g. [1]- Here
BITMAP_OPT_FULL_DAG (0x1) and BITMAP_OPT_HASH_CACHE (0x4) are shown as
top-level list).

The first commit fix those.

The second commit is about including the info of trailing checksum in the
bitmap-format documentation.

[1] https://git-scm.com/docs/bitmap-format#_on_disk_format

Abhradeep Chakraborty (2):
  bitmap-format.txt: fix some formatting issues
  bitmap-format.txt: add information for trailing checksum

 Documentation/technical/bitmap-format.txt | 100 +++++++++++-----------
 1 file changed, 49 insertions(+), 51 deletions(-)

base-commit: 2668e36

Submitted-As: https://lore.kernel.org/git/[email protected]

remote: create fetch.credentialsInUrl config

From: Derrick Stolee <[email protected]>

Users sometimes provide a "username:password" combination in their
plaintext URLs. Since Git stores these URLs in plaintext in the
.git/config file, this is a very insecure way of storing these
credentials. Credential managers are a more secure way of storing this
information.

System administrators might want to prevent this kind of use by users on
their machines.

Create a new "fetch.credentialsInUrl" config option and teach Git to
warn or die when seeing a URL with this kind of information. The warning
anonymizes the sensitive information of the URL to be clear about the
issue.

This change currently defaults the behavior to "allow" which does
nothing with these URLs. We can consider changing this behavior to
"warn" by default if we wish. At that time, we may want to add some
advice about setting fetch.credentialsInUrl=ignore for users who still
want to follow this pattern (and not receive the warning).

An earlier version of this change injected the logic into
url_normalize() in urlmatch.c. While most code paths that parse URLs
eventually normalize the URL, that normalization does not happen early
enough in the stack to avoid attempting connections to the URL first. By
inserting a check into the remote validation, we identify the issue
before making a connection. In the old code path, this was revealed by
testing the new t5601-clone.sh test under --stress, resulting in an
instance where the return code was 13 (SIGPIPE) instead of 128 from the
die().

However, we can reuse the parsing information from url_normalize() in
order to benefit from its well-worn parsing logic. We can use the struct
url_info that is created in that method to replace the password with
"<redacted>" in our error messages. This comes with a slight downside
that the normalized URL might look slightly different from the input URL
(for instance, the normalized version adds a closing slash). This should
not hinder users figuring out what the problem is and being able to fix
the issue.

As an attempt to ensure the parsing logic did not catch any
unintentional cases, I modified this change locally to to use the "die"
option by default. Running the test suite succeeds except for the
explicit username:password URLs used in t5550-http-fetch-dumb.sh and
t5541-http-push-smart.sh. This means that all other tested URLs did not
trigger this logic.

The tests show that the proper error messages appear (or do not
appear), but also count the number of error messages. When only warning,
each process validates the remote URL and outputs a warning. This
happens twice for clone, three times for fetch, and once for push.

Signed-off-by: Derrick Stolee <[email protected]>

Submitted-As: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]
In-Reply-To: https://lore.kernel.org/git/[email protected]

Integrate Scalar into the CI builds

During the review of the initial Scalar patch series, it was suggested to
include Scalar in Git's CI builds. Due to some conflicts, this was postponed
to a later patch series: This patch series.

Note that the changes to the GitHub workflow are somewhat transient in
nature: Based on the feedback I received on the Git mailing list, I see some
appetite for turning Scalar into a full-fledged top-level command in Git,
similar to gitk. Therefore my current plan is to do exactly that in the end
(and I already have patches lined up to that end). This will essentially
revert the ci/run-build-and-tests.sh change in this patch series.

This patch series is based on js/scalar-diagnose.

Johannes Schindelin (2):
  cmake: optionally build `scalar`, too
  ci: also run the `scalar` tests

 .github/workflows/main.yml          | 15 +++++++++++++++
 ci/run-build-and-tests.sh           |  2 ++
 ci/run-test-slice.sh                |  5 +++++
 contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++++
 4 files changed, 36 insertions(+)

base-commit: 15d8adc

Submitted-As: https://lore.kernel.org/git/[email protected]