[Buildroot] [PATCH 3/3] support/download/git: do not use git clone

Ricardo Martincoski ricardo.martincoski at datacom.ind.br
Tue Nov 1 19:33:54 UTC 2016


The logic of the script was completely rewritten based in some ideas
discussed during the review of [1].

Always using git init + git fetch has these advantages:
- git fetch works with all kinds of refs, while git clone supports only
  branches and tags without the ref/heads/ and ref/tags/ prefixes;
- a shallow fetch can be done for the head of those refs.

The remote is first asked for its references using git ls-remote saving
the output to a file.
This file is later on compared to the desired change set, determining if
it is a branch, tag, special ref or sha1.
A concern that arrives from this method is that the remote can change
between the git ls-remote and the git fetch, but in this case, once the
script creates a shallow fetch, the checkout does what it is expected:
- for a named reference (branch, tag or special ref), the fetch and
  checkout are successful using the "new" sha1 from the remote;
- for a sha1 of a branch, the fetch fails (because the sha1 it is not
  anymore at branch head), falling back to a successful full fetch;
- for a sha1 of a commit pointed by a tag (that should not be changed
  but it can be changed) the same behavior of a sha1 of a branch takes
  place;
- for a sha1 only accessible by a special refs, the checkout fails,
  falling back to an unsuccessful checkout after full fetch. This can be
  caused only be a error from a developer.

An analytical solution is used instead of a single awk line. It makes
each line of code simple: grep to check the entry is in the ls-remote
output and awk to actually get the reference to use.

[1] http://patchwork.ozlabs.org/patch/681841/

Signed-off-by: Ricardo Martincoski <ricardo.martincoski at datacom.ind.br>
---
 support/download/git | 91 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 60 insertions(+), 31 deletions(-)

diff --git a/support/download/git b/support/download/git
index f7eef15..1411d94 100755
--- a/support/download/git
+++ b/support/download/git
@@ -39,43 +39,72 @@ _git() {
     eval ${GIT} "${@}"
 }
 
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
-    printf "Doing shallow clone\n"
-    if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
-        git_done=1
-    else
-        printf "Shallow clone failed, falling back to doing a full clone\n"
+_git init ${verbose} "'${basename}'"
+
+pushd "${basename}" >/dev/null
+
+# Save the temporary file inside the .git directory that will be deleted after the checkout is done.
+tmpf=".git/ls-remote"
+_git ls-remote "'${repo}'" > "${tmpf}"
+
+do_a_shallow_fetch=0
+if grep "\<\(\|\(\|refs/\)\(heads\|tags\)/\)${cset}$" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a named reference\n"
+    # Support branches and tags in the simplified form.
+    # Support branches and tags and special refs in the complete form refs/heads/branch.
+    # When the name is ambiguous, the branch will be selected (by git fetch or git clone).
+    ref="${cset}"
+    do_a_shallow_fetch=1
+elif grep "^${cset}" "${tmpf}" >/dev/null 2>&1; then
+    printf "The desired version is a sha1\n"
+    if [ ${#cset} -lt 40 -a "1" != "$(awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}")" ]; then
+        printf "Ambiguous partial sha1\n"
+        awk "/^${cset}/{print \$1|\"sort -u | wc -l\"}" "${tmpf}"
+        exit 1
+    fi
+    # Accept full or unambiguous partial sha1. A sha1 can be referenced by many names.
+    # Prefer sha1 of commit pointed by a tag or sha1 of the tag itself,
+    # then sha1 pointed by any ref/*, and only then sha1 pointed by *HEAD.
+    ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\/tags\//{ print \$2; exit }" "${tmpf}")"
+    if [ -z "${ref}" ]; then
+        ref="$(awk -F'[\t^]' "/^${cset}.*\trefs\//{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -z "${ref}" ]; then
+        # sha1 is referenced by HEAD
+        ref="$(awk -F'[\t^]' "/^${cset}/{ print \$2; exit }" "${tmpf}")"
+    fi
+    if [ -n "${ref}" ]; then
+        printf "Fetching '%s' to get '%s'\n" "${ref}" "${cset}"
+        do_a_shallow_fetch=1
     fi
-fi
-if [ ${git_done} -eq 0 ]; then
-    printf "Doing full clone\n"
-    _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
 fi
 
-pushd "${basename}" >/dev/null
+git_fetch_done=0
+if [ ${do_a_shallow_fetch} -eq 1 ]; then
+    printf "Doing shallow fetch\n"
+    if _git fetch ${verbose} "${@}" --depth 1 "'${repo}'" "'${ref}:refs/buildroot/${cset}'" 2>&1; then
+        git_fetch_done=1
+    else
+        # It catches the case the remote changed after ls-remote.
+        printf "Shallow fetch failed, falling back to doing a full fetch\n"
+    fi
+fi
 
-# Try to get the special refs exposed by some forges (pull-requests for
-# github, changes for gerrit...). There is no easy way to know whether
-# the cset the user passed us is such a special ref or a tag or a sha1
-# or whatever else. We'll eventually fail at checking out that cset,
-# below, if there is an issue anyway. Since most of the cset we're gonna
-# have to clone are not such special refs, consign the output to oblivion
-# so as not to alarm unsuspecting users, but still trace it as a warning.
-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
-    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
+git_checkout_done=0
+if [ ${git_fetch_done} -eq 1 ]; then
+    if _git checkout -q "'refs/buildroot/${cset}'" 2>&1; then
+        git_checkout_done=1
+    else
+        printf "Checkout failed, falling back to doing a full fetch\n"
+    fi
 fi
 
-# Checkout the required changeset, so that we can update the required
-# submodules.
-_git checkout -q "'${cset}'"
+if [ ${git_checkout_done} -eq 0 ]; then
+    printf "Doing full fetch\n"
+    _git remote add origin "'${repo}'"
+    _git fetch ${verbose} "${@}" origin
+    _git checkout -q "'${cset}'"
+fi
 
 # Get date of commit to generate a reproducible archive.
 # %cD is RFC2822, so it's fully qualified, with TZ and all.
-- 
2.9.3




More information about the buildroot mailing list