Multi- to Monorepo Migration

Don’t like core? Set a different name through an envvar before running the script:

export MONOREPO_NAME=the-big-repo

No need to do anything. This script does not handle any master / main branch in any special way. It just merges whatever branches exist. Don’t have a “master” branch? None will be created.

Make sure your own computer has the right branch set up in its init.defaultBranch setting.

Large teams can’t afford to “stop the world” while a migration is in progress. You’ll be fixing stuff and pulling in new repositories as you go.

Here’s how to pull in an entirely new set of repositories:

/path/to/tomono --continue < my-new-repos.txt

Make sure you have your environment set up exactly the same as above. Particularly, you must be in the parent dir of the monorepo.

Tags are namespaced per remote, to avoid clashes. If your remote foo and bar both have a tag v1.0.0, your monorepo ends up with foo/v1.0.0 and bar/v1.0.0 pointing at their relevant commits.

If you don’t like this rewriting, you can fetch all tags from a specific remote to the top-level of the monorepo:

$ git fetch --tags foo

Be prepared to deal with any conflicts.

d="$(mktemp -d)"
echo "Setting up fresh multi-repos in $d"
cd "$d"

mkdir foo
(
    cd foo
    git init
    git commit -m "foo’s empty root" --allow-empty
    echo "This is foo" > i-am-foo.txt
    git add -A
    git commit -m "foo’s master"
    git tag v1.0
    git checkout -b branch-a
    echo "I am a new foo feature" > feature-a.txt
    git add -A
    git commit -m "foo’s feature branch A"
)

mkdir 中文
(
    cd 中文
    git init
    echo "你好" > 你好.txt
    git add -A
    git commit -m "中文的root"
    git tag v1.0
    git checkout -b branch-a
    echo "你好 from feature-a" > feature-a.txt
    git add -A
    git commit -m "new 中文 feature branch A"
    git branch branch-b master
    git checkout branch-b
    echo "I am an entirely new 中文 feature: B" > feature-b.txt
    git add -A
    git commit -m "中文’s feature branch B"
)

You now have two directories:

• foo (branches: master, branch-a)
• 中文 (branches: master, branch-a, branch-b)

Assuming the tomono script is in your $PATH, you can invoke it like this, from that same directory:

tomono <<EOF
$PWD/foo foo
$PWD/中文 中文
EOF

This will create a new directory, core, where you can find a git tree which looks somewhat like this:

*   b742af2 Merge 中文/branch-a (branch-a)
|\
| * c05c53c new 中文 feature branch A (中文/branch-a)
* |   a51d138 Merge foo/branch-a
|\ \
| * | ebb490a foo’s feature branch A (foo/branch-a)
* | | a08fa18 Root commit for monorepo branch branch-a
 / /
| | *   c53bf94 Merge 中文/branch-b (branch-b)
| | |\
| | | * 5e7f4f5 中文’s feature branch B (中文/branch-b)
| | |/
| |/|
| | * 2738327 Root commit for monorepo branch branch-b
| |
| | *   9a4b33a Merge 中文/master (HEAD -> master)
| | |\
| | |/
| |/|
| * | a9841a8 中文的root (tag: 中文/v1.0, 中文/master)
|  /
| *   b75840e Merge foo/master
| |\
| |/
|/|
* | 1515265 foo’s master (tag: foo/v1.0, foo/master)
* | f71fcde foo’s empty root
 /
* 7803cf5 Root commit for monorepo branch master

It’s possible that while you’re working on setting up your fresh monorepo, new changes have been pushed to the existing single repos:

(
    cd foo
    echo New changes >> i-am-foo.txt
    git commit -va -m 'New changes to foo'
)

Because their history was imported verbatim and nothing has been rewritten, you can import those changes into the monorepo.

First, fetch the changes from the remote:

$ cd core
$ git fetch foo

Now merge your changes using subtree merge:

git checkout master
git merge -X subtree=foo/ foo/master

And the updates should be reflected in the monorepo:

$ cat foo/i-am-foo.txt
This is foo
New changes

I used the branch master in this example, but any branch works the same way.

Now imagine you want to pull in a third repository into the monorepo:

mkdir zimlib
(
    cd zimlib
    git init
    echo "This is zim" > i-am-zim.txt
    git add -A
    git commit -m "zim’s master"
    git checkout -b branch-a
    echo "I am a new zim feature" > feature-a.txt
    git add -A
    git commit -m "zim’s feature branch A"
    # And some more weird stuff, to mess with you
    git checkout master
    git checkout -d
    echo top secret > james-bond.txt
    git add -A
    git commit -m "I am unreachable"
    git tag leaking-you HEAD
    git checkout --orphan empty-branch
    git rm --cached -r .
    git clean -dfx
    git commit -m "zim’s tricky empty orphan branch" --allow-empty
)

Continue importing it:

echo "$PWD/zimlib zim lib/zim" | tomono --continue

Note that we used a different name for this subrepo, inside the lib dir.

The result is that it gets imported into the existing monorepo, alongside the existing two projects:

$ cd core
$ git checkout master
Switched to branch 'master'
$ tree
.
├── foo
│   └── i-am-foo.txt
├── lib
│   └── zim
│       └── i-am-zim.txt
└── 中文
    └── 你好.txt

4 directories, 3 files
$ git checkout branch-a
Switched to branch 'branch-a'
$ tree
.
├── foo
│   ├── feature-a.txt
│   └── i-am-foo.txt
├── lib
│   └── zim
│       ├── feature-a.txt
│       └── i-am-zim.txt
└── 中文
    ├── feature-a.txt
    └── 你好.txt

4 directories, 6 files
$ head **/feature-a.txt
==> foo/feature-a.txt <==
I am a new foo feature

==> lib/zim/feature-a.txt <==
I am a new zim feature

==> 中文/feature-a.txt <==
你好 from feature-a

Every repository is fetched and fully handled individually, and sequentially:

1. fetch all the data related to this repository,
2. immediately check out and initialise every single branch which belongs to that repository.

git remote add "$reponame" "$repourl"
git config --add "remote.$reponame.fetch" "+refs/tags/*:refs/tags/$reponame/*"
git config "remote.$reponame.tagOpt" --no-tags
git fetch --atomic "$reponame"

<<list-branches>> | while read -r branch ; do
    <<handle-branch>>
done

References: list-branches, handle-branch

Used by: top-level

The remotes are configured to make sure that a default fetch always fetch all tags, and also puts them in their own namespace. The default refspec for tags is +refs/tags/*:refs/tags/*, as you can see that puts everything from the remote at the same level in your monorepo. Obviously that will cause clashes, so we add the reponame as an extra namespace.

The --no-tags option is the complement to --tags, which has that default refspec we don’t want. That’s why we disable it and roll our own, entirely.

In the context of a single repository, every branch is independently read into a subdirectory for that repository, and merged into the monorepo.

This is the money shot.

<<move-files-to-subdirectory>>
<<ensure-on-target-branch-in-monorepo>>

git read-tree --prefix "$repopath" "refs/remotes/$reponame/$branch"
tree="$(git write-tree)"
merge_commit="$(git commit-tree \
    "$tree" \
    -p "$branch" \
    -p "$move_commit" \
    -m "Merge $reponame/$branch")"
git reset -q "$merge_commit"

References: move-files-to-subdirectory, ensure-on-target-branch-in-monorepo

Used by: handle-remote

Source: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

We create a fresh directory for this script to run in, or continue on an existing one if the --continue flag is passed.

# Poor man’s arg parse :/
arg="${1-}"
: "${MONOREPO_NAME:=core}"

case "$arg" in
    "")
        if [[ -d "$MONOREPO_NAME" ]]; then
            >&2 echo "monorepo directory $MONOREPO_NAME already exists"
            exit 1
        fi
        mkdir "$MONOREPO_NAME"
        cd "$MONOREPO_NAME"
        git init
        ;;

    "--continue")
        if [[ ! -d "$MONOREPO_NAME" ]]; then
            >&2 echo "Asked to --continue, but monorepo directory $MONOREPO_NAME doesn’t exist"
            exit 1
        fi
        cd "$MONOREPO_NAME"
        if git status --porcelain | grep . ; then
            >&2 echo "Git status shows pending changes in the repo. Cannot --continue."
            exit 1
        fi
        # There isn’t anything special about --continue, really.
        ;;

    "--help" | "-h" | "help")
        cat <<EOF
Usage: tomono [--continue]

For more information, see the documentation at "https://tomono.0brg.net".
EOF
        exit 0
        ;;

    *)
        >&2 echo "Unexpected argument: $arg"
        >&2 echo
        >&2 echo "Usage: tomono [--continue]"
        exit 1
        ;;
esac

Used by: init

Most of this rigmarole is about UI, and preventing mistakes. As you can see, there is functionally no difference between continuing and starting fresh, beyond mkdir and git init. At the end of the day, every repo is read in greedily, and whether you do that on an existing monorepo, or a fresh one, doesn’t matter: every repo name you read in, is in fact itself like a --continue operation.

It’s horrible and kludgy but I just want to get something working out the door, for now.

I want a single branch name per line on stdout, for a single specific remote:

git branch -r --no-color --list "$reponame/*" --format "%(refname:lstrip=3)"

Used by: handle-remote

Initialization is what you’d expect from a shell script:

<<set-flags>>

<<prep-dir>>

empty_tree="$(git hash-object -t tree /dev/null)"

References: set-flags, prep-dir

Used by: top-level

On the other side, when done, update the working tree to whatever the current branch is to avoid any confusion:

git checkout .

Used by: top-level

If you have a branch and tag with the same name in a git repo, you will be familiar with this error:

warning: refname ’foo’ is ambiguous.

See #53. This happens whenever you refer to the tag or branch by its bare name, without specifying whether it’s a tag or a branch. To fix this, the monorepo script must always use refs/heads/... to specify the branch name.

Example:

mkdir duplicates
(
  cd duplicates
  git init -b check-dupes
  echo a > a
  echo b > b
  git add -A
  git commit -m commit1 a
  git tag check-dupes
  git commit -m commit2 b
)

We now have a duplicates repository with a branch and tag check-dupes, pointing at different revisions. After including it in the monorepo:

echo "$PWD/duplicates duplicates" | tomono --continue

We should get:

(
  cd core
  git checkout check-dupes
  # This file must exist
  diff -u duplicates/a <(echo a)
  # This file too
  diff -u duplicates/b <(echo b)
)