brendancleary.com

Want to do your part for software engineering research? Participate in the annual University of Victoria developer practices survey and help us to build a picture of how developers create software in 2013.

How do YOU develop software?.

• Problem

  • You are using GitHub to host your project and your using the GitHub Wiki to maintain your documentation. Now you want to create a release of your project including a current snapshot of the documentation. But because the documentation is separate from the code repo you have to somehow export the Wiki and then put that into the main repo each time you want to make a release. This is just plain messy and in the long term will probably lead to the documentation getting separated/lost or being duplicated in many places.

• Solution

  • Because the GitHub Wiki is itself a git repo we can include the Wiki in the main code repo as a submodule.
  • A submodule is conceptually a git repo inside another git repo, changes to each repo are independent, but the parent repo is linked to the child repo by a commit pointer.

• Benefits

  • We get what we want above, namely a standard way of including documentation in a release
  • We can take full advantage of the GitHub Wiki being a git repo, allowing us to point our documentation submodule at any version/branch of the Wiki we choose
  • Editing of the documentation can be done on the GitHub website or by developers in their local copy when they pull the documentation submodule

• Risks

  • Switching from a branch that includes the submodule to one that doesn’t can cause a few small issues which I note below.

This article describes how to include the GitHub Wiki as a submodule in a GitHub project (or any git repo for that matter).

Context

Lets say we have a GitHub repo called “project” and we want to include the associated Wiki as submodule in a dir called “docs”. We want the following dir structure:

- parent
    - docs

Setup – Clone submodule into existing repo

This section deals with setting up a submodule for the first time.

Get the address of your Wiki repo, it will be something like this:

https://github.com/username/project.wiki.git

Add the Wiki repo as a submodule “docs” to a dir within the “project” local copy:

$ git submodule add https://github.com/username/project.wiki.git docs

Commit the addition of the docs submodule to the parent repo:

$ git commit -m 'first commit with submodule docs'

Push the change to the remote server:

$ git push

Lets look at whats in “docs”

$ cd docs

If you look in the docs dir you’ll notice that its currently empty. You have to initialize the submodule and update it separately. Do the following:

$ cd..
$ git submodule init
$ git submodule update

The submodule is now setup and the docs dir should now have the Wiki docs in it.

Update the docs Submodule

One you have the submodule setup you can pretty much forget about it. However, if someone pushes changes to the parent repo that includes changes to the submodule you can pull them down as normal.

$git pull

This will pull down a new commit pointer for the submodule but it wont update the files in the submodule. Instead you need to merge the new submodule commit pointer (basically overwriting the existing submodule commit pointer)

$git merge origin/master

Now that the submodule pointer is pointing at the correct submodule commit you can do your submodule update

$ git submodule update

This will pull down the changes to the submodule up to the commit that came down in the parent pull

I know it seems like a lot but its actually pretty simple if you think about it, normally you will just use the following commands to pull and merge changes to the submodule:

$git pull
$git merge origin/"branch"
$git submodule update

Make changes to the docs submodule

If you want to make changes directly in the docs submodule for example manually editing the Wiki files or pulling down changes to the Wiki repo made by other developers. Its just like working with any repo except you do two commits, one in the submodule and one in the parent:

$ cd docs

Notice that docs is at a particular commit not on a branch, if you make changes you probably want to checkout the master or another branch.

$ git checkout master

Make your changes and commit as normal (note you are now committing to the submodule repo)

$ git commit -am "changed wiki files"

Now push to the remote, it is really really important that you remember to push submodule commits to the remote (see Gotcha 2).

$ git push

Now got back to the partent project and do another commit, that basically says I have updated the submodule which is in you, here is a pointer to the new version of the submodule

$ cd..
$ git commit -am "updated docs submodule"

Push the changes to the remote

$ git push

Cloning a project containing a submodule

If you are cloning a project that contains an existing submodule, all you need to do after cloning the repo as normal is to initialize the submodule:

$ git submodule init
$ git submodule update

Gotchas

1. Once you have initialized a submodule in a branch, if you checkout a branch that does not contain that submodule, git will see the directory as un-tracked. This may happen until all branches have been updated to contain the submodule. Its not a major problem, just don’t commit that un-tracked directory and you should be ok until the submodule gets merged into the branch. See the link below on how to deal with this.
2. If you commit changes to the submodule locally and don’t push those changes to the remote and then push commit and push changes to the parent the parent repo will now point at a Wiki repo commit that doesn’t exist on the remote. Luckily if its just documentation you probably wont break someone else’s local repo if you do this, but its still not a good idea.

Links

http://git-scm.com/book/en/Git-Tools-Submodules

I live in Outlook, I know its terribly unfashionable in the age of thin clients and the cloud but I just can’t give it up, for what I need it just works. I recently wrote this small tool to help me export rules that I define in outlook so that I can import them into Gmail as filters and have then run on the server rather than on the client. You can grab the code from GitHub by clicking here (Code) or download it below.

UPDATE – 9/7/2013
– Version 0.0.2 is current stable version
– 1500+ downloads on GitHub – Thank you.
– Added support for some new rule types see GitHub

DOWNLOAD

0.0.2 – supports more rule types (Stable version)
0.0.1 – only supports from move-to-folder rules

USAGE
– See instructions here http://bcleary.github.com/outlook-gmail-rules-export/

Normal service is resumed, which means I’ll probably be back here in a year or so 😉