SoundSoftware Code Site

svn-archive

svn-fast-export

*.pyc

.dotest

hg-fast-export.(sh|py) - mercurial to git converter using git-fast-import

=========================================================================

Legal

-----

Most hg-* scripts are licensed under the [MIT license]

(http://www.opensource.org/licenses/mit-license.php) and were written

by Rocco Rutte <pdmef@gmx.net> with hints and help from the git list and

\#mercurial on freenode. hg-reset.py is licensed under GPLv2 since it

copies some code from the mercurial sources.

The current maintainer is Frej Drejhammar <frej.drejhammar@gmail.com>.

Usage

-----

Using hg-fast-export is quite simple for a mercurial repository <repo>:

```

mkdir repo-git # or whatever

cd repo-git

git init

hg-fast-export.sh -r <repo>

```

Please note that hg-fast-export does not automatically check out the

newly imported repository. You probably want to follow up the import

with a `git checkout`-command.

Incremental imports to track hg repos is supported, too.

Using hg-reset it is quite simple within a git repository that is

hg-fast-export'ed from mercurial:

```

hg-reset.sh -R <revision>

```

will give hints on which branches need adjustment for starting over

again.

When a mercurial repository does not use utf-8 for encoding author

strings and commit messages the `-e <encoding>` command line option

can be used to force fast-export to convert incoming meta data from

<encoding> to utf-8. This encoding option is also applied to file names.

In some locales Mercurial uses different encodings for commit messages

and file names. In that case, you can use `--fe <encoding>` command line

option which overrides the -e option for file names.

As mercurial appears to be much less picky about the syntax of the

author information than git, an author mapping file can be given to

hg-fast-export to fix up malformed author strings. The file is

specified using the -A option. The file should contain lines of the

form `FromAuthor=ToAuthor`. The example authors.map below will

translate `User <garbage<user@example.com>` to `User <user@example.com>`.

```

-- Start of authors.map --

User <garbage<user@example.com>=User <user@example.com>

-- End of authors.map --

```

Tag and Branch Naming

---------------------

As Git and Mercurial have differ in what is a valid branch and tag

name the -B and -T options allow a mapping file to be specified to

rename branches and tags (respectively). The syntax of the mapping

file is the same as for the author mapping.

Notes/Limitations

-----------------

hg-fast-export supports multiple branches but only named branches with

exactly one head each. Otherwise commits to the tip of these heads

within the branch will get flattened into merge commits.

As each git-fast-import run creates a new pack file, it may be

required to repack the repository quite often for incremental imports

(especially when importing a small number of changesets per

incremental import).

The way the hg API and remote access protocol is designed it is not

possible to use hg-fast-export on remote repositories

(http/ssh). First clone the repository, then convert it.

Design

------

hg-fast-export.py was designed in a way that doesn't require a 2-pass

mechanism or any prior repository analysis: if just feeds what it

finds into git-fast-import. This also implies that it heavily relies

on strictly linear ordering of changesets from hg, i.e. its

append-only storage model so that changesets hg-fast-export already

saw never get modified.

Submitting Patches

------------------

Please use the issue-tracker at github

https://github.com/frej/fast-export to report bugs and submit

patches.

#!/usr/bin/env python

# Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.

# License: MIT <http://www.opensource.org/licenses/mit-license.php>

from mercurial import node

from hg2git import setup_repo,fixup_user,get_branch,get_changeset

from hg2git import load_cache,save_cache,get_git_sha1,set_default_branch,set_origin_name

from optparse import OptionParser

import re

import sys

import os

if sys.platform == "win32":

  # On Windows, sys.stdout is initially opened in text mode, which means that

  # when a LF (\n) character is written to sys.stdout, it will be converted

  # into CRLF (\r\n).  That makes git blow up, so use this platform-specific

  # code to change the mode of sys.stdout to binary.

  import msvcrt

  msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

# silly regex to catch Signed-off-by lines in log message

sob_re=re.compile('^Signed-[Oo]ff-[Bb]y: (.+)$')

# insert 'checkpoint' command after this many commits or none at all if 0

cfg_checkpoint_count=0

# write some progress message every this many file contents written

cfg_export_boundary=1000

def gitmode(flags):

  return 'l' in flags and '120000' or 'x' in flags and '100755' or '100644'

def wr_no_nl(msg=''):

  if msg:

    sys.stdout.write(msg)

def wr(msg=''):

  wr_no_nl(msg)

  sys.stdout.write('\n')

  #map(lambda x: sys.stderr.write('\t[%s]\n' % x),msg.split('\n'))

def checkpoint(count):

  count=count+1

  if cfg_checkpoint_count>0 and count%cfg_checkpoint_count==0:

    sys.stderr.write("Checkpoint after %d commits\n" % count)

    wr('checkpoint')

    wr()

  return count

def revnum_to_revref(rev, old_marks):

  """Convert an hg revnum to a git-fast-import rev reference (an SHA1

  or a mark)"""

  return old_marks.get(rev) or ':%d' % (rev+1)

def file_mismatch(f1,f2):

  """See if two revisions of a file are not equal."""

  return node.hex(f1)!=node.hex(f2)

def split_dict(dleft,dright,l=[],c=[],r=[],match=file_mismatch):

  """Loop over our repository and find all changed and missing files."""

  for left in dleft.keys():

    right=dright.get(left,None)

    if right==None:

      # we have the file but our parent hasn't: add to left set

      l.append(left)

    elif match(dleft[left],right) or gitmode(dleft.flags(left))!=gitmode(dright.flags(left)):

      # we have it but checksums mismatch: add to center set

      c.append(left)

  for right in dright.keys():

    left=dleft.get(right,None)

    if left==None:

      # if parent has file but we don't: add to right set

      r.append(right)

    # change is already handled when comparing child against parent

  return l,c,r

def get_filechanges(repo,revision,parents,mleft):

  """Given some repository and revision, find all changed/deleted files."""

  l,c,r=[],[],[]

  for p in parents:

    if p<0: continue

    mright=repo.changectx(p).manifest()

    l,c,r=split_dict(mleft,mright,l,c,r)

  l.sort()

  c.sort()

  r.sort()

  return l,c,r

def get_author(logmessage,committer,authors):

  """As git distincts between author and committer of a patch, try to

  extract author by detecting Signed-off-by lines.

  This walks from the end of the log message towards the top skipping

  empty lines. Upon the first non-empty line, it walks all Signed-off-by

  lines upwards to find the first one. For that (if found), it extracts

  authorship information the usual way (authors table, cleaning, etc.)

  If no Signed-off-by line is found, this defaults to the committer.

  This may sound stupid (and it somehow is), but in log messages we

  accidentially may have lines in the middle starting with

  "Signed-off-by: foo" and thus matching our detection regex. Prevent

  that."""

  loglines=logmessage.split('\n')

  i=len(loglines)

  # from tail walk to top skipping empty lines

  while i>=0:

    i-=1

    if len(loglines[i].strip())==0: continue

    break

  if i>=0:

    # walk further upwards to find first sob line, store in 'first'

    first=None

    while i>=0:

      m=sob_re.match(loglines[i])

      if m==None: break

      first=m

      i-=1

    # if the last non-empty line matches our Signed-Off-by regex: extract username

    if first!=None:

      r=fixup_user(first.group(1),authors)

      return r

  return committer

def export_file_contents(ctx,manifest,files,hgtags,encoding=''):

  count=0

  max=len(files)

  for file in files:

    # Skip .hgtags files. They only get us in trouble.

    if not hgtags and file == ".hgtags":

      sys.stderr.write('Skip %s\n' % (file))

      continue

    d=ctx.filectx(file).data()

    if encoding:

      filename=file.decode(encoding).encode('utf8')

    else:

      filename=file

    wr('M %s inline %s' % (gitmode(manifest.flags(file)),

                           strip_leading_slash(filename)))

    wr('data %d' % len(d)) # had some trouble with size()

    wr(d)

    count+=1

    if count%cfg_export_boundary==0:

      sys.stderr.write('Exported %d/%d files\n' % (count,max))

  if max>cfg_export_boundary:

    sys.stderr.write('Exported %d/%d files\n' % (count,max))

def sanitize_name(name,what="branch", mapping={}):

  """Sanitize input roughly according to git-check-ref-format(1)"""

  # NOTE: Do not update this transform to work around

  # incompatibilities on your platform. If you change it and it starts

  # modifying names which previously were not touched it will break

  # preexisting setups which are doing incremental imports.

  #

  # Use the -B and -T options to mangle branch and tag names

  # instead. If you have a source repository where this is too much

  # work to do manually, write a tool that does it for you.

  def dot(name):

    if name[0] == '.': return '_'+name[1:]

    return name

  n=mapping.get(name,name)

  p=re.compile('([[ ~^:?\\\\*]|\.\.)')

  n=p.sub('_', n)

  if n[-1] in ('/', '.'): n=n[:-1]+'_'

  n='/'.join(map(dot,n.split('/')))

  p=re.compile('_+')

  n=p.sub('_', n)

  if n!=name:

    sys.stderr.write('Warning: sanitized %s [%s] to [%s]\n' % (what,name,n))

  return n

def strip_leading_slash(filename):

  if filename[0] == '/':

    return filename[1:]

  return filename

def export_commit(ui,repo,revision,old_marks,max,count,authors,

                  branchesmap,sob,brmap,hgtags,encoding='',fn_encoding=''):

  def get_branchname(name):

    if brmap.has_key(name):

      return brmap[name]

    n=sanitize_name(name, "branch", branchesmap)

    brmap[name]=n

    return n

  (revnode,_,user,(time,timezone),files,desc,branch,_)=get_changeset(ui,repo,revision,authors,encoding)

  branch=get_branchname(branch)

  parents = [p for p in repo.changelog.parentrevs(revision) if p >= 0]

  if len(parents)==0 and revision != 0:

    wr('reset refs/heads/%s' % branch)

  wr('commit refs/heads/%s' % branch)

  wr('mark :%d' % (revision+1))

  if sob:

    wr('author %s %d %s' % (get_author(desc,user,authors),time,timezone))

  wr('committer %s %d %s' % (user,time,timezone))

  wr('data %d' % (len(desc)+1)) # wtf?

  wr(desc)

  wr()

  ctx=repo.changectx(str(revision))

  man=ctx.manifest()

  added,changed,removed,type=[],[],[],''

  if len(parents) == 0:

    # first revision: feed in full manifest

    added=man.keys()

    added.sort()

    type='full'

  else:

    wr('from %s' % revnum_to_revref(parents[0], old_marks))

    if len(parents) == 1:

      # later non-merge revision: feed in changed manifest

      # if we have exactly one parent, just take the changes from the

      # manifest without expensively comparing checksums

      f=repo.status(repo.lookup(parents[0]),revnode)[:3]

      added,changed,removed=f[1],f[0],f[2]

      type='simple delta'

    else: # a merge with two parents

      wr('merge %s' % revnum_to_revref(parents[1], old_marks))

      # later merge revision: feed in changed manifest

      # for many files comparing checksums is expensive so only do it for

      # merges where we really need it due to hg's revlog logic

      added,changed,removed=get_filechanges(repo,revision,parents,man)

      type='thorough delta'

  sys.stderr.write('%s: Exporting %s revision %d/%d with %d/%d/%d added/changed/removed files\n' %

      (branch,type,revision+1,max,len(added),len(changed),len(removed)))

  if fn_encoding:

    removed=[r.decode(fn_encoding).encode('utf8') for r in removed]

  removed=[strip_leading_slash(x) for x in removed]

  map(lambda r: wr('D %s' % r),removed)

  export_file_contents(ctx,man,added,hgtags,fn_encoding)

  export_file_contents(ctx,man,changed,hgtags,fn_encoding)

  wr()

  return checkpoint(count)

def export_note(ui,repo,revision,count,authors,encoding,is_first):

  (revnode,_,user,(time,timezone),_,_,_,_)=get_changeset(ui,repo,revision,authors,encoding)

  parents = [p for p in repo.changelog.parentrevs(revision) if p >= 0]

  wr('commit refs/notes/hg')

  wr('committer %s %d %s' % (user,time,timezone))

  wr('data 0')

  if is_first:

    wr('from refs/notes/hg^0')

  wr('N inline :%d' % (revision+1))

  hg_hash=repo.changectx(str(revision)).hex()

  wr('data %d' % (len(hg_hash)))

  wr_no_nl(hg_hash)

  wr()

  return checkpoint(count)

  wr('data %d' % (len(desc)+1)) # wtf?

  wr(desc)

  wr()

def export_tags(ui,repo,old_marks,mapping_cache,count,authors,tagsmap):

  l=repo.tagslist()

  for tag,node in l:

    # Remap the branch name

    tag=sanitize_name(tag,"tag",tagsmap)

    # ignore latest revision

    if tag=='tip': continue

    # ignore tags to nodes that are missing (ie, 'in the future')

    if node.encode('hex_codec') not in mapping_cache:

      sys.stderr.write('Tag %s refers to unseen node %s\n' % (tag, node.encode('hex_codec')))

      continue

    rev=int(mapping_cache[node.encode('hex_codec')])

    ref=revnum_to_revref(rev, old_marks)

    if ref==None:

      sys.stderr.write('Failed to find reference for creating tag'

          ' %s at r%d\n' % (tag,rev))

      continue

    sys.stderr.write('Exporting tag [%s] at [hg r%d] [git %s]\n' % (tag,rev,ref))

    wr('reset refs/tags/%s' % tag)

    wr('from %s' % ref)

    wr()

    count=checkpoint(count)

  return count

def load_mapping(name, filename):

  cache={}

  if not os.path.exists(filename):

    sys.stderr.write('Could not open mapping file [%s]\n' % (filename))

    return cache

  f=open(filename,'r')

  l=0

  a=0

  lre=re.compile('^([^=]+)[ ]*=[ ]*(.+)$')

  for line in f.readlines():

    l+=1

    line=line.strip()

    if line=='' or line[0]=='#':

      continue

    m=lre.match(line)

    if m==None:

      sys.stderr.write('Invalid file format in [%s], line %d\n' % (filename,l))

      continue

    # put key:value in cache, key without ^:

    cache[m.group(1).strip()]=m.group(2).strip()

    a+=1

  f.close()

  sys.stderr.write('Loaded %d %s\n' % (a, name))

  return cache

def branchtip(repo, heads):

  '''return the tipmost branch head in heads'''

  tip = heads[-1]

  for h in reversed(heads):

    if 'close' not in repo.changelog.read(h)[5]:

      tip = h

      break

  return tip

def verify_heads(ui,repo,cache,force,branchesmap):

  branches={}

  for bn, heads in repo.branchmap().iteritems():

    branches[bn] = branchtip(repo, heads)

  l=[(-repo.changelog.rev(n), n, t) for t, n in branches.items()]

  l.sort()

  # get list of hg's branches to verify, don't take all git has

  for _,_,b in l:

    b=get_branch(b)

    sanitized_name=sanitize_name(b,"branch",branchesmap)

    sha1=get_git_sha1(sanitized_name)

    c=cache.get(sanitized_name)

    if sha1!=c:

      sys.stderr.write('Error: Branch [%s] modified outside hg-fast-export:'

        '\n%s (repo) != %s (cache)\n' % (b,sha1,c))

      if not force: return False

  # verify that branch has exactly one head

  t={}

  for h in repo.heads():

    (_,_,_,_,_,_,branch,_)=get_changeset(ui,repo,h)

    if t.get(branch,False):

      sys.stderr.write('Error: repository has at least one unnamed head: hg r%s\n' %

          repo.changelog.rev(h))

      if not force: return False

    t[branch]=True

  return True

def hg2git(repourl,m,marksfile,mappingfile,headsfile,tipfile,

           authors={},branchesmap={},tagsmap={},

           sob=False,force=False,hgtags=False,notes=False,encoding='',fn_encoding=''):

  def check_cache(filename, contents):

    if len(contents) == 0:

      sys.stderr.write('Warning: %s does not contain any data, this will probably make an incremental import fail\n' % filename)

  _max=int(m)

  old_marks=load_cache(marksfile,lambda s: int(s)-1)

  mapping_cache=load_cache(mappingfile)

  heads_cache=load_cache(headsfile)

  state_cache=load_cache(tipfile)

  if len(state_cache) != 0:

    for (name, data) in [(marksfile, old_marks),

                         (mappingfile, mapping_cache),

                         (headsfile, state_cache)]:

      check_cache(name, data)

  ui,repo=setup_repo(repourl)

  if not verify_heads(ui,repo,heads_cache,force,branchesmap):

    return 1

  try:

    tip=repo.changelog.count()

  except AttributeError:

    tip=len(repo)

  min=int(state_cache.get('tip',0))

  max=_max

  if _max<0 or max>tip:

    max=tip

  for rev in range(0,max):

  	(revnode,_,_,_,_,_,_,_)=get_changeset(ui,repo,rev,authors)

  	mapping_cache[revnode.encode('hex_codec')] = str(rev)

  c=0

  brmap={}

  for rev in range(min,max):

    c=export_commit(ui,repo,rev,old_marks,max,c,authors,branchesmap,

                    sob,brmap,hgtags,encoding,fn_encoding)

  if notes:

    for rev in range(min,max):

      c=export_note(ui,repo,rev,c,authors, encoding, rev == min and min != 0)

  state_cache['tip']=max

  state_cache['repo']=repourl

  save_cache(tipfile,state_cache)

  save_cache(mappingfile,mapping_cache)

  c=export_tags(ui,repo,old_marks,mapping_cache,c,authors,tagsmap)

  sys.stderr.write('Issued %d commands\n' % c)

  return 0

if __name__=='__main__':

  def bail(parser,opt):

    sys.stderr.write('Error: No %s option given\n' % opt)

    parser.print_help()

    sys.exit(2)

  parser=OptionParser()

  parser.add_option("-m","--max",type="int",dest="max",

      help="Maximum hg revision to import")

  parser.add_option("--mapping",dest="mappingfile",

      help="File to read last run's hg-to-git SHA1 mapping")

  parser.add_option("--marks",dest="marksfile",

      help="File to read git-fast-import's marks from")

  parser.add_option("--heads",dest="headsfile",

      help="File to read last run's git heads from")

  parser.add_option("--status",dest="statusfile",

      help="File to read status from")

  parser.add_option("-r","--repo",dest="repourl",

      help="URL of repo to import")

  parser.add_option("-s",action="store_true",dest="sob",

      default=False,help="Enable parsing Signed-off-by lines")

  parser.add_option("--hgtags",action="store_true",dest="hgtags",

      default=False,help="Enable exporting .hgtags files")

  parser.add_option("-A","--authors",dest="authorfile",

      help="Read authormap from AUTHORFILE")

  parser.add_option("-B","--branches",dest="branchesfile",

      help="Read branch map from BRANCHESFILE")

  parser.add_option("-T","--tags",dest="tagsfile",

      help="Read tags map from TAGSFILE")

  parser.add_option("-f","--force",action="store_true",dest="force",

      default=False,help="Ignore validation errors by force")

  parser.add_option("-M","--default-branch",dest="default_branch",

      help="Set the default branch")

  parser.add_option("-o","--origin",dest="origin_name",

      help="use <name> as namespace to track upstream")

  parser.add_option("--hg-hash",action="store_true",dest="notes",

      default=False,help="Annotate commits with the hg hash as git notes in the hg namespace")

  parser.add_option("-e",dest="encoding",

      help="Assume commit and author strings retrieved from Mercurial are encoded in <encoding>")

  parser.add_option("--fe",dest="fn_encoding",

      help="Assume file names from Mercurial are encoded in <filename_encoding>")

  (options,args)=parser.parse_args()

  m=-1

  if options.max!=None: m=options.max

  if options.marksfile==None: bail(parser,'--marks')

  if options.mappingfile==None: bail(parser,'--mapping')

  if options.headsfile==None: bail(parser,'--heads')

  if options.statusfile==None: bail(parser,'--status')

  if options.repourl==None: bail(parser,'--repo')

  a={}

  if options.authorfile!=None:

    a=load_mapping('authors', options.authorfile)

  b={}

  if options.branchesfile!=None:

    b=load_mapping('branches', options.branchesfile)

  t={}

  if options.tagsfile!=None:

    t=load_mapping('tags', options.tagsfile)

  if options.default_branch!=None:

    set_default_branch(options.default_branch)

  if options.origin_name!=None:

    set_origin_name(options.origin_name)

  encoding=''

  if options.encoding!=None:

    encoding=options.encoding

  fn_encoding=encoding

  if options.fn_encoding!=None:

    fn_encoding=options.fn_encoding

  sys.exit(hg2git(options.repourl,m,options.marksfile,options.mappingfile,

                  options.headsfile, options.statusfile,

                  authors=a,branchesmap=b,tagsmap=t,

                  sob=options.sob,force=options.force,hgtags=options.hgtags,

                  notes=options.notes,encoding=encoding,fn_encoding=fn_encoding))

#!/bin/sh

# Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.

# License: MIT <http://www.opensource.org/licenses/mit-license.php>

ROOT="$(dirname "$(which "$0")")"

REPO=""

PFX="hg2git"

SFX_MAPPING="mapping"

SFX_MARKS="marks"

SFX_HEADS="heads"

SFX_STATE="state"

GFI_OPTS=""

PYTHON=${PYTHON:-python}

USAGE="[--quiet] [-r <repo>] [--force] [-m <max>] [-s] [--hgtags] [-A <file>] [-B <file>] [-T <file>] [-M <name>] [-o <name>] [--hg-hash] [-e <encoding>]"

LONG_USAGE="Import hg repository <repo> up to either tip or <max>

If <repo> is omitted, use last hg repository as obtained from state file,

GIT_DIR/$PFX-$SFX_STATE by default.

Note: The argument order matters.

Options:

	--quiet   Passed to git-fast-import(1)

	-r <repo> Mercurial repository to import

	--force   Ignore validation errors when converting, and pass --force

	          to git-fast-import(1)

	-m <max>  Maximum revision to import

	-s        Enable parsing Signed-off-by lines

	--hgtags  Enable exporting .hgtags files

	-A <file> Read author map from file

	          (Same as in git-svnimport(1) and git-cvsimport(1))

	-B <file> Read branch map from file

	-T <file> Read tags map from file

	-M <name> Set the default branch name (defaults to 'master')

	-o <name> Use <name> as branch namespace to track upstream (eg 'origin')

	--hg-hash Annotate commits with the hg hash as git notes in the

                  hg namespace.

	-e <encoding> Assume commit and author strings retrieved from

	              Mercurial are encoded in <encoding>

	--fe <filename_encoding> Assume filenames from Mercurial are encoded

	                         in <filename_encoding>

"

case "$1" in

    -h|--help)

      echo "usage: $(basename "$0") $USAGE"

      echo ""

      echo "$LONG_USAGE"

      exit 0

esac

IS_BARE=$(git rev-parse --is-bare-repository) \

    || (echo "Could not find git repo" ; exit 1)

if test "z$IS_BARE" != ztrue; then

   # This is not a bare repo, cd to the toplevel

   TOPLEVEL=$(git rev-parse --show-toplevel) \

       || (echo "Could not find git repo toplevel" ; exit 1)

   cd $TOPLEVEL || exit 1

fi

GIT_DIR=$(git rev-parse --git-dir) || (echo "Could not find git repo" ; exit 1)

while case "$#" in 0) break ;; esac

do

  case "$1" in

    -r|--r|--re|--rep|--repo)

      shift

      REPO="$1"

      ;;

    --q|--qu|--qui|--quie|--quiet)

      GFI_OPTS="$GFI_OPTS --quiet"

      ;;

    --force)

      # pass --force to git-fast-import and hg-fast-export.py

      GFI_OPTS="$GFI_OPTS --force"

      break

      ;;

    -*)

      # pass any other options down to hg2git.py

      break

      ;;

    *)

      break

      ;;

  esac

  shift

done

# for convenience: get default repo from state file

if [ x"$REPO" = x -a -f "$GIT_DIR/$PFX-$SFX_STATE" ] ; then

  REPO="`grep '^:repo ' "$GIT_DIR/$PFX-$SFX_STATE" | cut -d ' ' -f 2`"

  echo "Using last hg repository \"$REPO\""

fi

if [  -z "$REPO" ]; then

    echo "no repo given, use -r flag"

    exit 1

fi

# make sure we have a marks cache

if [ ! -f "$GIT_DIR/$PFX-$SFX_MARKS" ] ; then

  touch "$GIT_DIR/$PFX-$SFX_MARKS"

fi

# cleanup on exit

trap 'rm -f "$GIT_DIR/$PFX-$SFX_MARKS.old" "$GIT_DIR/$PFX-$SFX_MARKS.tmp"' 0

_err1=

_err2=

exec 3>&1

{ read -r _err1 || :; read -r _err2 || :; } <<-EOT

$(

  exec 4>&3 3>&1 1>&4 4>&-

  {

    _e1=0

    GIT_DIR="$GIT_DIR" $PYTHON "$ROOT/hg-fast-export.py" \

      --repo "$REPO" \

      --marks "$GIT_DIR/$PFX-$SFX_MARKS" \

      --mapping "$GIT_DIR/$PFX-$SFX_MAPPING" \

      --heads "$GIT_DIR/$PFX-$SFX_HEADS" \

      --status "$GIT_DIR/$PFX-$SFX_STATE" \

      "$@" 3>&- || _e1=$?

    echo $_e1 >&3

  } | \

  {

    _e2=0

    git fast-import $GFI_OPTS --export-marks="$GIT_DIR/$PFX-$SFX_MARKS.tmp" 3>&- || _e2=$?

    echo $_e2 >&3

  }

)

EOT

exec 3>&-

[ "$_err1" = 0 -a "$_err2" = 0 ] || exit 1

# move recent marks cache out of the way...

if [ -f "$GIT_DIR/$PFX-$SFX_MARKS" ] ; then

  mv "$GIT_DIR/$PFX-$SFX_MARKS" "$GIT_DIR/$PFX-$SFX_MARKS.old"

else

  touch "$GIT_DIR/$PFX-$SFX_MARKS.old"

fi

# ...to create a new merged one

cat "$GIT_DIR/$PFX-$SFX_MARKS.old" "$GIT_DIR/$PFX-$SFX_MARKS.tmp" \

| uniq > "$GIT_DIR/$PFX-$SFX_MARKS"

# save SHA1s of current heads for incremental imports

# and connectivity (plus sanity checking)

for head in `git branch | sed 's#^..##'` ; do

  id="`git rev-parse refs/heads/$head`"

  echo ":$head $id"

done > "$GIT_DIR/$PFX-$SFX_HEADS"

# check diff with color:

# ( for i in `find . -type f | grep -v '\.git'` ; do diff -u $i $REPO/$i ; done | cdiff ) | less -r

#!/usr/bin/env python

# Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.

# License: GPLv2

from mercurial import node

from hg2git import setup_repo,load_cache,get_changeset,get_git_sha1

from optparse import OptionParser

import sys

def heads(ui,repo,start=None,stop=None,max=None):

  # this is copied from mercurial/revlog.py and differs only in

  # accepting a max argument for xrange(startrev+1,...) defaulting

  # to the original repo.changelog.count()

  if start is None:

    start = node.nullid

  if stop is None:

    stop = []

  if max is None:

    max = repo.changelog.count()

  stoprevs = dict.fromkeys([repo.changelog.rev(n) for n in stop])

  startrev = repo.changelog.rev(start)

  reachable = {startrev: 1}

  heads = {startrev: 1}

  parentrevs = repo.changelog.parentrevs

  for r in xrange(startrev + 1, max):

    for p in parentrevs(r):

      if p in reachable:

        if r not in stoprevs:

          reachable[r] = 1

        heads[r] = 1

      if p in heads and p not in stoprevs:

        del heads[p]

  return [(repo.changelog.node(r),str(r)) for r in heads]

def get_branches(ui,repo,heads_cache,marks_cache,mapping_cache,max):

  h=heads(ui,repo,max=max)

  stale=dict.fromkeys(heads_cache)

  changed=[]

  unchanged=[]

  for node,rev in h:

    _,_,user,(_,_),_,desc,branch,_=get_changeset(ui,repo,rev)

    del stale[branch]

    git_sha1=get_git_sha1(branch)

    cache_sha1=marks_cache.get(str(int(rev)+1))

    if git_sha1!=None and git_sha1==cache_sha1:

      unchanged.append([branch,cache_sha1,rev,desc.split('\n')[0],user])

    else:

      changed.append([branch,cache_sha1,rev,desc.split('\n')[0],user])

  changed.sort()

  unchanged.sort()

  return stale,changed,unchanged

def get_tags(ui,repo,marks_cache,mapping_cache,max):

  l=repo.tagslist()

  good,bad=[],[]

  for tag,node in l:

    if tag=='tip': continue

    rev=int(mapping_cache[node.encode('hex_codec')])

    cache_sha1=marks_cache.get(str(int(rev)+1))

    _,_,user,(_,_),_,desc,branch,_=get_changeset(ui,repo,rev)

    if int(rev)>int(max):

      bad.append([tag,branch,cache_sha1,rev,desc.split('\n')[0],user])

    else:

      good.append([tag,branch,cache_sha1,rev,desc.split('\n')[0],user])

  good.sort()

  bad.sort()

  return good,bad

def mangle_mark(mark):

  return str(int(mark)-1)

if __name__=='__main__':

  def bail(parser,opt):

    sys.stderr.write('Error: No option %s given\n' % opt)

    parser.print_help()

    sys.exit(2)

  parser=OptionParser()

  parser.add_option("--marks",dest="marksfile",

      help="File to read git-fast-import's marks from")

  parser.add_option("--mapping",dest="mappingfile",

      help="File to read last run's hg-to-git SHA1 mapping")

  parser.add_option("--heads",dest="headsfile",

      help="File to read last run's git heads from")

  parser.add_option("--status",dest="statusfile",

      help="File to read status from")

  parser.add_option("-r","--repo",dest="repourl",

      help="URL of repo to import")

  parser.add_option("-R","--revision",type=int,dest="revision",

      help="Revision to reset to")

  (options,args)=parser.parse_args()

  if options.marksfile==None: bail(parser,'--marks option')

  if options.mappingfile==None: bail(parser,'--mapping option')

  if options.headsfile==None: bail(parser,'--heads option')

  if options.statusfile==None: bail(parser,'--status option')

  if options.repourl==None: bail(parser,'--repo option')

  if options.revision==None: bail(parser,'-R/--revision')

  heads_cache=load_cache(options.headsfile)

  marks_cache=load_cache(options.marksfile,mangle_mark)

  state_cache=load_cache(options.statusfile)

  mapping_cache = load_cache(options.mappingfile)

  l=int(state_cache.get('tip',options.revision))

  if options.revision+1>l:

    sys.stderr.write('Revision is beyond last revision imported: %d>%d\n' % (options.revision,l))

    sys.exit(1)

  ui,repo=setup_repo(options.repourl)

  stale,changed,unchanged=get_branches(ui,repo,heads_cache,marks_cache,mapping_cache,options.revision+1)

  good,bad=get_tags(ui,repo,marks_cache,mapping_cache,options.revision+1)

  print "Possibly stale branches:"

  map(lambda b: sys.stdout.write('\t%s\n' % b),stale.keys())

  print "Possibly stale tags:"

  map(lambda b: sys.stdout.write('\t%s on %s (r%s)\n' % (b[0],b[1],b[3])),bad)

  print "Unchanged branches:"

  map(lambda b: sys.stdout.write('\t%s (r%s)\n' % (b[0],b[2])),unchanged)

  print "Unchanged tags:"

  map(lambda b: sys.stdout.write('\t%s on %s (r%s)\n' % (b[0],b[1],b[3])),good)

  print "Reset branches in '%s' to:" % options.headsfile

  map(lambda b: sys.stdout.write('\t:%s %s\n\t\t(r%s: %s: %s)\n' % (b[0],b[1],b[2],b[4],b[3])),changed)

  print "Reset ':tip' in '%s' to '%d'" % (options.statusfile,options.revision)

#!/bin/sh

# Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.

# License: MIT <http://www.opensource.org/licenses/mit-license.php>

ROOT="`dirname $0`"

REPO=""

PFX="hg2git"

SFX_MARKS="marks"

SFX_MAPPING="mapping"

SFX_HEADS="heads"

SFX_STATE="state"

QUIET=""

PYTHON=${PYTHON:-python}

USAGE="[-r <repo>] -R <rev>"

LONG_USAGE="Print SHA1s of latest changes per branch up to <rev> useful

to reset import and restart at <rev>.

If <repo> is omitted, use last hg repository as obtained from state file,

GIT_DIR/$PFX-$SFX_STATE by default.

Options:

	-R	Hg revision to reset to

	-r	Mercurial repository to use

"

IS_BARE=$(git rev-parse --is-bare-repository) \

    || (echo "Could not find git repo" ; exit 1)

if test "z$IS_BARE" != ztrue; then

   # This is not a bare repo, cd to the toplevel

   TOPLEVEL=$(git rev-parse --show-toplevel) \

       || (echo "Could not find git repo toplevel" ; exit 1)

   cd $TOPLEVEL || exit 1

fi

GIT_DIR=$(git rev-parse --git-dir) || (echo "Could not find git repo" ; exit 1)

while case "$#" in 0) break ;; esac

do

  case "$1" in

    -r|--r|--re|--rep|--repo)

      shift

      REPO="$1"

      ;;

    -*)

      # pass any other options down to hg2git.py

      break

      ;;

    *)

      break

      ;;

  esac

  shift

done

# for convenience: get default repo from state file

if [ x"$REPO" = x -a -f "$GIT_DIR/$PFX-$SFX_STATE" ] ; then

  REPO="`grep '^:repo ' "$GIT_DIR/$PFX-$SFX_STATE" | cut -d ' ' -f 2`"

  echo "Using last hg repository \"$REPO\""

fi

# make sure we have a marks cache

if [ ! -f "$GIT_DIR/$PFX-$SFX_MARKS" ] ; then

  touch "$GIT_DIR/$PFX-$SFX_MARKS"

fi

GIT_DIR="$GIT_DIR" $PYTHON "$ROOT/hg-reset.py" \

  --repo "$REPO" \

  --marks "$GIT_DIR/$PFX-$SFX_MARKS" \

  --mapping "$GIT_DIR/$PFX-$SFX_MAPPING" \

  --heads "$GIT_DIR/$PFX-$SFX_HEADS" \

  --status "$GIT_DIR/$PFX-$SFX_STATE" \

  "$@"

exit $?

#!/usr/bin/env python

# Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.

# License: MIT <http://www.opensource.org/licenses/mit-license.php>

from mercurial import hg,util,ui,templatefilters

import re

import os

import sys

import subprocess

# default git branch name

cfg_master='master'

# default origin name

origin_name=''

# silly regex to see if user field has email address

user_re=re.compile('([^<]+) (<[^>]*>)$')

# silly regex to clean out user names

user_clean_re=re.compile('^["]([^"]+)["]$')

def set_default_branch(name):

  global cfg_master

  cfg_master = name

def set_origin_name(name):

  global origin_name

  origin_name = name

def setup_repo(url):

  try:

    myui=ui.ui(interactive=False)

  except TypeError:

    myui=ui.ui()

    myui.setconfig('ui', 'interactive', 'off')

  return myui,hg.repository(myui,url)

def fixup_user(user,authors):

  user=user.strip("\"")

  if authors!=None:

    # if we have an authors table, try to get mapping

    # by defaulting to the current value of 'user'

    user=authors.get(user,user)

  name,mail,m='','',user_re.match(user)

  if m==None:

    # if we don't have 'Name <mail>' syntax, extract name

    # and mail from hg helpers. this seems to work pretty well.

    # if email doesn't contain @, replace it with devnull@localhost

    name=templatefilters.person(user)

    mail='<%s>' % util.email(user)

    if '@' not in mail:

      mail = '<devnull@localhost>'

  else:

    # if we have 'Name <mail>' syntax, everything is fine :)

    name,mail=m.group(1),m.group(2)

  # remove any silly quoting from username

  m2=user_clean_re.match(name)

  if m2!=None:

    name=m2.group(1)

  return '%s %s' % (name,mail)

def get_branch(name):

  # 'HEAD' is the result of a bug in mutt's cvs->hg conversion,

  # other CVS imports may need it, too

  if name=='HEAD' or name=='default' or name=='':

    name=cfg_master

  if origin_name:

    return origin_name + '/' + name

  return name

def get_changeset(ui,repo,revision,authors={},encoding=''):

  node=repo.lookup(revision)

  (manifest,user,(time,timezone),files,desc,extra)=repo.changelog.read(node)

  if encoding:

    user=user.decode(encoding).encode('utf8')

    desc=desc.decode(encoding).encode('utf8')

  tz="%+03d%02d" % (-timezone / 3600, ((-timezone % 3600) / 60))

  branch=get_branch(extra.get('branch','master'))

  return (node,manifest,fixup_user(user,authors),(time,tz),files,desc,branch,extra)

def mangle_key(key):

  return key

def load_cache(filename,get_key=mangle_key):

  cache={}

  if not os.path.exists(filename):

    return cache

  f=open(filename,'r')

  l=0

  for line in f.readlines():

    l+=1

    fields=line.split(' ')

    if fields==None or not len(fields)==2 or fields[0][0]!=':':

      sys.stderr.write('Invalid file format in [%s], line %d\n' % (filename,l))

      continue

    # put key:value in cache, key without ^:

    cache[get_key(fields[0][1:])]=fields[1].split('\n')[0]

  f.close()

  return cache

def save_cache(filename,cache):

  f=open(filename,'w+')

  map(lambda x: f.write(':%s %s\n' % (str(x),str(cache.get(x)))),cache.keys())

  f.close()

def get_git_sha1(name,type='heads'):

  try:

    # use git-rev-parse to support packed refs

    ref="refs/%s/%s" % (type,name)

    l=subprocess.check_output(["git", "rev-parse", "--verify", "--quiet", ref])

    if l == None or len(l) == 0:

      return None

    return l[0:40]

  except subprocess.CalledProcessError:

    return None