EasyMercurial

h1. Communicating with a subprocess

h3. The problem

We want to run a subprocess (Hg) in such a way as to be able to talk to it interactively, pretending we are a terminal -- so we can deal with username/password interactions. The subprocess has some possibly platform-dependent test for whether its input is a terminal; if not, it will not ask for credentials (failing instead).

In practice, it appears the test comes down to a call to Python's sys.stdio.isatty() function. The documentation suggests you can override this test with e.g. --config ui.interactive=true on the command line; I'm not entirely sure.

We are using QProcess, which perhaps doesn't actually make matters any simpler to understand.

h3. Linux and OS/X

Two parts to the problem: how to give the subprocess a pseudoterminal, and how to make sure it doesn't get the controlling terminal of the parent process. (isatty returns true if the file descriptor is a terminal or pseudoterminal.)

QProcess can set the subprocess stdin to an existing file by name. So, we can call openpty() (nonstandard, nonrecommended BSD API) or carry out similar magic to allocate a pseudoterminal master/slave pair via /dev/ptmx or whatever, then pass the slave device name to QProcess::setStandardInputFile.

This seems to get confused if the parent process already has a controlling terminal. We need to detach from our controlling terminal first.

A method that seems to be recommended in some quarters is to fork, then setsid, then fork -- but we can't easily do this at the point of running the subprocess, because QProcess is handling that fork for us. (Would execvp not be easier?)

And we can't fork/setsid/fork at the start of our own process, because although this works fine on Linux, OS/X will reject it in subsequent Core API calls ("__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__").

So instead at startup we explicitly detach from the terminal using ioctl with TIOCNOTTY on fd 0 if isatty(0) returns true, or on the result of opening /dev/tty otherwise. That seems to work.

*Update:* Doesn't work as well as I thought. It's OK for the password, but for the username it tries to start using readline which confuses us completely.

h3. Windows

isatty on Windows returns true if the fd "is associated with a character device (a terminal, console, printer, or serial port)". (I'm not sure what it means by a terminal -- Windows doesn't seem to have them.)

It may be possible to override the isatty check with a config property as above -- but this doesn't solve things for us on Windows because Python's getpass uses putch and getch to write and read; these are low-level functions that interact directly with the console, not with stderr/stdin.

Options:

# Give the subprocess a console. QProcess explicitly uses the CREATE_NO_WINDOW flag to CreateProcess which sets no console handle on the subprocess, so we would have to stop using QProcess. A console in Windows is visible as a window -- there doesn't seem to be an equivalent to a pty -- not ideal.
# Send username/password to the subprocess on the command-line, perhaps using "MercurialKeyring":http://mercurial.selenic.com/wiki/KeyringExtension as well: @--config auth.default.username=plugh --config auth.default.password=plagh --config ui.interactive=false --config extensions.mercurial_keyring=@ (ugh, and not secure enough)
# Use the keyring extension and use the keyring directly beforehand, to set the password (this one, like the previous one, imply that we know in advance whether the password is needed -- perhaps we can check for the authorization required message when running noninteractively). This means we need to use the same keyring protocol, obviously -- I believe the Hg extension uses "Win32 CryptAPI":http://msdn.microsoft.com/en-us/library/ms947420.aspx via "Python Keyring":http://pypi.python.org/pypi/keyring
# Provide our own Mercurial extension which overrides ui.getpass in a way that is compatible with our purposes (e.g. perhaps showing its own dialog) -- is there really no such thing out there? There must be
# Call the whole thing off and rewrite in Python using the Hg API like any sensible person would have done in the first place.