EasyMercurial

h1. Communicating with a subprocess

h3. The problem

We want to run a subprocess (Hg) in such a way as to be able to talk to it interactively, pretending we are a terminal -- so we can deal with username/password interactions.  The subprocess has some possibly platform-dependent test for whether its input is a terminal; if not, it will not ask for credentials (failing instead).

In practice, it appears the test comes down to a call to Python's sys.stdio.isatty() function.  The documentation suggests you can override this test with e.g. --config ui.interactive=true on the command line; I'm not entirely sure.

We are using QProcess, which perhaps doesn't actually make matters any simpler to understand.

h3. Linux and OS/X

Two parts to the problem: how to give the subprocess a pseudoterminal, and how to make sure it doesn't get the controlling terminal of the parent process.  (isatty returns true if the file descriptor is a terminal or pseudoterminal.)

QProcess can set the subprocess stdin to an existing file by name.  So, we can call openpty() (nonstandard, nonrecommended BSD API) or carry out similar magic to allocate a pseudoterminal master/slave pair via /dev/ptmx or whatever, then pass the slave device name to QProcess::setStandardInputFile.

However, if our own process (the parent) was invoked from a terminal, it will already have a controlling terminal and openpty will just give us that terminal back again.  We need to detach from our controlling terminal first.

A method that seems to be recommended in some quarters is to fork, then setsid, then fork -- but we can't easily do this at the point of running the subprocess, because QProcess is handling that fork for us.  (Would execvp not be easier?)

And we can't fork/setsid/fork at the start of our own process, because although this works fine on Linux, OS/X will reject it in subsequent Core API calls ("__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__").

So instead at startup we explicitly detach from the terminal using ioctl with TIOCNOTTY on fd 0 if isatty(0) returns true, or on the result of opening /dev/tty otherwise.  That seems to work.

h3. Windows

isatty on Windows returns true if the fd "is associated with a character device (a terminal, console, printer, or serial port)".  I'm not sure what it means by a terminal -- Windows doesn't seem to have them.  A subprocess can be created (via CreateProcess) with a console, but QProcess explicitly uses the CREATE_NO_WINDOW flag which sets no console handle on the subprocess.  In any case a console in Windows is visible as a window, which probably isn't ideal.