How it works
So, we need the function we want to hook, "the original", and a function that we'd like to replace it with, "the replacement".
To hook a function, the first few bytes are overwritten with an instruction that jumps to the replacement function:
Now when ever the original function is called, the flow of control is diverted to the replacement.
This however, damages the original function. In order to be able to call the original from inside the replacement, a function that is equivalent to the original has to be created.
This is done by saving the instructions that were overwritten in the original function. They are copied to a special memory buffer that may contain executable machine code.
Following those instructions, we install another jump instruction that allows the remainder of the instructions in the original function to be executed. This is what the
original() method of a
nanohook::hook object returns.
Where nanohook may not work
These situations are very rare, but may exist in the wild.
If one of the instructions overwritten in the original function performs some kind of relative jump, the jump will now end up in the wrong place because the instruction that performs the jump is moved to a different address as part of the hooking process. If your replacement doesn't call the original function, this won't matter.
In future, I hope to be able to detect this case and either work around it or throw an exception when the hook is installed.
If subsequent instructions in the original function jump back to the instructions that have been replaced, bad things will happen. This problem is unlikely to be fixed in future, as it would require knowing where the end of the function is, which in general is very hard to determine. But again, if your replacement function doesn't call the original, this won't matter.
Modern compiler's can decide to inline functions for performance reasons. If you hook a function that the compiler has inlined in a particular location, that inlined 'copy' will not be redirected to your hook.