TinyMCE image manager: Error executing child request for rpc.aspx
The Moxiecode image manager plugin communicates with the server using rpc/default.aspx, which does nothing except transfer the request to rpc.aspx. This page normally responds in the JSON format, even when an error occurs. However, in some cases the error handling code itself gives an error, and then you get the error "The server response wasn't in JSON format, Error executing child request for rpc.aspx".
Calling rpc.aspx directly
The response was not in JSON format, because the page threw an exception and returned the error 500 page instead of JSON. The child request failed because rpc.aspx threw an exception. To debug this problem, we would like to see the exception. Since default.aspx does nothing else than transfer the request to rpc.aspx, we can get more detail by replacing default.aspx by rpc.aspx in our request.
Use the developer tools of your browser (press F12 on the keyboard) to see what request is failing. Then, open that link, replace default.aspx by rpc.aspx. This will give you the exception you are looking for.
In my case rpc.aspx gave an exception because it could write the error to the log file. IIS had no write permissions for the path I specified as log.path. I fixed this by giving the user NETWORK SERVICE write permissions to the log directory.