Inter-Process Communication using QSocketNotifier

sekigawa

This may relate to
"External control of the view position and magnification" recently posted by Ken.
Please refer to >> http://klayout.de/forum/comments.php?DiscussionID=521&page=1

I'm also studying some different scenarios where KLayout is externally controlled.
Incidentally, binding KLayout with a microscope is one of those scenarios ;-).
Thank Ken for sharing idea.

I believe those tasks can be generally categorized as "Inter-Process Communication with KLayout."
So I'm studying how to use Qt-binding mechanism provided by KLayout.
"Using the Qt binding: transforming KLayout into a HTTP server"
(Refer to >> http://www.klayout.de/rba_examples.html#example9)
is a good example to have KLayout work as a server.
Obviously the client is a web browser in this example.


My attempt is just opposite; to have KLayout work as a "client."
That is, a server is already running and KLayout with RBA attempts to connect
to the server with an extended menu selection.
Once the connection is established between the two, further communication starts.
In my first study, the server sends a message string to the client (KLayout /w RBA) and
the client attempts to echo-back the received message as it is.
As of now, 3 different in-coming channels are tested including
1) Tcp socket
2) Local socket
3) STDIN (KLayout is spawned by the server using QProcess)

To detect arrival of a message, a QSocketNotifier bundled with a proper descriptor
is to be used (as far as I understood) on the client side.
The table below shows the current status of my study.
On Linux, everything is fine. However, the socket notifier does not respond at all
on Windows. The behavior on Mac is more mysterious: I'm novice at OSX programming ;-(.

I'll appreciate useful comments and suggestions on this.

Kazzz


<<< Client Side :: KLayout /w RBA >>>
+========================================================+============+==============+=======+
OS and KLayout environment | QSocketNotifier bundled with...
| (for reading)
+========================================================+============+==============+=======+
OS KLayout Ruby KLayout's Qt | QTcpSocket | QLocalSocket | STDIN
+--------------------------------------------------------+------------+--------------+-------+
Ubuntu | | |
Linux 64-bit 0.23.5 1.9.2-p0 4.7.0 | O | O | O
+--------------------------------------------------------+------------+--------------+-------+
Windows7 32-bit 0.23.6 1.9.1-p430 4.8.2 | X | X | X
(installer | | |
package) | | |
+--------------------------------------------------------+------------+--------------+-------+
Mac OSX 64-bit 0.23.3 1.9.3-p484 4.8.4 | X | M | O
+========================================================+============+==============+=======+
Legend:
O = Socket notifier responds and works as intended
X = Socket notifier does not respond at all
M = Once activated, looses control over .setEnabled(true|false)


<<< Server Side :: Native Qt Application >>>
+========================================================
OS and Server program environment
+========================================================
OS Qt Compiler
+--------------------------------------------------------
Ubuntu
Linux 64-bit 4.7.0 g++ 4.4.4
+--------------------------------------------------------
Windows7 32-bit 4.8.6 VC++ 2010 (cl 16.00)
+--------------------------------------------------------
Mac OSX 64-bit 4.8.4 Clang
+========================================================

Matthias

Hello,

here is the formatted table again for convenience (you can use Markdown formatting: put four blanks in from of each line will make the text formatted as monospace code):

<<< Client Side :: KLayout /w RBA >>>
+========================================================+============+==============+=======+
            OS and KLayout environment                   |  QSocketNotifier bundled with...
                                                         |           (for reading)
+========================================================+============+==============+=======+
         OS          KLayout      Ruby     KLayout's Qt  | QTcpSocket | QLocalSocket | STDIN
+--------------------------------------------------------+------------+--------------+-------+
  Ubuntu                                                 |            |              |
  Linux     64-bit   0.23.5    1.9.2-p0      4.7.0       |      O     |      O       |    O
+--------------------------------------------------------+------------+--------------+-------+
  Windows7  32-bit   0.23.6    1.9.1-p430    4.8.2       |      X     |      X       |    X
                   (installer                            |            |              |
                     package)                            |            |              |
+--------------------------------------------------------+------------+--------------+-------+
  Mac OSX   64-bit   0.23.3    1.9.3-p484    4.8.4       |      X     |      M       |    O
+========================================================+============+==============+=======+
  Legend:
    O = Socket notifier responds and works as intended
    X = Socket notifier does not respond at all
    M = Once activated, looses control over .setEnabled(true|false)


<<< Server Side :: Native Qt Application >>>
+========================================================
          OS and Server program environment
+========================================================
         OS           Qt        Compiler
+--------------------------------------------------------
  Ubuntu
  Linux     64-bit   4.7.0    g++ 4.4.4
+--------------------------------------------------------
  Windows7  32-bit   4.8.6    VC++ 2010 (cl 16.00)
+--------------------------------------------------------
  Mac OSX   64-bit   4.8.4    Clang
+========================================================

Regarding the technical question embedded in this post: to be frank, I have not used QSocketNotifier on Windows myself. STDIN may not be available since KLayout is not compiled as a Console application on Windows. I can't tell why the QTcpSocket connection does not work - I think it requires a certain way the event loop is run. Maybe it needs to be configured to be listening to socket events on Windows.

If you are using QTcpSocket, you can use the signals provided by that object rather than QSocketNotifier. I guess this the preferred way. QTcpSocket gives you the "readyRead" signal (through QIODevice) for example. You can use that signal to detect that there are bytes available for reading.

Depending on the protocol, you may be able to avoid signals at all - if your server is responding quickly to your request, you can send a brief message with a command and wait for the return of the reply. A protocol for moving a microscope stage for example may look like this:

KLayout sends: [# of bytes] "MOVE_TO x y"
Controller responds: [# of bytes] "OK"

KLayout sends: [# of bytes] "STATUS?"
Controller responds: [# of bytes] "BUSY"
...
(the status request is repeated until finally:)
...
KLayout sends: [# of bytes] "STATUS?"
Controller responds: [# of bytes] "OK"

If each packet starts with a, lets say, a 16bit field with the number bytes in the message, the server will be able to stop reading immediately after having received these number of bytes and start sending the reply and vice versa. That way, there is no delay or waiting for data - just send a message and collect the result. Your application just needs to decide what to do while waiting for the operation to finish. You can poll the result in a QTimer event for example. Hence the application can stay active while the handshaking happens in the background.

That's just a bunch of suggestions - please comment.

Matthias

sekigawa

Hello,

Thank you for beautifying the messy table in the original posting.

I have noticed the presence of [signal] QIODevice::readyRead() in the native Qt tool kit.
However, I could not find the same in KLayout's Assistant Document.
Hence, I used [event] QSocketNotifier.activated() instead.
Below is the relevant part of my RBA script.

:
$sockTcp = QTcpSocket::new( RBA::Application.instance )
desc  = $sockTcp.socketDescriptor
state = $sockTcp.state
$sockTcp.connectToHost( $hostAddr, $hostPort, QIODevice_OpenModeFlag::ReadWrite.to_i )
$sockTcpNotifier = QSocketNotifier::new( desc, QSocketNotifier_Type::Read )
$sockTcpNotifier.activated(desc) do
    msg = sockTcpPullData()
    if ( msg != EmptyStr ) then
        sockTcpPushData( msg )
    end
end
:

With this, I got the warning below on Windows; not on Linux.
It looks the socket (descriptor) is already watched by somebody.

Warning: QSocketNotifier: Multiple socket notifiers for same socket 1528 and type Read

By guess work (assuming that [even] QIODevice::readyRead does exist but is dropped from the API document)
I modified as below. But, this resulted in "NoMethodError: undefined method `readyRead'".

:
$sockTcp = QTcpSocket::new( RBA::Application.instance )
desc  = $sockTcp.socketDescriptor
state = $sockTcp.state
$sockTcp.connectToHost( $hostAddr, $hostPort, QIODevice_OpenModeFlag::ReadWrite.to_i )
#$sockTcpNotifier = QSocketNotifier::new( desc, QSocketNotifier_Type::Read )
#$sockTcpNotifier.activated(desc) do
$sockTcp.readyRead do
    msg = sockTcpPullData()
    if ( msg != EmptyStr ) then
        sockTcpPushData( msg )
    end
end
:

I will try the approach suggested by your.
Thank you for your assistance and suggestions.

Kazzz

Matthias

Hello,

well, you're right ... the readyRead is missing. Thanks for mentioning this.

The documentation and the Ruby bindings are generated automatically from the Qt headers. So if the documentation does not mention it, there really is no signal.

I guess there is a bug in that generator script. I'll check that and provide a fix for the next minor release.

Regarding the warning: maybe you got it after running the script again? I'd have an explanation for that. If you run the script again, you create another QSocketNotifier. You assign that to the variable. But if the variable is already set, then there are two QSocketNotifier objects and that may cause the problem. The first one will go away when Ruby's garbage collector cleans up the memory, but this does not happen immediately.

To force an object to go away, you can explicitly destroy it before you create a new one

$sockTcpNotifier && $sockTcpNotifier.destroy
$sockTcpNotifier = QSocketNotifier::new(desc, QSocketNotifier_Type::Read) 

The destroy method will not destroy the Ruby object, but the underlying C++ object. The Ruby object is cleaned up later by the garbage collector.

Maybe that effect explains your observations on MacOS too. I assume that Windows, Linux and MacOS use different Ruby versions and the details of the garbage collection implementation various between Ruby versions.

BTW: the coming Python version behaves more predictably in that respect.

Matthias

sekigawa

Hello,

Thank you for your response.
I have first confirmed that the socket notifier is created only once in one session.
Also tested "$sockTcpNotifier && $sockTcpNotifier.destroy" but the warning
still shows up in the case of Windows. The root cause seems to be different.

Many thanks for adding QIODevice-related signals to the latest release 0.23.7.
As "QIODevice::readyRead" is now available, I have made changes where this event
can (should) be used instead of the socket notifier.
The table below shows the latest status of my trial.
IPC between KLayout (0.23.7) and a native C++ application is possible
using QTcpSocket on the 3 different platforms.

This is something I expected.

Kazzz

<<< Client Side :: KLayout /w RBA >>>
+========================================================+============+==============+=======+
            OS and KLayout environment                   | Inter-Process Communication channel
                                                         |           (for reading)
+========================================================+============+==============+=======+
         OS          KLayout      Ruby     KLayout's Qt  | QTcpSocket | QLocalSocket | STDIN
                                                         |     1)     |      2)      |   2)
+--------------------------------------------------------+------------+--------------+-------+
  Ubuntu                                                 |            |              |
  Linux     64-bit   0.23.7    1.9.2-p0      4.7.0       |      O     |      O       |    O
+--------------------------------------------------------+------------+--------------+-------+
  Windows7  32-bit   0.23.7    1.9.1-p430    4.8.2       |      O     |      X       |    X
                   (installer                            |            |              |
                     package)                            |            |              |
+--------------------------------------------------------+------------+--------------+-------+
  Mac OSX   64-bit   0.23.7*)  1.9.3-p484    4.8.4       |      O     |      M       |    O
+========================================================+============+==============+=======+
  Notes:
    *) Encountered some build errors while making "strm*" executable but "libklayout.so"
       and "klayout" were successfully built.

  Detection of data arrival by:
    1) QIODevice::readyRead
    2) QSocketNotifier bundled with a proper descriptor

  Legend:
    O = Socket notifier or QIODevice::readyRead responds and works as intended
    X = Socket notifier does not respond at all
    M = Once activated, looses control over .setEnabled(true|false)


<<< Server Side :: Native Qt Application >>>
+========================================================
          OS and Server program environment
+========================================================
         OS           Qt        Compiler
+--------------------------------------------------------
  Ubuntu
  Linux     64-bit   4.7.0    g++ 4.4.4
+--------------------------------------------------------
  Windows7  32-bit   4.8.6    VC++ 2010 (cl 16.00)
+--------------------------------------------------------
  Mac OSX   64-bit   5.3      Clang++ 600.0.51
+========================================================

Matthias

Hi Kazzz,

good to hear the update provides a solution. I am sorry for the missing signals initially - that would have made life easier. If you miss something again, just drop me a brief note and I'll have a look.

I think I know why the STDIN won't work on Windows (not built as a console application), but why QSocketNotifier won't work on Mac I have no clue. Since QTclSocket is the solution recommended by Qt, I hope that this is an acceptable workaround.

Best regards,

Matthias

sekigawa

Hello Matthias,

Yes, since the IPC over a TCP socket is the main method that is in my mind,
the latest release (0.23.7) provides an acceptable workaround.

Thank you and best regards,

Kazzz