Macro-development specific behaviour. Gui freezes due long computations.

wobbuuu

Hi Matthias,

I found that python script (*.lym) launched through hotkey (or simply clicked in menu) and through macro-development has different behaviour.
In script I redirect stdout to QTextEdit and use usual print statements to fill it with text. When I launch script through macro-development everything is all right but when I launch through hotkey or click on it in menu (window of macro-development closed) print occures at the end of execution but not during. This happens on linux and windows.

Second one, after execution of script ended I close gui of script and exit klayout, then bug report window appears. This is happens only in windows (windows 10 x64), maybe it is something specific but the interesting fact is that doesn't happen if I launch script through macro-development!
Are there any ideas what happens? I can provide some code if you need it.

Best Regards,

Anton

Matthias

Hi Anton,

yes, some code will be helpful.

Matthias

wobbuuu

Well, I've prepared an example which consists of 2 files: lym file and user interface. Moreover different ways of launch give significant difference in execution time.

Anton

Matthias

Hi Anton,

thanks for this nice test case. I know what you're talking about now. This answer is going to be a bit longer :-)

Different behaviour of IDE and menu

I can offer the following explanation for the different behaviour:

Basically, the concept of Qt-based (or any other application running a classical GUI framework) is that of a event loop. This means, there is one loop running which takes and dispatches events (mouse, painting, keyboard etc.) to the application. The application code is then supposed to implement the events in a "friendly" way. That is: executing quickly. Because, the application is essentially blocked while the event handling code executes.

The implementation you used basically violates this principle, because it runs a long loop. You do some printouts while the loop runs, but in a normal application you won't see them because paint events don't get dispatched while the loop runs. That explains why your QTextEdit is not getting updated when you run the code from the hotkey or menu. This is "normal application mode".

If you run it from the macro development IDE, you see these printouts, because the IDE will hook itself into the interpreter's execution engine. It needs to do so because that way it can handle breakpoints and allow you stopping the execution. It will also dispatch paint events and handle some mouse events - for example, the ones that target towards the IDE itself. Hence you can push the "Stop" button while your loop is busy. This explains why you see the updates and this is also the explanation why the run times are longer. The hooks will slow down the interpreter. If you close the IDE or you disable debugging, the interpreter runs at full speed, but also updates won't happen.

So now, what's the solution? Here are some suggestions. The final solution depends on the nature of your loop.

Use a timer and execute just a slice of operations

For example, if your loop is converting 10M elements, you could create an object which provides this conversion. This object needs to hold the data for your operation and the current status. Instead of running the whole conversion, you implement a method that converts let's say 10k elements and then stops. Essentially you will then instantiate your conversion objects and call the processing method 1000 times to complete the operation.

You can then set up a QTimer with a fast interval. Set the interval to 0ms - that means, Qt will issue the timer callback on each iteration of the event loop after it has processed the other events. In the timer callback you do the processing. You'll need some more logic to start and stop the timer and detect termination of the processing.

But over all, this is actually the scheme you are supposed to implement in classic UI style applications.

Here is a sketch:

class Converter:
  def __init__(self, things_it_needs_to_run):
    ...
  def finished(self):
    ... returns True if finished, False if not
  def perform(self):
    ...

# plain execution
converter = Converter(...)
while not converter.finished():
  converter.perform()

# within GUI
timer = QTimer()
timer.interval = 0
timer.timeout 

Make your loop process events

You can add QCoreApplication#processEvents calls to your loop. This will make your loop dispatch pending events, so you application will stay "alive". Don't call this method too often, because it will add some overhead. For example, call it every 1000th iteration.

This scheme looks simple and effective, but there are a couple of pitfalls. Your loop will basically replace the main event loop and you need to emulate everything that happens there. For example, you need to consider the case that someone closes the dialog window while your loop runs. You'll need to detect this case and terminate your loop in this case. You way also want to block user interactions to avoid this case. In can do so globally by telling processEvents not to handle user input events. But that means, you cannot implement some "Stop" button. If you want to do this, you'll need to disable the features you don't want users to use while the loop runs. Or you use event filters to ignore events that would interfere with your application.

Threads

Threads are beautifully dangerous. Threads can provide a way to execute operations in the background while the main thread does the GUI. This works nicely as long as your thread and the main thread don't try to access the same memory and interfere in unpredictable ways. The art is to minimize these interactions and to design them in a way they are thread-safe. You'll need to be familiar with synchronisation features such as mutexes to master this. Threads and GUI fit together nicely and badly: Qt signals are well dispatched across thread boundaries but if two threads access the same Qt object, you will encounter sporadic, hard to reproduce and even harder to debug crashes. A proper application design is required that avoids such interactions. A nicely separated worker thread and a task queue is a useful design pattern here.

Python offers threads, but I don't think they work with KLayout since I have not initialized Python's GIL (global interpreter lock). I have not tried it, so I cannot recommend this scheme. Apart from the issues with threads in general.

Separate processes

If you are able to wrap your processing code into a separate script (i.e. using KLayout in batch mode), you will be able to run this as a separate process. QProcess provides a nice abstraction. You can pick it's stdout in a timer event and display it inside the QTextEdit.

That's not the most efficient, but a very safe approach. But you need to export the input data to this process to a file and pull the results from another file too. I'd chose this approach to implement long-running operations that deliver a limited amount of output. Such as simulators or verification steps.

The crash you observe

This is probably due to a known issue with unregistered top level windows. It's hard to reproduce, so I cannot verify this.

But, you should be able to avoid this issue by deleting the window after you closed it:

org_stdout = sys.stdout
org_stderr = sys.stderr

edit = form.textEdit
sys.stdout = OutLog(edit, out=sys.stdout)
sys.stderr = OutLog(edit, out=sys.stderr, color=pya.QColor(255,0,0))

....
form.exec_()

sys.stdout = org_stdout
sys.stderr = org_stderr
edit = None

# this will make the form destroyed after execution has finished
form._destroy()

This method will instruct the Python code to destroy the underlying Qt objects. This should avoid the crash. Please note that it's mandatory to reset the stdout/stderr handlers so they will not access the dead edit object.

Some more remarks

Please note that running a script in the macro IDE does not mean it's run in some isolated environment. In fact, your script will modify the application's environment. That's what scripts were made for. But it needs to be made in a way that it acts "cooperative". Specifically:

• Tampering with the global stderr/stdout is not a good idea. You will basically modify the environment for all other scripts. Please consider using your own print implementation to print to the edit widget. At least please make sure the stderr/stdout handlers are reset back to their initial state once you're done.
• Your dialog is modal. Modality is application wide, so it will lock out the development IDE. That's why you can't interact with the IDE while your script runs. That's a basic limitation and I don't know whether there is a solution to this.
• There are subtle interactions between form-generated Qt objects and Python. Python does not always know when Qt objects get destroyed. Hence it may happen that you have Python objects pointing to dead Qt objects. That will usually cause a crash. It's better to clean up all dependencies once you're finished (like resetting the stderr/stdout handlers in the code above).

Kind regards,

Matthias

wobbuuu

Hello Matthias,

Wow! Thanks for this explanation and tips! I suppose I will implement one of your ideas. Also I'll try your suggestion about crash. I need some time to do these.

Best Regards,

Anton

wobbuuu

Hi Matthias,

Firstly, I'd like to mention that you were right about the crash, I won't use stdout for this anymore.:)

According to your answer I decided to use threads for my purpose. I went through some posts on stackoverflow looking for solution which I could implement out of the box but I didn't find any one. Basically I understand that it is better to use QThread instead of python's threading.Thread. Moreover someone write that it's better to use moveToThread of QObject method than QThread subclass. But anyway this doesn't work - I mean main thread go further and worker's thread can't print out messages (I send textEdit like an argument to worker). According to this post I suppose that pyqtSignal do the trick but I can't find it in documentation of Klayout's Qt classes. So what are you thinking about this? Part of my code which start worker thread mostly looks like in the last link except signals/slots.

Regards,

Anton

Matthias

Hi Anton,

I actually intended to discourage use of threads ...

KLayout does not use PyQt, so things are entirely different. Specifically there is no provision for defining and emitting signals - in a dynamic language they don't make sense. They are a C++ substitute for dynamic method binding which is available in a dynamic language by nature. So I did not provide them. Your application is the first one to require custom signals.

Events are an option, but I suspect there is an issue with the ownership of the event object (it needs to be released from Python's memory management), so I cannot recommend this approach too.

So basically, this is entirely new territory. It's an interesting attempt, but I don't believe this is feasible. I can try to make the current master branch fit for this approach, but that won't be soon.

So I'd definitely recommend one of the other approaches. Specifically since I doubt that Python itself is thread safe since I did not initialize and lock the GIL.

Matthias

wobbuuu

Hi Matthias,

I was actually hurt by my idea using threads, so I made process event loop like you suggested before. I suppose it is a good idea to post some code here, maybe it will be helpfull for someone. I simply implemented Timer class which "update" gui once a period which you choose by yourself.

import pya
import time

class Timer:
    def __init__(self):
        self.start = time.time()
        self.period = 0.1 # period of update in seconds
        self.last_update = self.start - self.period

    def update(self):
        self.current = time.time()
        if self.current > self.last_update + self.period:
            pya.QCoreApplication.processEvents()         
            self.last_update = self.current 

Somewhere in long loop computations timer should be updated.

timer = Timer()
for i in range(10 ** 7):
    timer.update()
    # compute smth

That works well for me, so it may be a decision if gui freezes due to long computations.

Best regards,

Anton

Matthias

Hi Anton,

that's a reasonable to do it. I'll look into the threading approach, but not with high priority.

KLayout uses a similar approach internally which is delivered by the progress reporting feature. But with C++ the overhead of processEvents is significant, so it's not always possible to deliver these calls without sacrificing performance. Plus with complex algorithms, it's not always possible to identify the main loops. That's why the GUI sometimes freezes. Threads can mitigate this issue, but I learned the hard way to be very cautious with them.

Regards,

Matthias