Python coming ...

Matthias

There has not been much core development recently, but I have not been idle.

So what's going on?

I received some requests through that forum and by mail whether support for Python instead of Ruby is feasible. The prospect was mainly the ability to reuse existing Python packages, for there appears to be a wider selection on Python, specifically in the area of scientific applications. Plus, Python appears to grow more popular in the EDA business, so people asked for a scripting language they are familiar with.

But I did not want to drop Ruby, since there are a lot of scripts outside already.

I mentioned in a forum entry that supporting both Ruby and Python will not be possible. I started to experiment a little and finally found: It's not impossible. Both scripting engines can coexist perfectly. So there is no reason for not supporting BOTH interpreters.

In the current snapshot there is actually support for both bindings already: the macro development IDE provides a switch in the console - you can enter Python or Ruby code. There is another tab for the macros, where Python macros can be created, stored, tested and even debugged. It perfectly fits. And it wasn't even very difficult.

The technical basis is a call interface abstraction within KLayout, which provides a common interface to scripting languages, called GSI (generic scripting interface). Language bindings use that interface to pass parameters and receive return values. The language implementation is basically just a link between the specific data model and the GSI model. In addition, the GSI model provides hooks for the debugger and documentation.

It turned out, that Python and Ruby are actually very similar regarding their data model. There are some differences (reference-count managed object lifetime vs. mark & sweep garbage collector) and Python in general implements a less uniform object model (heap-allocated classes and built-in classes). But some features are astoundingly similar (like debugger support). I had some trouble with cyclic references which I finally could resolve by using weak references. But now, the Python implementation works very well and debugger support for example is even better than for Ruby. In Python you can switch to a frame up in the stack and evaluate expressions within that context (I have not figured out yet how to change the evaluation context in Ruby to a context up in the stack).

The experiences I made allow some (biased) conclusions of mine. The comparison of languages is always a delicate issue. However I dare to make some points:

• In general, the difference between Python and Ruby is quite small - apart from the syntax. I personally prefer Ruby notation - it's more expressive in my view. That's often labelled "Perlish", but on the contrary, Python is strict in it's notation, so Python could be blamed "Tclish". Whether that is good or bad - I don't care. In the end, both languages provide similar features - a strict object model, duck typing, a huge and comprehensive runtime library and so on.
• Python's syntax is very simple and comprehensive on the other hand. You'll get used to using indents and "pass" (I doubted that to be frank). Pythons syntax highlighter configuration file is only 40% of that of Ruby and the highlighter has less pain to figure out the meaning of the characters. "Simple" probably means making less errors. But being a C++ programmer mainly, simplicity of notation is something of a luxury to me ...
• Ruby's metaprogramming capabilities are more expressed, for example by offering the mixin concept. Python describes some "protocols", but I have not seen something equivalent to the Enumerable module for example. In the end however, it's nothing that can't be done with Python too.
• You can dynamically extend instances with methods in Python too, but at the cost of instantiating bound method types. A little less elegant than just def'ing something on an instance. No need to defined methods on instances? Look at the UI designer sample and you know why that can be useful.
• I liked the way Python offers abstraction of the attribute behaviour through the descriptor concept. Ruby does not provide something similar - there is introspection, but you can't define the attribute behaviour in a way comparable with Python.
• Regular expressions are far easier to use in Ruby. One benefit of being "Perlish".
• Pythons strict (and clean) module hierarchy is a double-edged sword. On one hand, it's very easy locate your code in a hierarchy of modules. On the other hand, you have to employ a dummy module hierarchy if you want to mix C-level extensions with Python code.

There are some more points to discuss, definitely. I wonder whether I receive some replies to these biased comments. There are always language lovers and haters.

I guess Python will be quickly accepted as a script programming language for KLayout. But it will always coexist with Ruby and you can even use Ruby and Python macros or PCell's in parallel.

Ah and finally .. one surprising and somewhat disappointing fact: Python is not faster that Ruby. In fact, it's considerable slower (up to 5x). But I have not ran real benchmarks yet and I'm just using Python 2.7. Maybe 3.x is better.

Matthias

Matthias

Here's a screenshot demonstrating how Python and Ruby coexist in the macro development IDE:

Maks

Matthias -

this is great news. When do you expect to release the full support Klayout Python scripting?

Max

Matthias

Hi Max,

I am currently stabilizing the implementation and I plan to provide a preview version soon. I want to gather feedback from the community and ask for testing. So if you want to join, your support is very much welcome.

Thanks,

Matthias

Matthias

Finally! I provided a preview release for the Python support feature. It required some stabilization and I had some tough time building it on Windows. A full rebuild is now taking a whole night - bug fixing is tedious with such slow turnaround times.

The preview version is not a fully featured 0.24. It's more like 0.23 with Python. Please have a look at the documentation with the download links: http://klayout.de/python_preview.html.

Please give feedback and if you like it ... show your appreciation on the social network.

Enjoy,

Matthias

rellachs

Matthias,

this preview is great and it helps me a lot since I'm not familiar with Ruby. I tried the PCell example and it worked a treat. However, I dearly miss NumPy for any kind of array operations. Doing numerical stuff (like calculating coordinates) in Python without NumPy is just slow, inefficient and cumbersome.

When I try to install Numpy using the binaries from sourceforge it won't let me select a different Python root directory --> would install to the directory of my (generic) Python 2.7 distribution (EPD). Copy-pasting from the "site-packages" folder doesn't work either and is considered very bad practice anyway.

Would it be possible to get a step by step guide on how to install missing packages or even better another preview with NumPy already included?

Thanks in advance!

Matthias

Hi,

the solution depends in the system you are working with:

On Linux: just install the numpy package. Done.

On Windows:

• The evaluation binary is built against Python 3.4.2 with Visual Studio Express 2010. Fortunately that is a pretty common combination, so the next steps will be easier.

• Install Python 3.4.2 with VC2010 on your system from https://www.python.org/downloads/release/python-340

• Now get the NumPy package. The official way to is use "eazy_install numpy". This will try to build the library on your system which requires VC2010 and likely a lot more. I did not follow that approach.

• Instead, I downloaded and installed NumPy 1.9.1 from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy (DISCLAIMER: this site is a self-proclaimed "unofficial" repository. Please decide yourself whether you trust them).

• Now open KLayout and use the following script to point KLayout to this installation

  # Save this script as an "autorun" macro and adjust the path to your 
  # Python 3.4 installation
  import sys
  sys.path.append("c:/Python34/lib")
  sys.path.append("c:/Python34/lib/site-packages")
  

• With these preparations, you could now do "import numpy" to import NumPy. BUT: there is a tiny problem currently: sys.stderr is not initialized and the numpy module needs it. Hence you have to write for now:

  import sys
  sys.stderr = sys.stdout
  import numpy
  

I have tested that approach on Windows 7, 32 bit. I'll gather some feedback for the Python functionality and fix the stderr problem then.

Regards,

Matthias

rellachs

Hi Matthias,

realizing that you used Python 3.4.2 (not 2.7), appending the "site-packages"-folder to "sys.path" plus having stderr refer to stdout did the trick for me (on Windows), but it's still a lot of manual work.

I downloaded the binaries for the current version of NumPy from SourceForge (Python 3.4):
http://sourceforge.net/projects/numpy/files/NumPy/1.9.1/

--> numpy-1.9.1-win32-superpack-python3.4.exe

Then I used 7zip to extract the EXE-file: contains three different versions of NumPy... without SSE, with SSE2, with SSE3 --> select according to CPU and again use 7zip to extract the contents. You get two directories (PLATLIB and SCRIPTS); now copy "numpy" from PLATLIB to the "site-package"-directory and start the script with:

import os, sys
sys.stderr = sys.stdout
if sys.path.count(os.path.dirname(os.__file__)+"\\site-packages") == 0:
    sys.path.append(os.path.dirname(os.__file__)+"\\site-packages")
import numpy
...

It looks like it's working properly, however, my manually installed numpy fails its self-test:

numpy.test('full')

Even though I also managed to install the required package called "nose" from the repository you mentioned, all tests seem to fail with the same error:

AttributeError: 'pya._Channel' object has no attribute 'flush'

Comparing sys.stdout in KLayout (= pya._Channel) to sys.stdout using my generic Python 2.7 by executing

dir(sys.stdout)

... I find that there is a lot of properties missing. Bug or feature?

Regards, rellachs

btw: How do you do the fancy formatting in your posts?

Matthias

Hi rellachs,

This forum supports Markdown formatting - see http://daringfireball.net/projects/markdown/syntax for details about the syntax.

Regarding the issues with Python:

First of all, KLayout integrates Python which is something different from running Python. I cannot refer to an existing installation since that would create undesired installation dependencies. But KLayout uses the same basis than the official distribution, so KLayout can share the path with that installation.

The problem is twofold:

• You somehow have to able to install packages inside the Python 3.4 installation. On Linux you do that with eazy_install for example, but when I tried on Windows, the installer was looking for a MSVC compiler to build the packages and that is when I stopped. How to install Python libraries onto an existing installation on Windows is beyond my experience, so I can't give a lot of guidance. Ideally there is a Python distribution for 3.4.2 which is based on VC 2010 and comes with all the packages. You could install that one, but I am not aware of anyone. I can add some packages to the KLayout installer in the end, but since there is a big choice, I will never be able to satisfy everybody's needs. A way to install external modules is definitely required.

• The other part is KLayout itself. The sys.path trick provides a way to cross-reference packages, provided they are correctly installed with the Python interpreter. The stderr issue will go away in the released version. Regarding the channel classes - yes, KLayout is not supplying a full featured file stream as stdout. I redirect these channels to be able to display the output of Python scripts in the debugger window. The documentation says that one can use every class for stdout or stderr which provides write(), but apparently flush() is expected too (at least by some packages). I'll add that later. As far as I understand it's just the tests which require that, right?

Regards,

Matthias

rellachs

Hey Matthias,

I do understand the difference of KLayout's integrated Python functionality vs. a generic Python installation plus the trick of accessing the site-packages of the generic one from within KLayout. :-)

However, I wanted to point out that NumPy can be manually installed for KLayout's integrated Python by using the binaries from SourceForge following the abovementioned procedure - no separate Python distribution required. Same goes for e.g. the "nose"-package from the repository by Christian Gohlke you mentoined, so I assume a lot of other packages would work, too. But I still had to add the "site-packages" folder of the integrated Python environment to its "sys.path" since it's not in there by default for some reason --> wouldn't find any additional package even though they're located in the right spot. If you check the script-header within my last post, you'll find that I'm deducing the full path for the site-packages using the "os"-package that comes with Python.

Reading my own text I have to admit that it is indeed a bit misleading - sorry about that.

When I think about the variety of Python packages vs. KLayout, I guess a lot of people don't need that many additional packages. You probably want to do stuff like:

• automating / enhancing KLayout's GUI --> "pya" + maybe a GUI package (PyQT, wxPython, ...)
• create full featured layouts --> "pya"
• calculate (arrays of) coordinates for PCells --> NumPy (you don't need the full SciPy-Stack in most cases)

That's pretty much all I can think of right now. Yes, the "nose"-package is only needed for testing. (I wanted to check whether my copy-paste-installation of NumPy worked.)

...and thanks for the link regarding Markdown!

Cheers,

rellachs

Matthias

Hi rellachs,

thanks for the hint about the missing "site-packages". I'll add that to the final version too. I simple forgot that.

My reasoning was that having an external installation allows to check the new packages before cross-referencing them. But you're right, they can be packed with KLayout.

Regarding PyQt there is a real issue, since that creates an additional dependency (on the Qt version) and I have serious doubts how that will interact with the internals of the application (which is based on Qt too). I think that PyQt is intended to add GUI capabilities to a standalone interpreter, not access the internals of an application integrating Python. I was experimenting with qtruby4 which was working fine as long as I created independent components. As soon as I tried to access the internals of the application I ran into trouble and the application got unstable.

This is why I followed a different approach instead and exposed Qt (which I have anyway) to Python through the same mechanism the application API uses. That way the integration is consistent. The price to pay is an API which differs a little from the PyQt API, but I provide a rather complete binding of Qt 4.6 for the QtCore, QtGui, QtXml, QtSql and QtNet modules and partially for the QtDesigner module. The Ruby binding is described in detail here: http://www.klayout.de/doc/programming/qt_binding.html. For Python there is no separate documentation yet, but the translation is following the general rules mentioned in the Python documentation.

Regards,

Matthias

Masaki

Hi,
I want to use print() as a debugging tool for my python script writing.
But I found print() does not work in my windows.( it works well in Linux )

here is a my minimal script.

#I set PYTHONPATH=C:\Python34\lib;C:\Python34\DLLs;C:\Python34\lib\site-packages.

import pya
import os
import numpy as np

ly =pya.Layout()
ly.dbu = 0.00025
scalefactor = int(1/ly.dbu)
names = np.array(["top_cell","second_cell","third_cell"])
print(names,scalefactor) # not works on windows, but works on linux

log = open('logfile','w')
print(names,scalefactor,file=log) # this works both windows/linux
log.flush() #without this, buffer is not flushed before exiting. but it seems a detour.

Does anyone have an advice?

Matthias

Hi,

You mean you don't see any output in your Windows console? That is true since Klayout is compiled as a GUI application. You should see the output in the console window in the macro development IDE however.

Matthias

dion

It's so good. Now, I have to use boost_python to build python binding for klayout.

Masaki

Hi, Matthias,

Thank you, I used klayout from command prompt as klayout -zz -r script.py
I wonder why I thought the IDE is only for ruby... Thank you for great IDE for python users!

Best Regards,
Masaki

Matthias

Hi Masaki,

Sure the IDE supports Python. I won't do things halfway, you know :-)

And it's even a little better for Python - for example, you can access the variables in the local scope. For Ruby I have not found a solution for this yet.

Matthias