Unable to open a huge OASIS file

Comments

• Matthias

  January 2010 edited November -1

  Hi,

  the message basically says that the process went out of memory. I assume that you are using a 32bit executable which is limited to allocate a maximum of 4G memory. Please try to build a 64bit version. This should not exhibit this problem. I guess you will need a system with around 16G physical memory.

  If you already have a 64bit executable, please check if on your system a limit for the maximum process size is set ("limits -a").

  Best regards,

  Matthias
• phlion

  January 2010 edited November -1

  Hi Matthias,
  
  You are right, although we are running 64bit version of klayout, however
  we forgot to check the maximum memory limit of process, really appreciate
  for your hint and kindly support, thanks.
  
  Best Regards,
  --
  chi-hsiang-hung
• rrzzxx22

  November 2023 edited November 2023

  @Matthias is there a simple way on Windows 11 to do set unlimited maximum process size?
  I don't understand where to set the argument "limits -a" is it a linux only thing?

  I'm trying to open a giant oasis file, it won't work on 64 bit in editor. System has 100GB of ram it gets to about 80GB consumed then throws error "std::bad_alloc"

  The loading bar for "Reading OASIS file" stops each time at 3390 MB exactly, but memory usage continues to climb in the background. The Oasis file I'm trying to open is 3,477,014KB in size.

  testing it in the view only app.
  klayout_vo_app.exe

  In the view only mode it holds steady and makes progress at ~69GB of memory consumed, while slowly making progress reading the OASIS file. eventually it still crashes out though with the same error.
• Matthias

  November 2023 edited November 2023

  I don't think so. I guess that basically there is no limit on Windows. I understand that Windows kills a process when it goes to far. Maybe 80GB is the limit the system allows for every process under the given memory conditions.

  I'd recommend Linux anyway for large files. After all, Windows is a system made for running Office.

  How big is your original OASIS? Editor mode rules out, viewer mode may do, but expect the memory requirement to be something like two to three times the file size (uncompressed, without CBLOCKs).

  So big files won't do. You'll need other tools for that - those which can do read-on-demand and virtual memory management. This is the domain of commercial high-end tools.

  Matthias
• rrzzxx22

  November 2023

  @Matthias Thanks as always for taking the time to reply.

  The original OASIS file is 3.477GB or 3,477,014KB exactly.
  The PC has 100GB of ram.

  Does Linux natively do virtual memory management?

  I thought when a process needed more memory it just started requisitioning hard drive space and going really slow.

  I can't even open the giant file with the klayout_vo_app.exe (I'm assuming this is how you use "view only" mode?), it has the same crashing behavior as the editor.

  I found a smaller version of the files I need as separate cells instead of the giant master composite, so I can work with those, but I was hoping to see what's inside the giant file.
• Matthias

  November 2023

  Hi @rrzzxx22,

  That isn't really giant, but maybe it contains CBLOCKs. These are are compressed file sections. The real size could be much larger in that case. It's difficult to say what is going on without having the file at hand. Maybe you have some border case that triggers a particular issue, but again, I can't say what is going on.

  Linux does virtual memory management, but you need to have enough swap space. And yes, it will become slow, but there are chances it is still manageable.

  There are a few things you can try: under "File/Reader Options" on the "GDS+OASIS" tab you can configure that you only want to read certain layers. If the "layers or layer mapping" table is empty and you turn off "Read all layers", you will basically skip all geometrical content. You will not see any shapes, but the cell hierarchy. Maybe that already gives you an idea what is inside the file.

  There is a low-level serial OASIS dump tool that may also give some insights here: https://github.com/klayoutmatthias/dump_oas_gds2. But you need to be able to build it from sources. Linux is handy then.
  If you follow the dump, you can find out which layers are contained in the file, and whether there are particular features like CBLOCKs with a high compression ratio.

  Matthias