Global Placement very slow on some stystem

[titan73]

I use openroad with 2 systems:
- laptop HP EliteBook 840 G9 16 Go ram & intel i5 1245U 1.6 GHz with Ubuntu 22.04 under VM (Virtual Box) and Ubuntu 22.04
- LSF farm on REdHat 8.9 with a custom container based on oss-cas-suite some some other libraries

Between these 2 systems, I just noticed that under the VM on the laptop with Ubuntu the gui is slower.

When I run the implementation flow on these systems I got the followings durations:

Laptop / VM / Ubuntu: 22.04

TIME: Duration of 'global_placement' : 00h00m21s
TIME: Duration of 'clock_tree_synthesis' : 00h00m19s
TIME: Duration of 'global_route' : 00h00m13s
TIME: Duration of 'detailed_route' : 01h49m14s

Redhat 8.9 / Container:

TIME: Duration of 'global_placement' : 01h07m18s
TIME: Duration of 'clock_tree_synthesis' : 00h00m09s
TIME: Duration of 'global_route' : 00h00m07s
TIME: Duration of 'detailed_route' : 00h59m06s

I see for detailed_route that on LSF farm it is twice faster which is expected.
But for global_placement I see that the VM is 180 time faster than LSF on RedHat 8.9 which is huge !!!!

That 's not normal. What could explain such a difference? Is there some acceleration used in global placement that could work only on Ubuntu and not in LSF RedHat?

Something's wrong and I could find where it comes from.

[maliberty]

Do you get identical results in both cases? How much memory and how many cpus on each machine?

[jfshorter]

If the job was submitted in batch mode (non interactive) you should have a corresponding LSF summary print out at the very end of the job log file.
This will tell you how much memory the job claimed to have used and possibly the total host swap utilisation, but nothing else.

Worth verifying the job set up and tool versions fairly match between your local and server runs:

• Ensuring your local and the LSF job have matching design, constraints and tool builds/versions, plus disc quota.
  • Some enterprise NAS/storage filer solutions implement throttling.
  • Typically applies when your project quota or disc fill level is above 80% to protect hitting criticality too quickly.
• Double check the LSF bsub switches carefully:
  • Specify a maximum RAM amount using -M X (hard limit) and threads -n Y -R "span[hosts=1]" ensuring the commands are configured to only use that number of cores and constrain to a single hosts.
  • An unexplained aspect of LSF is that if you are using resource request conditions strings for memory requirements (e.g: -R "usage[mem > X]")
  • By default, rusage memory specification will too often allow your job to be dispatched directly in to swap, especially on a host that is already experiencing significant Out-Of-Memory (OOM) scheduling issues but not compute bound.
  • Also worth specifying as XGB if not already, better to slightly oversize and increase queue time than it spending most of its time OOM/awaiting disc.
  • Setting a hard limit will also protect your work from causing overspend and affecting others' jobs at the expense of partial/lost runs until you know the stable requirements.

Assuming the above aspects are not the issue, check host load and RAM/swap utilisation on your next run:

• Think about adding basic time {cmd} checks on any other steps you are running in your flow scripts to construct a crude gantt time chart.
• SSH in to the LSF host once the job is dispatched checking bjobs -w
• Use htop to verify the host is not over-subscribed + swap is not being used. Enable the OOM column to sort processes if this helps.
• A quick way to filter just for your processes is to relaunch with htop -u $USER
• You can also use uptime to get a crude measurement of the CPU thread utilisation in the past: 1m, 5m and 15m
• Check this against the number of cores & threads the host CPU has:
  • Get the CPU model name, cores count and thread count:
    • grep -m 3 -P '(model name|cpu cores)' /proc/cpuinfo = Model name and number of physical cores (excl threads)
    • ncpu = ThreadCount

If you happen to also run Intel based servers, I wish you good luck.
From experience treat any jobs running on architectures between Skylake - Ice Lake era micro-architectures (~Core ix 6xxxY, 2016 - Xeon Platinum 83xxY, 2019) and Sapphire Rapids:

• ThreadUtilisation ~= CPUCores = Busy host that will being noticeably slower than unloaded
• ThreadUtilisation > 2+(ThreadCount*ECC) = Thread exhaustion/oversubscribed, where ECC = effective core count: EC ~ 0.5 - 0.7

Appreciate there are several aspects I touched on here.
However, it helps for the process of elimination and to understand the true bottlenecks/pain points in any regression runs.

[titan73]

All figures are identical in both logs so yes.

On the VM: 1 cpu and 16 Go of ram

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 154
model name	: 12th Gen Intel(R) Core(TM) i5-1245U
stepping	: 4
cpu MHz		: 2496.002
cache size	: 12288 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 avx2 bmi2 invpcid rdseed clflushopt arat md_clear flush_l1d arch_capabilities
bugs		: apic_c1e spectre_v1 spectre_v2 spec_store_bypass swapgs bhi
bogomips	: 4992.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

On LSF I requested 4 cpus and 4 Go or ram (bsub -n 4 -R "rusage[mem=4000] span[hosts=1]") but the machine has 64 cpus and around 400 Go of available mem.

cpuinfo_lsf.txt

Global route does not use much memory (216 Mb).

START_TIME      FINISH_TIME     RUN_TIME    CPU_USED    MAX_MEM        
Aug 18 12:53    Aug 18 13:37    00:43:39    02:50:50    216 Mbytes

[maliberty]

Global placement does use threads so that could explain the difference. Its possible your smaller machine is thrashing due to insufficient memory.

[titan73]

I was also thinking about a swap issue due to lack of physical memory but I requested 4 Go which is a little (iirc the default minimum is around 40 Go and the farm does not complain we uses more than requested as long as it's below this 40 Go limit).
The LSF farm report only 216 Mbytes of max memory use when running the openroad script until the global placement.
Accord to /proc/memfinfo it has hundred of Go of physical memory.
So I don't know what's going on. Could it be some threads synchronization issue?

[titan73]

I've tested by requesting explicitly 40 Go without improvement.
Is there an option to disable multi-threading?

[titan73]

Tested:

set_thread_count 1
global_placement

with no change.

[maliberty]

You could run vtune on it to get a runtime profile. At this point its all guess work.

[titan73]

@maliberty How can I use vtune with OR? Is there a documentation available?

[maliberty]

vtune has its own docs. Usually you have to source something like /opt/intel/oneapi/vtune/latest/env/vars.sh and then run vtune -collect hotspots openroad .... to collect the data. vtune-gui to inspect the results.

The only thing OR specific is that drt will add some extra markers to the profile if you link with vtune. See the drt/CMakeLists.txt

[titan73]

Thanks for the info.
Actually I don't have the slowness issue with OR compiled on RHEL8.