So after talking to @padentomasello offline, we've discovered the issue is a bit simpler. There are two distinct issues that affect the default memory manager:

1. A JIT evaluation/memory pressure bug that was introduced when moving JIT heuristics into the memory manager. The new memory pressure measure returns 1.0 if the default JIT eval heuristic is met, and 0.0 otherwise. In the CPU backend, the memoryPressureThreshold (returned by getMemoryPressureThreshold()) defaults to 1.0, and the comparison checks if getMemoryPressure() >= getMemoryPressureThreshold(). In the OpenCL and CUDA backends, however, this check is only getMemoryPressure() > getMemoryPressureThreshold() (to be clear, this is >, not >=), so the JIT eval condition will never be met even if getMemoryPressure(), which means CUDA and OpenCL backends won't call kernel launches, leading to OOMs and odd behavior.
2. The behavior of garbage collection in the default memory manager has changed slightly in 3.7.0. The check for garbage collection in 3.6.4 during allocations was based on, among other things, a comparison between total_buffers and max_buffers. In 3.7.0, this is implicitly calling getMemoryPressure(), which is comparing lock_buffers and max_buffers.

tl;dr — we don't need the ratio idea presented in this PR and can use the existing getMemoryPressure() implementation, but we have to make sure that JIT eval checks are always checking that getMemoryPressure() >= getMemoryPressureThreshold(), and also that garbage collection checks at alloc time in the memory manager are consistent with 3.6.4 behavior.

cc @umar456 @9prady9

This gist should cover everything that needs changing for now.

@jacobkahn @umar456 Thank you for the feedback. I did not initially realize that getMemoryPressure was used outside of the memory manager. I have updated the PR to match @jacobkahn's gist.