Re memory, a 32bit program gets less than 4GB as the video and system buffers have to map into its addressing space.
The ASIO meter only shows peak usage for the disks and the most used core, not memory. It is most likely CPU that is saturated.
If you have a lot of plugins, there may be some that are very resource hungry. Try disabling those until the problems stop, whereupon you will know the culprits.
For recording, you may need to disable the plugins that require the most latency, whereas for mixing, you will probably need to use larger buffers.
Increasing sample rate will require more resources.