我如何让我的Fortran程序使用一定数量的RAM？(How can I get my Fortran program to use a certain amount of RAM?)

我如何让我的Fortran程序使用一定数量的RAM？(How can I get my Fortran program to use a certain amount of RAM?)

我正在尝试写一个Fortran程序，它会消耗大量的内存（因为背后的原因，请参阅本问题末尾的注释）。 我通过分配一个大小为3维的数组(n,n,n)然后解除分配它 - 继续增加n直到内存不足（这应该在使用〜16 GB内存时发生）。 不幸的是，在我看到系统资源高达16 GB之前，似乎我的程序耗尽内存。

这是我的示例代码：

1 program fill_mem 2 implicit none 3 integer, parameter :: ikind = selected_int_kind(8) 4 integer, parameter :: rkind = 8 5 6 integer(kind = ikind) :: nfiles = 100 7 integer(kind = ikind) :: n = 1200 8 integer(kind = ikind) :: i, nn 9 10 real(kind = rkind), allocatable :: real_arr(:,:,:) 11 12 character(500) :: sysline 13 14 15 call system('echo ''***no_allocation***'' > outfile') 16 call system('ps aux | grep fill_mem.exe >> outfile') 17 !call system('smem | grep fill_mem.exe >> sm.out') 18 allocate(real_arr(n, n, n)) 19 20 nn = 100000 21 do i = 1,nn 22 deallocate(real_arr) 23 n = n + 10 24 print*, 'n = ', n 25 allocate(real_arr(n, n, n)) 26 call system('echo ''*************'' >> outfile') 27 write(sysline, *) 'allocation', i, '... n = ', n 28 29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%' 30 31 call system(trim(adjustl('echo '//sysline//'>> outfile'))) 32 call system('ps aux | grep fill_mem.exe >> outfile') 33 enddo 34 35 end program fill_mem

这里是示例输出：

1 ***no_allocation*** 2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe 3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 5 ************* 6 allocation 1 ... n = 1210 7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe 8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 10 ************* 11 allocation 2 ... n = 1220 12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe 13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 15 ************* 16 allocation 3 ... n = 1230 17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 20 ************* 21 allocation 4 ... n = 1240 22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 25 ************* 26 allocation 5 ... n = 1250 27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe 30 ************* 31 allocation 6 ... n = 1260 32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 35 ************* 36 allocation 7 ... n = 1270 37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe

现在，我看到VSZ部分达到了〜15 GB，所以我假设当我试图解决更多问题时，它会失败

Operating system error: Cannot allocate memory Allocation would exceed memory limit

因为没有那么多的RAM。 尽管RSS为什么远远低于这个水平？ 当我真正查看我的系统资源时，我发现大约有140 MB用完了（我正在Linux虚拟机中运行此操作系统，并通过Windows监视系统资源 - 尽管如此，我还是给了GM 16 GB的RAM，所以我应该看看虚拟机内存不断增加，直到达到16 GB的标记为止，虚拟机具有VT-x /嵌套分页/ PAE / NX，因此它应该像使用本机操作系统一样使用物理体系结构）。

任何人都可以解释为什么我没有看到我的程序实际使用完整的16 GB RAM，以及如何编写我的代码来保存这些我在RAM中创建的阵列 - 完全利用我的可用硬件？

注意：我试图编写一个读取大量内存的示例程序的原因是我正在处理的数据占用了大约14 GB的ascii文本空间。 在本程序的整个过程中，我将需要处理大量数据，因此我希望一次性读取所有数据，然后在整个程序期间从RAM中引用它。 为了确保我正确地做到了这一点，我试图编写一个简单的程序，它将一次存储一个非常大的数组（〜15 GB）。

I am trying to write a Fortran program which will eat up a lot of memory (for the reasoning behind this, please see the note at the end of this question). I am doing this by allocating a 3 dimensional array of size (n,n,n) and then deallocating it - continually increasing n until I run out of memory (this should happen when ~16 GB of memory is used). Unfortunately, it seems as if my program is running out of memory long before I see the system resources get up to 16 GB.

Here is my sample code:

1 program fill_mem 2 implicit none 3 integer, parameter :: ikind = selected_int_kind(8) 4 integer, parameter :: rkind = 8 5 6 integer(kind = ikind) :: nfiles = 100 7 integer(kind = ikind) :: n = 1200 8 integer(kind = ikind) :: i, nn 9 10 real(kind = rkind), allocatable :: real_arr(:,:,:) 11 12 character(500) :: sysline 13 14 15 call system('echo ''***no_allocation***'' > outfile') 16 call system('ps aux | grep fill_mem.exe >> outfile') 17 !call system('smem | grep fill_mem.exe >> sm.out') 18 allocate(real_arr(n, n, n)) 19 20 nn = 100000 21 do i = 1,nn 22 deallocate(real_arr) 23 n = n + 10 24 print*, 'n = ', n 25 allocate(real_arr(n, n, n)) 26 call system('echo ''*************'' >> outfile') 27 write(sysline, *) 'allocation', i, '... n = ', n 28 29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%' 30 31 call system(trim(adjustl('echo '//sysline//'>> outfile'))) 32 call system('ps aux | grep fill_mem.exe >> outfile') 33 enddo 34 35 end program fill_mem

and here is the sample output:

1 ***no_allocation*** 2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe 3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 5 ************* 6 allocation 1 ... n = 1210 7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe 8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 10 ************* 11 allocation 2 ... n = 1220 12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe 13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 15 ************* 16 allocation 3 ... n = 1230 17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 20 ************* 21 allocation 4 ... n = 1240 22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 25 ************* 26 allocation 5 ... n = 1250 27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe 30 ************* 31 allocation 6 ... n = 1260 32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 35 ************* 36 allocation 7 ... n = 1270 37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe

Now, I see that the VSZ portion gets up to ~15 GB so I am assuming when I try to address more, it fails with

Operating system error: Cannot allocate memory Allocation would exceed memory limit

because there is not that much RAM. Why is it that RSS is so far below that, though? When I actually look on my system resources I see about 140 MB being used up (I am running this in a Linux VM and monitoring the system resources through Windows - I have given the GM 16 GB of RAM to use though, so I should see the VM memory increasing until it reaches the 16 GB mark - for what it's worth, the VM has VT-x/Nested Paging/PAE/NX so it should use the physical architecture just like the native OS).

Can anyone explain why I do not see my program actually using up the full 16 GB of RAM and how I can write my code to keep these arrays I am creating in RAM - fully utilizing my available hardware?

NOTE: The reason I am trying to write a sample program which reads a lot of memory is that I am working with data which takes up around 14 GB of space in ascii text. I will need to be working with data A LOT throughout the course of this program, so I want to read it all in at once and then reference it from RAM throughout the duration of the program. To make sure I am doing this correctly, I am trying to write a simple program which will store a very large array (~15 GB) in memory all at once.

最满意答案

（注意：Fortran标准没有说明应该如何实现这样的事情，下面的描述指的是如何在当前操作系统上实现Fortran编译器。）

当你执行一个ALLOCATE语句（或者等价地，在C，FWIW中调用malloc（）时），实际上并不是预留物理内存，而只是为你的进程映射地址空间。 这就是VSZ上升的原因，但不是RSS。 实际上，仅当您第一次访问内存时（通常以页面大小粒度，即大多数当前hw上的4 KB），才会为您的进程预留物理内存。 所以只有当你开始把一些数据放到你的数组里时，RSS才会开始攀升。 例如像一个声明

real_arr = 42.

应该把你的RSS提升到VSZ的附近。

(Caveat: The Fortran standard doesn't say how such thing ought to be implemented etc., the description below refers to how Fortran compilers are typically implemented on current operating systems.)

When you execute an ALLOCATE statement (or equivalently, calling malloc() in C, FWIW), you're not actually reserving physical memory, but only mapping address space for your process. That's why the VSZ goes up, but not the RSS. Actually reserving physical memory for your process happens only when you first access the memory (typically at page size granularity, that is, 4 KB on most current hw). So only once you start putting some data into your array does the RSS begin to climb. E.g. a statement like

real_arr = 42.

ought to bump up your RSS to the vicinity of the VSZ.

我如何让我的Fortran程序使用一定数量的RAM？(How can I get my Fortran program to use a certain amount of RAM?)

我正在尝试写一个Fortran程序，它会消耗大量的内存（因为背后的原因，请参阅本问题末尾的注释）。 我通过分配一个大小为3维的数组(n,n,n)然后解除分配它 - 继续增加n直到内存不足（这应该在使用〜16 GB内存时发生）。 不幸的是，在我看到系统资源高达16 GB之前，似乎我的程序耗尽内存。

这是我的示例代码：

1 program fill_mem 2 implicit none 3 integer, parameter :: ikind = selected_int_kind(8) 4 integer, parameter :: rkind = 8 5 6 integer(kind = ikind) :: nfiles = 100 7 integer(kind = ikind) :: n = 1200 8 integer(kind = ikind) :: i, nn 9 10 real(kind = rkind), allocatable :: real_arr(:,:,:) 11 12 character(500) :: sysline 13 14 15 call system('echo ''***no_allocation***'' > outfile') 16 call system('ps aux | grep fill_mem.exe >> outfile') 17 !call system('smem | grep fill_mem.exe >> sm.out') 18 allocate(real_arr(n, n, n)) 19 20 nn = 100000 21 do i = 1,nn 22 deallocate(real_arr) 23 n = n + 10 24 print*, 'n = ', n 25 allocate(real_arr(n, n, n)) 26 call system('echo ''*************'' >> outfile') 27 write(sysline, *) 'allocation', i, '... n = ', n 28 29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%' 30 31 call system(trim(adjustl('echo '//sysline//'>> outfile'))) 32 call system('ps aux | grep fill_mem.exe >> outfile') 33 enddo 34 35 end program fill_mem

这里是示例输出：

1 ***no_allocation*** 2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe 3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 5 ************* 6 allocation 1 ... n = 1210 7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe 8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 10 ************* 11 allocation 2 ... n = 1220 12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe 13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 15 ************* 16 allocation 3 ... n = 1230 17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 20 ************* 21 allocation 4 ... n = 1240 22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 25 ************* 26 allocation 5 ... n = 1250 27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe 30 ************* 31 allocation 6 ... n = 1260 32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 35 ************* 36 allocation 7 ... n = 1270 37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe

现在，我看到VSZ部分达到了〜15 GB，所以我假设当我试图解决更多问题时，它会失败

Operating system error: Cannot allocate memory Allocation would exceed memory limit

因为没有那么多的RAM。 尽管RSS为什么远远低于这个水平？ 当我真正查看我的系统资源时，我发现大约有140 MB用完了（我正在Linux虚拟机中运行此操作系统，并通过Windows监视系统资源 - 尽管如此，我还是给了GM 16 GB的RAM，所以我应该看看虚拟机内存不断增加，直到达到16 GB的标记为止，虚拟机具有VT-x /嵌套分页/ PAE / NX，因此它应该像使用本机操作系统一样使用物理体系结构）。

任何人都可以解释为什么我没有看到我的程序实际使用完整的16 GB RAM，以及如何编写我的代码来保存这些我在RAM中创建的阵列 - 完全利用我的可用硬件？

注意：我试图编写一个读取大量内存的示例程序的原因是我正在处理的数据占用了大约14 GB的ascii文本空间。 在本程序的整个过程中，我将需要处理大量数据，因此我希望一次性读取所有数据，然后在整个程序期间从RAM中引用它。 为了确保我正确地做到了这一点，我试图编写一个简单的程序，它将一次存储一个非常大的数组（〜15 GB）。

I am trying to write a Fortran program which will eat up a lot of memory (for the reasoning behind this, please see the note at the end of this question). I am doing this by allocating a 3 dimensional array of size (n,n,n) and then deallocating it - continually increasing n until I run out of memory (this should happen when ~16 GB of memory is used). Unfortunately, it seems as if my program is running out of memory long before I see the system resources get up to 16 GB.

Here is my sample code:

1 program fill_mem 2 implicit none 3 integer, parameter :: ikind = selected_int_kind(8) 4 integer, parameter :: rkind = 8 5 6 integer(kind = ikind) :: nfiles = 100 7 integer(kind = ikind) :: n = 1200 8 integer(kind = ikind) :: i, nn 9 10 real(kind = rkind), allocatable :: real_arr(:,:,:) 11 12 character(500) :: sysline 13 14 15 call system('echo ''***no_allocation***'' > outfile') 16 call system('ps aux | grep fill_mem.exe >> outfile') 17 !call system('smem | grep fill_mem.exe >> sm.out') 18 allocate(real_arr(n, n, n)) 19 20 nn = 100000 21 do i = 1,nn 22 deallocate(real_arr) 23 n = n + 10 24 print*, 'n = ', n 25 allocate(real_arr(n, n, n)) 26 call system('echo ''*************'' >> outfile') 27 write(sysline, *) 'allocation', i, '... n = ', n 28 29 write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%' 30 31 call system(trim(adjustl('echo '//sysline//'>> outfile'))) 32 call system('ps aux | grep fill_mem.exe >> outfile') 33 enddo 34 35 end program fill_mem

and here is the sample output:

1 ***no_allocation*** 2 1000 12350 0.0 0.0 12780 760 pts/1 S+ 13:32 0:00 ./fill_mem.exe 3 1000 12352 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 4 1000 12354 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 5 ************* 6 allocation 1 ... n = 1210 7 1000 12350 0.0 0.0 13853104 796 pts/1 S+ 13:32 0:00 ./fill_mem.exe 8 1000 12357 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 9 1000 12359 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 10 ************* 11 allocation 2 ... n = 1220 12 1000 12350 0.0 0.0 14199096 952 pts/1 S+ 13:32 0:00 ./fill_mem.exe 13 1000 12362 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 14 1000 12364 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 15 ************* 16 allocation 3 ... n = 1230 17 1000 12350 0.0 0.0 14550804 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 18 1000 12367 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 19 1000 12369 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 20 ************* 21 allocation 4 ... n = 1240 22 1000 12350 0.0 0.0 14908284 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 23 1000 12372 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 24 1000 12374 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 25 ************* 26 allocation 5 ... n = 1250 27 1000 12350 0.0 0.0 15271572 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 28 1000 12377 0.0 0.0 4400 612 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 29 1000 12379 0.0 0.0 9384 916 pts/1 S+ 13:32 0:00 grep fill_mem.exe 30 ************* 31 allocation 6 ... n = 1260 32 1000 12350 0.0 0.0 15640720 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 33 1000 12382 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 34 1000 12384 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe 35 ************* 36 allocation 7 ... n = 1270 37 1000 12350 0.0 0.0 16015776 956 pts/1 S+ 13:32 0:00 ./fill_mem.exe 38 1000 12387 0.0 0.0 4400 616 pts/1 S+ 13:32 0:00 sh -c ps aux | grep fill_mem.exe >> outfile 39 1000 12389 0.0 0.0 9384 920 pts/1 S+ 13:32 0:00 grep fill_mem.exe

Now, I see that the VSZ portion gets up to ~15 GB so I am assuming when I try to address more, it fails with

Operating system error: Cannot allocate memory Allocation would exceed memory limit

because there is not that much RAM. Why is it that RSS is so far below that, though? When I actually look on my system resources I see about 140 MB being used up (I am running this in a Linux VM and monitoring the system resources through Windows - I have given the GM 16 GB of RAM to use though, so I should see the VM memory increasing until it reaches the 16 GB mark - for what it's worth, the VM has VT-x/Nested Paging/PAE/NX so it should use the physical architecture just like the native OS).

Can anyone explain why I do not see my program actually using up the full 16 GB of RAM and how I can write my code to keep these arrays I am creating in RAM - fully utilizing my available hardware?

NOTE: The reason I am trying to write a sample program which reads a lot of memory is that I am working with data which takes up around 14 GB of space in ascii text. I will need to be working with data A LOT throughout the course of this program, so I want to read it all in at once and then reference it from RAM throughout the duration of the program. To make sure I am doing this correctly, I am trying to write a simple program which will store a very large array (~15 GB) in memory all at once.

最满意答案

（注意：Fortran标准没有说明应该如何实现这样的事情，下面的描述指的是如何在当前操作系统上实现Fortran编译器。）

当你执行一个ALLOCATE语句（或者等价地，在C，FWIW中调用malloc（）时），实际上并不是预留物理内存，而只是为你的进程映射地址空间。 这就是VSZ上升的原因，但不是RSS。 实际上，仅当您第一次访问内存时（通常以页面大小粒度，即大多数当前hw上的4 KB），才会为您的进程预留物理内存。 所以只有当你开始把一些数据放到你的数组里时，RSS才会开始攀升。 例如像一个声明

real_arr = 42.

应该把你的RSS提升到VSZ的附近。

(Caveat: The Fortran standard doesn't say how such thing ought to be implemented etc., the description below refers to how Fortran compilers are typically implemented on current operating systems.)

When you execute an ALLOCATE statement (or equivalently, calling malloc() in C, FWIW), you're not actually reserving physical memory, but only mapping address space for your process. That's why the VSZ goes up, but not the RSS. Actually reserving physical memory for your process happens only when you first access the memory (typically at page size granularity, that is, 4 KB on most current hw). So only once you start putting some data into your array does the RSS begin to climb. E.g. a statement like

real_arr = 42.

ought to bump up your RSS to the vicinity of the VSZ.