Question

My question is, perhaps, a poorly worded one and stems from my amateurish understanding of memory management.

My concern is this: I have a Perl script that forks many times. As I understand from the fork page in perldoc, copy-on-write is being implemented. Each of the children then calls system(), forking again, to call an external program. The data from the external program is read back into the child, and dumped as a Storable file to be reaped and processed by the parent once all children have exited.

What concerns me is my perceived volatility of this situation. Consider, what I see in my mind, the worst case scenario: For each of the children, as soon as new data arrives, the entire copy-on-write memory becomes, well, copied. If this is the case, I am going to quickly run into memory problems after creating a few forks.

But alternatively, does copy-on-write only copy the smallest chunk of memory that contains the needed data? Then what is this quanta of memory? How is its size set?

I am uncertain as to whether the specifics of what I am asking are language dependent or dependent on some lower-level process.

Answer 1

Memory is organized in pages, generally 4K each (this can be set to different values, and is hardware-specific, but that s the norm on Intel platforms with standard operating systems). When a child process writes into a copy-on-write page, it will be copied.

Answer 2

Yes, forking will increase your memory footprint. If this is a problem, use a module like Parallel::ProcManager or Forks::Super that can throttle the number of active background processes. Limiting the number of active forks is also a good idea when your processes are CPU bound, I/O bound, or have the potential to overuse any other limited resource on your machine.

use Forks::Super MAX_PROC => 10, ON_BUSY => block;

...
$pid = fork();        # blocks if there are already 10 child processes
...                   # unblocks when one of the children finishes

友情链接