ShareThis
You’ve probably heard the managers asking questions like; how much free memory does the system have? Why is the amount of free memory so low? Where did the entire RAM go?
This article will provide you the right answers that will make the managers and customers smile, and understand the Linux memory consumption concept.
The standard Linux utilities for memory consumption report are “free” and “top”. The “free” utility displays the amount of free and used RAM, and “top” utility also provides a per-process memory usage breakdown. These utilities may often report that the free amount of memory left in the system is rather lower than you would expect, especially after the system was up and running for a while. You may conclude that there are problems related to memory leaks, or that your available RAM is being used inefficiently or even that your system does not have enough RAM to run correctly. Whilst these issues could be real problems with your system, it doesn’t always have to be case.
Linux memory consumption concept
Linux memory consumption concept is all about efficiency. The system’s RAM is a resource that is meant to be used; 100% of it (if possible), all the time (if possible).
Linux utilizes unused RAM to cache data and filesystem meta-data from slower storage devices (Flash or disk) because fetching the information from the RAM is much quicker: There are no bottlenecks such as slow physical media, slow buses or device clocks, and not decompression is required.
Assuming there are no memory leaks, the reason that memory report tools report low amount of free memory is because the RAM is considered to be wasted if it isn’t used. This concept may require some time to digest, because conventional thinking may lead to the conclusion that an efficient system is a system with a lot of free memory. This is not entirely correct. In Linux case, the kernel tries to utilize the most of the RAM to improve the system performance. Keeping the cache means that if the kernel or a task needs the same data again, there’s a good chance it will still be in the fast cache in memory.
Getting the amount of free memory
As mentioned before, there are two standard utilities that report the amount of free memory. For RT Embedded projects, these utilities are provided by “Busybox”, and their output might be slightly different than standard distributions due to memory consumption restraints. Here’s an example for the output of “free” utility:
# free |
The numbers shown are in Kilo-Bytes, meaning that this system has total memory of about 120MB, 41MB is used and 77MB is free. The buffers column displays the amount of memory that was used for filesystem meta-data (such as the filesystem tree information, file location information, etc.). System with hard disks or flash drives may also have swap memory. This snapshot was taken from a system with a single flash without a disk, and therefore, the total swap memory is 0.
Here’s an example of the output from “top”, taken from Fedora distribution:
top - 15:58:22 up 21 min, 2 users, load average: 0.00, 0.03, 0.15 |
The output shows similar memory reports fields as the “free” utility as well as per-proces detailed information. For the purpose of this article, the following columns are relevant:
- VIRT: Virtual Image (KB). The total amount of virtual memory used by the task. It includes code, data and shared libraries as well as pages that have been swapped out.
- RES: Resident size (KB). The non-swapped physical memory a task has used.
- %MEM: Memory usage (RES). A task’s currently used share of available RAM.
The detailed output of “top” may interest you when you are debugging memory leaks. In this case, the leaky task Resident size (and %MEM) will rise over time. Otherwise, the output of “free” is sufficient.
Getting more information from the proc filesystem
The kernel provides a special proc file called meminfo, which displays the memory consumption figures in details. The meminfo file may differ between distributions, especially between Embedded Linux and Desktop Linux. Here’s a typical output of this proc file, from the same system that was used for the “free” snapshot:
# cat /proc/meminfo |
The information fields which are available in meminfo proc file:
- MemTotal – Total amount of physical RAM, in kilobytes.
- MemFree – The amount of physical RAM, in kilobytes, left unused by the system.
- Buffers – The amount of physical RAM, in kilobytes, used for file buffers.
- Cached – The amount of physical RAM, in kilobytes, used as cache memory.
- Active – The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
- Inactive – The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
- LowTotal and LowFree – The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
- Mapped – The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
- Slab – The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
- Committed_AS – The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory.
- PageTables – The total amount of memory, in kilobytes, dedicated to the lowest page table level.
- VMallocTotal – The total amount of memory, in kilobytes, of total allocated virtual address space.
- VMallocUsed – The total amount of memory, in kilobytes, of used virtual address space.
- VMallocChunk – The largest contiguous block of memory, in kilobytes, of available virtual address space.
Getting the absolute free memory number
Your manager wants to know how much free memory the system has. What he really wants to know actually is the minimum amount of memory that the system requires in order to function properly. Let’s call this number “The absolute free memory number”. Given this number and the amount of actual installed memory, the manager also wants to know how much RAM is left for the customers’ customizations, or whether a smaller RAM device can be used in order to reduce the hardware cost (or even both…).
The absolute free memory number is not only the value of MemFree, but the sum of the MemFree, Buffers and Cached fields. It means that the majority of the used memory by Buffers and Cached can be reclaimed in case a task will require it, while reducing the amount of buffers and cached. Using unused (clean) cache will not impact the system performance. However, degradation in the system performance will appear once starting to free used cache. In some point (usually in the last free 1MB, will be discussed soon) the degradation will be very noticeable and the system will become very slow. It happens because live tasks are swapped out of the RAM, and execution of commands requires flash/disk access and decompression. If even more RAM is required, the kernel will initiate the Out-Of-Memory Killer and start killing tasks that have the highest “bad” score (See the article about Linux processes for more details). In this stage, the system is not in a stable state and its functionality is not guaranteed. If your system reaches this state on a regular basis, you may either have a serious memory leak, or the actual physical RAM installed is insufficient.
Flush and invalidate system caches
It is possible to artificially increase the value of MemFree (and make your manager happy) by flushing and invalidating the clean (unused) cache and filesystem meta-data. This operation is non-destructive and it does not free the currently used caches (like running processes). Prior to this operation, the “sync” command needs to be run first in order to make sure all cached objects are synchronized.
- To free pagecache:
– echo 1 > /proc/sys/vm/drop_caches - To free dentries and inodes:
– echo 2 > /proc/sys/vm/drop_caches - To free pagecache, dentries and inodes (basically everything):
– echo 3 > /proc/sys/vm/drop_caches
Here is the output of the same meminfo file, before and after all the caches were freed:
| Memory status before dropping caches | Memory status after dropping caches |
MemTotal: 119248 kB |
MemTotal: 119248 kB |
We can see that we gained almost 8MB in of RAM in MemFree field (and also in the “free” report) just by dropping the caches, and that the amount of Buffers and Cached has been reduced.
Other kernel tunables regarding memory
The Kernel provides run-time tunables that can change its behavior in extreme cases:
- /proc/sys/vm/min_free_kbytes: Used to force the Linux VM to keep a minimum number of kilobytes free. This number’s value is around 1MB. The Kernel will start swapping out and killing processes in order to meet this requirement.
- /proc/sys/vm/overcommit_memory: Controls over-commit of system memory, possibly allowing processes to allocate (but not use) more memory than is actually available.
Resources:
http://www.linuxinsight.com/proc_sys_vm_hierarchy.html
https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/en-US/Reference_Guide/s2-proc-meminfo.html
http://wiki.linuxquestions.org/wiki/FAQ_-_Linux_problems
http://linux.die.net/man/1/top
http://linux.die.net/man/1/free
| Check out the ads, there could be something that may interest you there. The ads revenue helps me to pay for the domain and storage. |
It’s posts like this that keep me coming back and checking this site regularly, thanks for the info!
Hello Hai!
We include Slab too to calculate Total Free Memory. Don’t we ?
Actually, I just faced the problem you mentioned here – Out of Memory where the Out-Of-Memory Killer started killing processes and finally reboots my system
Do you think we could modify /proc/sys/vm/min_free_kbytes a to smaller value ?
Thanks,
Uma
It depends. If there is a memory leak somewhere, it will just delay the end. Otherwise, you may lower the value only if there are no other choices (i.e. unload unneeded applications/drivers, optimize memory usage). You may also need to consider installing a larger RAM device in the next generation of your system.
Thank you!
Hi
I’m execute echo 1 > /proc/sys/vm/drop_caches to free pagecache
After execute that command, my server is hung and must be restart. Please explain me how that can be occur ?
Hello Defry, please use sync command before dropping the caches.
Hello,
I think , In some circumstances, we should also care about Mapped.
Because when some process calls mmap with MAP_LOCKED, that memory is not available from other processes, but counted in Cached.
How do you think?
Here is how to test:
### Creating 1GB file ‘foo’ with dd ###
dd if=/dev/zero of=foo bs=1M count=1024
### save this as mmap.c ###
#include
#include
#include
#include
#include
#include
#include
#include
char* p;
char* p2;
int size;
static void cleanup( int nr )
{
printf( “sig\n” );
munmap( p2, size );
}
int main( int argc, char* argv[] )
{
int fd;
int i;
fd = open( argv[ 1 ], O_RDWR );
signal(SIGHUP, cleanup);
signal(SIGINT, cleanup);
signal(SIGTERM, cleanup);
signal(SIGSTOP, cleanup);
printf( “mapping %s … please wait.\n”, argv[ 1 ] );
size = 1024 * 1024 * 1024;
/*p = mmap( NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0 );*/
p = mmap( NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, fd, 0 );
p2=p;
for( i = 0; i < size; ++i )
{
*p = 0×61;
++p;
}
printf( "mapped\n" );
sleep( 3000 );
return 0;
}
### compile ###
gcc mmap.c -o mmap
### run ###
./mmap foo
check /proc/meminfo from another console.
Thanks.
Sorry, source code is here: http://www.jumperz.net/tools/mmap.c.txt
Save this as “mmap.c”.
Memory mapped areas need to be considered in some cases, like the example you provided. However, in most cases, this is not the common use. The memory mapped areas are usually used to map common hardware areas or shared memory segments. Each process that maps the same memory area will report it in his map file. If you’ll add all the reported shared memory segments, you might end up with a wrong number because there is only one physical area.
Thank you for your reply.
>However, in most cases, this is not the common use
I agree.