The ARC cache is similar to the buffer cache, but just dedecated to ZFS, so there is generally nothing to worry about it.

On ZOL 0.6.3(didn't check other version yet), it's set max to use half of memory on the host.
The difference between buffer cache memory and ARC cache one is that for regular application, the first one is immediately available to allocation while the ARC cache one is not. In general ZFS monitors the free RAM available and when too low, it releases RAM to other consumers.
According to you application, you may want to set ZFS ARC maximum.

 

Set ZFS ARC MAX, zfs_arc_max

This setting is done in the /etc/modprobe.d/zfs.conf file.

For example, should you want the ARC never to exceed 24 GB, add this line:

options zfs zfs_arc_max=25769803776

The number is for my server(32GB memory), which is a file server, no other application running on it. So leave 8GB to OS and cluster file system.

This paramater is the most popular one that people like to tune their ZFS. Similarily, you can set other zfs tuning parameters in zfs.conf too. The list of tunable parameters you can find in

/sys/module/zfs/parameters

Note: Before you touch other parameters, I suggest you read this article first

ZFS tunable parameters, though it is for Open Solaris ZFS, still worth to read.

Get ZFS ARC statistics

To get the current ARC size and various other ARC statistics, run this command:

cat /proc/spl/kstat/zfs/arcstats


The size metric will show the current size of the ARC.

Tool to use for ZFS ARC monitoring

There is a tool called arcstat.pl on Solaris, what about ZOL?

It's called arcstat.py, right in the zfs linux package

# rpm -qf /usr/bin/arcstat.py
zfs-0.6.3-1.el6.x86_64

One example:

# arcstat.py 10 10
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
13:23:43     2     0      0     0    0     0    0     0    0    24G   24G  
13:23:53  7.2K  2.8K     38     0    0  2.8K  100     0    0    23G   24G  
13:24:03  7.5K  2.8K     37     0    0  2.8K  100     0    0    24G   24G  
13:24:13  7.9K  2.9K     36     0    0  2.9K  100     0    0    24G   24G  
13:24:23  6.3K  2.5K     40     0    0  2.5K  100     0    0    24G   24G  
13:24:33  7.1K  2.8K     39     0    0  2.8K  100     0    0    24G   24G  

Usage examples

# arcstat.py -h
Usage: arcstat.py [-hvx] [-f fields] [-o file] [-s string] [interval [count]]

     -h : Print this help message
     -v : List all possible field headers and definitions
     -x : Print extended stats
     -f : Specify specific fields to print (see -v)
     -o : Redirect output to the specified file
     -s : Override default field separator with custom character or string

Examples:
    arcstat.py -o /tmp/a.log 2 10
    arcstat.py -s "," -o /tmp/a.log 2 10
    arcstat.py -v
    arcstat.py -f time,hit%,dh%,ph%,mh% 1

Free up ARC

Idealy, ZFS will release ARC memory reservation when needed by other application. If you want to force it to do so, then you have to export zfs pool which is using ARC, umount a zfs filesystem may not force zfs release ARC reserve.

zpool export <zfspool>