ubuntu中flashcache使用教程

当前存在多种性能价格不一的存储设备，以价格高性能好的存储设备来加速价格低性能较差的存储设备，是一种提升系统整体性能的方案。flashcache便可以做这件事情。

源码：https://github.com/facebookarchive/flashcache

系统信息

nvm@nvm-pc:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

flashcache安装

源码安装（不可行）

最初使用源码安装，但是后面编译报错（不管是ubuntu16还是ubuntu18），网上找了很多资料，也没有解决。这个过程中也发现，原来ubuntu库中已经有这个package了。本节记录错误，直接安装的可以跳过本节。

下载

apt install -y git make gcc
git clone git@github.com:facebookarchive/flashcache.git

安装

cd flashcache
make
make install

问题

1. make时候可能会报如下错误

Makefile:975: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
  CC [M]  /home/nvm/flashcache/src/flashcache_conf.o
  CC [M]  /home/nvm/flashcache/src/flashcache_main.o
/home/nvm/flashcache/src/flashcache_main.c: In function ‘dm_io_async_bvec_pl’:
/home/nvm/flashcache/src/flashcache_main.c:119:6: error: ‘struct dm_io_request’ has no member named ‘bi_rw’
  iorq.bi_rw = rw;
      ^
/home/nvm/flashcache/src/flashcache_main.c: In function ‘dm_io_async_bvec’:
/home/nvm/flashcache/src/flashcache_main.c:143:6: error: ‘struct dm_io_request’ has no member named ‘bi_rw’
  iorq.bi_rw = rw;
      ^
/home/nvm/flashcache/src/flashcache_main.c: In function ‘flashcache_map’:
/home/nvm/flashcache/src/flashcache_main.c:2055:39: error: ‘struct bio’ has no member named ‘bi_rw’
 #define bio_barrier(bio)        ((bio)->bi_rw & REQ_FLUSH)
                                       ^
/home/nvm/flashcache/src/flashcache_main.c:2095:6: note: in expansion of macro ‘bio_barrier’
  if (bio_barrier(bio))
      ^
/home/nvm/flashcache/src/flashcache_main.c:2055:49: error: ‘REQ_FLUSH’ undeclared (first use in this function)
 #define bio_barrier(bio)        ((bio)->bi_rw & REQ_FLUSH)

解决：安装libelf-dev

sudo apt install libelf-dev

然后继续报错

/home/nvm/flashcache/src/flashcache_main.c: In function ‘dm_io_async_bvec_pl’:
/home/nvm/flashcache/src/flashcache_main.c:119:6: error: ‘struct dm_io_request’ has no member named ‘bi_rw’
  iorq.bi_rw = rw;
      ^
/home/nvm/flashcache/src/flashcache_main.c: In function ‘dm_io_async_bvec’:
/home/nvm/flashcache/src/flashcache_main.c:143:6: error: ‘struct dm_io_request’ has no member named ‘bi_rw’
  iorq.bi_rw = rw;
      ^
/home/nvm/flashcache/src/flashcache_main.c: In function ‘flashcache_map’:
/home/nvm/flashcache/src/flashcache_main.c:2055:39: error: ‘struct bio’ has no member named ‘bi_rw’
 #define bio_barrier(bio)        ((bio)->bi_rw & REQ_FLUSH)
                                       ^
/home/nvm/flashcache/src/flashcache_main.c:2095:6: note: in expansion of macro ‘bio_barrier’
  if (bio_barrier(bio))
      ^
/home/nvm/flashcache/src/flashcache_main.c:2055:49: error: ‘REQ_FLUSH’ undeclared (first use in this function)
 #define bio_barrier(bio)        ((bio)->bi_rw & REQ_FLUSH)

网上查询了相关问题，发现从kernel4.8起，struct dm_io_request就没有bi_rw成员了，WFT…当前我使用的内核是4.15，flashcache自然不能正常编译通过。而flashcache很久没有更新了，官方git上bug至今也没有修复（不知道是不是自己搜索能力优先，有解决办法的麻烦告知下，谢谢～）

apt安装（适合ubuntu16及以下）

1. 包安装

sudo apt install flashcache-dkms flashcache-utils

注：在ubuntu18中会提示找不到这两个包，应该是ubuntu18中apt源还没有。所以建议在ubuntu16及以下版本中测试flashcache

2. 模块加载

sudo modprobe flashcache

3. 检查是否成功

$ sudo lsmod |grep flashcache
flashcache            102400  0

4. 设置开机自动加载（根据需要设置）

echo 'flashcache' | sudo tee --append /etc/modules

flashcache使用

相关命令

与flashcache相关的命令好几个，由于flashcache是通过device mapper来做的设备映射和缓存，所以对flashcache device的操作命令也有dmsetup。具体使用可以不输入任何参数回车查看

• flashcache_create
• flashcache_destroy
• flashcache_load
• flashcache_setioctl
• dmsetup

使用步骤

1. 查看硬盘信息 $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme1n1 259:5 0 13.4G 0 disk sda 8:0 0 223.6G 0 disk

这里我们要用nvme1n1来加速sda盘

2. 创建混合存储

以下两种创建方式选择一种即可

2.1 通过设备名创建

使用flashcache_create命令（具体参数项可以不加任何参数回车查看），myflashcache是混合存储名字，自己定义，两个设备前一个是加速盘，后一个是被加速盘，设备可以是一个分区，也可以是整块磁盘

注意：创建需要root权限，需加sudo，否则会报错

$ sudo flashcache_create -v -p back myflashcache /dev/nvme1n1 /dev/sda
cachedev myflashcache, ssd_devname /dev/nvme1n1, disk_devname /dev/sda cache mode WRITE_BACK
block_size 8, md_block_size 8, cache_size 0
Flashcache metadata will use 44MB of your 2994MB main memory

2.2 通过UUID创建

• 格式化被加速盘，获取UUID（a538ffef-52b6-454f-aae7-f1fcd1b171d2）

$ sudo mkfs.f2fs -f -t f2fs /dev/sda
$ ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root  9 1月   8 22:36 a538ffef-52b6-454f-aae7-f1fcd1b171d2 -> ../../sda

注意：没有格式化的设备没有UUID，所以需要先格式化。创建混合存储设备之后，混合存储设备后续可以再进行格式化，不需要重新解绑获取UUID

• 创建 flashcache 混合存储

$ sudo flashcache_create -v -p back myflashcache /dev/nvme1n1 /dev/disk/by-uuid/a538ffef-52b6-454f-aae7-f1fcd1b171d2
# 输出
cachedev myflashcache, ssd_devname /dev/nvme1n1, disk_devname /dev/disk/by-uuid/5f31415c-6c49-4d0a-b344-86bede1722f5 cache mode WRITE_BACK
block_size 8, md_block_size 8, cache_size 0
Flashcache metadata will use 73MB of your 7827MB main memory
Flashcache Module already loaded
version string "git commit: 3.1.3+git20150701
"
Creating FlashCache Volume : "echo 0 468862128 flashcache /dev/disk/by-uuid/5f31415c-6c49-4d0a-b344-86bede1722f5 /dev/nvme1n1 myflashcache 1 2 8 0 512 0 0 8 | dmsetup create myflashcache"

3. 查看flashcache device

$ lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme1n1        259:5    0  13.4G  0 disk 
└─myflashcache 253:0    0 223.6G  0 dm   
sda              8:0    0 223.6G  0 disk 
└─myflashcache 253:0    0 223.6G  0 dm 

4. 使用

sudo mount /dev/mapper/myflashcache /test
sudo umount /test

此时，myflashcache相当于一个存储设备，可以像操作通常设备一样操作它

查看混合存储状态

在使用一段时间后，我们可以看flashcache的缓存状态，可以以此为依据来调整flashcache的策略。

$ sudo dmsetup status myflashcache
0 468862128 flashcache stats: 
    reads(2895), writes(30735)
    read hits(2632), read hit percent(90)
    write hits(2082) write hit percent(6)
    dirty write hits(0) dirty write hit percent(0)
    replacement(263), write replacement(28653)
    write invalidates(0), read invalidates(0)
    pending enqueues(0), pending inval(0)
    metadata dirties(30735), metadata cleans(30653)
    metadata batch(60723) metadata ssd writes(665)
    cleanings(30653) fallow cleanings(0)
    no room(0) front merge(30010) back merge(514)
    force_clean_block(0)
    disk reads(263), disk writes(30653) ssd reads(33285) ssd writes(31660)
    uncached reads(0), uncached writes(0), uncached IO requeue(0)
    disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
    uncached sequential reads(0), uncached sequential writes(0)
    pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
    lru hot blocks(1751296), lru warm blocks(1751296)
    lru promotions(0), lru demotions(0)

查看混合存储相关信息

相关信息存储在/proc/flashcache/<加速盘+被加速盘>/目录下面，可以通过cat查看具体信息

$ ls /proc/flashcache/nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2/
flashcache_errors  flashcache_iosize_hist  flashcache_pidlists  flashcache_stats

$ cat /proc/flashcache/nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2/flashcache_errors 
disk_read_errors=0 disk_write_errors=0 ssd_read_errors=0 ssd_write_errors=0 memory_alloc_errors=0

也可以通过sysctl查看flashcache的信息：

$ sudo sysctl -a | grep flashcache
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.cache_all = 1
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.clean_on_read_miss = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.clean_on_write_miss = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.dirty_thresh_pct = 20
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.do_pid_expiry = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.do_sync = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.fallow_clean_speed = 2
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.fallow_delay = 900
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.fast_remove = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.io_latency_hist = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.lru_hot_pct = 75
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.lru_promote_thresh = 2
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.max_clean_ios_set = 2
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.max_clean_ios_total = 4
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.max_pids = 100
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.new_style_write_merge = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.pid_expiry_secs = 60
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.reclaim_policy = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.skip_seq_thresh_kb = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.stop_sync = 0
dev.flashcache.nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2.zero_stats = 0
$ ls /proc/sys/dev/flashcache/nvme1n1+a538ffef-52b6-454f-aae7-f1fcd1b171d2/
cache_all            do_pid_expiry       fast_remove         max_clean_ios_set      pid_expiry_secs     zero_stats
clean_on_read_miss   do_sync             io_latency_hist     max_clean_ios_total    reclaim_policy
clean_on_write_miss  fallow_clean_speed  lru_hot_pct         max_pids               skip_seq_thresh_kb
dirty_thresh_pct     fallow_delay        lru_promote_thresh  new_style_write_merge  stop_sync

更改缓存模式

flashcache支持三种缓存模式：

• Writeback : 写入时先写到Cache中，同时将对应block的元数据dirty bit，但是并不会立即写入后备存储器
• Writethrough : 写入时将数据同时写入cache和后备存储器，后备存储器写完，才算写操作完成
• Writearound : 写入的时候，绕过Cache，直接写入后备存储器，即加速盘只当读缓存

下面这张图可以形象说明三者区别：

在创建缓存时可以设置缓存模式，输入flashcache_create命令查看，其中-b参数用于设置缓存模式

# flashcache_create
Usage: flashcache_create [-v] [-p back|thru|around] [-w] [-b block size] [-m md block size] [-s cache size] [-a associativity] cachedev ssd_devname disk_devname
Usage : flashcache_create Cache Mode back|thru|around is required argument
Usage : flashcache_create Default units for -b, -m, -s are sectors, or specify in k/M/G. Default associativity is 512.

解绑和删除

注意：解绑之前先把混合设备数据转移防止丢失！

sudo umount /test
sudo dmsetup remove myflashcache
sudo flashcache_destroy /dev/nvme1n1

参考：
http://www.yangguanjun.com/2018/03/08/ssd-enhance-sata-with-flashcache/
https://sujunmin.github.io/blog/2017/08/31/Ubuntu%2016.04%20%E5%AE%89%E8%A3%9D%E8%88%87%E4%BD%BF%E7%94%A8%20Flashcache%20%E6%B5%81%E7%A8%8B%E7%B4%80%E9%8C%84/