Home › High Performance › Files › MooseFS Distributed File System Installation and Configuration Guide

MooseFS Distributed File System Installation and Configuration Guide

Categories: Files 2015-12-20 root Comments (0)
       Before starting the configuration, let’s go over the lab environment. I used 5 machines for local testing: one master, two chunkservers, and two clients. Also, disable the firewall to avoid affecting the experiment results:
 
master:192.168.20.210
chunkserver1:192.168.20.211
chunkserver2:192.168.20.212
client：192.168.20.213
client：192.168.20.214

         Actually, three machines are enough. MooseFS does not strictly require that the master, chunkserver, and client cannot run on the same machine, and I have indeed tried this without any issues. However, considering system security and stability, it is recommended to run the master on a separate machine with proper backups. The chunkserver and client placements can be decided based on specific needs.
 

1. Environment Setup (All machines must do this):

Install common libraries

yum -y install gcc zlib-devel zlib fuse fuse-devel fuse-libs ntp

Download and extract MooseFS:
 
wget http://mirror.cnop.net/moosefs/moosefs-2.0.81-1.tar.gz
or
wget http://mirror2.cnop.net/moosefs/moosefs-2.0.81-1.tar.gz
tar -zxvf  moosefs-2.0.81-1.tar.gz
cd moosefs-2.0.81
 
Create the mfs user and group:
 
useradd  mfs -s /sbin/nologin
 
Install MooseFS:
 
./configure –prefix=/usr/local/mfs   –with-default-user=mfs –with-default-group=mfs
make && make install
 
MooeFS System Overview:
    The MooseFS system is very easy to configure. Let me show you the magic of MooseFS. First, a brief look at the MooseFS directory structure. If you used the same compile parameters as I did, entering /usr/local/mfs will reveal a few clean directories: bin, etc, sbin, share, and var.
 
bin: Directory for MooseFS client tools
sbin: Directory for MooseFS server tools
etc: Directory for MooseFS configuration files, including master and chunkserver
share: Help files directory, self-explanatory
var: Data files directory.
 
    

2. Master Configuration
After the first step, enter the relevant directory:

     cd  /usr/local/mfs/etc/mfs

   
The file we need is mfsmaster.cfg under etc, and the tool is mfsmaster under sbin. Let’s look at mfsmaster.cfg first.
cp mfsmaster.cfg.dist mfsmaster.cfg
vi mfsmaster.cfg
 
# WORKING_USER = mfs
# WORKING_GROUP = mfs
 
# LOCK_FILE = /var/run/mfs/mfsmaster.pid
# DATA_PATH = /usr/local/mfs/var/mfs
# SYSLOG_IDENT = mfsmaster
 
# BACK_LOGS = 50
 
# REPLICATIONS_DELAY_INIT = 300
# REPLICATIONS_DELAY_DISCONNECT = 3600
 
# MATOCS_LISTEN_HOST = *
# MATOCS_LISTEN_PORT = 9420
 
# MATOCU_LISTEN_HOST = *
# MATOCU_LISTEN_PORT = 9421
 
# CHUNKS_LOOP_TIME = 300
# CHUNKS_DEL_LIMIT = 100
# CHUNKS_REP_LIMIT = 15
 
There seem to be quite a few options, but for the first time we only need to pay attention to one line.

# MATOCS_LISTEN_HOST = * Change to MATOCS_LISTEN_HOST = 192.168.20.210

 
Remove the #, replace the * with the master’s IP. It’s that simple. Save and exit. A note on MooseFS configuration files: although they are all commented out with # by default (meaning they don’t take effect), the parameters recorded in the unedited configuration file are the default values the system runs with. Keep this in mind.
 
Start the master

cd /usr/local/mfs/var/mfs
cp metadata.mfs.empty metadata.mfs

cd /usr/local/mfs/etc/mfs
cp mfsexports.cfg.dist mfsexports.cfg
cp mfstopology.cfg.dist mfstopology.cfg

/usr/local/mfs/sbin/mfsmaster

 
Note: The cp commands above must be executed, otherwise you will get an error saying metadata.mfs cannot be found. The Metadata file records file sizes, attributes, locations, and also covers non-regular file systems like directories, sockets, pipes, and devices.

After that, you can check the system log to see the master’s running status.
 

The command to stop the master is as follows:

/usr/local/mfs/sbin/mfsmaster stop

 

Tip: If you find typing such long commands every time annoying, you can add it to your PATH:

export PATH=/usr/local/mfs/sbin:/usr/local/mfs/sbin:$PATH

After that, controlling the master directly with mfsmaster or mfsmaster -s is much more convenient, isn’t it?
For startup on boot, we can add the following line to /etc/rc.local (adjust according to your actual situation):

 

vi  /etc/rc.local
/usr/local/mfs/sbin/mfsmaster
/usr/local/mfs/sbin/mfscgiserv

Starting the management interface:

 /usr/local/mfs/sbin/mfscgiserv

Browser access:

http://192.168.20.210:9425

You can then access it.

Adding ports to the firewall:

 /sbin/iptables -I INPUT -p tcp –dport 9421 -j ACCEPT
/etc/rc.d/init.d/iptables save
/sbin/iptables -I INPUT -p tcp –dport 9419 -j ACCEPT
/etc/rc.d/init.d/iptables save
/sbin/iptables -I INPUT -p tcp –dport 9420 -j ACCEPT
/etc/rc.d/init.d/iptables save
/sbin/iptables -I INPUT -p tcp –dport 9425 -j ACCEPT
/etc/rc.d/init.d/iptables save
service iptables restart

 
3. Chunkserver Configuration (the server that actually stores data)
    Software installation is not repeated here; please refer to step one. Here we take one machine, 192.168.20.211, as an example.The files used are mfschunkserver.cfg and mfshdd.cfg under the etc directory (if the files don’t exist after software installation, you can copy the corresponding .dist files in the same directory and rename them). First, let’s look at mfschunkserver.cfg:

cd /usr/local/mfs/etc/mfs
cp mfschunkserver.cfg.dist mfschunkserver.cfg
vi mfschunkserver.cfg

 
# WORKING_USER = mfs
# WORKING_GROUP = mfs
 
# DATA_PATH = /usr/local/mfs/var/mfs
# LOCK_FILE = /var/run/mfs/mfschunkserver.pid
# SYSLOG_IDENT = mfschunkserver
 
# BACK_LOGS = 50
 
# MASTER_RECONNECTION_DELAY = 30
 
# MASTER_HOST = mfsmaster
# MASTER_PORT = 9420
 
# MASTER_TIMEOUT = 60
 
# CSSERV_LISTEN_HOST = *
# CSSERV_LISTEN_PORT = 9422
 
# CSSERV_TIMEOUT = 60
 
# CSTOCS_TIMEOUT = 60
 
# HDD_CONF_FILENAME = /usr/local/mfs/etc/mfshdd.cfg
There are quite a few options, no fewer than in mfsmaster.cfg, but don’t worry, we only care about the following line:

# MASTER_HOST = mfsmaster change to MASTER_HOST = 192.168.20.210

 
Now let’s look at mfshdd.cfg, which as the name implies is the MFS hard drive configuration:
 
cd /usr/local/mfs/etc/mfs
cp mfshdd.cfg.dist mfshdd.cfg

Create a directory for storing data and set permissions:

mkdir /data
chown -R mfs.mfs  /data

Editmfshdd.cfg

vi mfshdd.cfg

You can ignore the default comments, as they don’t exist on our machine.Here we add one line at the bottom:

/data

This partition is a separate hard drive on my machine specifically used for storing data.

 

Starting chunkserver
 

/usr/local/mfs/sbin/mfschunkserver
 

To start mfschunkserver on boot, add the following:

vi /etc/rc.local

/usr/local/mfs/sbin/mfschunkserver

4. mfsmetalogger Configuration (on any machine except the master server):
 
cd /usr/local/mfs/etc/mfs
cp mfsmetalogger.cfg.dist  mfsmetalogger.cfg
vi mfsmetalogger.cfg
 
 
Change the IP after MASTER_HOST to the primary server IP
Startup:
/usr/local/mfs/sbin/mfsmetalogger

V. Configuring the Client

       Strictly speaking, the client doesn’t require much configuration; it’s essentially just an application. The client can be any server that needs to use the distributed system and has mfsmount installed. For the client, MooseFS differs only in its mount method; once mounted, it is no different from a regular hard drive partition. So how do you mount it?

       Here we take the server 192.168.20.213 as an example. We won’t repeat the MooseFS software installation steps here; please refer to the first step. The command to use is mfsmount, which is located in the bin directory.

# mkdir /data
#/usr/local/mfs/bin/mfsmount /data -H 192.168.20.210

 

Note: if the client mount directory and the storage server are on the same machine, the directories must not be the same. The mfsmount command commonly uses two parameters:-H specifies the server.The last argument specifies the local mount directory.

        Mount the other machine, 192.168.20.214, in the same way.
        From now on, you can read and write normally to the /data directory on 213. Anything you write will be distributed across multiple other chunkserver machines to ensure data safety.
Add to startup on boot:

vi /etc/rc.local
Add the following information:
/usr/local/mfs/bin/mfsmount /data -H 192.168.20.210

VI. Mounting the Trash Can Directory:
 

Create the trash can mount directory:
mkdir /home/trash
Mount the trash can:
/usr/local/mfs/bin/mfsmount -o mfsmeta -H 192.168.20.210 /home/trash

       The trash directory contains files deleted to the recycle bin. If you move them to the undel subdirectory, the deleted files will be restored to the original MooseFS file system. As shown below, after executing the last line, you can see the file has been restored in the file system:

       Only the administrator has permission to access MFSMETA (user uid 0, usually root). Inside the MFSMETA directory, besides the trash and trash/undel directories, there is a third directory called reserved. This directory contains files that have been deleted but still have open handles. Once the user closes these opened files, the files in the reserved directory will be deleted, and the file data will be immediately removed. The naming convention for files in the reserved directory is the same as in the trash directory.
 

Common Commands (Client Operations):

Check the number of file copies:

/usr/local/mfs/bin/mfsgetgoal /data

Check how many copies the /data directory has.

Set the number of file copies:

/usr/local/mfs/bin/mfssetgoal  -r 3 /data

Set the directory /data to keep 3 copies.

Set deletion delay:

/usr/local/mfs/bin/mfsrsettrashtime 10 -r /data

Data in the recycle bin will be cleared 10 seconds after deletion.

Check the deletion delay time:

/usr/local/mfs/bin/mfsgettrashtime /data

Related Posts

Leave a Comment

Your email address will not be published.