一、背景知识
二、机器分配
| IP | Roles | Hostname | 备注 |
| 222.31.76.209 | client | localhost.localdomain | |
| 222.31.76.178 | mds&monitor | ceph_mds | |
| 222.31.76.74 | osd | ceph_osd0 | |
| 222.31.76.67 | osd | ceph_osd1 | |
| 222.31.76.235 | osd | ceph_osd2 |
内核版本：linux-3.8.8.tar.xz （stable2013-04-17）
ceph版本：ceph-0.60.tar.gz （01-Apr-201317:42）
三、编译与配置
checking for uuid_parse in -luuid... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libuuid not found
See `config.log' for more details.
checking for add_key in -lkeyutils... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libkeyutils not found
See `config.log' for more details.
checking for CRYPTOPP... no
checking for library containing _ZTIN8CryptoPP14CBC_EncryptionE... no
checking for NSS... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no suitable crypto library found
See `config.log' for more details.
checking for CRYPTOPP... no
checking for library containing _ZTIN8CryptoPP14CBC_EncryptionE... -lcryptopp
checking for NSS... no
configure : using cryptopp for cryptography
checking for FCGX_Init in -lfcgi... no
checking for fuse_main in -lfuse... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no FUSE found (use --without-fuse to disable)
See `config.log' for more details.
checking for fuse_getgroups... no
checking jni.h usability... no
checking jni.h presence... no
checking for jni.h... no
checking for LIBEDIT... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: No usable version of libedit found.
See `config.log' for more details.
checking atomic_ops.h usability... no
checking atomic_ops.h presence... no
checking for atomic_ops.h... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no libatomic-ops found (use --without-libatomic-ops to disable)
See `config.log' for more details.
checking for snappy_compress in -lsnappy... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libsnappy not found
See `config.log' for more details.
checking for leveldb_open in -lleveldb... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libleveldb not found
See `config.log' for more details.
checking for io_submit in -laio... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libaio not found
See `config.log' for more details.
checking boost /spirit /include /classic_core.hpp usability... no
checking boost /spirit /include /classic_core.hpp presence... no
checking for boost /spirit /include /classic_core.hpp... no
checking boost /spirit.hpp usability... no
checking boost /spirit.hpp presence... no
checking for boost /spirit.hpp... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: "Can't find boost spirit headers "
See `config.log' for more details.
checking whether to check for GCC pthread /shared inconsistencies... yes
checking whether -pthread is sufficient with -shared... yes
configure : creating . /config.status
config.status : creating Makefile
config.status : creating scripts /gtest -config
config.status : creating build -aux /config.h
config.status : executing depfiles commands
config.status : executing libtool commands
CXXLD osdmaptool
CXX ceph_dencoder -ceph_dencoder.o
test /encoding /ceph_dencoder.cc : In function 'int main(int, const char**)' :
test /encoding /ceph_dencoder.cc : 196 : note : variable tracking size limit exceeded with -fvar -tracking -assignments, retrying without
CXX ceph_dencoder -rgw_dencoder.o
In file included from rgw /rgw_dencoder.cc : 6 :
rgw/rgw_acl_s3.h:9:19: error: expat.h: No such file or directory
In file included from rgw /rgw_acl_s3.h : 12,
from rgw /rgw_dencoder.cc : 6 :
rgw/rgw_xml.h:62: error: 'XML_Parser' does not name a type
make[ 3] : * * * [ceph_dencoder -rgw_dencoder.o] Error 1
make[ 3] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[ 1] : * * * [all] Error 2
make[ 1] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
make: *** [all-recursive] Error 1
CXXLD cephfs
CXXLD librados -config
CXXLD ceph -fuse
CCLD rbd -fuse
CCLD mount.ceph
CXXLD rbd
CXXLD rados
CXXLD ceph -syn
make[ 3] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[ 1] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
Making all in man
make[1]: Entering directory `/cwn/ceph/ceph-0.60/man'
make[ 1] : Nothing to be done for `all '.
make[1]: Leaving directory `/cwn/ceph/ceph-0.60/man'
libtool : finish : PATH = "/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/sbin" ldconfig -n /usr /local /lib /rados -classes
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Libraries have been installed in :
/usr /local /lib /rados -classes
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the ` -LLIBDIR '
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH ' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to ` /etc /ld.so.conf '
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/usr/local/lib/ceph" || /bin/mkdir -p "/usr/local/lib/ceph"
/usr/bin/install -c ceph_common.sh ' /usr /local /lib /ceph '
make[4]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[ 3] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[ 1] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /src '
Making install in man
make[1]: Entering directory `/cwn/ceph/ceph-0.60/man'
make[ 2] : Entering directory ` /cwn /ceph /ceph - 0. 60 /man '
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/share/man/man8" || /bin /mkdir -p "/usr/local/share/man/man8"
/usr /bin /install -c -m 644 ceph -osd. 8 ceph -mds. 8 ceph -mon. 8 mkcephfs. 8 ceph -fuse. 8 ceph -syn. 8 crushtool. 8 osdmaptool. 8 monmaptool. 8 ceph -conf. 8 ceph -run. 8 ceph. 8mount.ceph. 8 radosgw. 8 radosgw -admin. 8 ceph -authtool. 8 rados. 8 librados -config. 8 rbd. 8 ceph -clsinfo. 8 ceph -debugpack. 8 cephfs. 8 ceph -dencoder. 8 ceph -rbdnamer. 8 rbd-fuse. 8 '/usr/local/share/man/man8'
make[ 2] : Leaving directory ` /cwn /ceph /ceph - 0. 60 /man '
make[1]: Leaving directory `/cwn/ceph/ceph-0.60/man'
;
; Sample ceph ceph.conf file.
;
; This file defines cluster membership, the various locations
; that Ceph stores data, and any other runtime options.
; If a 'host' is defined for a daemon, the init.d start /stop script will
; verify that it matches the hostname ( or else ignore it). If it is
; not defined, it is assumed that the daemon is intended to start on
; the current host (e.g., in a setup with a startup.conf on each
; node).
; The variables $type, $id and $name are available to use in paths
; $type = The type of daemon, possible values : mon, mds and osd
; $id = The ID of the daemon, for mon.alpha, $id will be alpha
; $name = $type.$id
; For example :
; osd. 0
; $type = osd
; $id = 0
; $name = osd. 0
; mon.beta
; $type = mon
; $id = beta
; $name = mon.beta
; global
[global]
; enable secure authentication
; auth supported = cephx
; allow ourselves to open a lot of files
max open files = 131072
; set log file
log file = /var /log /ceph /$name.log
; log_to_syslog = true ; uncomment this line to log to syslog
; set up pid files
pid file = /var /run /ceph /$name.pid
; If you want to run a IPv6 cluster, set this to true. Dual -stack isn 't possible
;ms bind ipv6 = true
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /data/mon$id
; If you are using for example the RADOS Gateway and want to have your newly created
; pools a higher replication level, you can set a default
;osd pool default size = 3
; You can also specify a CRUSH rule for new pools
; Wiki: http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH
;osd pool default crush rule = 0
; Timing is critical for monitors, but if you want to allow the clocks to drift a
; bit more, you can specify the max drift.
;mon clock drift allowed = 1
; Tell the monitor to backoff from this warning for 30 seconds
;mon clock drift warn backoff = 30
; logging, for debugging monitor crashes, in order of
; their likelihood of being helpful :)
debug ms = 1
;debug mon = 20
;debug paxos = 20
;debug auth = 20
[mon.0]
host = ceph_mds
mon addr = 222.31.76.178:6789
; mds
; You need at least one. Define two to get a standby.
[mds]
; where the mds keeps it's secret encryption keys
keyring = /data /keyring.$name
; mds logging to debug issues.
;debug ms = 1
;debug mds = 20
[mds.alpha]
host = ceph_mds
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
sudo = true
; This is where the osd expects its data
osd data = /data /osd$id
; Ideally, make the journal a separate disk or partition.
; 1 - 10GB should be enough; more if you have fast or many
; disks. You can use a file under the osd data dir if need be
; (e.g. /data /$name /journal), but it will be slower than a
; separate disk or partition.
; This is an example of a file -based journal.
osd journal = /data /$name /journal
osd journal size = 1000 ; journal size, in megabytes
; If you want to run the journal on a tmpfs (don 't), disable DirectIO
;journal dio = false
; You can change the number of recovery operations to speed up recovery
; or slow it down if your machines can't handle it
; osd recovery max active = 3
; osd logging to debug osd issues, in order of likelihood of being
; helpful
;debug ms = 1
;debug osd = 20
;debug filestore = 20
;debug journal = 20
; ### The below options only apply if you're using mkcephfs
; ### and the devs options
; The filesystem used on the volumes
osd mkfs type = btrfs
; If you want to specify some other mount options, you can do so.
; for other filesystems use 'osd mount options $fstype'
osd mount options btrfs = rw,noatime
; The options used to format the filesystem via mkfs.$fstype
; for other filesystems use 'osd mkfs options $fstype'
; osd mkfs options btrfs =
[osd. 0]
host = ceph_osd0
; if 'devs' is not specified, you 're responsible for
; setting up the 'osd data ' dir.
btrfs devs = /dev/mapper/VolGroup-lv_ceph
[osd.1]
host = ceph_osd1
btrfs devs = /dev/mapper/VolGroup-lv_ceph
[osd.2]
host = ceph_osd2
btrfs devs = /dev/mapper/VolGroup-lv_ceph
4. 配置网络
5.
创建文件系统并启动。以下命令在监控节点进行！[ /usr /local /etc /ceph /fetch_config /tmp /fetched.ceph.conf. 2693]
The authenticity of host 'ceph_mds (127.0.0.1)' can 't be established.
RSA key fingerprint is a7:c8:b8:2e:86:ea:89:ff:11:93:e9:29:68:b5:7c:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph_mds ' (RSA) to the list of known hosts.
ceph.conf 100% 4436 4.3KB/s 00:00
temp dir is /tmp/mkcephfs.tIHQnX8vkw
preparing monmap in /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool --create --clobber --add 0 222.31.76.178:6789 --print /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: monmap file /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: generated fsid f998ee83-9eba-4de2-94e3-14f235ef840c
epoch 0
fsid f998ee83-9eba-4de2-94e3-14f235ef840c
last_changed 2013-05-31 08:22:52.972189
created 2013-05-31 08:22:52.972189
0: 222.31.76.178:6789/0 mon.0
/usr/local/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.tIHQnX8vkw/monmap (1 monitors)
=== osd.0 ===
pushing conf and monmap to ceph_osd0:/tmp/mkfs.ceph.0b3c65941572123eb704d9d614411fc1
scp: /etc/ceph/ceph.conf: No such file or directory
cp /usr /local /etc /ceph /ceph.conf /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd0 : /usr /local /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd0 : /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd1 : /usr /local /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd1 : /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd2 : /usr /local /etc /ceph /ceph.conf
scp /usr /local /etc /ceph /ceph.conf root@ceph_osd2 : /etc /ceph /ceph.conf
temp dir is /tmp /mkcephfs.hz1EcPJjtu
preparing monmap in /tmp /mkcephfs.hz1EcPJjtu /monmap
/usr /local /bin /monmaptool --create --clobber --add 0 222. 31. 76. 178 : 6789 --print /tmp /mkcephfs.hz1EcPJjtu /monmap
/usr /local /bin /monmaptool : monmap file /tmp /mkcephfs.hz1EcPJjtu /monmap
/usr /local /bin /monmaptool : generated fsid 62fdb8b1 - 8d98 - 42f2 - 9cef -b95e2ad7bd43
epoch 0
fsid 62fdb8b1 - 8d98 - 42f2 - 9cef -b95e2ad7bd43
last_changed 2013 - 05 - 31 08 : 39 : 48. 198656
created 2013 - 05 - 31 08 : 39 : 48. 198656
0 : 222. 31. 76. 178 : 6789 / 0 mon. 0
/usr /local /bin /monmaptool : writing epoch 0 to /tmp /mkcephfs.hz1EcPJjtu /monmap ( 1 monitors)
== = osd. 0 == =
pushing conf and monmap to ceph_osd0 : /tmp /mkfs.ceph. 2e991ed41f1cdca1149725615a96d0be
umount : /data /osd0 : not mounted
umount : /dev /mapper /VolGroup -lv_ceph : not mounted
WARNING ! - Btrfs Btrfs v0. 20 -rc1 IS EXPERIMENTAL
WARNING ! - see http : / /btrfs.wiki.kernel.org before using
fs created label (null) on /dev /mapper /VolGroup -lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0. 20 -rc1
2013-05-31 12:39:04.073438 7f02cd9ac760 -1 filestore(/data/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013 - 05 - 31 12 : 39 : 04. 362010 7f02cd9ac760 - 1 created object store /data /osd0 journal /data /osd. 0 /journal for osd. 0 fsid 62fdb8b1 - 8d98 - 42f2 - 9cef -b95e2ad7bd43
2013-05-31 12:39:04.362074 7f02cd9ac760 -1 auth: error reading file: /data/osd0/keyring: can 't open /data/osd0/keyring: (2) No such file or directory
2013-05-31 12:39:04.362280 7f02cd9ac760 -1 created new key in keyring /data/osd0/keyring
collecting osd.0 key
=== osd.1 ===
pushing conf and monmap to ceph_osd1:/tmp/mkfs.ceph.9a9f67ff6e7516b415d30f0a89bfe0dd
umount: /data/osd1: not mounted
umount: /dev/mapper/VolGroup-lv_ceph: not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-31 08:39:13.237718 7ff0a2fe4760 -1 filestore(/data/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-31 08:39:13.524175 7ff0a2fe4760 -1 created object store /data/osd1 journal /data/osd.1/journal for osd.1 fsid 62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
2013-05-31 08:39:13.524241 7ff0a2fe4760 -1 auth: error reading file: /data/osd1/keyring: can't open /data /osd1 /keyring : ( 2) No such file or directory
2013 - 05 - 31 08 : 39 : 13. 524430 7ff0a2fe4760 - 1 created new key in keyring /data /osd1 /keyring
collecting osd. 1 key
== = osd. 2 == =
pushing conf and monmap to ceph_osd2 : /tmp /mkfs.ceph. 51a8af4b24b311fcc2d47eed2cd714ca
umount : /data /osd2 : not mounted
umount : /dev /mapper /VolGroup -lv_ceph : not mounted
WARNING ! - Btrfs Btrfs v0. 20 -rc1 IS EXPERIMENTAL
WARNING ! - see http : / /btrfs.wiki.kernel.org before using
fs created label (null) on /dev /mapper /VolGroup -lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0. 20 -rc1
2013 - 05 - 31 09 : 01 : 49. 371853 7ff422eb1760 - 1 filestore( /data /osd2) could not find 23c2fcde /osd_superblock / 0 / / - 1 in index : ( 2) No such file or directory
2013 - 05 - 31 09 : 01 : 49. 583061 7ff422eb1760 - 1 created object store /data /osd2 journal /data /osd. 2 /journal for osd. 2 fsid 62fdb8b1 - 8d98 - 42f2 - 9cef -b95e2ad7bd43
2013 - 05 - 31 09 : 01 : 49. 583123 7ff422eb1760 - 1 auth : error reading file : /data /osd2 /keyring : can 't open /data/osd2/keyring: (2) No such file or directory
2013-05-31 09:01:49.583312 7ff422eb1760 -1 created new key in keyring /data/osd2/keyring
collecting osd.2 key
=== mds.alpha ===
creating private key for mds.alpha keyring /data/keyring.mds.alpha
creating /data/keyring.mds.alpha
bufferlist::write_file(/data/keyring.mds.alpha): failed to open file: (2) No such file or directory
could not write /data/keyring.mds.alpha
can't open /data/keyring.mds.alpha: can 't open /data/keyring.mds.alpha: (2) No such file or directory
failed: '/usr/local/sbin/mkcephfs -d /tmp/mkcephfs.hz1EcPJjtu --init-daemon mds.alpha'
temp dir is /tmp /mkcephfs.v9vb0zOmJ5
preparing monmap in /tmp /mkcephfs.v9vb0zOmJ5 /monmap
/usr /local /bin /monmaptool --create --clobber --add 0 222. 31. 76. 178 : 6789 --print /tmp /mkcephfs.v9vb0zOmJ5 /monmap
/usr /local /bin /monmaptool : monmap file /tmp /mkcephfs.v9vb0zOmJ5 /monmap
/usr /local /bin /monmaptool : generated fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
epoch 0
fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
last_changed 2013 - 05 - 31 08 : 50 : 21. 797571
created 2013 - 05 - 31 08 : 50 : 21. 797571
0 : 222. 31. 76. 178 : 6789 / 0 mon. 0
/usr /local /bin /monmaptool : writing epoch 0 to /tmp /mkcephfs.v9vb0zOmJ5 /monmap ( 1 monitors)
== = osd. 0 == =
pushing conf and monmap to ceph_osd0 : /tmp /mkfs.ceph. 8912ed2e34cfd2477c2549354c03faa3
umount : /dev /mapper /VolGroup -lv_ceph : not mounted
WARNING ! - Btrfs Btrfs v0. 20 -rc1 IS EXPERIMENTAL
WARNING ! - see http : / /btrfs.wiki.kernel.org before using
fs created label (null) on /dev /mapper /VolGroup -lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0. 20 -rc1
2013 - 05 - 31 12 : 49 : 36. 548329 7f67d293e760 - 1 journal check : ondisk fsid 919417f1 -0a79 - 4463 - 903c - 3fc9df8ca0f8 doesn 't match expected 3b3d2772-4981-46fd-bbcd-b11957c77d47, invalid (someone else's ?) journal
2013 - 05 - 31 12 : 49 : 36. 953666 7f67d293e760 - 1 filestore( /data /osd0) could not find 23c2fcde /osd_superblock / 0 / / - 1 in index : ( 2) No such file or directory
2013 - 05 - 31 12 : 49 : 37. 244334 7f67d293e760 - 1 created object store /data /osd0 journal /data /osd. 0 /journal for osd. 0 fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
2013 - 05 - 31 12 : 49 : 37. 244397 7f67d293e760 - 1 auth : error reading file : /data /osd0 /keyring : can 't open /data/osd0/keyring: (2) No such file or directory
2013-05-31 12:49:37.244580 7f67d293e760 -1 created new key in keyring /data/osd0/keyring
collecting osd.0 key
=== osd.1 ===
pushing conf and monmap to ceph_osd1:/tmp/mkfs.ceph.69d388555243635efea3c5976d001b64
umount: /dev/mapper/VolGroup-lv_ceph: not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-31 08:49:45.012858 7f82a3d52760 -1 journal check: ondisk fsid 28f23b77-6f77-47b3-b946-7eda652d4488 doesn't match expected 65a75a4f -b639 - 4eab - 91d6 -00c985118862, invalid (someone else 's?) journal
2013-05-31 08:49:45.407962 7f82a3d52760 -1 filestore(/data/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-31 08:49:45.696990 7f82a3d52760 -1 created object store /data/osd1 journal /data/osd.1/journal for osd.1 fsid 652b09fb-bbbf-424c-bd49-8218d75465ba
2013-05-31 08:49:45.697052 7f82a3d52760 -1 auth: error reading file: /data/osd1/keyring: can't open /data /osd1 /keyring : ( 2) No such file or directory
2013 - 05 - 31 08 : 49 : 45. 697238 7f82a3d52760 - 1 created new key in keyring /data /osd1 /keyring
collecting osd. 1 key
== = osd. 2 == =
pushing conf and monmap to ceph_osd2 : /tmp /mkfs.ceph. 686b9d63c840a05a6eed5b5781f10b27
umount : /dev /mapper /VolGroup -lv_ceph : not mounted
WARNING ! - Btrfs Btrfs v0. 20 -rc1 IS EXPERIMENTAL
WARNING ! - see http : / /btrfs.wiki.kernel.org before using
fs created label (null) on /dev /mapper /VolGroup -lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0. 20 -rc1
2013 - 05 - 31 09 : 12 : 20. 708733 7fa54ae8f760 - 1 journal check : ondisk fsid dc21285e - 3bde - 4f53 - 9424 -d059540ab920 doesn 't match expected cae83f10-d633-48d1-b324-a64849eca974, invalid (someone else's ?) journal
2013 - 05 - 31 09 : 12 : 21. 057154 7fa54ae8f760 - 1 filestore( /data /osd2) could not find 23c2fcde /osd_superblock / 0 / / - 1 in index : ( 2) No such file or directory
2013 - 05 - 31 09 : 12 : 21. 253689 7fa54ae8f760 - 1 created object store /data /osd2 journal /data /osd. 2 /journal for osd. 2 fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
2013 - 05 - 31 09 : 12 : 21. 253749 7fa54ae8f760 - 1 auth : error reading file : /data /osd2 /keyring : can 't open /data/osd2/keyring: (2) No such file or directory
2013-05-31 09:12:21.253931 7fa54ae8f760 -1 created new key in keyring /data/osd2/keyring
collecting osd.2 key
=== mds.alpha ===
creating private key for mds.alpha keyring /data/keyring.mds.alpha
creating /data/keyring.mds.alpha
Building generic osdmap from /tmp/mkcephfs.v9vb0zOmJ5/conf
/usr/local/bin/osdmaptool: osdmap file ' /tmp /mkcephfs.v9vb0zOmJ5 /osdmap '
/usr/local/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.v9vb0zOmJ5/osdmap
Generating admin key at /tmp/mkcephfs.v9vb0zOmJ5/keyring.admin
creating /tmp/mkcephfs.v9vb0zOmJ5/keyring.admin
Building initial monitor keyring
added entity mds.alpha auth auth(auid = 18446744073709551615 key=AQCXnKhRiL/QHhAA091/MQGD25V54smKBz959w== with 0 caps)
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQDhK6hROEuRDhAA9uCsjB++Szh8sJy3CUgeoA== with 0 caps)
added entity osd.1 auth auth(auid = 18446744073709551615 key=AQBpnKhR0EKMKRAAzNWvZgkDWrPSuZaSttBdsw== with 0 caps)
added entity osd.2 auth auth(auid = 18446744073709551615 key=AQC1oahReP4fDxAAR0R0HTNfVbs6VMybLIU9qg== with 0 caps)
=== mon.0 ===
/usr/local/bin/ceph-mon: created monfs at /data/mon0 for mon.0
placing client.admin keyring in /etc/ceph/keyring
== = mon. 0 == =
Starting Ceph mon. 0 on ceph_mds...
starting mon. 0 rank 0 at 222. 31. 76. 178 : 6789 / 0 mon_data /data /mon0 fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
== = mds.alpha == =
Starting Ceph mds.alpha on ceph_mds...
starting mds.alpha at : / 0
== = osd. 0 == =
Mounting Btrfs on ceph_osd0 : /data /osd0
Scanning for Btrfs filesystems
Starting Ceph osd. 0 on ceph_osd0...
starting osd. 0 at : / 0 osd_data /data /osd0 /data /osd. 0 /journal
== = osd. 1 == =
Mounting Btrfs on ceph_osd1 : /data /osd1
Scanning for Btrfs filesystems
Starting Ceph osd. 1 on ceph_osd1...
starting osd. 1 at : / 0 osd_data /data /osd1 /data /osd. 1 /journal
== = osd. 2 == =
Mounting Btrfs on ceph_osd2 : /data /osd2
Scanning for Btrfs filesystems
Starting Ceph osd. 2 on ceph_osd2...
starting osd. 2 at : / 0 osd_data /data /osd2 /data /osd. 2 /journal
health HEALTH_OK
monmap e1 : 1 mons at { 0 = 222. 31. 76. 178 : 6789 / 0}, election epoch 2, quorum 0 0
osdmap e7 : 3 osds : 3 up, 3 in
pgmap v432 : 768 pgs : 768 active +clean; 9518 bytes data, 16876 KB used, 293 GB / 300 GB avail
mdsmap e4 : 1 / 1 / 1 up { 0 =alpha =up :active}
GLOBAL :
SIZE AVAIL RAW USED %RAW USED
300M 293M 16876 0
POOLS :
NAME ID USED %USED OBJECTS
data 0 0 0 0
metadata 1 9518 0 21
rbd 2 0 0 0
6. 客户端挂载
mount : wrong fs type, bad option, bad superblock on ceph_mds : /,
missing codepage or helper program, or other error
( for several filesystems (e.g. nfs, cifs) you might
need a /sbin /mount. <type > helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so
ceph : Unknown symbol ceph_create_client (err 0)
ceph : Unknown symbol ceph_calc_pg_primary (err 0)
ceph : Unknown symbol ceph_osdc_release_request (err 0)
ceph : Unknown symbol ceph_con_open (err 0)
ceph : Unknown symbol ceph_flags_to_mode (err 0)
ceph : Unknown symbol ceph_msg_last_put (err 0)
ceph : Unknown symbol ceph_caps_for_mode (err 0)
ceph : Unknown symbol ceph_copy_page_vector_to_user (err 0)
ceph : Unknown symbol ceph_msg_new (err 0)
ceph : Unknown symbol ceph_msg_type_name (err 0)
ceph : Unknown symbol ceph_pagelist_truncate (err 0)
ceph : Unknown symbol ceph_release_page_vector (err 0)
ceph : Unknown symbol ceph_check_fsid (err 0)
ceph : Unknown symbol ceph_pagelist_reserve (err 0)
ceph : Unknown symbol ceph_pagelist_append (err 0)
ceph : Unknown symbol ceph_calc_object_layout (err 0)
ceph : Unknown symbol ceph_get_direct_page_vector (err 0)
ceph : Unknown symbol ceph_osdc_wait_request (err 0)
ceph : Unknown symbol ceph_osdc_new_request (err 0)
ceph : Unknown symbol ceph_pagelist_set_cursor (err 0)
ceph : Unknown symbol ceph_calc_file_object_mapping (err 0)
ceph : Unknown symbol ceph_monc_got_mdsmap (err 0)
ceph : Unknown symbol ceph_osdc_readpages (err 0)
ceph : Unknown symbol ceph_con_send (err 0)
ceph : Unknown symbol ceph_zero_page_vector_range (err 0)
ceph : Unknown symbol ceph_osdc_start_request (err 0)
ceph : Unknown symbol ceph_compare_options (err 0)
ceph : Unknown symbol ceph_msg_dump (err 0)
ceph : Unknown symbol ceph_buffer_new (err 0)
ceph : Unknown symbol ceph_put_page_vector (err 0)
ceph : Unknown symbol ceph_pagelist_release (err 0)
ceph : Unknown symbol ceph_osdc_sync (err 0)
ceph : Unknown symbol ceph_destroy_client (err 0)
ceph : Unknown symbol ceph_copy_user_to_page_vector (err 0)
ceph : Unknown symbol __ceph_open_session (err 0)
ceph : Unknown symbol ceph_alloc_page_vector (err 0)
ceph : Unknown symbol ceph_monc_do_statfs (err 0)
ceph : Unknown symbol ceph_monc_validate_auth (err 0)
ceph : Unknown symbol ceph_osdc_writepages (err 0)
ceph : Unknown symbol ceph_parse_options (err 0)
ceph : Unknown symbol ceph_str_hash (err 0)
ceph : Unknown symbol ceph_pr_addr (err 0)
ceph : Unknown symbol ceph_buffer_release (err 0)
ceph : Unknown symbol ceph_con_init (err 0)
ceph : Unknown symbol ceph_destroy_options (err 0)
ceph : Unknown symbol ceph_con_close (err 0)
ceph : Unknown symbol ceph_msgr_flush (err 0)
Key type ceph registered
libceph : loaded (mon /osd proto 15 / 24, osdmap 5 / 6 5 / 6)
ceph : loaded (mds proto 32)
libceph : parse_ips bad ip 'ceph_mds'
ceph : loaded (mds proto 32)
libceph : parse_ips bad ip 'ceph_mds'
Key type ceph unregistered
Key type ceph registered
libceph : loaded (mon /osd proto 15 / 24, osdmap 5 / 6 5 / 6)
ceph : loaded (mds proto 32)
libceph : parse_ips bad ip 'ceph_mds'
libceph: no secret set (for auth_x protocol)
libceph: error -22 on auth protocol 2 init
libceph : client4102 fsid 652b09fb -bbbf - 424c -bd49 - 8218d75465ba
最终查明原因，是因为mount时还需要输入用户名和密钥，具体mount命令为：
parsing options : name =admin,secret =AQCXnKhRgMltJRAAi0WMqr +atKFPaIV4Aja4hQ ==
[client. admin]
key = AQCXnKhRgMltJRAAi0WMqr+atKFPaIV4Aja4hQ==
查看客户端的挂载情况：
Filesystem Size Used Avail Use % Mounted on
/dev /mapper /VolGroup -lv_root
50G 13G 35G 27 % /
tmpfs 2.0G 0 2.0G 0 % /dev /shm
/dev /sda1 477M 48M 405M 11 % /boot
/dev /mapper /VolGroup -lv_home
405G 71M 385G 1 % /home
222.31.76.178:/ 300G 6.1G 294G 3% /mnt/ceph
allow = %everyone
到此Ceph的安装配置就已全部完成，可以在客户端的/mnt/ceph目录下使用Ceph分布式文件系统。
---------------------------------------------------------------------