具体的错误信息如下:
2020-01-04T14:09:55.276+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] MongoDB starting : pid=17716 port=37017 dbpath=/usr/local/mongodb/data/db 64-bit host=localhost.localdomain
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] db version v3.4.18
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] git version: 4410706bef6463369ea2f42399e9843903b31923
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] allocator: tcmalloc
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] modules: none
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] build environment:
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] distmod: rhel62
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] distarch: x86_64
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] target_arch: x86_64
2020-01-04T14:09:55.304+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/bin/mongodb.conf", net: { maxIncomingConnections: 5000, port: 37017 }, operationProfiling: { mode: "slowOp", slowOpThresholdMs: 200 }, processManagement: { fork: true, pidFilePath: "/usr/local/mongodb/data/mongodb.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/usr/local/mongodb/data/db", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, syncPeriodSecs: 60.0, wiredTiger: { collectionConfig: { blockCompressor: "snappy" }, engineConfig: { cacheSizeGB: 2.0, directoryForIndexes: true, journalCompressor: "snappy", statisticsLogDelaySecs: 0 }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, logRotate: "rename", path: "/usr/local/mongodb/data/logs/mongod.log", timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } }
2020-01-04T14:09:55.304+0800 W - [initandlisten] Detected unclean shutdown - /usr/local/mongodb/data/db/mongod.lock is not empty.
2020-01-04T14:09:55.315+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten]
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten]
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten] ** WARNING: The configured WiredTiger cache size is more than 80% of available RAM.
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/faq-memory-diagnostics-wt
2020-01-04T14:09:55.315+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=2048M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-01-04T14:09:55.325+0800 E STORAGE [initandlisten] WiredTiger error (0) [1578118195:325249][17716:0x7fed9e17bdc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 638970963 doesn't match expected checksum of 187512981
2020-01-04T14:09:55.325+0800 E STORAGE [initandlisten] WiredTiger error (0) [1578118195:325298][17716:0x7fed9e17bdc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2020-01-04T14:09:55.325+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1578118195:325325][17716:0x7fed9e17bdc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-01-04T14:09:55.325+0800 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 365
2020-01-04T14:09:55.325+0800 I - [initandlisten]
***aborting after fassert() failure
2020-01-04T14:09:55.386+0800 F - [initandlisten] Got signal: 6 (Aborted).
0x55b8b9bafaa1 0x55b8b9baecb9 0x55b8b9baf19d 0x7fed9ce6c7e0 0x7fed9cafb4f5 0x7fed9cafccd5 0x55b8b8e4022d 0x55b8b98b3266 0x55b8b8e4a982 0x55b8b8e4ab9e 0x55b8b8e4ae00 0x55b8ba4bdd8f 0x55b8ba4be2db 0x55b8ba4ba8cd 0x55b8ba4bf4a7 0x55b8ba4d8ed3 0x55b8ba5124eb 0x55b8ba59ceeb 0x55b8ba59d42d 0x55b8ba59d6ec 0x55b8ba520c51 0x55b8ba5931d8 0x55b8ba55d0be 0x55b8ba55d19b 0x55b8ba50e98b 0x55b8b98977da 0x55b8b988fe85 0x55b8b9781567 0x55b8b8e2c5de 0x55b8b8e4c1bb 0x7fed9cae7d20 0x55b8b8eac0d1
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55B8B8621000","o":"158EAA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B8B8621000","o":"158DCB9"},{"b":"55B8B8621000","o":"158E19D"},{"b":"7FED9CE5D000","o":"F7E0"},{"b":"7FED9CAC9000","o":"324F5","s":"gsignal"},{"b":"7FED9CAC9000","o":"33CD5","s":"abort"},{"b":"55B8B8621000","o":"81F22D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B8B8621000","o":"1292266"},{"b":"55B8B8621000","o":"829982","s":"__wt_eventv"},{"b":"55B8B8621000","o":"829B9E","s":"__wt_err"},{"b":"55B8B8621000","o":"829E00","s":"__wt_panic"},{"b":"55B8B8621000","o":"1E9CD8F","s":"__wt_block_extlist_read"},{"b":"55B8B8621000","o":"1E9D2DB","s":"__wt_block_extlist_read_avail"},{"b":"55B8B8621000","o":"1E998CD","s":"__wt_block_checkpoint_load"},{"b":"55B8B8621000","o":"1E9E4A7"},{"b":"55B8B8621000","o":"1EB7ED3","s":"__wt_btree_open"},{"b":"55B8B8621000","o":"1EF14EB","s":"__wt_conn_btree_open"},{"b":"55B8B8621000","o":"1F7BEEB","s":"__wt_session_get_btree"},{"b":"55B8B8621000","o":"1F7C42D","s":"__wt_session_get_btree"},{"b":"55B8B8621000","o":"1F7C6EC","s":"__wt_session_get_btree_ckpt"},{"b":"55B8B8621000","o":"1EFFC51","s":"__wt_curfile_open"},{"b":"55B8B8621000","o":"1F721D8"},{"b":"55B8B8621000","o":"1F3C0BE","s":"__wt_metadata_cursor_open"},{"b":"55B8B8621000","o":"1F3C19B","s":"__wt_metadata_cursor"},{"b":"55B8B8621000","o":"1EED98B","s":"wiredtiger_open"},{"b":"55B8B8621000","o":"12767DA","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55B8B8621000","o":"126EE85"},{"b":"55B8B8621000","o":"1160567","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55B8B8621000","o":"80B5DE"},{"b":"55B8B8621000","o":"82B1BB","s":"main"},{"b":"7FED9CAC9000","o":"1ED20","s":"__libc_start_main"},{"b":"55B8B8621000","o":"88B0D1"}],"processInfo":{ "mongodbVersion" : "3.4.18", "gitVersion" : "4410706bef6463369ea2f42399e9843903b31923", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.el6.x86_64", "version" : "#1 SMP Tue Jun 19 21:26:04 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55B8B8621000", "elfType" : 3, "buildId" : "A20A22CC07DF3D0508531D5DF7E2E07A664EF73B" }, { "b" : "7FFCBE23F000", "elfType" : 3, "buildId" : "A527E0E44603433B61EA716072C7A30506AD5000" }, { "b" : "7FED9DD05000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BECFB85A8BC084042D5BF2BA9E66325CE798B659" }, { "b" : "7FED9D920000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "CBDA444A7109874C5350AE9CEEF3F82F749B347F" }, { "b" : "7FED9D718000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "552CEC3216281CCFD7FA6432C723D50163255823" }, { "b" : "7FED9D514000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2AF795BFFD122309BA3359FEBABB5D0967403D17" }, { "b" : "7FED9D290000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4AAEE970B045D8BF946578B9C7F3AB5CDE9AB44A" }, { "b" : "7FED9D07A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7FED9CE5D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4EA475CD3FD3B69B6C95D9381FA74B36DB4992EF" }, { "b" : "7FED9CAC9000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "BCA7789C2EA8E28CB7CE553E183AC7E7EE36F8A2" }, { "b" : "7FED9DF71000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, { "b" : "7FED9C885000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, { "b" : "7FED9C59E000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, { "b" : "7FED9C39A000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, { "b" : "7FED9C16E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, { "b" : "7FED9BF58000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FED9BD4D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, { "b" : "7FED9BB4A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7FED9B930000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4786A2A5D30B121601958E84D643C70C13C4FBA5" }, { "b" : "7FED9B711000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b8b9bafaa1]
mongod(+0x158DCB9) [0x55b8b9baecb9]
mongod(+0x158E19D) [0x55b8b9baf19d]
libpthread.so.0(+0xF7E0) [0x7fed9ce6c7e0]
libc.so.6(gsignal+0x35) [0x7fed9cafb4f5]
libc.so.6(abort+0x175) [0x7fed9cafccd5]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b8b8e4022d]
mongod(+0x1292266) [0x55b8b98b3266]
mongod(__wt_eventv+0x3D7) [0x55b8b8e4a982]
mongod(__wt_err+0x9D) [0x55b8b8e4ab9e]
mongod(__wt_panic+0x2E) [0x55b8b8e4ae00]
mongod(__wt_block_extlist_read+0x8F) [0x55b8ba4bdd8f]
mongod(__wt_block_extlist_read_avail+0x2B) [0x55b8ba4be2db]
mongod(__wt_block_checkpoint_load+0x26D) [0x55b8ba4ba8cd]
mongod(+0x1E9E4A7) [0x55b8ba4bf4a7]
mongod(__wt_btree_open+0xB53) [0x55b8ba4d8ed3]
mongod(__wt_conn_btree_open+0x18B) [0x55b8ba5124eb]
mongod(__wt_session_get_btree+0xFB) [0x55b8ba59ceeb]
mongod(__wt_session_get_btree+0x63D) [0x55b8ba59d42d]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55b8ba59d6ec]
mongod(__wt_curfile_open+0x161) [0x55b8ba520c51]
mongod(+0x1F721D8) [0x55b8ba5931d8]
mongod(__wt_metadata_cursor_open+0x6E) [0x55b8ba55d0be]
mongod(__wt_metadata_cursor+0x4B) [0x55b8ba55d19b]
mongod(wiredtiger_open+0x196B) [0x55b8ba50e98b]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x73A) [0x55b8b98977da]
mongod(+0x126EE85) [0x55b8b988fe85]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x55b8b9781567]
mongod(+0x80B5DE) [0x55b8b8e2c5de]
mongod(main+0x96B) [0x55b8b8e4c1bb]
libc.so.6(__libc_start_main+0x100) [0x7fed9cae7d20]
mongod(+0x88B0D1) [0x55b8b8eac0d1]
----- END BACKTRACE -----
通过查看官方的jira,发现此问题还挺多,具体链接如下:
https://jira.mongodb.org/browse/SERVER-28666
https://jira.mongodb.org/browse/SERVER-31237
https://jira.mongodb.org/browse/SERVER-27894
https://jira.mongodb.org/browse/SERVER-19815
目前官方的建议是从其他节点同步或者从备份中恢复。
通过此次故障,得出像WiredTiger.wt这样重要并不大的文件,多做备份,避免不必要的麻烦