ClickHouse介绍

• ClickHouse是由俄罗斯的Yandex公司开发的一款快速、可扩展的列式数据库管理系统。它专门针对OLAP场景设计，在海量数据分析和查询方面具有出色的性能表现，同时还能支持实时数据插入和更新。
• ClickHouse采用了面向列的存储方式，具有较高的压缩比和查询速度。它支持SQL语言，并提供了一系列丰富的函数库，可满足各种复杂的数据分析需求。同时，ClickHouse还支持数据分区和分片，可实现横向扩展
• ClickHouse可在Linux和Windows等操作系统上运行。它还支持RESTful API和JDBC等多种接口，方便与其他系统进行集成。由于其出色的性能和可扩展性，ClickHouse被许多企业用于大规模数据处理和分析，如Yandex、CloudFlare、Kenshoo等。在国内，ClickHouse也被各大厂青睐和使用，例如京东、腾讯、哔哩哔哩、字节跳动等
• 从我个人两年多的使用经验来说，它对于中小型公司，只有部分业务涉及大数据处理的场景，也很适用，我们可以使用单体ClickHouse作为大数据存储和分析组件，后续可以根据业务增长自由扩展
• 在使用ClickHouse时，我们可以尽情的存储我们的海量数据，也可以毫不忌讳的直接把海量数据使用SQL进行计算和统计
• 有人这么形容clickhouse，说它是一个手动拼装的跑车，性能强劲但需要手动去维修维护它
• 目前遇到比较头疼的一个问题，就是断电重启导致的文件损坏问题，虽然在使用其他数据库时也遇到过，但是感觉异常断电时，它比MySQL等更容易出问题，毕竟压缩存储要求它一直在计算和合并文件

问题介绍

• 办公室有一台二手服务器，作为平时开发测试使用。由于机器没放在机房，会偶发断电
• 异常断电后，文件系统是有出问题的可能的，尤其是一些不断在读写合并的文件
• 春节后，发现clickhouse启动不了，使用systemctl status clickhouse-server返回如下

● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
   Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since 二 2023-02-21 15:46:41 CST; 10s ago
  Process: 9688 ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid (code=exited, status=70)
 Main PID: 9688 (code=exited, status=70)

2月 21 15:46:41 localhost.localdomain systemd[1]: clickhouse-server.service: main process exited, code=exited, status=70/n/a
2月 21 15:46:41 localhost.localdomain systemd[1]: Unit clickhouse-server.service entered failed state.
2月 21 15:46:41 localhost.localdomain systemd[1]: clickhouse-server.service failed.

• 对于具体的报错信息，还是要查看clickhouse的报错日志，默认位置在/var/log/clickhouse-server/，错误日志如下：

2023.02.21 15:47:12.351823 [ 9848 ] {} <Error> system.query_thread_log (95febdac-d99a-4bdb-9431-653a75f3a34b): Detaching broken part /var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394084_2980 (size: 0.00 B). If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.02.21 15:47:12.365662 [ 9787 ] {} <Error> Application: Caught exception while loading metadata: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/], Stack trace (when copying this message, always include the lines below):

0. std::runtime_error::runtime_error(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b4107cd in ?
1. std::__1::system_error::system_error(std::__1::error_code, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b41a437 in ?
2. ? @ 0x145458c1 in /usr/bin/clickhouse
3. ? @ 0x1b3bf356 in /usr/bin/clickhouse
4. ? @ 0x1b3bd762 in /usr/bin/clickhouse
5. std::__1::__fs::filesystem::directory_iterator::directory_iterator(std::__1::__fs::filesystem::path const&, std::__1::error_code*, std::__1::__fs::filesystem::directory_options) @ 0x1b3bd581 in /usr/bin/clickhouse
6. DB::DiskLocalDirectoryIterator::DiskLocalDirectoryIterator(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x14552124 in /usr/bin/clickhouse
7. DB::DiskLocal::iterateDirectory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x145483d7 in /usr/bin/clickhouse
8. DB::MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1552dd06 in /usr/bin/clickhouse
9. DB::MergeTreeData::createPart(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreePartInfo const&, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::IMergeTreeDataPart const*) const @ 0x1547a871 in /usr/bin/clickhouse
10. ? @ 0x154c98ac in /usr/bin/clickhouse
11. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xaf6546a in /usr/bin/clickhouse
12. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xaf674a4 in /usr/bin/clickhouse
13. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xaf62837 in /usr/bin/clickhouse
14. ? @ 0xaf662fd in /usr/bin/clickhouse
15. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
16. __clone @ 0xfe96d in /usr/lib64/libc-2.17.so

Cannot print extra info for Poco::Exception (version 22.2.2.1)
2023.02.21 15:47:12.366162 [ 9787 ] {} <Information> Application: Shutting down storages.
2023.02.21 15:47:12.366178 [ 9787 ] {} <Information> Context: Shutdown disk default
2023.02.21 15:47:12.391451 [ 9787 ] {} <Debug> Application: Shut down storages.
2023.02.21 15:47:13.199583 [ 9787 ] {} <Debug> Application: Destroyed global context.
2023.02.21 15:47:13.202239 [ 9787 ] {} <Error> Application: filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/]
2023.02.21 15:47:13.202291 [ 9787 ] {} <Information> Application: shutting down
2023.02.21 15:47:13.202308 [ 9787 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.02.21 15:47:13.203474 [ 9788 ] {} <Trace> BaseDaemon: Received signal -2
2023.02.21 15:47:13.203546 [ 9788 ] {} <Information> BaseDaemon: Stop SignalListener thread
2023.02.21 15:47:13.276203 [ 9786 ] {} <Information> Application: Child process exited normally with code 70.

• 关键报错日志是这一句：<Error> Application: filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/]，这个文件夹损坏，结构需要清理

解决办法

• 百度了一堆，确实能搜到解决方式，牵涉到磁盘挂载、修复之类的，试了下没用，识别不到磁盘，可能是我哪里搞错了，暂时舍弃
• 还有一个办法，给文件重命名，试了下，使用mv old_file_name new_file_name重命名，直接死机了
• 重启服务器后，我使用文件传输工具查看修改文件名，竟然成功了。想了下这个损坏的文件，应该不能随便移动，应该使用rename
• 虽然文件命名成功了，但是clickhouse还是无法启动，还是类似报错，应该是clickhouse还是会使用这个文件
• 想着怎么才能让clickhouse忽略这个文件夹，试着把上面层级的文件夹95f改了下，改成95f1，启动服务后发现可以了