问题一:
ERROR c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - dump address /192.168.1.50:3306 has an error, retrying. caused by
com.alibaba.otter.canal.parse.exception.CanalParseException: can\'t find start position for example
原因:meta.dat 中保存的位点信息和数据库的位点信息不一致;导致canal抓取不到数据库的动作;
解决方案:删除meta.dat删除,再重启canal,问题解决;
集群操作:进入canal对应的zookeeper集群下,删除节点/otter/canal/destinations/xxxxx/1001/cursor ;重启canal即可恢复;
问题二:
java.lang.OutOfMemoryError: Java heap space
canal消费端挂了太久,在zk对应conf下节点的
/otter/canal/destinations/test_db/1001/cursor 位点信息是很早以前,导致重启canal时,从很早以前的位点开始消费,导致canal服务器内存爆掉
监听数据库变更,只有TransactionBegin/TransactionEnd,没有拿到数据的EventType;
原因可能是canal.instance.filter.black.regex=.*\\..*导致,改canal.instance.filter.black.regex=再重启试试;
问题三:
ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:fdyb_db[com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: com.google.common.collect.ComputationException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`mysql`.`pds_4490277`
Caused by: com.google.common.collect.ComputationException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`mysql`.`pds_4490277`
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`mysql`.`pds_4490277`
Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SELECT command denied to user \'cy_canal\'@\'11.217.0.224\' for table \'pds_4490277\', sqlState=42000, sqlStateMarker=#]
with command: desc `mysql`.`pds_4490277`
分析:mysql系统表权限较高,canal读该表的binlog失败,位点无法移动
解决:将配置项中黑名单加上mysql下的所有表:canal.instance.filter.black.regex = mysql\\..* ,修改后canal集群不需要重启即可恢复;
其它注意点:检查下CanalConnector是否调用subscribe(filter)方法;有的话,filter需要和instance.properties的canal.instance.filter.regex一致,否则subscribe的filter会覆盖instance的配置,如果subscribe的filter是.*\\..*,那么相当于你消费了所有的更新数据。
问题四:
现象:数据库修改后,canal应用感知不到binlog,数据无法正常消费处理;
定位:1.查看canal服务器,canal应用,zk服务器的日志,确认无异常;2.查看mysql,es服务器,无异常,3.查看canal服务器,canal应用配置项,发现canal服务器的canal.properties有问题;
原因:canal.properties中配置了canal.ip和canal.zkServers,如果是zk集群模式下的canal配置了canal.ip,则会优先按IP连接canal服务器,从而让zk功能失效,位点文件则会保存到本地;一旦本地位点文件出现问题,各方无错误日志,问题就很难排查;
解决:将canal.ip配置项置为空,关掉canal服务器,canal应用,删除zk上的节点,重启canal服务器,canal应用,问题解决;
问题五:
canal启动,创建数据库的脚本为
CREATE DATABASE /*!32312 IF NOT EXISTS*/`db_name` /*!40100 DEFAULT CHARACTER SET utf8mb4 */
canal启动报错,无法处理.
2019-06-13 16:18:59.318 [destination = example , address = /Ip:port , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.tsdb.MemoryTableMeta - parse faield : CREATE DATABASE /*!32312 IF NOT EXISTS*/`db_name` /*!40100 DEFAULT CHARACTER SET utf8mb4 */
com.alibaba.fastsql.sql.parser.ParserException: illegal name, pos 40, line 1, column 17, token HINT
at com.alibaba.fastsql.sql.parser.SQLExprParser.name(SQLExprParser.java:1812) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.dialect.mysql.parser.MySqlStatementParser.parseCreateDatabase(MySqlStatementParser.java:6543) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.dialect.mysql.parser.MySqlStatementParser.parseCreate(MySqlStatementParser.java:292) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.parser.SQLStatementParser.parseStatementList(SQLStatementParser.java:228) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.SQLUtils.parseStatements(SQLUtils.java:536) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.fastsql.sql.repository.SchemaRepository.console(SchemaRepository.java:439) ~[fastsql-2.0.0_preview_855.jar:2.0.0_preview_855]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.MemoryTableMeta.apply(MemoryTableMeta.java:81) ~[canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.applyHistoryOnMemory(DatabaseTableMeta.java:463) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.rollback(DatabaseTableMeta.java:174) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.processTableMeta(AbstractMysqlEventParser.java:106) [canal.parse-1.1.3.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:192) [canal.parse-1.1.3.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192]
查询tsdb的meta_history,把相关的创建语句进行修改,去掉 /* 这些字符, 重启OK。
问题六:
rocketmq自动创建topic的队列配置为4,
在canal的具体的一个instance里面配置的instance.properties,其中 canal.mq.partitionsNum=16,错误是无限重试发送。
修复方式是canal.mq.partitionsNum=4, (但是在测试环境没有复现)
注意事项:
数据库操作存在batch update,
canal发送到rocketmq的数据,data字段存在的数据有可能是多条,且数量不一(例如:批量更新了10,可能发了3条消息,一条消息包括1条数据,1条消息包括3条数据,1条消息包括6条数据)。
问题七:
2020-09-28 11:31:31.938 [destination = mall-canal_priceDB , address = priceDB.master.com/192.168.0.223:3320 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:mall-canal_priceDB[com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`oriceDB`.`t_settlement_price`
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`priceDB`.`t_settlement_price`
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: fetch failed by table meta:`priceDB`.`t_settlement_price`
Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user \'xxxx\'@\'xx.xx.xx.xx\' for table \'t_settlement_price\', sqlState=42000, sqlStateMarker=#]
with command: show create table `priceDB`.`t_settlement_price`
at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.query(MysqlQueryExecutor.java:61)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.query(MysqlConnection.java:106)
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.TableMetaCache.getTableMeta(TableMetaCache.java:177)
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.getTableMeta(LogEventConvert.java:950)
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEventForTableMeta(LogEventConvert.java:479)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:280)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$SimpleParserStage.onEvent(MysqlMultiStageCoprocessor.java:246)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
原因:数据帐号没有show command执行权限,查询不到table meta数据,重新授权即可
转自:
https://www.jianshu.com/p/3e5c289a2228
https://blog.csdn.net/my201110lc/article/details/77885720