kettle 批量导入到 doris 报错。
187*****902 发布于2021-10-15 浏览:558 回复:2
0
收藏

版本  V0.14

我们生产环境的问题,救急!!!!

kettle 抓取数据以后 导入到 doris 使用的是mysql协议。 每次导入到 500条的 时候 必报错。请问是什么原因?

2021-10-15 15:27:43,872 WARN (thrift-server-pool-3|546) [Coordinator.updateFragmentExecStatus():1369] one instance report fail, query_id=b85d66d45c3c4441-8f2ccf697660bf3a instance_id=b85d66d45c3c4441-8f2ccf697660bf3b
2021-10-15 15:27:43,872 WARN (thrift-server-pool-3|546) [Coordinator.updateStatus():670] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: b85d66d45c3c4441-8f2ccf697660bf3a, instance id: b85d66d45c3c4441-8f2ccf697660bf3b
2021-10-15 15:27:43,873 WARN (doris-mysql-nio-pool-153|570) [StmtExecutor.handleInsertStmt():846] insert failed: already stopped, skip waiting for close. cancelled/!eos: : 1/0
2021-10-15 15:27:43,873 WARN (doris-mysql-nio-pool-153|570) [StmtExecutor.handleInsertStmt():894] handle insert stmt fail: insert_b85d66d45c3c4441-8f2ccf697660bf3a
org.apache.doris.common.DdlException: errCode = 2, detailMessage = already stopped, skip waiting for close. cancelled/!eos: : 1/0
at org.apache.doris.common.ErrorReport.reportDdlException(ErrorReport.java:67) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.handleInsertStmt(StmtExecutor.java:847) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:326) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:243) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:199) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:337) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:535) ~[palo-fe.jar:3.4.0]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:3.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-15 15:27:43,875 WARN (doris-mysql-nio-pool-153|570) [StmtExecutor.execute():380] errors when abort txn
org.apache.doris.transaction.TransactionNotFoundException: errCode = 2, detailMessage = transaction not found
at org.apache.doris.transaction.DatabaseTransactionMgr.abortTransaction(DatabaseTransactionMgr.java:949) ~[palo-fe.jar:3.4.0]
at org.apache.doris.transaction.GlobalTransactionMgr.abortTransaction(GlobalTransactionMgr.java:210) ~[palo-fe.jar:3.4.0]
at org.apache.doris.transaction.GlobalTransactionMgr.abortTransaction(GlobalTransactionMgr.java:205) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:376) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:243) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:199) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:337) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:535) ~[palo-fe.jar:3.4.0]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:3.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-15 15:27:43,880 WARN (thrift-server-pool-1|151) [Coordinator.updateFragmentExecStatus():1369] one instance report fail, query_id=ce663cab6e747bc-a0bf4b95266cd3fb instance_id=ce663cab6e747bc-a0bf4b95266cd3fc
2021-10-15 15:27:43,880 WARN (thrift-server-pool-1|151) [Coordinator.updateStatus():670] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: ce663cab6e747bc-a0bf4b95266cd3fb, instance id: ce663cab6e747bc-a0bf4b95266cd3fc
2021-10-15 15:27:43,880 WARN (doris-mysql-nio-pool-152|569) [StmtExecutor.handleInsertStmt():846] insert failed: already stopped, skip waiting for close. cancelled/!eos: : 1/0
2021-10-15 15:27:43,881 WARN (doris-mysql-nio-pool-152|569) [StmtExecutor.handleInsertStmt():894] handle insert stmt fail: insert_ce663cab6e747bc-a0bf4b95266cd3fb
org.apache.doris.common.DdlException: errCode = 2, detailMessage = already stopped, skip waiting for close. cancelled/!eos: : 1/0
at org.apache.doris.common.ErrorReport.reportDdlException(ErrorReport.java:67) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.handleInsertStmt(StmtExecutor.java:847) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:326) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:243) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:199) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:337) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:535) ~[palo-fe.jar:3.4.0]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:3.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-15 15:27:43,882 WARN (doris-mysql-nio-pool-152|569) [StmtExecutor.execute():380] errors when abort txn
org.apache.doris.transaction.TransactionNotFoundException: errCode = 2, detailMessage = transaction not found
at org.apache.doris.transaction.DatabaseTransactionMgr.abortTransaction(DatabaseTransactionMgr.java:949) ~[palo-fe.jar:3.4.0]
at org.apache.doris.transaction.GlobalTransactionMgr.abortTransaction(GlobalTransactionMgr.java:210) ~[palo-fe.jar:3.4.0]
at org.apache.doris.transaction.GlobalTransactionMgr.abortTransaction(GlobalTransactionMgr.java:205) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:376) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:243) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:199) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:337) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:535) ~[palo-fe.jar:3.4.0]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:3.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]

收藏
点赞
0
个赞
共2条回复 最后由187*****902回复于2021-10-20
#3187*****902回复于2021-10-20

好的 已经解决了,刚开始以为使用 sql insert 批量插入没有啥问题,后来仔细看了下文档,不支持这种方式的频繁导入。

0
#2IamStrangers回复于2021-10-18

doris暂不支持实时数据导入,使用时需要按批导入,在业务侧自行控制导入频率,建议5秒间隔以上

0
快速回复
TOP
切换版块