我这边用sparkstreaming 消费kafka 后通过stream load 塞入doris, 之前出过版本数据过多问题,我把时间窗口最长设置了5分钟,长的多达半小时,但是平均每天还是得出现一次某个be版本过多导致的问题,查询时报errCode = 2, detailMessage = Internal Error: read storage fail. res=-3111, tablet=1592760.881279058.c14619d5a986013a-2489382bd30d41a7, backend=172.16.0.144
我已经把分区buckets降为了2,而且数据只会插入某个月的分区。 我们doris应该没这么脆弱吧,之前我用的hive也没这问题,有什么调优的参数吗,我现在是
max_cumulative_compaction_num_singleton_deltas=500 compaction_task_num_per_disk=5 这2个参数设置了一下。不知道有什么好的建议。
CREATE TABLE binlog_cars.`mv_inf_car_bin` (
`id` bigint(20) NOT NULL COMMENT "id",
....
`_create_time` datetime NULL COMMENT ""
) ENGINE=OLAP
UNIQUE KEY(`id`, `_ts`, `_dt`)
COMMENT "mv_inf_car_bin"
PARTITION BY RANGE(`_dt`)
(PARTITION m202107 VALUES [('2021-07-01'), ('2021-08-01')),
PARTITION m202108 VALUES [('2021-08-01'), ('2021-09-01')),
PARTITION m202109 VALUES [('2021-09-01'), ('2021-10-01')))
DISTRIBUTED BY HASH(`id`) BUCKETS 2
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-1",
"dynamic_partition.end" = "2",
"dynamic_partition.prefix" = "m",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "2",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
-3111 不是版本过多的错误,-215 或者-235 才是。
你这个需要到对应的BE,搜下 -3111 ,然后看下上下文的具体错误了。