跳到主要内容

Databend Releases

This page provides information about recent features, enhancements, and bug fixes for Databend.

Feb 22, 2024 (v1.2.344)

What's Changed

Exciting New Features ✨

  • feat: Enable async backtrace for opendal by @Xuanwo in #14384
  • feat: Allow users to configure tcp keepalive for opendal by @Xuanwo in #14385
  • feat: Remove the need of setting allow_anonymous by @Xuanwo in #14398
  • feat: support outer to inner join by @xudong963 in #14401
  • feat: more constrain for file format string options. by @youngsofun in #14374
  • feat: support functions nvl and nvl2 by @PsiACE in #14421
  • feat: add json delete by name and index by @akoshchiy in #14397
  • feat: Add COS location support by @Xuanwo in #14435
  • feat(query): support session setting scope LOCAL by @everpcpc in #14441
  • feat: allow to config task session parameters. by @ZhiHanZ in #14446
  • feat(query): copy into allow disable_variant_check for parquet file by @b41sh in #14444
  • feat: add create or replace database support by @lichuang in #14449
  • feat: add basic support for show user functions by @PsiACE in #14440
  • feat: meta-service: customize chunk size when export by @drmingdrmer in #14467
  • feat: pushdown limit to single window by @ariesdevil in #14460
  • feat: Improve merge into performance, Update Column Only Optimization by @JackTan25 in #14429
  • feat(planner): support predicates move around by @Dousir9 in #14338
  • feat(rbac): ensure the builtin roles on auth by @flaneur2020 in #14451
  • feat: treat path in uri as prefix. by @youngsofun in #14420
  • feat: add structlog by @andylokandy in #14495
  • feat: add create or replace table support by @lichuang in #14508
  • feat: use parquet-rs as an alternative in fuse engine by @SkyFan2002 in #14268
  • feat: allow to config cloud control plane grpc timeout by @ZhiHanZ in #14522
  • feat: support window function in subquery by @xudong963 in #14518
  • chore(query): add system.tables query speed test by @TCeason in #14517
  • feat(query): agg-hashtable-singleton by @Freejww in #14524
  • feat: system table add query_log.log_type_name and cluster.cluster_id by @BohuTANG in #14545
  • feat: add column create_on for user functions table by @PsiACE in #14540
  • feat: support single join to inner join by @xudong963 in #14442
  • feat: add create or replace stage support by @lichuang in #14548
  • feat(geo): add geometry data type by @ariesdevil in #14470
  • feat(query): support early filtering for more join types by @Dousir9 in #14525
  • feat: add runtime filter stats info to profile by @xudong963 in #14569
  • feat: add create or replace connection support by @lichuang in #14580
  • feat: add create or replace user support by @lichuang in #14583
  • feat: add #- json operator by @akoshchiy in #14584
  • feat: add support for create or replace function by @PsiACE in #14581
  • feat: add create or replace stream support by @lichuang in #14596
  • feat: add create or replace view support by @lichuang in #14599
  • feat: Add KVPbApi: protobuf support for meta-service by @drmingdrmer in #14605
  • feat: use materialized cte for standard stream by @zhyass in #14561
  • feat: add create or replace file format support by @lichuang in #14611
  • feat: add create or replace agg index support by @lichuang in #14636
  • feat: add create or replace virtual column support by @lichuang in #14649
  • feat: add create or replace password policy support by @lichuang in #14654
  • feat: add create or replace masking policy support by @lichuang in #14656
  • feat: add create or replace network policy support by @lichuang in #14658
  • feat: add create or replace share endpoint support by @lichuang in #14660
  • refactor: enables the bloom runtime filter to be turned on adaptively by @xudong963 in #14686

Thoughtful Bug Fix 🔧

  • fix(sqlsmith): fix generate nested not null types caused panic by @b41sh in #14392
  • fix: optimize vacuum drop table call schema gc_drop_tables API in batch mode by @lichuang in #14395
  • fix: min/max of default col values in snapshot should adhere to rules by @dantengsky in #14368
  • fix: fix row number concat by @JackTan25 in #14416
  • fix(query): onwership/role info serialize to pb by @TCeason in #14393
  • fix: incorrect column id in inner_project schema by @SkyFan2002 in #14419
  • fix(query): fix pow and add strcmp domain by @Freejww in #14418
  • fix: stream cannot be accessed correctly by @zhyass in #14433
  • fix(query): global contain stage and udf priv by @TCeason in #14344
  • fix(query): add utf8 validation in deser string by @sundy-li in #14443
  • fix: flaky test by @zhyass in #14454
  • fix(query): config value for session settings by @everpcpc in #14475
  • fix: throw err when cast Decimal overflow by @guojidan in #14489
  • fix(query): replace mock_s3 with mock_aws by @zhang2014 in #14498
  • fix: should use is_nested while populating in-memory cache by @dantengsky in #14503
  • fix: wrong result with special stream alias by @zhyass in #14494
  • fix: fix update/delete using subquery only support boolean data type by @lichuang in #14413
  • fix(query): privilege type forward compat by @TCeason in #14501
  • fix(query): add range for memory ratio settings by @everpcpc in #14513
  • fix: refactor merge into insert spectify default values by @JackTan25 in #14477
  • fix(query): fix variant function with invalid jsonb panic by @b41sh in #14520
  • fix: Smaller random data by @JackTan25 in #14521
  • fix: Remove test by @JackTan25 in #14526
  • fix: table maybe locked when recluster by @zhyass in #14510
  • fix: fix ranking function out of bounds when calculate partition end by @ariesdevil in #14535
  • fix(query): do not check exists for unsetting by @everpcpc in #14536
  • fix(query): add warning for unset unknown setting by @everpcpc in #14537
  • fix(executor): fix some display name for profile plans by @zhang2014 in #14523
  • fix(ddl): fix create table bitmap and variant with default value by @b41sh in #14472
  • fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14554
  • fix: runtime filters add to wrong table by @xudong963 in #14552
  • fix(query): revoke need consider GrantObject::Database/Table by @TCeason in #14567
  • fix(query): fix double panic if enable_experimental_aggregate_hashtable by @zhang2014 in #14576
  • fix: fix comment for observe_append_send_spent by @domyway in #14586
  • fix: insert binary by @andylokandy in #14556
  • fix(query): fix parse unsigned integer with number by @b41sh in #14590
  • fix: incorrect schema may be used while executing replace-into by @sundy-li in #14592
  • fix(ast): unexpected cut in not null by @andylokandy in #14604
  • fix(query): try fix oom with large sort limit by @zhang2014 in #14629
  • fix: otlp logger runtime for graceful shutdown by @everpcpc in #14539
  • fix(executor): fix broken profile graph for cluster mode by @zhang2014 in #14666
  • fix: Use stat instead of list for checking storage by @Xuanwo in #14680
  • fix: fix get by keypath with null value by @b41sh in #14684
  • fix(query): account_admin should support grant ownership to new role after drop an created role by @TCeason in #14597
  • fix(query): fix unnest variant array panic by @b41sh in #14688
  • fix(query): Pass ci Test when enable enable_experimental_aggregate_hashtable . by @sundy-li in #14544

Code Refactor 🎉

  • refactor: StringType::Scalar: Vec<u8> -> String by @andylokandy in #14349
  • refactor: Add trait FromToProtoEnum by @drmingdrmer in #14423
  • refactor: add kv-api::Key::ValueType by @drmingdrmer in #14425
  • refactor: drop unused blocks when using ranking window function by @ariesdevil in #14428
  • refactor(executor): refactor processor profiling by @zhang2014 in #14377
  • refactor(executor): remove old profile framework by @zhang2014 in #14468
  • refactor: replace OneTable with DummyTableScan by @xudong963 in #14461
  • refactor: remove udf from system functions table by @PsiACE in #14471
  • refactor: Using reqwest-hickory-resolver as global dns client by @Xuanwo in #14476
  • refactor: meta-service: purge log in another task by @drmingdrmer in #14519
  • refactor: meta-service purge log in small chunks to avoid blocking by @drmingdrmer in #14527
  • refactor(query): account_admin role generate in query by @TCeason in #14514
  • refactor(query): disallowed tokio::spawn method by @zhang2014 in #14563
  • refactor: refine UdfMgr by @drmingdrmer in #14589
  • refactor: get_udf() does not need to specify Seq by @drmingdrmer in #14613
  • refactor: Add KVPbApi::list_pb(), simplify UdfMgr::get_udfs() by @drmingdrmer in #14623
  • refactor: Simplify UdfMgr::add_udf() by @drmingdrmer in #14631
  • refactor: UdfApi::drop_udf() return the dropped record instead of returning ErrorCode by @drmingdrmer in #14640
  • refactor(query): support additional param to set the error rate in approx_count_distinct function by @sundy-li in #14609
  • refactor: Remove UdfApi, use UdfMgr directly by @drmingdrmer in #14642
  • refactor(query): disallowed tokio spawn and block_on by @zhang2014 in #14639
  • refactor(executor): add graph information to prepare adding points by @dqhl76 in #14681

Build/Testing/CI Infra Changes 🔌

  • ci: fix previous tag for nightly release by @everpcpc in #14478

Others 📒

  • chore: upgrade crate shlex to version 1.3.0 by @dantengsky in #14404
  • chore: move meta-data type Id from crate meta-api to meta-app by @drmingdrmer in #14408
  • chore: Add debug log for read parts numbers by @Xuanwo in #14396
  • chore(planner): fix push down filter scan by @Dousir9 in #14415
  • chore: modify the max block count in compact by @zhyass in #14424
  • chore: enable new setting data_retention_time_in_days by @dantengsky in #14402
  • chore: fix cache variable name by @xx01cyx in #14430
  • chore: remove binary collatoin by @andylokandy in #14409
  • chore: bump rust toolchain for dev container by @leiysky in #14436
  • chore: bump rust version of build container by @leiysky in #14437
  • chore: fix build tool installation by @leiysky in #14438
  • chore: bump rust toolchain to nightly-2024-01-23 by @leiysky in #14434
  • chore: Forbidden non equal merge into distributed by @JackTan25 in #14426
  • chore(query): reduce get ownership by @TCeason in #14427
  • chore: simplify IndexScalar by @andylokandy in #14432
  • chore: make cost factor configurable by @leiysky in #14445
  • chore: set peer_start to current row when do ranking functions by @ariesdevil in #14450
  • chore: only wrap logical get with distinct to eliminate duplicates rows for subquery in single mode by @xudong963 in #14452
  • chore: support mark join commutation and fix left mark join by @xudong963 in #14455
  • chore(query): add buffer size in spill write by @sundy-li in #14466
  • chore: avoid wildcard match on data type by @andylokandy in #14457
  • chore: fix cyclic deps by @SkyFan2002 in #14483
  • chore(query): fix hash join spill block size by @Dousir9 in #14484
  • chore(query): bump backtrace-rs version by @zhang2014 in #14485
  • chore: correct ci-benchmark TPC-H Q5 by @Dousir9 in #14491
  • chore: update Domain::Map by @andylokandy in #14314
  • chore(query): update opendal version and add retry io timeout by @zhang2014 in #14496
  • chore(executor): add unit for profile statistics by @zhang2014 in #14499
  • chore: clean license message by @ZhiHanZ in #14509
  • chore: support row fetcher with window function by @ariesdevil in #14492
  • chore(query): mark obsoleted session settings in config by @everpcpc in #14512
  • chore: remove unused codes by @zhyass in #14511
  • chore: sleep 5ms before next batch purge by @drmingdrmer in #14530
  • chore(query): make column as an optional keyword in add/modify/drop by @sundy-li in #14533
  • chore: fix openraft version: use v0.9.0-alpha.5 instead of v0.9.0-alpha.4 by @drmingdrmer in #14538
  • chore(query): need refactor system table test into benchmark by @TCeason in #14546
  • chore: refactor pattern plan with Matcher by @leiysky in #14549
  • chore: remove get_table_index method by @xudong963 in #14557
  • chore: make runtime filter for broadcast rework by @xudong963 in #14559
  • chore(ci): add system tables/columns benchmark cloud by @TCeason in #14555
  • chore: format reuse.benchmark_query_meta.yml by @dantengsky in #14575
  • chore: improve parallel run for merge into by @JackTan25 in #14574
  • chore(executor): using human format for explain analyze profile by @zhang2014 in #14578
  • chore(ci): fix benchmark labels for local by @everpcpc in #14603
  • chore: add error message on refresh table info by @flaneur2020 in #14606
  • chore: refine hash join code by @Dousir9 in #14571
  • chore(ci): fix benchmark scripts by @everpcpc in #14608
  • chore: make materialized cte scan_progress accurate by @xudong963 in #14602
  • chore: Bump OpenDAL to pinned tag version by @Xuanwo in #14601
  • chore(ci): benchmark system table query speed with a large test data by @TCeason in #14612
  • chore(ci): load benchmark data from hits_100m_obfuscated_v1 by @everpcpc in #14619
  • chore(ci): remove dummy dataset for benchmark by @everpcpc in #14620
  • chore(ci): fix hits benchmark schema by @everpcpc in #14621
  • chore(ci): tmp disable local hits benchmark by @everpcpc in #14625
  • chore(ci): fix internal test reduce bendsql warn by @TCeason in #14627
  • chore(ci): fix download benchmark results by @everpcpc in #14628
  • chore(ci): fix get dataset from benchmark result file by @everpcpc in #14630
  • chore: add table index to explain join by @xudong963 in #14632
  • chore(ci): fix download benchmark result by @everpcpc in #14633
  • chore(ci): fix new line in benchmark comment by @everpcpc in #14638
  • chore(query): improve hash join code by @Dousir9 in #14614
  • chore: reduce small block writes (improve merge into write performance) by @JackTan25 in #14637
  • chore: move KVPbApi impl to seperate files by @drmingdrmer in #14634
  • chore(ci): fix comment benchmark result by @everpcpc in #14643
  • chore: refine plan id generation by @leiysky in #14644
  • chore(ci): fix runner provider for release benchmark by @everpcpc in #14646
  • chore(meta): remove tenant for table info by @zhang2014 in #14647
  • chore(ci): fix artifacts for benchmark archive by @everpcpc in #14652
  • chore: remove spawn_blocking for merge into and replace into by @JackTan25 in #14664
  • chore(query): update async trait version by @zhang2014 in #14667
  • chore: add dev deps for proj crate by @ariesdevil in #14670
  • chore: add dev dep sqlite binary for proj crate by @ariesdevil in #14673
  • chore(executor): remove useless warn log by @zhang2014 in #14672
  • chore: install liblzma-dev for build-tool by @everpcpc in #14678
  • chore: upgrade arrow-format by @ariesdevil in #14675
  • chore: optimize q17 join order by @xudong963 in #14624
  • chore(query): max_decoding_message_size by @sundy-li in #14687

New Contributors

  • @xx01cyx made their first contribution in #14430
  • @domyway made their first contribution in #14586

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.307...v1.2.344

Jan 25, 2024 (v1.2.307)

What's Changed

Exciting New Features ✨

  • feat: Add huggingface support by @Xuanwo in #14304
  • feat(query): add float64 version of distance function overload by @sundy-li in #14311
  • feat: CopyIntoLocation support output. by @youngsofun in #14315
  • feat: query level data cache metrics by @dantengsky in #14313
  • feat: CSV format add option binary_format and output_header. by @youngsofun in #14329
  • feat(planner): implement cost-based optimization for distributed query by @leiysky in #14318
  • feat(query): add function MONTHS_BETWEEN by @sundy-li in #14352
  • feat: support runtime filter for right join and string join key by @xudong963 in #14359
  • feat(query): admin api add system info by @everpcpc in #14360
  • feat: target build optimization for merge into by @JackTan25 in #14066
  • feat: support min_max runtime filter for string type by @xudong963 in #14372
  • feat: add task_depents and task_dependents_enable function by @ZhiHanZ in #14383
  • feat: add data_retention_time_in_days_max config and data_retention_time_in_days setting by @BohuTANG in #14388
  • chore(query): add index scalar by @sundy-li in #14362
  • feat: standard stream by @zhyass in #14272

Thoughtful Bug Fix 🔧

  • fix(query): add comment in table meta by @sundy-li in #14321
  • fix(query): grant all to role_name need succ by @TCeason in #14333
  • fix(query): missing push binary column by @everpcpc in #14342
  • fix: fix full outer join bug by @lichuang in #14347
  • fix: clear the derived columns after flattening by @xudong963 in #14353
  • fix: flaky test by @zhyass in #14366
  • fix: keys in stage_attachment.file_format_options should be case-insensitive by @youngsofun in #14357
  • fix: unexpected '/' in unload path. by @youngsofun in #14375
  • fix: fix update using using constant in predicate from subquery panic by @lichuang in #14367
  • fix(query): fix inner type of nested type can not be set as not null by @b41sh in #14379
  • fix(query): fix json cast to string with quote by @b41sh in #14389
  • fix(query): use serde_json maybe cause old query can not access data that generate by new query by @TCeason in #14380

Code Refactor 🎉

  • refactor: add cardinality to CteScan explain by @xudong963 in #14310
  • refactor: add BinaryColumn, BinaryColumnBuilder by @andylokandy in #14306
  • refactor(query): delete owner in table/db meta by @TCeason in #14307
  • refactor(query): built-in roles inside meta by @TCeason in #14320
  • refactor(query): built-in roles inside meta by @TCeason in #14339
  • refactor: optimize SchemaApi::create_table() by @drmingdrmer in #14341
  • refactor: meta-client should not block by RPC by @drmingdrmer in #14355
  • refactor: create-table fetch latest state after backoff sleep by @drmingdrmer in #14364
  • refactor: do not use TxnPutRequest|TxnDeleteRequest.prev_value, always return previous value by @drmingdrmer in #14371
  • refactor: adjsut file format options. by @youngsofun in #14335

Build/Testing/CI Infra Changes 🔌

  • ci: ci-cloud build both amd64 & arm64 by @everpcpc in #14317

Others 📒

  • chore: update testing parquet assets for stage test by @andylokandy in #14303
  • chore(query): forbiden revoke ownership statement by @TCeason in #14301
  • chore: fix system table ctx warning and log more warnings to log by @BohuTANG in #14309
  • chore: fix mark scan, test performance. by @JackTan25 in #14319
  • chore(ci): do not release macos build by @everpcpc in #14326
  • chore(query): add log prefix filter config by @zhang2014 in #14330
  • chore: tweak fuse compat test by @dantengsky in #14322
  • chore(ci): fix bendsql version for macos by @everpcpc in #14351
  • chore: test binary with udf by @everpcpc in #14354
  • chore: add a setting to enforce broadcast join by @xudong963 in #14361
  • chore(planner): refine cascades optimizer by @leiysky in #14350
  • chore(query): improve agg index range comparison by @Dousir9 in #14378
  • chore(planner): improve selectivity estimator by @Dousir9 in #14386
  • chore: set auto_compaction_imperfect_blocks_threshold to 50 by @zhyass in #14390

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.292...v1.2.307

Jan 11, 2024 (v1.2.292)

What's Changed

Exciting New Features ✨

  • feat: return latest leader addr if request is forwarded by @drmingdrmer in #14288
  • feat: add task sql check by @ZhiHanZ in #14292
  • feat(query): support adaptive filter reorder by @Dousir9 in #14295
  • feat: support semi join to inner join and optimize tpch q18 by @xudong963 in #14221

Thoughtful Bug Fix 🔧

  • fix(query): remove topk optimization in parquet2 by @sundy-li in #14297

Code Refactor 🎉

  • refactor: use binary for non-utf8 string type by @andylokandy in #14235

Others 📒

  • chore: add select tests for agg index with order by by @ariesdevil in #14289
  • chore(flaky): fix flaky 18_0002_ownership_cover by @TCeason in #14300

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.290...v1.2.292

Jan 10, 2024 (v1.2.290)

What's Changed

Exciting New Features ✨

  • feat: show full databases add owner field by @guojidan in #14184
  • feat(query): Add array_reduce function with lambda expression by @Freejww in #14143
  • feat(query): ownership cover stage/udf by @TCeason in #14217
  • feat: distributed execution of update statement by @SkyFan2002 in #13971
  • feat(query): Support check password policy when login by @b41sh in #14124
  • feat: add owner field into show full tables by @guojidan in #14219
  • feat: push filter down below the window by @lichuang in #14108
  • feat: support explain fragments for update by @SkyFan2002 in #14227
  • feat: cast between string and binary by @andylokandy in #14247
  • feat(query): string to ts support parse ISO 8601 tz format by @TCeason in #14253
  • feat: add json concat by @akoshchiy in #14251
  • feat: CSV add option empty_field_as. by @youngsofun in #14269
  • feat(planner): support greedy join order algorithm by @Dousir9 in #14240
  • feat: auto refresh ReadOnlyAttach-ed table schema by @dantengsky in #14241
  • feat(query): add basic sort support for agg index rewrite by @ariesdevil in #14270
  • feat(query): support transfer grant by @TCeason in #14238

Thoughtful Bug Fix 🔧

  • fix: load parquet with missing_field_as get wrong results. by @youngsofun in #14207
  • fix: disable distributed plan after disabling join reorder by @xudong963 in #14218
  • fix: fix EmptyMap domain as undefine by @lichuang in #14226
  • fix: aggregate and window function as the argument of unnest return unable to get field named by @Freejww in #14233
  • fix(query): fix missing hash fields for NonZeroU64 by @sundy-li in #14245
  • fix: fix count rewrite in agg index by @ariesdevil in #14256
  • fix(query): fix copy into float value cast to integer by @b41sh in #14257
  • fix(query): Add LegacyScalar & LegacyColumn by @sundy-li in #14264
  • fix(rbac): revoke privilege by @flaneur2020 in #14278

Code Refactor 🎉

  • refactor: Bump OpenDAL to 0.44 by @Xuanwo in #14202
  • refactor: Define features provided by raft server and features required by raft client by @drmingdrmer in #14214
  • refactor(query): Determine whether to check the ownership based on the object type by @TCeason in #14210
  • refactor: refactor NativeDeserializeDataTransform. by @RinChanNOWWW in #14213
  • refactor(schema): re-arrange conversion between arrow2 by @andylokandy in #14175
  • refactor: Replace KVApi::get() and KVApi::mget_kv() with get_kv_stream() by @drmingdrmer in #14255
  • refactor: remove from message::Request: Get(r), MGet(r), List(r) and StreamGet(r). by @drmingdrmer in #14258
  • refactor: Endpoint: add parse(); make fields private by @drmingdrmer in #14265
  • refactor: common-exception crate should be independent of meta-types by @drmingdrmer in #14275
  • refactor: move GrpcHelper to crate common-types by @drmingdrmer in #14277

Build/Testing/CI Infra Changes 🔌

  • ci: docker image tag latest only on stable release by @everpcpc in #14266

Others 📒

  • chore: Pin hive_metastore to tagged version 0.0.2 by @Xuanwo in #14201
  • chore: disable distributed execution of merge-into when spilling is enabled by @dantengsky in #14178
  • chore: modify the max block count in compact by @zhyass in #14222
  • chore(query): support selection for render_error by @Dousir9 in #14220
  • chore(executor): add more tests for executor schedule by @dqhl76 in #14225
  • chore(query): fix first error row by @Dousir9 in #14228
  • chore: test Mergeinto runtime filter performance and wizard by @JackTan25 in #14212
  • chore: refine the error message for management api by @BohuTANG in #14242
  • chore(query): fix password policy typos by @b41sh in #14246
  • chore: refactor the hook by @BohuTANG in #14250
  • chore: update doc about compatibility changes by @drmingdrmer in #14260
  • chore(planner): add TPC-DS join order sqllogictest by @Dousir9 in #14244
  • chore(ci): fix docker push tags by @everpcpc in #14273
  • chore: use raw sql as task SQL instead of pretty one by @ZhiHanZ in #14281
  • chore(query): AlterUdf and AlterShareTenants needs alter global priv by @TCeason in #14284

New Contributors

  • @Freejww made their first contribution in #14143

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.279...v1.2.290

Jan 2, 2024 (v1.2.279)

What's Changed

Exciting New Features ✨

  • feat: support abs for decimal types by @sundy-li in #14195

Thoughtful Bug Fix 🔧

  • chore(query): refresh hook does not return an error by @b41sh in #14194

Code Refactor 🎉

  • refactor: use Binaryfuse16 and disable feature uniform-random by @xudong963 in #14187
  • refactor: Implement exponential backoff in meta-service txn retries by @drmingdrmer in #14196
  • refactor: Remove Compatible error by @drmingdrmer in #14198

Build/Testing/CI Infra Changes 🔌

  • ci: add ee management mode test by @zhyass in #14188

Others 📒

  • chore(query): better handling of nullable columns during filter execution by @Dousir9 in #14197

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.275...v1.2.279

Dec 30, 2023 (v1.2.275)

What's Changed

Exciting New Features ✨

  • feat(query): Support create/alter/drop/desc/show password policy by @b41sh in #14012
  • feat: tweak auto compaction strategy by @zhyass in #13577
  • feat: support reading partitioned Delta table. by @youngsofun in #14084
  • feat: create function support lambda by @guojidan in #14104
  • feat: make Top-N sort can be spilled. by @RinChanNOWWW in #14131
  • feat: support to build DAG on task DDL by @ZhiHanZ in #14141
  • feat: copy from parquet support missing_field_as. by @youngsofun in #14149
  • feat: Bump OpenDAL to v0.43.0 by @Xuanwo in #13978
  • feat(expr): add Binary type by @andylokandy in #14142
  • feat(rbac): treat the empty ownership as owned by ACCOUNT_ADMIN instead of PUBLIC by @flaneur2020 in #14112
  • feat: support more runtime filter types by @xudong963 in #13993
  • feat: add stream_status http api by @zhyass in #14185
  • feat(query): new filter execution framework by @Dousir9 in #13846

Thoughtful Bug Fix 🔧

  • fix(executor): fix broken profile graph for union all by @zhang2014 in #14107
  • fix(query): add udf name(in meta) into UdfServerCall by @TCeason in #14105
  • fix: select stream limit get error by @zhyass in #14086
  • fix(query): grant object use table/db id by @TCeason in #14016
  • fix: hash join hang by @xudong963 in #14157
  • fix(query): ownership only support grant or revoke to role by @TCeason in #14160
  • fix: trim block metas pruned by runtime filter by @dantengsky in #14166
  • fix(expr): fix cast map column validity during evaluation by @andylokandy in #14113
  • fix(query): fix parse string to JSON value by @b41sh in #14177
  • fix(executor): fix broken query profiling graph by @zhang2014 in #14183
  • fix(query): fix cast domain values did not perform rounding operation by @b41sh in #14158

Code Refactor 🎉

  • refactor: refactor error message return for drop table and set options by @JackTan25 in #14078
  • refactor: refactor create agg index by @ariesdevil in #14085
  • refactor: refine if enable join spill by @xudong963 in #14130
  • refactor: integrate HeapMerger to TransformMultiSortMerge. by @RinChanNOWWW in #14065
  • refactor: make sync agg index default by @ariesdevil in #14140
  • refactor: delegate backoff handling to Openraft by @drmingdrmer in #14154
  • refactor: move to meta-service handshake into pool by @drmingdrmer in #14161
  • refactor: build_block method by @xudong963 in #14164
  • refactor(planner): Correct usage of exchange operator by @leiysky in #14163

Build/Testing/CI Infra Changes 🔌

  • ci: add lychee links checker with cache by @PsiACE in #14100

Documentation 📔

  • docs: replace the wrong links in README by @dqhl76 in #14090
  • docs: add compatibility to meta/README.md by @drmingdrmer in #14128

Others 📒

  • chore(query): desugar round/truncate by @sundy-li in #14087
  • chore(query): Rename table not allow modify catalog or database by @TCeason in #14098
  • chore: add more comments for auto compact and remove the settings table goldenfile test by @BohuTANG in #14101
  • chore: handle error returned from resolve_data_source() by @lichuang in #14097
  • chore(query): add log on slow http query request by @flaneur2020 in #14082
  • chore: the final merge sort run outputs blocks directly. by @RinChanNOWWW in #14091
  • chore(query): improve concat boolean types by @Dousir9 in #14096
  • chore(query): round the results for decimal division by @sundy-li in #14109
  • chore: refine enable_compact_after_write, enable_recluster_after_write need deprecated by @BohuTANG in #14114
  • chore: rename CompactOptions::limit to num_segment_limit by @dantengsky in #14116
  • chore: add limit for compact hook by @zhyass in #14115
  • chore: remove the compact hook warn message and rename auto_compaction_threshold to auto_compaction_imperfect_blocks_threshold by @BohuTANG in #14117
  • chore: remove flaky test(cluster) in 01_0013_system_metrics.test by @BohuTANG in #14123
  • chore(planner): support enumerating required properties by @leiysky in #14121
  • chore(ast): support expr in the position of function parameter by @andylokandy in #14122
  • chore: improve the error message by GPT4 by @BohuTANG in #14133
  • chore(expr): cleanup code for kernels/group_by by @andylokandy in #14136
  • chore: remove the unused setting:enable_recluster_after_write by @BohuTANG in #14146
  • chore(query): virtual columns not pushdown if not created by @b41sh in #14147
  • chore: remove onlyif mysql in tests by @xudong963 in #14170
  • chore(query): remove some onlyif mysql in test by @TCeason in #14168
  • chore(executor): add error when dump running graph by @zhang2014 in #14173
  • chore(executor): add tests for schedule queue by @dqhl76 in #14180
  • chore: rename hash join probe state name by @xudong963 in #14186

New Contributors

  • @dqhl76 made their first contribution in #14090

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.262...v1.2.275

Dec 20, 2023 (v1.2.262)

What's Changed

Exciting New Features ✨

  • feat: auto enable change tracking when create stream by @zhyass in #13858
  • feat: support qualify clause by @ariesdevil in #13829
  • feat: matched only and insert only pipeline separate by @JackTan25 in #13680
  • feat: Load credential while role_arn has been specified by @Xuanwo in #13875
  • feat(query): support take_ranges by @Dousir9 in #13878
  • feat: add json_path_match & @?,@@ operators by @akoshchiy in #13906
  • feat: add merge status by @JackTan25 in #13903
  • feat: support connection in stage related infer_schema by @guojidan in #13890
  • feat: iceberg table engine. by @youngsofun in #13835
  • feat(query): add TransformFilter by @Dousir9 in #13922
  • feat: mask admon_key_id && secret_access_key by @guojidan in #13916
  • feat(query): Add warnings in the query response by @flaneur2020 in #13918
  • feat: Try to use volo thrift for async support by @Xuanwo in #13921
  • feat(cluster): support compression for query flight data by @zhang2014 in #13934
  • feat(planner): support timeout termination for query optimization by @leiysky in #13955
  • feat: add internal stream columns by @zhyass in #13960
  • feat: add table engine delta. by @youngsofun in #13949
  • feat: add new processor TransformSortSpill to handle external sort. by @RinChanNOWWW in #13938
  • feat: add kill query when extend lock failure by @zhyass in #13980
  • feat(executor): support output query plan profiles to logs by @zhang2014 in #13945
  • feat(query): add config for query profile log by @everpcpc in #13987
  • feat: enable sort spilling when memory reach the limit. by @RinChanNOWWW in #13989
  • feat: support maximum check by @guojidan in #13992
  • feat(meta): support use id get table/db_name by @TCeason in #14006
  • feat: show locks by @zhyass in #14004