Databend Releases
This page provides information about recent features, enhancements, and bug fixes for Databend.
Jan 2, 2025 (v1.2.680)
What's Changed
Exciting New Features ✨
- feat(query): optimize min/max aggregation use accurate column stats by @sundy-li in #16270
- feat: new table option
data_retention_period_in_hours
by @dantengsky in #16266 - feat(query): add config udf_server_allow_insecure by @everpcpc in #16290
- feat(query): Inverted index search function support options by @b41sh in #16256
- feat(geo): add geography data type by @forsaken628 in #16286
- feat(query): Support create/drop dictionary ddl. by @Winnie-Hong0927 in #16280
- feat: add endpoint /v1/session/logout and record client session ID in meta. by @youngsofun in #16281
- feat(query): ST_GEOGRAPHYFROMEWKT by @forsaken628 in #16302
- feat: support temporary table by @SkyFan2002 in #16250
- feat(query): Add jq set returning function for processing Variant data with jq filters by @maxjustus in #16288
- feat: Add
KVPbApi::get_id_and_value()
by @drmingdrmer in #16325 - feat: Check essential permissions when creating table with external location by @dantengsky in #16315
- feat(query): add is_local to admin /v1/tables info by @everpcpc in #16329
- feat(query): add is_external to /v1/tables info by @everpcpc in #16336
- feat(query): add plan cache by @sundy-li in #16333
- feat: login and renew return session_token_validity_in_secs too. by @youngsofun in #16313
- feat(query): add discovery nodes api by @zhang2014 in #16353
- feat: hilbert clustering by @zhyass in #16296
- feat: add error stack to ErrorCode by @andylokandy in #16343
- feat: support vacuum temporary table by @SkyFan2002 in #16364
- feat: http handler support temp table. by @youngsofun in #16375
- feat: batch size hint of stream consumption by @zhyass in #16372
- feat(query): make external server parallel by batch by @sundy-li in #16390
- feat: estimate selectivity by table sample by @xudong963 in #16362
- feat: introduce an intermediate version that is compatible with vacuum2 by @SkyFan2002 in #16354
- feat(query): Procedure (Part1) by @TCeason in #16348
- feat: add limit push down rule by @xudong963 in #16403
- feat: discovery_nodes not need auth. by @youngsofun in #16392
- feat(query): Support SHOW VARIABLES by @TCeason in #16409
- feat: compact after merge sort by @zhyass in #16401
- feat(query): Support access Redis data from dictionaries via the
dict_get
function. by @Winnie-Hong0927 in #16389 - feat: continue vacuum drop table on per-table cleanup failures by @SkyFan2002 in #16424
- feat: Load DATABEND_ENTERPRISE_LICENSE_EMBEDDED while building by @Xuanwo in #16440
- feat(query): Support access Mysql data from dictionaries via the
dict_get
function. by @Winnie-Hong0927 in #16444 - feat: Auto compact(re-cluster) for multiple table insertion statement by @SkyFan2002 in #16443
- feat(query): Procedure Part2 support arguments by @TCeason in #16453
- feat(query): refactor window spill by @Dousir9 in #16448
- feat(query): push rank limit into aggregate partial node by @sundy-li in #16466
- feat: add eliminate union optimizer rule by @xudong963 in #16478
- feat(query): parallel k way merge sort by @forsaken628 in #16340
- feat: add fast path for cross join to optimize tpcds q23 (~50x) by @xudong963 in #16504
- feat(catalog): ignore error when listing databases by @BohuTANG in #16514
- feat: limit stage name to plain_identifier by @youngsofun in #16522
- feat(query): Initial implementation of postgres-compatible number
to_char
function by @forsaken628 in #16507 - feat(query): Procedure support drop if exists and create or replace by @TCeason in #16500
- feat(query): DATE_ADD Functions to support week as a unit by @TracyZYJ in #16530
- feat: http handler add X-DATABEND-VERSION in each response. by @youngsofun in #16518
- feat: short sql length setting by @arkzuse in #16502
- feat: unset table options by @dantengsky in #16544
- feat(query): window partition by spill to disk by @forsaken628 in #16441
- feat(query): Simplified version Pattern S/L/D/G for number
to_char
by @forsaken628 in #16569 - feat(query): add json array lambda functions by @sundy-li in #16573
- feat(query): support
describe user
statement by @b41sh in #16579 - feat: copy option 'pattern' support variable. by @youngsofun in #16525
- feat(query): support json array functions by @b41sh in #16571
- feat(query): add date_diff function by @chagelo in #16560
- feat(query): Reduce serialisation memory usage when spilling to local disk by @forsaken628 in #16580
- feat(query): Support
show dictionaries
DQL. by @Winnie-Hong0927 in #16602 - feat: support sampling table with block and row level simultaneously by @xudong963 in #16613
- feat: add unload options
overwrite
,include_query_id
anduse_raw_path
. by @youngsofun in #16614 - feat(function): Add mode Aggregate function. by @Freejww in #16627
- feat(query): Support use parquet format when spilling by @forsaken628 in #16612
- feat(query): support
json_object_insert
function by @b41sh in #16636 - feat: http handler support request forwarding. by @youngsofun in #16637
- feat: Implemented a new sql
explain analyze graphical
by @Maricaya in #16543 - feat: add deterministic block level sampling for small datasets by @xudong963 in #16670
- feat: support explain decorrelated plan by @xudong963 in #16681
- feat: add ARRAY_SIZE alias for ARRAY_LENGTH by @ding-young in #16685
- feat(query): support
json_object_delete
andjson_object_pick
function by @b41sh in #16682 - feat: Bump OpenDAL to 0.50.1 by @Xuanwo in #16661
- feat: support map lambda functions by @Dragonliu2018 in #16683
- feat(query): add api /v1/verify by @everpcpc in #16665
- feat: support calculating table storage size(including time travel) by @SkyFan2002 in #16698
- feat(query): add table function udf_echo && expose more metrics of udf client by @sundy-li in #16696
- feat(query): Settings clause by @TCeason in #16669
- feat: implement
is_not_null
selectivity based on null count in stats by @xudong963 in #16730 - feat: HTTP handler use cookie to pass session id to and from clients. by @youngsofun in #16735
- feat: impl opendal::HttpFetch for StorageHttpClient by @everpcpc in #16738
- feat(query): function about convert_timezone #16177 by @florann in #16181
- feat(functions): add new function: map_pick by @hanxuanliang in #15573
- feat(metrics): new storage_http_requests_count by @everpcpc in #16769
- feat(functions): add map_insert function by @hanxuanliang in #15567
- feat: filter null value before join by @xudong963 in #16722
- feat(query): Support rename dictionary ddl. by @Winnie-Hong0927 in #16754
- feat(query): add last_day, previous_day, next_day function by @Freejww in #16740
- feat: implement StringColumn using StringViewArray by @andylokandy in #16610
- feat: Implement WAL-based RaftLog storage by @drmingdrmer in #16776
- feat(query): expand trim/ltrim/rtrim by @TCeason in #16802
- feat: support pushdown predicate into iceberg engine by @sundy-li in #16650
- feat(query): TopN window operator by @forsaken628 in #16726
- feat(query): support SHOW DROP DATABASES by @TCeason in #16811
- feat: uuid function from v4 to v7 by @BohuTANG in #16827
- feat: refine the rule
PushDownFilterWindowTopN
in case top n is equal to 0. by @forsaken628 in #16830 - feat: add aws glue as an iceberg connection type by @Rowlandev in #16824
- feat(query): add distributed pruning settings by @zhang2014 in #16864
- feat: databend-meta adds more metrics about raft-log by @drmingdrmer in #16884
- feat(query): speed up fetching redis data from dictionaries via mget. by @Dragonliu2018 in #16766
- feat(query): add new function
TO_TIMESTAMP<int, scale>
by @TCeason in #16924 - feat: disable specifying copy options when create stage. by @youngsofun in #16925
- feat(doc): Introducing Databend Guru on Gurubase.io by @kursataktas in #16773
- feat(query): Support geometry relation functions by @b41sh in #16927
- feat(query): virtual column allow cast to other type by @b41sh in #16903
- feat: copy support option COLUMN_MATCH_MODE by @youngsofun in #16963
- feat(query): speed up fetching mysql data from dictionaries via batch processing. by @Dragonliu2018 in #16948
- feat: Support USE CATALOG syntax and current_catalog() function by @flashmouse in #16926
- feat(query): Support
st_collect
aggregate function by @b41sh in #16996 - feat(query): Support
flight_connection_max_retry_times
andflight_connection_retry_interval
setting by @b41sh in #16856 - feat: show dropped table and data_retention_time_in_hours in fuse_tim… by @SkyFan2002 in #17015
- feat(rbac): add GrantObject::Warehouse by @TCeason in #17029
- feat(query): partition sort spill by @forsaken628 in #16987
- feat: support vacuum leaked table data by @SkyFan2002 in #17022
- feat(query):
to_binary
function support variant, bitmap, geometry, geography types by @b41sh in #17026 - feat(query): add interval type by @TCeason in #16990
- feat: add key prefix count assertions to transactions by @drmingdrmer in #17053
- feat(query): null type infer as Nullabel(String) by default in create as statement by @TCeason in #17052
- feat(query): support global network policy by @everpcpc in #17050
- feat(query): add config network_policy_whitelist by @everpcpc in #17065
- feat(query): pre-check cast type by @andylokandy in #16836
- feat(functions): add jaro_winkler string similarity function by @maxjustus in #16993
- feat(query): add config jwks_refresh_interval & jwks_refresh_timeout by @everpcpc in #17087
- feat: databend-meta transaction support generic bool-expression and else-if chain by @drmingdrmer in #17064
- feat(query): refactor vacuum temp files by @sundy-li in #17089
- refactor: Serialize iceberg table inside engine options to reduce catalog requests by @Xuanwo in #17049
- feat(query): support interval function by @TCeason in #17093
- feat: new settings
stream_consume_batch_size_hint
by @dantengsky in #17102 - feat(query): support javascript/python script User Defined Aggregate Function by @forsaken628 in #17054
- feat: databend-meta watch API provides
initial_flush
field by @drmingdrmer in #17098 - feat(query): support javascript/python script User Defined Aggregate Function by @forsaken628 in #17108
- feat(query): support interval 'string' by @TCeason in #17105
- feat(query): Add
external_block_rows
metrics by @b41sh in #17116 - feat: support vacuum leaked table data (#17022)-Revert by @dantengsky in #17123
Thoughtful Bug Fix 🔧
- fix: max_wait_time in http handler not work when result set is large … by @youngsofun in #16267
- fix(query): enable distribute query for system log table by @zhang2014 in #16300
- fix(query): fix child_has_window by @sundy-li in #16301
- fix: uri path used as both path and root. by @youngsofun in #16321
- fix: vacuum dry run should carry on when target objects are missing by @dantengsky in #16322
- fix(query): fix session leak in ctas by @zhang2014 in #16337
- fix: When Replacing column mask, old
name->id->value
should be completely deleted by @drmingdrmer in #16328 - fix: reduce sample possibility to avoid flaky test by @xudong963 in #16330
- fix(query): fix IFNULL by @sundy-li in #16339
- fix(query): fix wrong result for join system tables in cluster mode by @zhang2014 in #16342
- fix(query): fix try cast by @sundy-li in #16341
- fix(query): fix binary default string by @sundy-li in #16345
- fix: push filter down join with materialized cte by @xudong963 in #16349
- fix(query): fix aggregate panic in cluster mode by @zhang2014 in #16319
- fix(query): fix incorrect alias of select items by @sundy-li in #16368
- fix: separate sessions for temporary tables with memory engine by @SkyFan2002 in #16374
- fix(executor): enable backpressure for union all by @zhang2014 in #16388
- fix: hilbert_index function panic by @zhyass in #16393
- fix(iceberg): Handling options like "s3.region" by @Xuanwo in #16404
- fix: non-default catalog should not call get_table_function by @Xuanwo in #16410
- fix(query): fix stack overflow for optimizer by @zhang2014 in #16420
- fix: add custom (de)serialization methods for special float value by @dqhl76 in #16258
- fix: table lock flaky test by @zhyass in #16429
- fix(query): The columns permission of the system library is misjudged. by @TCeason in #16431
- fix: restore parser backrtace when map_res failed by @andylokandy in #16434
- fix(iceberg): Table desc should allow nested db_name by @Xuanwo in #16414
- fix: split if block too big during append by @zhyass in #16435
- fix: flaky test 09_0036_merge_into_without_dist by @dantengsky in #16457
- fix: wrong order by result by @xudong963 in #16422
- fix: Multi-thread recreate view/table gives error by @zhyass in #16465
- fix: log query detail and profile with identical format by @everpcpc in #16469
- fix: revert log format breakage from #16249 by @andylokandy in #16470
- fix: typo for query & profile logging config by @everpcpc in #16476
- chore: add extra bracket for
and
andor
to make explain clear by @xudong963 in #16494 - fix(query): fix planner cache cause modify async function does not take effect by @b41sh in #16510
- fix:
ignore_result
doesn't have an effect on union by @xudong963 in #16515 - fix(query): fix copy into table with disable distributed copy into by @zhang2014 in #16529
- fix(query): fix column leaf_index by @Dousir9 in #16537
- fix: Only when all tables are included, the db can be dropped by @drmingdrmer in #16539
- fix: move TableInfo building out of SchemaApi by @drmingdrmer in #16548
- fix: identify failed vacuum db by id by @drmingdrmer in #16553
- fix(query): fix json object compare cause panic by @b41sh in #16555
- fix: direct io syscall not available on mac by @forsaken628 in #16574
- fix: window in scalar subquery returns wrong results by @xudong963 in #16567
- fix: avoid eliminating the left child of union in
RuleEliminateUnion
by @xudong963 in #16584 - fix(query): fix inverted index panic caused by null value by @b41sh in #16576
- fix(query): fix cannot obtain stack trace when segmentation fault by @zhang2014 in #16566
- fix: merge into panic by @zhyass in #16581
- fix(query): list_dictionaries should not ignore db_id by @TCeason in #16587
- fix(iceberg): Make sure iceberg table has been loaded by @Xuanwo in #16593
- fix: Disable
prewhere
for iceberg table for now by @Xuanwo in #16597 - fix(query): fix left outer join wrong result by @Dousir9 in #16601
- fix: subquery returns wrong results under cluster by @xudong963 in #16606
- fix(query): ntile frame should be unbounded by @TCeason in #16604
- fix: lock mgr may failed to release lock in time by @dantengsky in #16611
- fix(query): support subquery in pivot by @Dragonliu2018 in #16631
- fix(query): check_timestamp/date -> clamp_timestamp/date by @TCeason in #16634
- fix: deadlock in table lock by @zhyass in #16632
- fix(query): sort spilling use arrow file format painc by @forsaken628 in #16658
- fix(query): create table need fail if storage format is invalid by @TCeason in #16663
- fix(query): forbid explain explain statement by @TCeason in #16654
- fix(storage): fix refresh virtual column using async api by @b41sh in #16656
- fix(query): insert with long sql parse error by @sundy-li in #16678
- fix(ci): flaky test by @zhyass in #16664
- fix: insert long sql again by @sundy-li in #16684
- fix(query): sort spilling may hang by @forsaken628 in #16672
- fix:
JEAllocator
should properly handle nullptr returned by @dantengsky in #16692 - fix(metrics): use f64::MAX as histogram bound to display +Inf properly by @flaneur2020 in #16701
- fix(query): use custom connector for udf client by @everpcpc in #16697
- fix: MetaGrpcClient deadlock when drop by @drmingdrmer in #16727
- fix: date_add/sub need respect timezone by @TCeason in #16721
- fix(query): remove single quotes for system.columns comment by @TCeason in #16732
- fix: HTTP handle lost rows when result set is wide and large. by @youngsofun in #16748
- fix: keep column statistics of all NULL column by @dantengsky in #16753
- fix:
Files::remove_file_in_batch
should not swallow errors by @dantengsky in #16761 - fix(query): add tls config for udf client by @everpcpc in #16782
- fix: token and cookie session id miss match. by @youngsofun in #16786
- fix: clustering_information panic if string domain is none by @zhyass in #16792
- fix(query): fix broken file log layout by @zhang2014 in #16803
- fix: re-ensure dir before upgrading by @drmingdrmer in #16805
- fix: purge may not work on tables after a flash back operation by @SkyFan2002 in #16812
- fix(storage): recluster endless loop by @zhyass in #16831
- fix: incorrect table data disk cache key by @dantengsky in #16837
- fix(query): fix group by with alias column can't bind the column by @b41sh in #16804
- fix(cluster): fix the possibility of connection leak when the cluster state is broken. by @zhang2014 in #16842
- fix(query): fix aggregator_groups_builder to work with string view by @sundy-li in #16843
- fix(query): check storage request method with arg list-type by @everpcpc in #16849
- fix: replace risky
dma_buffer_as_vec
implementations by @forsaken628 in #16829 - fix(query): fix and check total_buffer_len and total_bytes_len by @sundy-li in #16854
- fix: raft-log wont write when exceeding 256M by @drmingdrmer in #16876
- fix(query): fix distinct set-returning function by @b41sh in #16883
- fix(query): fix incorrect total_bytes_len in string view by @sundy-li in #16877
- fix: StreamVersionMismatched when using CTAS in multi statement trans… by @SkyFan2002 in #16889
- fix(ci): flaky test by @zhyass in #16898
- fix(storge): write progress in compact hook by @zhyass in #16901
- fix: ensure atomicity in create_lock_revision by @zhyass in #16907
- fix: dropped temporary table should not be shown in system.temporary_tables by @SkyFan2002 in #16911
- fix(query): join predict use cast_expr_to_non_null_boolean by @sundy-li in #16937
- fix(ci): flaky test by @zhyass in #16933
- fix(query): add_hours function may panic if the argument is too big by @TCeason in #16929
- fix: lock attach table's snapshot location by @dantengsky in #16952
- fix(ci): fix release dbg upload path by @zhang2014 in #16958
- fix(query): make memory engine as non-local table by @sundy-li in #16955
- fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #16974
- fix: fix segfault in list_domain by @forsaken628 in #16979
- fix(query): keep remaining_predicates when filtering grouping sets by @sundy-li in #16971
- fix(query): fix bug while mysql external dictionary table contains null values by @Dragonliu2018 in #16978
- fix(sqlparser): fix order by followed by scalar by @chagelo in #16967
- fix(query): group by item allow set returning functions by @b41sh in #16986
- fix: http handler set cookie only when cookie_enabled=true. by @youngsofun in #16992
- fix: need_sticky field was incorrectly set to true. by @youngsofun in #16997
- fix(ci): fix benchmark ci failure by @zhang2014 in #17005
- fix: insert and mutation progress by @zhyass in #17014
- fix(query): reuse connection to fix dictionary mysql flaky test by @b41sh in #17016
- fix(ci): flaky test by @zhyass in #17030
- fix: filter push down union in r cte by @xudong963 in #17031
- fix: drop channel if partition receiver finished by @dqhl76 in #17037
- fix: handle trailing zeros in Raft log WAL with EXT4 writeback mode by @drmingdrmer in #17042
- fix(query): fix read small parquet files in cluster mode by @sundy-li in #17063
- fix(query): fix thread leak by @zhang2014 in #17077
- fix(query): fix thread leak and remove all usage by @dqhl76 in #17081
- fix: change tracking stream failed by @zhyass in #17072
- fix(query): fix join on not null column cause panic by @b41sh in #17069
- fix: create as query respect ddl_column_type_nullable by @TCeason in #17071
- fix: check if the table option is valid according to the engine by @SkyFan2002 in #17076
- fix(ci): flaky test by @zhyass in #17090
- fix(query): double panic if broken state in block builder by @zhang2014 in #17091
- fix(query): Correct sign extension handling in months_days_micros struct by @TCeason in #17086
- fix(query): union all panic in mysql client by @zhyass in #17095
- fix: name resolution bug in case-sensitive mode by @zhyass in #17097
- fix(query): fix a number of bugs in the spill config. by @forsaken628 in #17103
- fix(query): uri path should not be percent-encoded by @youngsofun in #17109
- fix(query): fix default values of
SpillConfig
may not take effect by @forsaken628 in #17113 - fix(query): table field data type in
show create table
incorrect by @forsaken628 in #17112 - fix(query): fix udf script return type is arrow
DataType::LargeList
caused failed by @b41sh in #17127 - fix: ignore case when matching function name by @notauserx in #16912
- fix: analyze table contain keyword bug by @zhyass in #17131
- fix(query): Disabled the shrink_scalar optimization for cast expressions during the resolve procedure args by @TCeason in #17119
- fix(log): prefix_filter not working. by @youngsofun in #17136
Code Refactor 🎉
- refactor: replace absolute expiration time with relative TTL in schema API by @drmingdrmer in #16268
- refactor(query): remove lru cache meter by @zhang2014 in #16260
- refactor: Use global http client to share the connection pool by @Xuanwo in #16276
- refactor(query): refactor hash join spill by @Dousir9 in #15746
- refactor: change
DatabaseInfo.meta
toSeqV<DatabaseMeta>
by @drmingdrmer in #16295 - refactor: change value of meta-service key
DatabaseNameIdent
fromDatabaseId
toId<DatabaseId>
by @drmingdrmer in #16299 - refactor: adopt
get_pb()
for get_background_task by @drmingdrmer in #16308 - refactor: Reduce parallelism when vacuuming dropped tables by @dantengsky in #16312
- refactor(query): add setting external_server_request_retry_times by @sundy-li in #16307
- refactor(query): refactor bind set returning functions by @b41sh in #16316
- refactor(query): add config enable_meta_data_upgrade_json_to_pb_from_v307 by @TCeason in #16306
- refactor: add associated type kvapi::Value::KeyType by @drmingdrmer in #16320
- refactor: kv-app-errors does not need serde by @drmingdrmer in #16324
- refactor(query): refactor bind async functions by @b41sh in #16331
- refactor: Catalog Manager API improvements for create and drop operations by @drmingdrmer in #16338
- refactor: Add BaseApi to abstract usually used meta-service access pattern by @drmingdrmer in #16351
- refactor: simplify SchemaApi::list_indexes(), get_index() list_background_jobs() by @drmingdrmer in #16357
- chore(query): optimize window sort by @sundy-li in #16355
- chore(query): reduce hash join memory usage by @Dousir9 in #16359
- refactor: simplify drop_index() with NameIdValueApi by @drmingdrmer in #16366
- refactor: simplify dictionary API by @drmingdrmer in #16371
- refactor: Add item count check to
KVPbApi::get_pb_stream()
by @drmingdrmer in #16377 - refactor(query): optimize show drop tables by @TCeason in #16370
- chore(query): Simplified sort simple rows implementation by @forsaken628 in #16382
- refactor(query): refactor some codes of parquet2 by @sundy-li in #16386
- refactor: remove all share related functions by @drmingdrmer in #16383
- refactor: replace get_pb_value() with get_pb() by @drmingdrmer in #16395
- refactor: Add NameValueApi, simplify virtual column APIs by @drmingdrmer in #16398
- refactor: simplify catalog API by @drmingdrmer in #16406
- refactor: Simplify table lock API. by @drmingdrmer in #16413
- chore(query): introduce EmptyResultScan plan by @sundy-li in #16411
- refactor: remove unused
TableInfo.tenant
by @drmingdrmer in #16416 - refactor: simplify least-visible-time schema API, add LeastVisibleTimeIdent by @drmingdrmer in #16418
- refactor: add mget_id_value_compat() by @drmingdrmer in #16425
- refactor(query): first check privilege in SystemEngine get_full_data by @TCeason in #16421
- chore(query): limit max concurrency of spill io requests by @zhang2014 in #16442
- chore(query): make string to int respect behavior like PG by @sundy-li in #16428
- refactor: simplify vacuum drop table handling by @drmingdrmer in #16446
- chore: use logforth to replace fern by @andylokandy in #16249
- refactor(executor): use event cause to refactor shuffle processor by @zhang2014 in #16445
- refactor: cleanup logic to get tables to vacuum by @drmingdrmer in #16450
- refactor: enhancements of HTTP client session. by @youngsofun in #16452
- refactor(query): sort kernel optimization by @sundy-li in #16458
- refactor(storage): improve inverted index read fst file first to reduce load index by @b41sh in #16385
- refactor: refactor vacuum by @drmingdrmer in #16454
- chore: simplify get_sequence() and get_sequence_next_value() by @drmingdrmer in #16463
- refactor: add DatabaseMeta.gc_in_progress by @drmingdrmer in #16464
- refactor(query): use pipeline exchange to refactor window scatter by @zhang2014 in #16471
- refactor(query): simplify expression kernel by @sundy-li in #16482
- refactor: add KVPbCrudApi as abstraction layer for commonly used CRUD operations by @drmingdrmer in #16479
- refactor: add KVPbCrudApi::crud_try_insert() and crud_try_upsert() by @drmingdrmer in #16484
- refactor: SchemaApi::list_databases() by @drmingdrmer in #16485
- refactor: SchemaApi::get_database_history() by @drmingdrmer in #16487
- refactor: test SchemaApi::get_tenant_history_database() by @drmingdrmer in #16488
- refactor(query): refactor license manager by @zhang2014 in #16489
- refactor(query): refactor license manager by @zhang2014 in #16492
- refactor: simplify
DroppedId
. Replace manual comparison with string comparison by @drmingdrmer in #16495 - refactor: update dependency datafusion-orc. by @youngsofun in #16491
- refactor: get_tableinfos_by_ids() by @drmingdrmer in #16503
- chore(query): refactor like function by @sundy-li in #16493
- chore: simplify SchemaApi::truncate_table() by @drmingdrmer in #16506
- refactor: remove get_pb_value() by @drmingdrmer in #16516
- refactor: Output whole error message with source while retry by @Xuanwo in #16519
- refactor: unify path of write segment by @SkyFan2002 in #16517
- refactor: Improve gc_dropped_table_by_id() method by @drmingdrmer in #16528
- refactor: dropped table listing and GC by @drmingdrmer in #16531
- refactor: gc_dropped_db_by_id() by @drmingdrmer in #16538
- refactor: SchemaAPI::do_get_table_history by @drmingdrmer in #16540
- refactor:
DroppedId
for listing db/tables for gc by @drmingdrmer in #16542 - refactor:
get_history_tables_for_gc()
should not returnTableInfo
, but just table name, id and values by @drmingdrmer in #16545 - refactor(storage): improve inverted index match phrase query by @b41sh in #16547
- refactor: Automatically chunk large key sets in KVPbApi methods by @drmingdrmer in #16561
- refactor: simplify get_tables_history() by @drmingdrmer in #16572
- refactor(iceberg): Use iceberg file IO instead of our own operator by @Xuanwo in #16577
- refactor: add SchemaApi::get_table_in_db by @drmingdrmer in #16582
- refactor: SchemaApi::get_table_meta_history() by @drmingdrmer in #16586
- refactor: fix error message. by @youngsofun in #16578
- refactor(storage): inverted index use merge io to read data by @b41sh in #16589
- refactor: isolate session_id and session states of diff users. by @youngsofun in #16592
- refactor: Unify the set and unset structures by @TCeason in #16622
- refactor(query): make delta table support partition prunning by @sundy-li in #16621
- refactor(storage): inverted index support calculate the score by @b41sh in #16609
- refactor: some http APIs do not need sticky. by @youngsofun in #16644
- refactor: delete user_procedure and refactor procedure interpreter by @TCeason in #16532
- refactor(query): use arrow-flight to exchange datas by @sundy-li in #16657
- refactor: refine cast variant to map by @andylokandy in #16691
- refactor: streamline system.temp_files table scan by @dantengsky in #16659
- refactor: Align versions for arrow, opendal, hyper, tonic, prost by @Xuanwo in #16704
- refactor: refine column statistic map serialization by @dantengsky in #16728
- refactor: MetaGrpcClient needs to specify timeout to auto-reconnect to restarted server by @drmingdrmer in #16733
- refactor: replace direct GitHub dependencies with published crates.io versions by @drmingdrmer in #16700
- refactor: Enforce all deps must be declared at root by @Xuanwo in #16741
- refactor: Remove not used
async-std
by @Xuanwo in #16745 - refactor: change blocking pruning process to non-blocking channel by @dqhl76 in #16718
- refactor: ensure last-purged log-id cursor in exported from databend-meta service by @drmingdrmer in #16759
- refactor: Consolidate
MetaStorageError
variants by @drmingdrmer in #16760 - refactor(storage): refactor read virtual columns by @b41sh in #16758
- refactor: remove
location_prefix
, which is no longer supported by @dantengsky in #16757 - refactor(base): add stacktrace to replace backtrace by @zhang2014 in #16643
- refactor: when upgrading, do not import already purged garbage log entries by @drmingdrmer in #16809
- refactor: align metrics field names with current behavior by @drmingdrmer in #16826
- refactor(base): support stacktrace in cluster mode by @zhang2014 in #16821
- refactor: refactor cte binder and fix materialized cte used in subquery by @xudong963 in #16785
- refactor: remove EmbeddedMeta by @drmingdrmer in #16847
- refactor: remove unused sled related codes by @drmingdrmer in #16857
- refactor: make ListKV non-blocking by returning stream directly by @drmingdrmer in #16868
- refactor: optimize query system.tables when query single table by @TCeason in #16869
- refactor(query): refactor geometry functions by @b41sh in #16870
- refactor: use jiff crate replace chrono in date/timestamp func by @TCeason in #16787
- refactor: add new raft-log metrics to "metactl status" response by @drmingdrmer in #16899
- refactor: remove obsolete gRPC read API by @drmingdrmer in #16909
- refactor: deprecate: Operation::AsIs and will be removed by @drmingdrmer in #16913
- refactor(sqlsmith): refactor sqlsmith using http client by @b41sh in #16890
- refactor: remove ValueRef and use Value instead by @andylokandy in #16891
- refactor: make test_watch_expired_events() less sensitive to inaccurate time by @drmingdrmer in #16966
- refactor: use temp table to refactor materialized cte by @xudong963 in #16900
- refactor: set path of cookie. by @youngsofun in #17008
- refactor: replace
RangeMap
withspan-map
for better range lookup by @drmingdrmer in #17006 - refactor: revert pr #16869 by @TCeason in #17012
- refactor: auto split large mget_database_names_by_ids into chunks with
KVPbApi::get_pb_values_vec()
by @drmingdrmer in #17011 - refactor: mget db names and table names remove duplicate ids to avoid too many parameters by @b41sh in #17013
- refactor: integrate fuse table block pruning into pipeline by @dqhl76 in #16841
- refactor: optimize query system.tables when query single table (#16869) by @TCeason in #17021
- refactor(cluster): refactor cluster for dyn cluster [part 1] by @zhang2014 in #17023
- refactor(query): refactor udf server batch rows by @b41sh in #17092
- refactor: set name for all runtime worker threads. by @youngsofun in #17084
- refactor(query): refactor udf server connection reuse endpoint by @b41sh in #17106
- refactor: simplify alter and drop cluster key logic by @zhyass in #17128
- refactor: integrate segment pruning into pipeline by @dqhl76 in #17126
Build/Testing/CI Infra Changes 🔌
- ci: Add iceberg tpch sqllogictests by @Xuanwo in #16598
- ci: Enable incremental back to see how it works by @Xuanwo in #16817
- ci: fix flaky test by @youngsofun in #16945
- ci: ignore RUSTSEC-2024-0402. by @youngsofun in #16998
Documentation 📔
- docs: add stream to readme by @BohuTANG in #16810
Others 📒
- chore(query): add more snapshot/segment serde tests by @sundy-li in #16273
- chore: Use tagged iceberg crates by @Xuanwo in #16278
- chore: add a setting to decide if table sample is deterministic by @xudong963 in #16275
- chore(ci): tmp disable sharing test by @everpcpc in #16285
- chore(ci): flaky test by @zhyass in #16207
- chore: replace condition and if_then with txn in txn methods by @drmingdrmer in #16289
- chore(query): bump ast 0.0.3 by @sundy-li in #16283
- chore(query): avoid create query context each auth by @zhang2014 in #16279
- chore(query): remove unused parquet2 codes by @sundy-li in #16282
- chore: add setting auto_compaction_segments_limit by @zhyass in #16298
- chore: remove the log about if histogram exists by @xudong963 in #16303
- chore: simplify create_data_mask by @drmingdrmer in #16309
- chore(ci): adjust release time by @everpcpc in #16334
- chore(query): explain ast work like explain syntax by @sundy-li in #16335
- chore: support temporary table with memory engine by @SkyFan2002 in #16332
- chore(query): improve project set by @Dousir9 in #16326
- chore: remove unused serde impl by @drmingdrmer in #16347
- chore: remove empty reply struct by @drmingdrmer in #16350
- chore(query): fix window index out of range by @Dousir9 in #16352
- chore: separate test run for temp_table suite by @SkyFan2002 in #16358
- chore(code): update the version of gix-path in Cargo.lock by @Dousir9 in #16365
- chore(ci): add internal test by @TCeason in #16369
- chore: improve Debug and Display for DataId and TIdent by @drmingdrmer in #16379
- chore: add logging for embedded meta test by @drmingdrmer in #16384
- chore(ci): fix cargo clippy by @TCeason in #16397
- chore: add error stack to cmd entrence by @andylokandy in #16391
- chore(query): add external server request profile by @zhang2014 in #16400
- chore(query): refactor udf retry by @sundy-li in #16405
- chore(query): add connect duration metric for udf by @zhang2014 in #16407
- chore: add join type for explain analyze partial by @xudong963 in #16408
- chore: bump quinn-proto to 0.11.8 for cargo audit pass by @dqhl76 in #16419
- chore(planner): improve infer filter by @Dousir9 in #16361
- chore: log remote addr when meta-service raft API recevies a message by @drmingdrmer in #16438
- chore(query): add column: system.tables.total_columns by @TCeason in #16447
- chore(ci): fix dataset domain by @everpcpc in #16455
- chore(ci): release with musl by @everpcpc in #16460
- chore(executor): support for data exchange between different processors within the pipeline by @zhang2014 in #16467
- chore(executor): more edge detail for executing graph by @zhang2014 in #16468
- chore(ci): remove hdfs release by @everpcpc in #16477
- chore(query): null first behavior by @sundy-li in #16475
- chore: update nom-rule by @andylokandy in #16483
- chore: bump proj4rs 0.1.4 by @b41sh in #16474
- chore: remove logs from block writer by @dantengsky in #16497
- chore: simplify get_gc_table_info() by @drmingdrmer in #16499
- chore(query): replace to datafuse-extras/dtparse by @TCeason in #16509
- chore(storage): hook recluster avoid scan all segments by @zhyass in #16498
- chore: update source repo for async-compression by @everpcpc in #16513
- chore(query): move recursion & backtrace crate to datafuse-extras by @zhang2014 in #16520
- chore(query): add more asserts by @sundy-li in #16536
- chore: update repo for workflows by @everpcpc in #16546
- chore(storage): refactor compact source by @zhyass in #16527
- chore(query): SHOW/DESC is not DDL by @TCeason in #16583
- chore: check lock revision before watch delete by @zhyass in #16590
- chore(ci): flaky test by @zhyass in #16596
- chore(query): improve variable parser by @sundy-li in #16599
- chore: refine table sample tests by @xudong963 in #16600
- chore(query): add more logs on aggregation by @sundy-li in #16552
- chore: reset recluster block size by @zhyass in #16623
- chore(query): add http address info to cluster node info by @zhang2014 in #16620
- chore: drop guard when acquire lock failed by @zhyass in #16616
- chore: revert "lock mgr may failed to release lock in time (#16611)" by @dantengsky in #16630
- chore(query): reduce unsafe codes in kernels by @sundy-li in #16633
- chore(query): add warn log if vacuum temporary file is error by @zhang2014 in #16646
- chore: resolve todo in dynamic sample by @xudong963 in #16629
- chore(query): enable loser tree merge sort default by @forsaken628 in #16649
- chore(query): replace
arrow2
witharrow-rs
inserialize_column
anddeserialize_column
by @forsaken628 in #16655 - chore(query): remove hash join build detail status by @zhang2014 in #16674
- chore: upon startup, meta-service output human-readable error if trying to open future version on-disk data by @drmingdrmer in #16679
- chore(query): modify rbac test result display by @TCeason in #16686
- chore: adjust default value of setting
max_storage_io_requests
by @SkyFan2002 in #16680 - chore(query): returns nan for aggregate_kurtosis and aggregate_skewness by @sundy-li in #16688
- chore: display port number on error from bind to port failure by @lyang24 in #16695
- chore: remove
SortColumnDescription.is_nullable
by @forsaken628 in #16699 - chore: remove transaction keyword by @SkyFan2002 in #16707
- chore: add timeout for
detect_region
operation to prevent hanging request by @dantengsky in #16709 - chore(query): add current_query_id and database id for admin api by @zhang2014 in #16703
- chore(query): index scalar support variant by @sundy-li in #16719
- chore: display log.file.limit and prefix_filter by @TCeason in #16713
- chore: avoid returning error during computing binary comparison selectivity by @xudong963 in #16725
- chore: Simplifying BlockMetaInfo Definitions by @forsaken628 in #16716
- chore: tweak cargo feature "jemalloc" by @dantengsky in #16693
- chore: remove flaky test by @TCeason in #16739
- chore: refine vacuum by @dantengsky in #16743
- chore: Remove not used dep
config
by @Xuanwo in #16746 - chore(query): Introduce VecExt and SliceExt to check unsafe bounds in debug mode by @sundy-li in #16747
- chore: make VACUUM TEMPORARY FILES killable by @dantengsky in #16751
- chore: tweak batch deletion concurrency by @dantengsky in #16770
- chore: refine raft-types import path by @drmingdrmer in #16764
- chore: remove obsolete version by @everpcpc in #16789
- chore: add snapshot location cache for attached table by @dantengsky in #16795
- chore: update meta-service version info and compatibility doc by @drmingdrmer in #16800
- chore(query): embedded public key of license by @zhang2014 in #16813
- chore(query): refactor common/arrow removes arrow/io crates by @sundy-li in #16808
- chore: purge dropped view metadata during vacuum by @dantengsky in #16819
- chore(query): keep queries detail and profile log format by @zhang2014 in #16822
- chore(ci): test with different license by @everpcpc in #16815
- chore(base): rollback to kernel signal handler after dump backtrace by @zhang2014 in #16828
- chore: remove unused codes by @zhyass in #16834
- chore: fix typo in
ColumnCacheKeyBuilder
by @dantengsky in #16838 - chore(cluster): disable parallel commit of cluster tasks by @zhang2014 in #16851
- chore(query): remove SliceExt by @sundy-li in #16853
- chore(ci): separate build profile by @everpcpc in #16855
- chore(query): remove useless new line for query log by @zhang2014 in #16860
- chore: add some log by @SkyFan2002 in #16863
- chore(query): improve distributed runtime filter by @Dousir9 in #16862
- chore: enable stderr logging for
table_meta_inspector
by @dantengsky in #16865 - chore(query): remove common/arrow crate by @sundy-li in #16846
- chore(ci): adjust runner size by @everpcpc in #16879
- chore(settings): enable spill by @Dousir9 in #16888
- chore: remove unused config_id from RaftConfig by @drmingdrmer in #16902
- chore(ci): flaky test by @zhyass in #16917
- chore(ci): enable overflow-checks in ci profile by @sundy-li in #16895
- chore(ci): flaky test by @zhyass in #16919
- chore(query): add snapshot logs in read partitions by @sundy-li in #16918
- chore: bump rustls version 0.23.18 by @TCeason in #16936
- chore(query): spilt binary symbol by @zhang2014 in #16861
- chore:
k way merge sort
add boundary check to prevent process crash by @forsaken628 in #16934 - chore(ci): remove aarch64 musl in release by @zhang2014 in #16946
- chore(planner): improve cardinality estimation by @Dousir9 in #16938
- chore(planner): refine project set cardinality by @Dousir9 in #16944
- chore: support recursive cte with normal cte by @xudong963 in #16932
- chore(binder): add enable_materialized_cte settings by @Dousir9 in #16950
- chore(storage): hook compact avoid scan all segments by @zhyass in #16954
- chore(storage): do compact before recluster during compact hook by @zhyass in #16949
- chore(ci): upgrade bendsql by @everpcpc in #16965
- chore(storage): refine error message for stream read offset snapshot by @zhyass in #16964
- chore: bump ruzstd from 0.7.2 to 0.7.3 by @dqhl76 in #16969
- chore(ci): install bendsql from packages by @everpcpc in #16970
- chore(planner): improve physical join by @Dousir9 in #16962
- chore(ci): fix some flaky test by @TCeason in #16980
- chore(query): improve decimal aggregate state by @sundy-li in #16983
- chore(planner): refine physical join by @Dousir9 in #16988
- chore: adjust fuse_time_travel_size() by @SkyFan2002 in #16995
- chore(ci): use new dsn for create database by @zhang2014 in #16968
- chore(ci): rust toolchain 2024-11-28 build image by @sundy-li in #17001
- chore(query): fix topk in native by @sundy-li in #17003
- chore(query): bump up rust toolchain to 2024-11-28 by @sundy-li in #16981
- chore: tweak attach table refreshing by @dantengsky in #17019
- chore(query): remove old aggregate codes by @sundy-li in #17025
- chore: add more test to meta-service txn value-matching by @drmingdrmer in #17033
- chore(cluster): disable parallel commit for cluster task by @zhang2014 in #17034
- chore: improve error handling in partition sending loop by @dqhl76 in #17035
- chore(ci): remove musl release by @everpcpc in #17041
- chore(query): improve comparison functions by @sundy-li in #17032
- chore(binder): fix bind materialized cte by @Dousir9 in #17046
- chore: refmt crash log message by @TCeason in #17059
- chore: bump dtparse by @TCeason in #17073
- chore: remove unused dev by @zhyass in #17088
- chore(query): Revert "feat(query): support javascript/python script User Defined Aggregate Function" by @sundy-li in #17107
- chore: add more log by @SkyFan2002 in #17110
- chore(planner): fix except and intersect operation by @Dousir9 in #17111
- chore(query): add fuzz tests by @sundy-li in #17100
- chore(query): bump federated mysql version by @TCeason in #17118
- chore(query): reduce-functions-code-expand by @sundy-li in #17114
- chore(query): improve hook vacuum temp after query by @sundy-li in #17125
- chore(query): bump enum-as-inner 0.6 by @forsaken628 in #17120
- chore: rename default_cluster_key_id to cluster_key_seq by @zhyass in #17135
- chore(query): fix udf metrics by @sundy-li in #17134
New Contributors
- @arkzuse made their first contribution in #16502
- @chagelo made their first contribution in #16560
- @Dragonliu2018 made their first contribution in #16631
- @Maricaya made their first contribution in #16543
- @ding-young made their first contribution in #16685
- @lyang24 made their first contribution in #16695
- @florann made their first contribution in #16181
- @Rowlandev made their first contribution in #16824
- @kursataktas made their first contribution in #16773
- @flashmouse made their first contribution in #16926
- @notauserx made their first contribution in #16912
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.615...v1.2.680
Aug 19, 2024 (v1.2.615)
What's Changed
Exciting New Features ✨
- feat: support sample for table scan by @xudong963 in #16216
- feat(meta): Support create dictionary meta api by @Winnie-Hong0927 in #16178
- feat: auth by refresh and session tokens. by @youngsofun in #16220
- feat: http handler support set variable. by @youngsofun in #16239
- feat(query): support
json_object_agg
function by @b41sh in #16238 - feat: support block level table sample by @xudong963 in #16237
- feat(query): add identifier function from session variable by @sundy-li in #16264
Thoughtful Bug Fix 🔧
- fix(query): cume_dist() always return unbounded frame by @TCeason in #16242
- fix(query): fix alter user set
MUST_CHANGE_PASSWORD
by @b41sh in #16243
Code Refactor 🎉
- refactor(query): remove cache solt by @zhang2014 in #16233
- refactor: optimize share table performance by @lichuang in #16218
- refactor(query): don't retry transform on each block by @sundy-li in #16235
- refactor(metactl): change to subcommands & support transfer-leader by @everpcpc in #16254
- refactor(deps): Bump OpenDAL and other crates by @Xuanwo in #16251
- refactor(query): remove named cache by @zhang2014 in #16245
- refactor(query): decrease information_schema.tables call union by @TCeason in #16246
- refactor: introduce new method
accurate_columns_ranges
toTable
by @dantengsky in #16247
Others 📒
- chore: remove admin api for testing for old version SM by @drmingdrmer in #16236
- chore: disable table info refreshing for
ai_to_sql
by @dantengsky in #16240 - chore: impl Display for
LicenseInfo
andJWTClaims<T>
by @drmingdrmer in #16244 - chore: update command for metactl in tests by @everpcpc in #16263
- chore: more logs for attached table refreshing by @dantengsky in #16261
- chore: refactoring some functions by @b41sh in #16265
- chore: comment out fake_time test by @dantengsky in #16269
- chore(query): revert #16192 by @Dousir9 in #16271
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.609...v1.2.615
Aug 13, 2024 (v1.2.609)
What's Changed
Exciting New Features ✨
- feat: support recursive cte used in multiple places by @xudong963 in #15835
- feat(query): support distinct from for hash join by @Dousir9 in #15838
- feat: Implement Default for CatalogInfo by @Xuanwo in #15844
- feat(query): enable udf in udf call by @sundy-li in #15839
- feat(query): support row fetch for merge into by @Dousir9 in #15859
- feat: support filling uuid when loading data and uuid is missing/empty in some rows. by @youngsofun in #15871
- feat: stream consuming for
copy into LOCATION from
stmt by @dantengsky in #15882 - feat: publish databend-common-ast to crates.io by @andylokandy in #15905
- feat: Persist Meta-Service State Machine on Disk by @drmingdrmer in #15772
- feat: failsafe tool
fuse_amend
by @dantengsky in #15929 - feat: refactor create share endpoint ddl by @lichuang in #15937
- feat: add built-in udfs in config by @BohuTANG in #15938
- feat(planner): support pruning columns for merge into when insert only by @Dousir9 in #15948
- feat: clickhouse handler no longer support insert with format. by @youngsofun in #15952
- feat(query): Array lambda function support outer scope columns by @b41sh in #15957
- feat: support real-time retrieval of profiles from admin API (part 1) by @dqhl76 in #15958
- feat: refactor create database from share ddl by @lichuang in #15950
- feat: pretty print backtrace by @andylokandy in #15913
- feat: allow tuple field name to contain '_'. by @youngsofun in #15965
- feat(query): alter user support modify current password by @TCeason in #15962
- feat: update sort algorithm using loser tree for multi sort merge by @forsaken628 in #15869
- feat: orc support missing fields. by @youngsofun in #15970
- feat: orc add option missing_field_as. by @youngsofun in #15974
- feat(executor): support error profiling for cluster mode by @zhang2014 in #15969
- feat(query): implement HAVERSINE by @kkk25641463 in #15971
- feat(query): implement st_length/st_distance by @kkk25641463 in #15982
- feat: config option for syncing disk cache data by @dantengsky in #15984
- feat: add tenant&queryid to udf client by @BohuTANG in #15987
- feat: Reducing column.clone overhead in transform aggregate by introducing InputColumns by @forsaken628 in #15991
- feat: support real-time retrieval of profiles from admin API (part 2) by @dqhl76 in #15975
- feat: parquet add option
missing_field_as
. by @youngsofun in #15993 - feat: http handler limit body size of each response to 10MB. by @youngsofun in #15960
- feat: add NameResolutionSuggest to enhancement table name case sensitive error by @forsaken628 in #15889
- feat: refactor share spec location and format by @lichuang in #15989
- feat(query): add setting sort_spilling_batch_bytes by @sundy-li in #16019
- feat: change nan/inf to NaN/Infinity in http handler results. by @youngsofun in #16017
- feat: pretty print backtrace on panic by @andylokandy in #16024
- feat(ast): add
TokenKind::{Get, Put}
by @andylokandy in #16020 - feat(query): add new setting enable_dst_hour_fix by @TCeason in #16022
- feat: Change 'inf' to 'Infinity' when unloading CSV/TSV by @youngsofun in #16028
- feat(query): ST_CONTAINS by @kkk25641463 in #15994
- refactor: Refactor aggregate hashtable, replace &[Column] with InputColumns by @forsaken628 in #16038
- feat(query): ST_SETSRID/ST_NPOINTS by @kkk25641463 in #16035
- feat: new table function
set_cache_capacity
by @dantengsky in #16016 - feat: support must change password option for create user by @b41sh in #16031
- feat: snapshots generated in multi txn has same prev_snapshot_id by @SkyFan2002 in #16044
- feat(query): add option [ IGNORE | RESPECT NULLS ] functionality for window rank function(first_value, last_value, nth_value) by @TCeason in #15919
- feat: add function name to udf client header by @everpcpc in #16053
- feat: back compatibility of old share db by @lichuang in #16056
- feat: log file with local time by @everpcpc in #16064
- feat: add share integration test by @lichuang in #16051
- feat(query): impl LocalShuffle by @Freejww in #16055
- feat(query): add setting enable_strict_datetime_parser by @TCeason in #16067
- feat: improve selectivity accuracy by @xudong963 in #16069
- feat: optimize loser tree, disable peek top2 by @forsaken628 in #15979
- feat(storage): add table function clustering_statistics by @zhyass in #16081
- feat: limit auto cast when loading from column store. by @youngsofun in #16082
- feat(query): PartitionsShuffleKind::ConsistentHash support by @forsaken628 in #16094
- feat(http): expose queries queue length to /v1/status by @flaneur2020 in #16112
- feat(storage): refactor recluster by @zhyass in #16070
- feat(query): support json null for nullable response by @everpcpc in #16120
- feat(query): hash partitioning in window by @Freejww in #16090
- feat: generate accurate histogram by @xudong963 in #16018
- feat(query): add sql variable support by @sundy-li in #16134
- feat(query): support parse create dictionary stmt by @Winnie-Hong0927 in #16137
- feat(storage): refactor compact by @zhyass in #16119
- feat: add histogram info to fuse_statistic table by @xudong963 in #16141
- feat(query): add user function admin api by @zhang2014 in #16146
- feat: add
is_attach
column tosystem.tables
and disable index creation for attache tables by @dantengsky in #16166 - feat(planner): unify execution of DML statements (MERGE, UPDATE, DELETE) by @Dousir9 in #16060
- feat(query): Support
json_array_agg
function by @b41sh in #16169 - feat: add share catalog by @lichuang in #16172
- feat: explain analyze output partial infos to debug cardinality estimator by @xudong963 in #16185
- feat: transfer leader command for meta-service by @drmingdrmer in #16198
- feat: meta-service transfer_leader: add response and default value by @drmingdrmer in #16201
- feat: allow casting int to uint for string functions by @andylokandy in #16210
- feat(query): Support materialized view match algorithm by @b41sh in #16023
- feat: grant view to share by @lichuang in #16186
- feat(query): ST_TRANSFORM by @kkk25641463 in #15992
- feat: introduce new setting
enable_last_snapshot_location_hint
by @dantengsky in #16226
Thoughtful Bug Fix 🔧
- fix(query): list user stage not check privilege by @TCeason in #15800
- fix: fix sequence used in function calls by @lichuang in #15814
- fix: multi-tbl insert with lateral flattern panics by @dantengsky in #15822
- fix: uuid() as default value result in copy error. by @youngsofun in #15837
- fix: function uuid() should be non_deterministic. by @youngsofun in #15840
- fix: stream illegal after source rename by @zhyass in #15843
- fix: parse comment in raw value by @andylokandy in #15834
- fix(query): to_timestmap should always return err if parse err by @TCeason in #15850
- fix(query): decimal div overflow by @sundy-li in #15856
- fix(query): fix sum rewriter by @sundy-li in #15870
- fix: merge into unresolved conflict by @zhyass in #15884
- fix(query): window function out of bounds access panic by @shamb0 in #15860
- fix: stream error after alter table by @zhyass in #15907
- fix(query): fix expr stack overflow [part 1] by @zhang2014 in #15790
- fix(jwks): refresh the jwks store on key id not found by @flaneur2020 in #15921
- fix(query): fix cluster mode stack overflow by @zhang2014 in #15927
- fix: remove VisitorWithParent, add more delete/update by subquery test by @lichuang in #15526
- fix: no-recursive cte reference itself by @xudong963 in #15941
- fix(query): fix stackoverflow for ci release mode by @zhang2014 in #15944
- fix(query): window panic when max_block_size is small by @TCeason in #15949
- fix: when upgrade snapshot v002 to v003, put kvs after expire index by @drmingdrmer in #15947
- fix(meta): fix reserved number err by @TCeason in #15955
- fix(storage): stream show columns maybe error by @zhyass in #15964
- fix: update orc-rust to fix error when read columns of array<float>. by @youngsofun in #15968
- fix(query): fix tuple inner fields name with quotes by @b41sh in #15973
- fix(query): udf function support lambda arguments by @b41sh in #15981
- fix: wrong arrow schema when fuse engine read parquet by @SkyFan2002 in #15997
- fix(query): fix drop table column with quotes by @b41sh in #16006
- fix(query): fix incorrect fast_memcmp function by @sundy-li in #16008
- fix(query): coalesce continue loop when arg is null by @TCeason in #16002
- fix(query): unify udf allow list validator by @sundy-li in #16012
- fix: arrow schema in parquet reader by @SkyFan2002 in #16004
- fix(query): fix create table as query with null type by @b41sh in #16041
- fix(query): fix ambiguous time by @TCeason in #16046
- fix: output format missing '-' for '-Infinity'. by @youngsofun in #16052
- fix(query): if role not store ownership skip revoke ownership from role by @TCeason in #16047
- fix(query): pad information_schema.views.view_definition by @TCeason in #16065
- fix: bitmap with empty buffer stands for empty. by @youngsofun in #16066
- fix(query): if timestamp is some, no need to init parsed by @TCeason in #16078
- fix(query): fix read decimal by @sundy-li in #16080
- fix(query): fix session manager dead lock if call instance status by @zhang2014 in #16088
- fix: fix native decompress binary offsets out of bounds by @b41sh in #16085
- fix(query): fix incorrect derive_stats for window and limit plan by @sundy-li in #16087
- fix: wrong prev snapshot id by @SkyFan2002 in #16092
- fix: error message in data. by @youngsofun in #16089
- fix(storage): recluster maybe endless loop by @zhyass in #16096
- fix(query): fix subquery with function failed by @b41sh in #16103
- fix(query): cast string to decimal respect numeric_cast_option setting by @sundy-li in #16101
- fix: tenant tables with stream by @zhyass in #16115
- fix(query): fix stackoverflow in collect_statistics by @zhang2014 in #16123
- fix(query): revert window hash partition by @Freejww in #16129
- fix(query): fix window hash partition optimization by @Freejww in #16136
- fix: deadlock with export() and build_snapshot() by @drmingdrmer in #16138
- fix: vacuum dropped table in parallel by @dantengsky in #16139
- fix: drop attached table that can no longer reach table data by @dantengsky in #16125
- fix(query): revert find_eq_and_or_filter by @TCeason in #16148
- fix(query): fix add column default value is indeterministic expression by @b41sh in #16153
- fix(query): fix in operator convert to subquery cast failed by @b41sh in #16159
- fix: explain analyze lack of statistics by @dqhl76 in #16162
- fix: meta-service: data inconsistency risk by @drmingdrmer in #16175
- fix(query): reduce get table meta call by @TCeason in #16170
- fix: properly ignore column statistic from meta v2 if string contains non-utf8 by @andylokandy in #16180
- fix(query): support optimize or filter in system.tables by @TCeason in #16165
- fix(query): fix wasm udf runtime create with code by @b41sh in #16191
- fix: skip column statistics entirely if deserialization failed by @andylokandy in #16192
- fix(query): to_year/monday... should respect dst by @TCeason in #16195
- fix(query): rounding decimal multiply results by @sundy-li in #16196
- fix: backward compatibility of table statistics by @dantengsky in #16200
- fix(query): remove deserialize_cluster_stats by @sundy-li in #16204
- fix(query): fix index out of bounds with constant expr by @zhang2014 in #16206
- fix(query): cannot use fully qualified names with views by @TCeason in #16223
- fix(query): fix unexpected panic message by @zhang2014 in #16221
- fix: block_entry.memory_size by @forsaken628 in #16230
Code Refactor 🎉
- refactor: move databend-meta binaries to separate dir by @drmingdrmer in #15803
- refactor: adaptive entries count for
AppendEntriesRequest
by @drmingdrmer in #15805 - refactor: add meta-meta compat test by @drmingdrmer in #15823
- refactor: Migrate catalog info into table info by @Xuanwo in #15857
- refactor(query): refactor merge into pipeline by @Freejww in #15891
- refactor(query): remove async for query binder by @zhang2014 in #15900
- refactor: CatalogInfo has been stored in TableInfo by @Xuanwo in #15902
- refactor: Make CatalogCreator Accept Arc<CatalogInfo> by @Xuanwo in #15906
- refactor: simplify adding transform to pipeline by @dantengsky in #15873
- refactor: improve modulo predicate selectivity by @xudong963 in #15917
- refactor: optimize gen_columns_statistics() for scalar. by @youngsofun in #15909
- refactor: optimize building bloom index for scalar. by @youngsofun in #15910
- refactor: attach table read only by default by @dantengsky in #15922
- refactor(cluster): refactor flight actions and add flight secret by @zhang2014 in #15930
- refactor: speed up ColumnBuilder::repeat(Scalar::Null). by @youngsofun in #15939
- refactor: Bump arrow to 52 by @Xuanwo in #15943
- refactor: fuse statistics table function by @xudong963 in #15954
- refactor: optimize OrcChunkReader. by @youngsofun in #15967
- refactor: Use iceberg-rust to replace icelake by @Xuanwo in #15951
- refactor: unify transaction related code by @SkyFan2002 in #15966
- refactor: rename metrics & tweak system.caches table by @dantengsky in #15996
- refactor: upgrade QuotaMgr to using protobuf by @forsaken628 in #15858
- refactor(query): use shuffle on distributed merge into by @Freejww in #15946
- refactor: add history record for queries_profiling table by @dqhl76 in #16014
- refactor: optimize take_by_slices_limit_from_blocks by @forsaken628 in #15978
- refactor(query): add crash handler for databend-query by @zhang2014 in #16054
- refactor: Simplify HashMethod.build_keys_state function signature by replacing &[(Column, DataType)] with InputColumns by @forsaken628 in #16050
- refactor: meta-service wait at most 1 second to shutdown by @drmingdrmer in #16104
- refactor(query): Optimize aggregate function arg_min_max by @forsaken628 in #16109
- refactor(query): If table_id is specified, use it directly by @TCeason in #16098
- refactor: upgrade to Openraft-0.10.0-alpha.2 by @drmingdrmer in #16091
- refactor: Bump OpenDAL to 0.48.0 by @Xuanwo in #16147
- refactor: tweak
table_statistics
of trait Table by @dantengsky in #16152 - refactor: add retry logic for flight service by @dqhl76 in #16097
- refactor: refector ShareMeta struct by @lichuang in #16100
- refactor(query): Optimize udf js runtime to avoid lock blocking by @b41sh in #16174
- refactor(query): if projection not query stat/owner will not get table_stats/ownership when access system.tables by @TCeason in #16183
- refactor: improve network error handling. by @drmingdrmer in #16194
- refactor(query): remove useless meter by @zhang2014 in #16202
- refactor(query): remove useless hash builder by @zhang2014 in #16203
- chore: remove unused minmax by @zhyass in #16197
- refactor: display unix timestamp for human by @drmingdrmer in #16212
- refactor: support unset session|global by @TCeason in #16214
- refactor: Use native-tls as default by @Xuanwo in #16199
- refactor: fuse table funcs by @dantengsky in #16149
- refactor(query): use bigint to handle the fallback of decimal op overflow by @sundy-li in #16215
- refactor: CrudMgr::update should return only when the value is changed by @drmingdrmer in #16225
Build/Testing/CI Infra Changes 🔌
- ci: code reuse and get raid of streaming load. by @youngsofun in #15811
- ci: test load parquet unloaded by databend. by @youngsofun in #15813
- ci: longer timeout for test_stateless_standalone. by @youngsofun in #15936
- ci: Build runner for nightly-2024-07-02 by @Xuanwo in #16033
Others 📒
- chore: recluster final ignore error by @zhyass in #15815
- chore: reduce recluster depth threshold by @zhyass in #15819
- chore(query): remove redundant code in merge into by @Freejww in #15812
- chore: normalize compat test script names by @drmingdrmer in #15817
- chore: fix scripts in meta-meta compatibility test by @drmingdrmer in #15826
- chore(ci): add reporter for meta chaos test by @everpcpc in #15829
- chore: bump arrow-udf by @sundy-li in #15836
- chore(query): remove addRowNumber if source is physical table by @Freejww in #15828
- chore(query): show view just display view name by @TCeason in #15841
- chore: Fix build of arrow-udf by @Xuanwo in #15846
- chore: change meta chaos io delay param by @lichuang in #15852
- chore(ci): fix duplicate failure artifact name by @everpcpc in #15854
- chore: disable recursive cte test for mysql by @xudong963 in #15853
- chore(query): add retry logs by @sundy-li in #15862
- chore: refactor name and match expression by @lichuang in #15863
- chore: update logcall by @andylokandy in #15831
- chore: install gdb for query service image by @everpcpc in #15875
- chore(query): string_to_date/ts should not ignore error by @TCeason in #15878
- chore: fix recursive cte hang by @xudong963 in #15883
- chore(executor): avoid lock poisoning for pipeline executor by @zhang2014 in #15887
- chore: add log to show if enable distributed optimization by @xudong963 in #15890
- chore: print the select plan of insert query by @xudong963 in #15892
- chore(query): improve error msg by @sundy-li in #15874
- chore: replace patched codespan-reporting with rspack-codespan-reporting by @andylokandy in #15894
- chore(query): udf allow list only judge host match by @sundy-li in #15896
- chore: change meta chaos io delay param by @lichuang in #15897
- chore: Bump minitrace for new opentelementry support by @Xuanwo in #15700
- chore(ci): switch runner to aws by @everpcpc in #15903
- chore: improve spiller buffer by @xudong963 in #15904
- chore: copy into default value using
RemoteExpr
by @b41sh in #15893 - chore: update domain for datasets & benchmark by @everpcpc in #15920
- chore: remove open sharding binary and test by @lichuang in #15923
- chore: add test case of issue 15791 by @lichuang in #15931
- chore(storage): limit compact num when recluster by @zhyass in #15926
- chore: fix param typo in function process_left_or_full_join_null_block by @lichuang in #15933
- chore(planner): refactor join equi conditions by @Dousir9 in #15924
- chore(planner): refine join_order_changed check in merge into by @Dousir9 in #15934
- chore: remove streaming_load api. by @youngsofun in #15935
- chore(query): readd some tests by @sundy-li in #15940
- chore: add issue case for pr15941 by @xudong963 in #15945
- chore(query): enable none lazy pruner in lazy read by @sundy-li in #15942
- chore(docs): update domain for databend.rs by @everpcpc in #15959
- chore: Bump hive_metastore to 0.1.0 by @Xuanwo in #15976
- chore: move display_ident() to ast by @andylokandy in #15980
- chore(query): show user functions should display built-in udfs by @TCeason in #15990
- chore(query): fix cluster ci failure if set license by @zhang2014 in #15999
- chore(ci): add license for cluster test by @everpcpc in #15995
- chore: update nom-rule by @andylokandy in #16009
- chore: orc and parquet use option missing_field_as. by @youngsofun in #16007
- chore: add repo link in databend-common-ast lib by @lewiszlw in #16010
- chore(query): disallow agg/window/udfcall in insert expr by @sundy-li in #16015
- chore: remove unused ShareEndpointManager and ShareTableConfig by @lichuang in #16013
- chore: add
errors
column and more tests forsystem.queries_profiling
table by @dqhl76 in #16029 - chore(query): add full expr in error message by @sundy-li in #16032
- chore: temporarily disable a flaky test by @dqhl76 in #16042
- chore: add privilege checking for
fuse_amend
by @dantengsky in #16045 - chore: add
node
column to tablesystem.malloc_stats_totals
by @dantengsky in #16043 - chore(query): enable distributed merge into by default by @Dousir9 in #16059
- chore(ci): setup license with mask by @everpcpc in #16061
- chore(query): information_schema.tables support display view engine by @TCeason in #16058
- chore: fix share test integration missing query binary by @lichuang in #16071
- chore: Bump jsonb version 0.4.1 by @b41sh in #16073
- chore(query): better error msg in check_function by @sundy-li in #16075
- chore(query): fix merge into virtual computed field by @Dousir9 in #16086
- chore: update rust toolchain to nightly-2024-07-02 by @andylokandy in #16026
- chore(code): update the version of object_store in Cargo.lock by @Dousir9 in #16110
- chore(query): disable inner columns in prewhere by @Dousir9 in #16108
- chore(query): add sigaltstack for signal handler by @zhang2014 in #16122
- chore(query): limit frames size for capture backtrace by @zhang2014 in #16118
- chore(query): remove useless box processor impl by @zhang2014 in #16128
- chore: resolve lints by @andylokandy in #16127
- chore: fake time test by @dantengsky in #16130
- chore: resolve lints (part 2) by @andylokandy in #16133
- chore(http): exposes max_running_query_executed_time in /v1/status by @flaneur2020 in #16131
- chore(query): optimize tables query speed by @TCeason in #16144
- chore(ci): add buildkitd config & fix typos by @everpcpc in #16151
- chore: reset table_lock_expire_secs default value by @zhyass in #16154
- chore(query): move udf admin api to management mode by @zhang2014 in #16155
- chore: improve parser by @andylokandy in #16156
- chore: reduce the sleep range during lock holder by @zhyass in #16158
- chore(ci): split internal test sql by @TCeason in #16167
- chore(query): revert 16097 by @zhang2014 in #16171
- chore(ci): reset table lock expire in sqllogic by @zhyass in #16160
- chore(query): skip empty block for udf script by @zhang2014 in #16182
- chore(ci): disable musl build by @everpcpc in #16205
- chore: refine TimedFutre wrapper by @drmingdrmer in #16211
- chore(query): change stream table is_local to false by @Dousir9 in #16208
- chore(ci): try fix main ci err by @TCeason in #16224
- chore: refine meta-service by @drmingdrmer in #16229
- chore(planner): fix distributed merge into by @Dousir9 in #16228
New Contributors
- @lewiszlw made their first contribution in #16010
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.530...v1.2.609
Jun 16, 2024 (v1.2.530)
What's Changed
Exciting New Features ✨
- feat(query): implement ST_XMAX by @kkk25641463 in #15406
- feat(executor): record query profiling for standalone error query by @zhang2014 in #15425
- feat(query): add external_server_request_batch_rows setting by @sundy-li in #15440
- feat(function): Implement map_size function by @forsaken628 in #15444
- feat(query): add created_time column to processes table by @TCeason in #15448
- feat(function): Implement map_cat function by @shamb0 in #15348
- feat: support adjust query's priority by @dqhl76 in #15352
- feat(executor): record errors info into profiling log by @zhang2014 in #15472
- feat: optimize vacuum drop table, execute in parallel by @lichuang in #15478
- feat: http handler add endpoint /v1/login/. by @youngsofun in #15483
- feat(functions): add map_contains_key scalar function by @hanxuanliang in #15465
- feat(query): build image limit python 3.12.2 by @sundy-li in #15501
- feat(query): ensure git is installed in pyenv by @sundy-li in #15505
- feat(query): pyenv 3.12.2 by @sundy-li in #15512
- feat(query): support unset session setting by @TCeason in #15510
- feat: add version metrics for meta & query by @everpcpc in #15522
- feat: support explain and profile multi table insert statemennt by @SkyFan2002 in #15518
- feat(query): add query query_hash and query_parameterized_hash to query log by @TCeason in #15531
- feat: add meta key_num metrics by @lichuang in #15536
- feat(query): bump arrow-udf version to enhance types supported by JS UDFs by @maxjustus in #15529
- feat: resolve conflict when multi table insert commit by @SkyFan2002 in #15503
- feat(query): udf script support load from stage location by @sundy-li in #15542
- feat(metasrv): add metrics for snapshot key numbers by @everpcpc in #15545
- feat: select support consume stream by @zhyass in #15541
- feat(query): show users support display roles by @TCeason in #15544
- feat: remove XML input format. by @youngsofun in #15540
- feat(query): create table support add inverted index by @b41sh in #15547
- feat: fuzzy table data disk cache key reload by @dantengsky in #15566
- feat(query): variant type as map value support bloom filter by @b41sh in #15562
- feat(metasrv): add metrics for db_size by @everpcpc in #15595
- feat: support millisecond level task scheduling by @ZhiHanZ in #15599
- feat(query): show grants support where/limit option by @TCeason in #15607
- feat: otlp logging support http endpoint by @everpcpc in #15636
- feat(planner): support ExpressionScan and CacheScan by @Dousir9 in #15561
- feat(query): implement ST_YMAX/ST_XMIN/ST_YMIN by @kkk25641463 in #15594
- feat(query): add python udf script runtime by @sundy-li in #15494
- feat: compact and recluster add table lock by @zhyass in #15632
- feat(query): support disable vacuum temporary files after query by @zhang2014 in #15673
- feat(query): Role/UserInfo add field create_on and udpate_on by @TCeason in #15661
- feat: unload parquet with version as
created_by
. by @youngsofun in #15680 - feat(metrics): add process collector by @flaneur2020 in #15591
- feat: add meta-service config sled_max_cache_size_mb by @drmingdrmer in #15709
- feat(query): add async retry transform by @sundy-li in #15647
- feat: basic support input file format ORC. by @youngsofun in #15720
- feat: recluster unclustered blocks by @zhyass in #15623
- feat: add task_history function by @ZhiHanZ in #15731
- feat(query): show indexes add inverted indexes by @b41sh in #15733
- feat: Bump to opendal v0.47.0 rc by @Xuanwo in #15745
- feat: Add http config for all storage services by @Xuanwo in #15757
- feat(query): add inverted index size to block meta by @b41sh in #15752
- feat: add meta verifier binary by @lichuang in #15756
- feat(query): add shuffle before window plan by @sundy-li in #15726
- feat: parquet support option null_if by @youngsofun in #15776
- feat: rm CREDENTIALS as alias of CONNECTION. by @youngsofun in #15780
- feat(function): Implement map_delete function by @shamb0 in #15480
- feat(query): support try_to_timestmap(string, string), try_to_date(string, string) by @TCeason in #15775
- feat: add io delay chaos meta test scripts by @lichuang in #15769
- feat: support recursive cte by @xudong963 in #15675
- feat(ci): add logging test for query by @everpcpc in #15792
- feat: support dynamic enable of rust backtrace by @dqhl76 in #15749
Thoughtful Bug Fix 🔧
- fix(query): fix cse projection by @sundy-li in #15409
- fix(storage): change tracking with alias update row_version bug in merge into by @zhyass in #15412
- fix(executor): fix broken merge into source profiling by @zhang2014 in #15415
- fix(query): ensure all inputs pull all data that less than max partition count in new agg singleton by @Freejww in #15418
- fix(query): fix
array_concat
function domain panic by @b41sh in #15424 - fix: ExistsTable should has_result_set. by @youngsofun in #15431
- fix: max_files not work for transform on ndjson/csv/tsv. by @youngsofun in #15441
- fix(functions): improve array_prepend and array_append by @andylokandy in #15437
- fix(executor): fix queries executor cannot kill problem and reactivate kill test assertion by @dqhl76 in #15443
- fix(query): fix create table with cluster by get map type panic by @b41sh in #15475
- fix(query): RoleInfo from_pb should not err by @TCeason in #15496
- fix: fix group by cube with join panic assert by @lichuang in #15515
- fix: Enable opendal backtrace output to help debug by @Xuanwo in #15521
- fix:
explain ast
of multi table insert stmt by @dantengsky in #15527 - fix: Prevent premature snapshot commit on unfinished writes by @drmingdrmer in #15534
- fix(query): fix view flaky test with different dialect by @b41sh in #15533
- fix(query): fix qualify display by @sundy-li in #15537
- fix: FlightData must contain valid data header by @Xuanwo in #15559
- fix: http handler kill not working. by @youngsofun in #15558
- fix: Avoid duplicate temp file name for install-snapshot requests by @drmingdrmer in #15565
- fix(query): fix shrink_d256 for negative value by @sundy-li in #15574
- fix(query): fix bloom fitler using
is_null
andnot
function by @b41sh in #15577 - fix(query): fix filter with nested
not
expr return wrong result by @b41sh in #15580 - fix: fix normalize disjunctive filter by @xudong963 in #15587
- fix: consume stream inside explicit txn by @dantengsky in #15582
- fix: TxnManager with state AutoCommit should not be saved in http manager. by @youngsofun in #15602
- fix(query): fix subquery writer in udf by @sundy-li in #15610
- fix: analyze table error by @zhyass in #15614
- fix: update derive-vistor to unblock rustc upgrage by @andylokandy in #15625
- fix: allow http client retry when lost response with next_uri=/final. by @youngsofun in #15624
- fix: subquery correlated column is derived column by @xudong963 in #15629
- fix(query): unify optimize plan call in one place by @sundy-li in #15630
- fix: missing service label for otlp logs by @everpcpc in #15639
- fix(query): delete len check in mget_database/table_names_by_ids by @TCeason in #15640
- fix(udf): use Debug to display transport error on UDF server by @flaneur2020 in #15646
- fix(query): show grants also need to check user roles by @TCeason in #15650
- fix(query): fix current date/time functions return constant value in a statement. by @b41sh in #15659
- fix: merge into split by @dantengsky in #15644
- fix: make sure lock guard is dropped before next recluster by @zhyass in #15662
- fix(query): db owner role can access all tables under this db by @TCeason in #15634
- fix(cluster): fix complex plan serde in cluster actions by @zhang2014 in #15672
- fix: potential missing index creation in merge process by @dantengsky in #15674
- fix(query): fix result cache in empty result by @sundy-li in #15683
- fix: display expr considering associativity by @andylokandy in #15685
- fix: fix create table error but table id uncommttied by @lichuang in #15665
- fix: failed to select blocks for recluster by @zhyass in #15667
- fix: add unit test of concurrent commit table meta data by @lichuang in #15703
- fix(executor): fix lost pipeline on finished callback by @zhang2014 in #15693
- fix(executor): fix query log lost for http kill by @zhang2014 in #15715
- fix:
fuse_snapshot
only show the latest snapshot for external table by @dantengsky in #15721 - fix: fix union and lazy materialization by @xudong963 in #15727
- fix(ast): assert ast display and reparse by @andylokandy in #15666
- fix: Handle Duplicate Tombstones in Meta-Service List Operation by @drmingdrmer in #15736
- fix: rebuild missing bloom index while pruning by @dantengsky in #15738
- fix(storage): Fix a potential data loss while concurrent write by @Xuanwo in #15759
- fix: unload csv file too large. by @youngsofun in #15758
- fix: off by one mistake in auto compaction by @dantengsky in #15761
- fix(query): fix decimal cmp by @sundy-li in #15764
- fix: lazy materialization does not support set operation by @xudong963 in #15754
- fix: auto cast fail error in copy should show the related columns #15768 by @youngsofun in #15770
- fix: http handler handle panic when starting query. by @youngsofun in #15785
- fix(ci): meta chaos dockerfile path by @everpcpc in #15787
- fix: add settings enable_streaming_load, default 0. by @youngsofun in #15794
- fix(query): create ownership obejct only check current_role privilege by @TCeason in #15751
Code Refactor 🎉
- refactor: list stage use stream to avoid OOM. by @youngsofun in #15405
- refactor: Minimize lock duration during raft-log export by @drmingdrmer in #15417
- refactor: move storage/fuse/io/files.rs to storage/common/io/files.rs by @BohuTANG in #15428
- refactor: define CatalogIdIdent with TIdent by @drmingdrmer in #15429
- refactor: define CatalogIdToNameIdent with TIdent by @drmingdrmer in #15432
- refactor: set enable_experimental_rbac_check default 1 by @TCeason in #15436
- refactor(query): enable little vacuum after query finish by @zhang2014 in #15434
- refactor: remove
MergeIntoSource
operator by @xudong963 in #15435 - refactor: ScalarVisitor refactor replace_predicate_column by @TCeason in #15439
- refactor: use scalarvisitor replace predicate recursive by @TCeason in #15449
- refactor(query): remove useless settings by @zhang2014 in #15461
- refactor(query): add query id for query log by @zhang2014 in #15466
- refactor: Bump OpenDAL 0.46, arrow 51, tonic 0.11, reqwest 0.12, hyper 1, http 1 by @Xuanwo in #15442
- refactor:
SessionContext
doesn't requireArc
by @drmingdrmer in #15491 - refactor: remove project operator by @xudong963 in #15479
- refactor(query): revert dev_setup by @sundy-li in #15516
- refactor: remove databend-common-meta-app from databend-common-ast by @andylokandy in #15509
- refactor: remove databend-common-base from databend-common-ast by @andylokandy in #15519
- refactor(query): allow udf server in insert source by @sundy-li in #15523
- refactor: remove redundant lifetime on fmt::Formatter by @andylokandy in #15528
- refactor:
SessionContext::current_tenant
should not be changed at runtime by @drmingdrmer in #15504 - refactor: migrate TSV input format to new framework. by @youngsofun in #15506
- refactor(cluster): refactor flight service actions by @zhang2014 in #15419
- refactor: Improve logging and snapshot management by @drmingdrmer in #15571
- refactor: Embed key number into snapshot id by @drmingdrmer in #15588
- refactor: Remove
install_snapshot
v0 meta-service API by @drmingdrmer in #15590 - refactor: Enhance Meta-Service Cluster Management by @drmingdrmer in #15601
- refactor: Decouple Snapshot Write and Commit Operations by @drmingdrmer in #15603
- refactor: polish memory usage when copy into parquet. by @youngsofun in #15596
- refactor(storage): bloom index filter use
fold_with_domain
insteadfold
by @b41sh in #15619 - refactor: refine cast reloperator error info by @xudong963 in #15648
- refactor: refactor merge into optimizer and fix unexpected distribution plan by @xudong963 in #15507
- refactor(query): no need to deserialize all userinfo when create user by @TCeason in #15671
- refactor: enable mem-profiling for meta-service by @drmingdrmer in #15705
- refactor: delete raft log one by one. by @drmingdrmer in #15723
- refactor: more explicit error confirmation for http query route error. by @youngsofun in #15621
- refactor: split None and Owner/Grantobject by @TCeason in #15739
- refactor(cluster): record query id for cluster mode log by @zhang2014 in #15609
- refactor: GrantEntry not store ownership privilege by @TCeason in #15730
- refactor: add workspace crates to Cargo.toml by @andylokandy in #15747
- refactor: remove unsafe impl Send+Sync for GlobalRegistry by @drmingdrmer in #15782
- refactor: remove meta-service V001 data version by @drmingdrmer in #15778
- refactor(ast): eliminate clone when displaying expr by @zhang2014 in #15773
- refactor: clustering_information support specify cluster keys by @zhyass in #15783
- refactor: add non-async version of mete-client API request_sync by @drmingdrmer in #15786
Build/Testing/CI Infra Changes 🔌
- ci: Bump version to 2024-02-08 (the same commit with 1.78) by @Xuanwo in #15455
- ci: tcph/tpcds use copy instead of streaming load for prepare data. by @youngsofun in #15681
Documentation 📔
- docs: delete repeat summary by @baerwang in #15744
Others 📒
- chore: Vendor parquet2 instead by @Xuanwo in #15422
- chore(query): refactor metrics by @sundy-li in #15421
- chore: remove unused file by @drmingdrmer in #15438
- chore(storage): allow append when change tracking was enabled by @zhyass in #15427
- chore: Bump to rust 2024-02-08 (also 1.78) by @Xuanwo in #15456
- chore: ignore stream column compare in block compact mutator by @zhyass in #15446
- chore: polish log for copy. by @youngsofun in #15463
- chore: improve show streams performance by @zhyass in #15460
- chore(ci): update deprecated actions by @everpcpc in #15470
- chore: change flood info log to debug by @BohuTANG in #15473
- chore: remove breakline for log by @BohuTANG in #15476
- chore: remove incorrect usage of
Box::leak
by @dantengsky in #15481 - chore: fix copy/spill log message by @BohuTANG in #15482
- chore: Upgrade opentelemetry and remove metrics crate by @Xuanwo in #15484
- chore: add KVPbApi::get_pb_stream() but it is not yet ready for production use by @drmingdrmer in #15471
- chore: log file_name name for short by @BohuTANG in #15487
- chore: Bump poem to 3.0 by @Xuanwo in #15486
- chore: fix some flood logs with batch progress by @BohuTANG in #15492
- chore(meta): tracking query id for meta log by @zhang2014 in #15497
- chore: Increase default io timeout for behavior changes by @Xuanwo in #15493
- chore(query): tracking query log for mysql handler by @zhang2014 in #15495
- chore: polish error message NDJSON. by @youngsofun in #15502
- chore(query): pyenv to 3.9 by @sundy-li in #15511
- chore: recluster disable sort spill by @zhyass in #15490
- chore: change load_file_metadata_expire_hours default from 24*7 to 12 hours by @BohuTANG in #15514
- chore(ci): use python3.12 as build tool base by @everpcpc in #15520
- chore: adjust elapsed time of various log messages by @dantengsky in #15517
- chore: Upgrade minitrace related crates to 0.6.5 by @Xuanwo in #15530
- chore: reduce snapshot building progress logging by @drmingdrmer in #15532
- chore: convert to arrow column remove ignore inside nullable by @b41sh in #15477
- chore: leave all un-committed data to purge/vacuum by @dantengsky in #15535
- chore: mask the config sensitive field for display by @BohuTANG in #15543
- chore: Remove tokio-console since we don't need it by @Xuanwo in #15546
- chore: Bump databend client and geozero to latest version by @Xuanwo in #15550
- chore: Bump rustls to 0.22 by @Xuanwo in #15552
- chore: remove unused micromarshal patch by @PsiACE in #15551
- chore(executor): set plain_statistics to false for spill time profiling statistics by @zhang2014 in #15555
- chore: change the short_sql from 64bytes to 10KB by @BohuTANG in #15554
- chore: support aborting table navigation by @dantengsky in #15549
- chore(query): add more heavy actions query type by @zhang2014 in #15556
- chore(storage): remove abort operation by @zhyass in #15557
- chore(query): all queryies should generate hash by @TCeason in #15560
- chore: Remove sentry entirely and bump databend client by @Xuanwo in #15563
- chore: Reduce the duplicated clap and gix by @Xuanwo in #15564
- chore(ast): unify all of
{escape,unescape,quote,unquote}\_{string/at_string/ident}
by @andylokandy in #15548 - chore: short_sql from 10KB to 30KB by @BohuTANG in #15575
- chore: disk cache load refine and reduce the info flood logs by @BohuTANG in #15576
- chore: Bump jsonb version 0.4.0 by @b41sh in #15589
- chore: fix error capture of mulit statement transaction in mysql session by @SkyFan2002 in #15583
- chore: udpate doc for meta-service compat change by @drmingdrmer in #15593
- chore(deps): bump rustls from 0.22.2 to 0.22.4 by @dependabot in #15586
- chore: add more logs for vacuum temp files by @BohuTANG in #15597
- chore: Refine state machine v002 by @drmingdrmer in #15608
- chore(deps): bump gix from 0.62.0 to 0.63.0 by @dependabot in #15615
- chore: more commit logs by @dantengsky in #15617
- chore: prepare for SMV003: move shared code to separate dir by @drmingdrmer in #15622
- chore(query): add sort plan before window plan by @sundy-li in #15616
- chore(query): fix filter reorder check by @Dousir9 in #15613
- chore: reduce the short_sql to 128bytes by @BohuTANG in #15626
- chore: remove
databend-common-{io, exception}
from databend-common-ast by @andylokandy in #15618 - chore(query): add timeout for open telemetry logger by @zhang2014 in #15627
- chore: spill out short_sql to common/base/src/string.rs by @BohuTANG in #15633
- chore: add setting compact_max_block_selection by @zhyass in #15641
- chore(executor): add some time tracking log by @zhang2014 in #15651
- chore: minimize #[derive(skip)] by @andylokandy in #15654
- chore(planner): remove z3 by @Dousir9 in #15663
- chore: re-enable musl-g++ link for musl build image by @dantengsky in #15679
- chore(query): fix replace into from distributed select plan by @sundy-li in #15657
- chore: fix wrong print status by @zhyass in #15684
- chore(query): add assert on explain parser by @sundy-li in #15658
- chore(ci): add build for udf feature by @everpcpc in #15682
- chore(ci): fix release workflow by @everpcpc in #15687
- chore(ci): fix sqllogic test for release by @everpcpc in #15688
- chore(ci): fix pack for release by @everpcpc in #15689
- chore(ci): fix artifact upload/download for gcs by @everpcpc in #15691
- chore: add unit test of vacuum orphan table meta data by @lichuang in #15690
- chore(ci): fix checkout and version on release build by @everpcpc in #15692
- chore(ci): fix distribution for release by @everpcpc in #15695
- chore(ci): fix notify & publish release by @everpcpc in #15696
- chore(ci): fix release notify template by @everpcpc in #15697
- chore(query): add create query in admin tables api by @zhang2014 in #15694
- chore(ci): fix publish service image by @everpcpc in #15698
- chore(ci): fix run benchmark after release by @everpcpc in #15699
- chore(ci): ci-cloud build with python-udf by @everpcpc in #15701
- chore(ci): only provide debian based service image by @everpcpc in #15704
- chore(ci): fix download artifacts for docker_all_in_one by @everpcpc in #15708
- chore: more log for table history traversing by @dantengsky in #15710
- chore(executor): revert #15693 by @zhang2014 in #15711
- chore(ci): fix download artifacts for cloud build by @everpcpc in #15712
- chore(ci): fix dockerfile for cloud service image build by @everpcpc in #15713
- chore(ci): fix checking release status from jobs by @everpcpc in #15714
- chore(planner): fix merge predicates with different data types by @Dousir9 in #15725
- chore: add databend user to docker service images by @everpcpc in #15728
- chore(query): fix pipeline display by @sundy-li in #15729
- chore(ci): fix download artifacts for local benchmark by @everpcpc in #15735
- chore(ci): setup python3.12 for local benchmark by @everpcpc in #15737
- chore(ci): fix check release status by @everpcpc in #15748
- chore(ci): fix release notify icon by @everpcpc in #15750
- chore: output raft log index range for purge logging by @drmingdrmer in #15740
- chore: add more test case about create table fail by @lichuang in #15765
- chore: add error info when table name is misleading by @xudong963 in #15771
- chore: Pin opendal to released version by @Xuanwo in #15766
- chore: limit the count of set operation by @xudong963 in #15777
- chore: replace manually incr/decr with couter guard by @drmingdrmer in #15784
- chore(query): remove useless async keyword by @zhang2014 in #15781
- chore: add more alias name tests for recursive cte by @xudong963 in #15788
- chore: remove code about access share table into from meta service by @lichuang in #15793
- chore: remove meta chaos from release action by @lichuang in #15796
- chore(query): substr support function domain by @zhyass in #15795
- chore: update doc about meta compatibility by @drmingdrmer in #15799
New Contributors
- @forsaken628 made their first contribution in #15444
- @hanxuanliang made their first contribution in #15465
- @maxjustus made their first contribution in #15529
- @baerwang made their first contribution in #15744
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.453...v1.2.530
May 6, 2024 (v1.2.452)
What's Changed
Exciting New Features ✨
- feat(query): implement ST_GEOMPOINTFROMGEOHASH by @kkk25641463 in #15182
- feat(rbac): drop role will transfer role owns object to account_admin by @TCeason in #15154
- feat(query): implement ST_MAKEPOLYGON by @kkk25641463 in #15194
- feat: distinguish early close from kill/cancel with new LogType::Closed. by @youngsofun in #15206
- feat: add EXECUTE IMMEDIATE statement by @andylokandy in #15144
- feat(http): get client ip from headers by @flaneur2020 in #15205
- feat(query): support alter table comment by @TCeason in #15227
- feat(query): match function support multiple fields with boost by @b41sh in #15196
- feat: support insert with cte by @xudong963 in #15241
- feat: add setting
script_max_steps
by @andylokandy in #15244 - feat: atomic CTAS by @dantengsky in #15186
- feat(query): integrate "like" to filter execution by @Dousir9 in #15233
- feat(query): add user option: disabled by @TCeason in #15249
- feat(query): Inverted index support set filters and index record options by @b41sh in #15254
- feat: add sequence meta api by @lichuang in #15247
- feat(query): add new setting parse_datetime_ignore_remainder. by @TCeason in #15260
- feat: http handler response add field
has_result_set
and stateStarting
. by @youngsofun in #15258 - feat: support cte in update selection by @xudong963 in #15276
- feat: impl create dynamic table sql parser and planner by @zhyass in #15250
- feat: support cte in delete by @xudong963 in #15284
- feat(query): implement ST_ASGEOJSON by @kkk25641463 in #15214
- feat(query): support
map_keys
andmap_values
function by @b41sh in #15291 - feat(query): implement TO_GEOMETRY/TRY_TO_GEOMETRY function by @kkk25641463 in #15302
- feat(query): 14925, support udf wasm by @shamb0 in #15107
- feat(query): top k syntax support by @TCeason in #15318
- feat(query): implement st_asewkb/st_aswkb/st_asewkt/st_aswkt/st_geohash by @kkk25641463 in #15325
- feat(query): support with clause in copy into by @sundy-li in #15343
- feat(query): inverted index support json type by @b41sh in #15267
- feat(query): support compute quota feature by @zhang2014 in #15336
- feat: Introduce enterprise storage quota feature by @Xuanwo in #15338
- feat(query): inverted index use empty position data when query not contain phrase terms by @b41sh in #15362
- feat(query): implement ST_DIMENSION function by @kkk25641463 in #15347
- feat(query): implement ST_ENDPOINT by @kkk25641463 in #15366
- feat(query): implement ST_POINTN by @kkk25641463 in #15373
- feat(query): support explain insert with insert source by @TCeason in #15357
- feat(query): implement ST_SRID by @kkk25641463 in #15377
- feat(query): implement ST_X/ST_Y/ST_STARTPOINT by @kkk25641463 in #15384
- feat: Add: stream based snapshot API install_snapshot_v2() to meta-service by @drmingdrmer in #15395
Thoughtful Bug Fix 🔧
- fix(query): set setting case insensitive by @TCeason in #15199
- fix(query): show grants panic when role/user has immute db/table privilege by @TCeason in #15188
- fix: geos-sys compile error #15200 by @drmingdrmer in #15209
- fix(query): alter modify col comment need modify when type not modify by @TCeason in #15217
- fix(query): fix negative session_running_acquired_queries by @zhang2014 in #15223
- fix: fix map get return double nested nullable by @ariesdevil in #15230
- fix: show locks in current tenant by @zhyass in #15252
- fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #15229
- fix: fix delete from subquery and one more condition bug by @lichuang in #15279
- fix(query): fix udf name with upper case letters by @b41sh in #15300
- fix: don't constant fold giant expr by @andylokandy in #15311
- fix(query): fix decimal op loss precision by @sundy-li in #15313
- fix(query): NDJSON copy into allow cast bool and number to string by @b41sh in #15308
- fix: incorrect overlap checking (replace into) by @dantengsky in #15315
- fix(typo): refering -> referring by @andylokandy in #15323
- fix: insert Plan should not have schema. by @youngsofun in #15330
- fix(executor): queries exeuctor points calc incorrectly by @dqhl76 in #15349
- fix(query): show processlit host is null in HTTPQuery by @TCeason in #15363
- fix: check all partitions for small files instead of the last only. by @youngsofun in #15360
- fix: structlog should print non-zero-rooted tree by @andylokandy in #15370
- fix: when listing-kv, limit the right bound if possible by @drmingdrmer in #15372
- fix: list stage returns at most 15000 files. by @youngsofun in #15385
- fix(query): spill block need consider scalar by @TCeason in #15387
- fix: Respond with
SnapshotMismatch
error instead of internal error by @drmingdrmer in #15391 - fix: solve some inconsistencies between code and comments by @YichiZhang0613 in #15394
Code Refactor 🎉
- refactor(query): remove useless code (system.tracing) by @zhang2014 in #15185
- refactor: explain merge into by @xudong963 in #15195
- refactor: simplify
DatabaseFactory
, just static dispatch by @drmingdrmer in #15203 - refactor(query): move api and metrics to servers mod by @zhang2014 in #15210
- refactor: add view system tables by @TCeason in #15184
- refactor: introduce
UninitTenant
andTIdentRaw
by @drmingdrmer in #15207 - refactor: define DatabaseNameIdent with TIdent by @drmingdrmer in #15235
- refactor: ScalarVisitor refactor push_down_prewhere by @TCeason in #15215
- refactor: replace String with Tenant in xxxPlan by @drmingdrmer in #15240
- refactor: avoid using PREFIX directly. use method to build list-prefix by @drmingdrmer in #15243
- refactor: remove unused
list_all_tables()
by @drmingdrmer in #15251 - refactor: remove kvapi::Key impl for String by @drmingdrmer in #15253
- refactor: grant_ownership retry logic by @TCeason in #15275
- refactor: Extract
kvapi::KeyCodec
trait fromkvapi::Key
by @drmingdrmer in #15262 - refactor: Introduce generic parameter
N
toTIdent
by @drmingdrmer in #15271 - refactor: define
ShareConsumer
withTIdent
by @drmingdrmer in #15280 - refactor: define VirtualColumnNameIdent with TIdent by @drmingdrmer in #15285
- refactor: define DbIdListKey with TIdent, and rename it to DatabaseIdHistoryIdent by @drmingdrmer in #15287
- refactor: define TenantQuotaIdent with TIdent by @drmingdrmer in #15289
- refactor: Define TableLockKey with TIdent by @drmingdrmer in #15319
- refactor: TIdent add a flag to indicate if the tenant should be encoded into key by @drmingdrmer in #15321
- refactor: define BackgroundJobIdIdent with TIdent, with HAS_TENANT disabled by @drmingdrmer in #15327
- refactor: http handler check has_result_set() before get schema of plan. by @youngsofun in #15329
- refactor: Define DataMaskIdIdent with TIdent by @drmingdrmer in #15331
- refactor: add right bound when listing expired keys to avoid unnecessary data copy by @drmingdrmer in #15334
- refactor: Upgrade Openraft to 0.9.7 by @drmingdrmer in #15339
- refactor: improve install-snapshot performance by @drmingdrmer in #15345
- refactor: move snapshot building io to other task. by @drmingdrmer in #15351
- refactor: only has password policy update user login res by @TCeason in #15356
- refactor: Define TenantOwnershiptObject with TIdent by @drmingdrmer in #15358
- refactor: define TenantUserIdent with TIdent by @drmingdrmer in #15361
- refactor: session_ctx.client_ip refactor to
Option<String>
by @TCeason in #15368 - refactor: move DirName to separate file; add peek() to KeyParser by @drmingdrmer in #15365
- refactor: Define StageFileIdent with TIdent by @drmingdrmer in #15371
- refactor: remove stage use stream to avoid OOM. by @youngsofun in #15378
- refactor: sync create inverted index by @b41sh in #15379
- refactor: dispatch load to 16 threads when installing snapshot by @drmingdrmer in #15381
- refactor: get_table_by_id() returns an
Option<Seqv<TableMeta>>
by @drmingdrmer in #15388 - refactor: remove get_table_name_by_id() by @drmingdrmer in #15389
Others 📒
- chore(executor): add time limit daemon for queries executor by @dqhl76 in #15174
- chore(query): max_running_queries from 0 to 8, enable the max running queries by @zhang2014 in #15190
- chore(query): add session_running_acquired_queries metric by @zhang2014 in #15193
- chore(executor): add end_of_stream flag for executor stream by @zhang2014 in #15202
- chore(planner): refine explain read size by @Dousir9 in #15204
- chore: change parquet_fast_read_bytes setting from 0 to 16MB by @BohuTANG in #15212
- chore(planner): fix decorrelate exists subquery by @Dousir9 in #15208
- chore: old op LogicalGet modify to new op name Scan by @TracyZYJ in #15222
- chore(query): parser error need to log error by @sundy-li in #15219
- chore: vacuum drop table for external location table by @BohuTANG in #15220
- chore(query): fix nullable array domain by @sundy-li in #15224
- chore(query): add query queued duration for query log by @zhang2014 in #15225
- chore(query): remove useless function map:get by @sundy-li in #15245
- chore: update sqllogictests crate to 0.19.1 by @finchxx in #15242
- chore(query): support like push down not by @Dousir9 in #15268
- chore(executor): add node level error tracking by @dqhl76 in #15239
- chore(query): refine set sql_dialect error message by @Dousir9 in #15282
- chore(query): bump arrow-udf to extra lib by @sundy-li in #15283
- chore(planner): improve delete explain by @Dousir9 in #15290
- chore: Enhance Flexibility of Dynamic Table Creation by Allowing Options in Any Order by @zhyass in #15297
- chore: add license to cargo.toml by @BohuTANG in #15299
- chore(planner): fix can_filter_null by @Dousir9 in #15298
- chore: improve distributed plan by @xudong963 in #15296
- chore(query): simplify stage list by @sundy-li in #15303
- chore: enable distributed optimization for subquery in delete by @xudong963 in #15288
- chore: correct stream error message by @zhyass in #15309
- chore: tune auto-compact parameters triggered by insertions by @dantengsky in #15261
- chore(query): improve delete pruner by @sundy-li in #15316
- chore(planner): fix merge into statistics by @Dousir9 in #15320
- chore(planner): add disable merge into join reorder settings by @Dousir9 in #15322
- chore: fix
RulePushDownLimitWindow
id by @xudong963 in #15326 - chore(query): add merge ratio in sort spill by @sundy-li in #15332
- chore: refine runtime filter log by @xudong963 in #15335
- chore: add err message to help capture new agg ht hang by @Freejww in #15344
- chore: enhance runtime filter log by @xudong963 in #15350
- chore(planner): support fast return for hash join spill by @Dousir9 in #15364
- chore(ci): fix flaky test; by @TCeason in #15369
- chore: add more thest to prefix_right_bound() by @drmingdrmer in #15374
- chore: purge inverted index by @dantengsky in #15354
- chore(query): reuse the script runtime during pipeline builder by @sundy-li in #15375
- chore: adjust row count assertion in table compaction by @dantengsky in #15380
- chore: if a txn log is too long, display only the first 5 items by @drmingdrmer in #15392
- chore: Fix opendal upstream commit missing by @Xuanwo in #15393
New Contributors
- @finchxx made their first contribution in #15242
- @YichiZhang0613 made their first contribution in #15229
- @shamb0 made their first contribution in #15107
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.410...v1.2.452
Apr 8, 2024 (v1.2.410)
What's Changed
Exciting New Features ✨
- feat(query): support settings admin api for global level settings by @zhang2014 in #14901
- feat: add a new ratio config instead of table_data_deserialized_data_bytes by @chienguo in #14896
- feat(ast): add parser for script by @andylokandy in #14891
- feat: support cross join spilling by @xudong963 in #14887
- feat: add more function alias names by @sundy-li in #14924
- feat(query): support spill for new agg hashtable by @Freejww in #14905
- feat(query): Support refresh inverted index by @b41sh in #14912
- feat: ndjson support option null_if. by @youngsofun in #14938
- feat(query): add statement_queued_timeout_in_seconds setting for queries queue by @zhang2014 in #14945
- feat: change tracking enabled table support merge into by @zhyass in #14900
- feat: histogram aggregate function by @suimenno3002 in #14839
- feat(query): add metrics for session queue manager by @everpcpc in #14966
- feat(query): add error_or function by @sundy-li in #14980
- feat: purge duplicated files (copy into table) by @dantengsky in #14962
- feat(query): support parse mulity
modify column? ty
by @TracyZYJ in #14979 - feat: support prql by @ncuwaln in #14922
- chore(query): support domain contains in string type by @sundy-li in #15023
- feat(query): check view's inner table privilege when create or alter view by @TCeason in #14986
- feat: add buffer for spiller by @xudong963 in #15021
- feat: add show views & desc view by @blackstar-baba in #14926
- feat: support pagination on task_history by @ZhiHanZ in #15047
- feat(ast): add parser for template sql by @andylokandy in #15026
- refactor: SHOW GRANTS should not display drop on table/database by @TCeason in #14931
- feat(query): Support query match inverted index by @b41sh in #14997
- feat: Implement concurrent stat by @Xuanwo in #15083
- feat(metasrv): add network metrics rpc_delay_ms by @everpcpc in #15084
- feat: create stream at navigation point by @zhyass in #15062
- feat(query): implement st_makeline() function by @kkk25641463 in #15059
- feat(planner): improve push down filter join by @Dousir9 in #14872
- feat: support otel context propagation for databend-query by @ZhiHanZ in #15096
- feat(query): add query_result_cache_min_execute_secs by @sundy-li in #15101
- feat(query): Support hook refresh inverted index after insert by @b41sh in #15094
- feat: time travel support at stream by @zhyass in #15116
- feat(query): implement st_geometryfromewkb function by @kkk25641463 in #15118
- feat: refactor ndjson input format. by @youngsofun in #14943
- feat(script): add script compiler by @andylokandy in #15113
- feat: support multi-table insert by @SkyFan2002 in #15002
- feat(query): to_timestamp support parser string to ts without timezone by @TCeason in #15131
- feat: Enhance show views by @blackstar-baba in #15109
- feat: support expr in values clause of multi table insert by @SkyFan2002 in #15147
- feat(query): implement ST_GEOMFROMGEOHASH by @kkk25641463 in #15156
- feat(query): show create table support display inverted index by @b41sh in #15169
- feat(query): support CHANGES clause by @zhyass in #15163
- feat: add inverted index pruning status by @BohuTANG in #15175
- feat: Support to add random seed on random engine by @blackstar-baba in #15167
- feat(planner): push down not in filter expression by @Dousir9 in #15168
- feat: time travel support at offset by @zhyass in #15177
Thoughtful Bug Fix 🔧
- fix(ci): fix internal benchmark by @TCeason in #14897
- fix: Display of
merge into .. insert *
by @dantengsky in #14892 - fix:
check_and_upgrade_to_pb()
should return MetaError, add quota by @drmingdrmer in #14899 - fix: Display of compact AST by @SkyFan2002 in #14914
- fix(query): fix native topk_sorter by @sundy-li in #14939
- fix: stream source can't get the same tablemeta by @zhyass in #14935
- fix: cluster table with change tracking enabled append panic by @zhyass in #14956
- fix: enable runtime filter when
join_spilling_memory_ratio
!=0 by @xudong963 in #14959 - fix: fix stream explain bug for merge into by @JackTan25 in #14969
- fix(query): Need return error udf_name when derialize from pb failed. by @TCeason in #14964
- fix: try fix compute stddev for decimal type overflow by @ariesdevil in #14952
- fix: csv read error with skip_header. by @youngsofun in #14981
- fix: fix incorrect agg spill in new agg hashtable by @Freejww in #14995
- fix: wrong row id by @youngsofun in #15018
- fix: trim leading '/'s for files specified in copy opt by @dantengsky in #15056
- fix: stream get wrong result if change tracking is changed by @zhyass in #15072
- fix(ast): fix incorrect display for ast (part 1) by @andylokandy in #15069
- fix: privilege check for insert statement by @SkyFan2002 in #15075
- fix(query): fix attach_query_str by @sundy-li in #15066
- fix: fix plan_sql error can not write query_log by @BohuTANG in #15077
- fix: improve error hints when select * from xx.yy by @ZhengLin-Li in #14663
- fix: Support udf for copy table by @JackTan25 in #15078
- fix(ast): fix incorrect display for ast (part 2) by @andylokandy in #15076
- fix: make block data type be consistent with hash method by @xudong963 in #15089
- fix: add prql description by @ncuwaln in #15091
- fix: sqllogicaltest datatype mismatch in
expr_in_window.test
by @c8ef in #15090 - fix: schema mismatch in
named_window_basic.test
by @c8ef in #15099 - fix: test duplicate in
window_basic.test
by @c8ef in #15100 - fix(query): verify_privilege need check privilege one by one by @TCeason in #15105
- fix(query):user option DEFAULT_ROLE ignore ascii_case by @TCeason in #15137
- fix(query): tuple field name can not contain special char by @TCeason in #15126
- fix(query): new agg hashtable hung with little data on singleton by @Freejww in #15140
- fix(query): show query with name consider quote by @TCeason in #15138
- fix: forbid end at stream syntax by @zhyass in #15173
- fix(storage): Don't return empty credential while loading failed by @Xuanwo in #15181
Code Refactor 🎉
- refactor: upgrade to openraft 0.9.0 by @drmingdrmer in #14907
- refactor: use kvapi::Key to define meta-service key for QuotaMgr by @drmingdrmer in #14915
- refactor: reduce hash join build spilling files by @xudong963 in #14919
- refactor: define kvapi::Key for PasswordPolicyMgr by @drmingdrmer in #14920
- refactor: reduce meta call by @TCeason in #14917
- refactor: add NetworkPolicyIdent as NetworkPolicy meta-service key by @drmingdrmer in #14933
- refactor: http handler plan sql async. by @youngsofun in #14908
- refactor: Add
TIdent
for all meta-service key that share the same form of<PREFIX>/<TENANT>/<NAME>
by @drmingdrmer in #14940 - refactor: add generic CrudMgr to implement various simple CRUD manager by @drmingdrmer in #14988
- refactor(query): use project plan for copy into select by @zhang2014 in #14977
- refactor: use CrudMgr to impl PasswordPolicyMgr by @drmingdrmer in #14994
- refactor: Add ExistError and UnknownError, TenantResource does not need to define error type any more by @drmingdrmer in #14996
- refactor: Use CrudMgr to impl FileFormatMgr by @drmingdrmer in #14999
- refactor: Use CrudMgr to impl ConnectionMgr by @drmingdrmer in #15003
- refactor(query): add copy into location physical plan by @zhang2014 in #15010
- refactor(query): csv reader support prefetch by @youngsofun in #14983
- refactor: alter view only need to modify table option by @TCeason in #15012
- refactor: truncate support build pipeline by @zhyass in #15027
- refactor: move id-generator from crate meta-api to meta-app by @drmingdrmer in #15036
- refactor: remove innecessary serde by @drmingdrmer in #15037
- refactor(query): use scan table physical plan for copy into table from stage by @zhang2014 in #15016
- refactor: remove innecessary serde by @drmingdrmer in #15040
- refactor: remove innecessary serde by @drmingdrmer in #15042
- refactor: spead up csv reader. by @youngsofun in #15043
- refactor(query): remove metrics if moved to the profile by @zhang2014 in #15054
- refactor: reduce hash join probe spilled files by @xudong963 in #14942
- refactor: separate key and value: add
BackgroundTaskCreator
by @drmingdrmer in #15063 - refactor: remove serde from CatalogId and CatalogNameIdent by @drmingdrmer in #15068
- refactor: remove serde from kvapi::Key types by @drmingdrmer in #15070
- refactor: use type
Tenant
forBackgroundJobIdent.tenant
by @drmingdrmer in #15074 - refactor: BackgroundTaskIdent use Tenant by @drmingdrmer in #15080
- refactor: make the block pruning more clear by @BohuTANG in #15079
- refactor: change PartInfo is_lazy to LazyLevel by @BohuTANG in #15081
- refactor:
QueryFragmentsPlanPacket
should include just setting changes by @drmingdrmer in #15095 - refactor(executor): refactor queries executor to prepare by @dqhl76 in #14965
- refactor: show grants support display ObjectId by @TCeason in #15064
- refactor: replace
NonEmptyString
withTenant
by @drmingdrmer in #15103 - refactor: replace String with Tenant by @drmingdrmer in #15119
- refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdListIdent and DataMaskNameIdent by @drmingdrmer in #15123
- refactor: remove count-table for tenant. It is not used at all by @drmingdrmer in #15134
- refactor: use Tenat in VirtualColumnNameIdent by @drmingdrmer in #15135
- refactor(query): record physical plan level metrics into profiling log by @zhang2014 in #15092
- refactor: define ShareEndpointIdent with TIdent by @drmingdrmer in #15136
- refactor: add peer info to network error context by @drmingdrmer in #15139
- refactor: add Tenant to ShareConsumer by @drmingdrmer in #15142
- chore: client should not depend on the final response. by @youngsofun in #15146
- refactor(executor): globalize
QueriesPipelineExecutor
creation method by @dqhl76 in #15129 - refactor(planner): remove statistics collection from bind phase by @Dousir9 in #15112
- refactor(query): only intercept dml and queries in queries queue by @zhang2014 in #15143
- refactor: user can execute show grants query itself by @TCeason in #15149
- refactor: generate inverted indexs for each blocks by @b41sh in #15150
- refactor: remove dependency on parquet2: Part I by @SkyFan2002 in #15158
- refactor: use
Tenant
forListBackgroundTasksReq
by @drmingdrmer in #15176 - refactor: move TIdent into separate files by @drmingdrmer in #15179
Others 📒
- chore(storage): split mutation_log file by @zhyass in #14903
- chore(query): create user/role name can not contain single/double quote by @TCeason in #14902
- chore(ci): benchmark local need set meta client timeout by @TCeason in #14910
- chore: extend time tolerance for meta-service test by @drmingdrmer in #14927
- chore: add step logs for hash join build side by @xudong963 in #14930
- chore: use shorter name for management API by @drmingdrmer in #14937
- chore(ci): add pr summary check by @everpcpc in #14944
- chore(ci): fix set failure for pr check by @everpcpc in #14946
- chore(ci): fix description check by @everpcpc in #14947
- chore(ci): fix event for pr assistant by @everpcpc in #14948
- chore(query): improve push_column_with_selection by @Dousir9 in #14954
- chore(query): add disable system tables by @zhang2014 in #14958
- chore(planner): fix distributed query plan by @Dousir9 in #14951
- chore: tweak txn syntax by @SkyFan2002 in #14960
- chore(query): add settings level for admin api by @zhang2014 in #14967
- chore(ast): CreatOption::None -> CreateOption::Create by @andylokandy in #14941
- chore: improve histogram‘s implement and comments by @suimenno3002 in #14976
- chore: number range for settings by @BohuTANG in #14985
- chore: group limit optimization in new hashtable by @sundy-li in #14989
- chore: add integration test for runtime filter by @xudong963 in #14992
- chore: upgrade openraft to 0.9.1 by @drmingdrmer in #15006
- chore: max_running_queries from 8 to 0, disable the max running queries by @BohuTANG in #15013
- chore: tweak deletion batch size & log messages by @dantengsky in #15005
- chore: rollback partial tests by @JackTan25 in #15015
- chore: add benchmark for loading data on cloud by @everpcpc in #15017
- chore: add a setting for inlist to join by @xudong963 in #15024
- chore: unify TIdent key format by @drmingdrmer in #15025
- chore: the fist http resp wait longer to tolerant old bendsql clients. by @youngsofun in #15022
- chore: add log about presign elapsed by @flaneur2020 in #14990
- chore: fix sql seq for benchmark report by @everpcpc in #15030
- chore: print query detail for bench & fix prepare database for load bench by @everpcpc in #15032
- chore: default enalbe merge into by @BohuTANG in #15034
- chore: run load test with new label by @everpcpc in #15035
- chore(query): add spill profile for join by @zhang2014 in #15044
- chore(http): tune log on getting /v1/query/:id/kill by @flaneur2020 in #14987
- chore: default enable aggregate/sort/join spill by @BohuTANG in #15038
- chore: do not drop database first for bench by @everpcpc in #15041
- chore(query): improve group by @sundy-li in #15031
- chore(planner): fix invalid aggregate physical plan by @Dousir9 in #15046
- chore: adjust the default value for table lock timeout by @zhyass in #15055
- chore: use Tenant::new_literal() and new_nonempty() to avoid creating empty Tenant by @drmingdrmer in #15061
- chore(ci): fix benchmark job dependency by @everpcpc in #15087
- chore: remove rebinding merge into for insert-only by @xudong963 in #15086
- chore: clean z3 and meta dir by @zhyass in #15098
- chore(executor): remove duplicate profile statistics by @zhang2014 in #15115
- chore(query): errorcode use snake shape format by @TCeason in #15122
- chore: treat seq as place holder when navigate by @zhyass in #15124
- chore(planner): make optimize function async by @Dousir9 in #15121
- chore: adjust stack size of flaky ut by @SkyFan2002 in #15130
- chore: to avoid UT stackoverflow, disable async_backtrace of
TypeChecker::resolve
by @dantengsky in #15133 - chore(query): refactor new transform partition bucket for new aggregation hashtable by @Freejww in #15093
- chore: enable inlist to join in cluster by @xudong963 in #15108
- chore: remove no longer used message encoding benchmark by @dantengsky in #15148
- chore(query): fix order for scalar cmp with null by @sundy-li in #15145
- chore: fix typos by @Dousir9 in #15153
- chore(query): use i64 min max value as histogram bucket min max limit by @zhang2014 in #15152
- chore(query): turn on new agg hashtable by @Freejww in #15155
- chore(query): row-fetcher-cache by @sundy-li in #15159
- chore: add CopyInto to query queue by @BohuTANG in #15160
- chore: refine the query/02_function to query/functions by @BohuTANG in #15161
- chore: refine the query queue is_heavy_action func by @BohuTANG in #15166
- chore: fix window plan in lazy topn by @sundy-li in #15165
- chore(deps): bump h2 from 0.3.24 to 0.3.26 by @dependabot in #15172
- chore(query): fix boolean type comparison by @Dousir9 in #15171
- chore: return inserted rows for each table in multi table insert by @SkyFan2002 in #15178
New Contributors
- @suimenno3002 made their first contribution in #14839
- @ncuwaln made their first contribution in #14922
- @blackstar-baba made their first contribution in #14926
- @kkk25641463 made their first contribution in #15059
- @c8ef made their first contribution in #15090
Full Changelog: https://github.com/databendlabs/databend/compare/v1.2.371...v1.2.410
Mar 11, 2024 (v1.2.371)
What's Changed
Exciting New Features ✨
- feat: add div0 divnull method by @sundy-li in #14693
- feat(query): grant object visibility check by @TCeason in #14458
- feat: add a new
kvapi::Key
type:Tenant
by @drmingdrmer in #14704 - feat(query): privilege access check need consider if exists by @TCeason in #14700
- feat: add kvapi::Value to define meta-service value behavior by @drmingdrmer in #14708
- feat(geo): support geometry type for create table by @ariesdevil in #14615
- feat(query): support vacuum temporary files by @zhang2014 in #14690
- feat: add
parent()
method to kvapi::Key to describe the hierarchical structure of meta-service data by @drmingdrmer in #14716 - feat(query): integrate hash join with new filter framework by @Dousir9 in #14689
- feat: add kvapi::ValueType::dependency_keys() to retrieve relation such as table-name-to-table-id and table-id-to-table-meta by @drmingdrmer in #14735
- feat: support multi-statement transactions by @SkyFan2002 in #14562
- feat: DESC[RIBE] [TABLE] name by @dracoooooo in #14736
- feat(query): new implementation of analyze table by @sundy-li in #14725
- feat: add Analyze for merge into by @JackTan25 in #14763
- feat(executor): implement time slicing for shared executor scheduling across queries by @dqhl76 in #14770
- feat(sqlsmith): Support generating binary and geometry values by @b41sh in #14758
- feat: get_client_info() returns the server time in milliseconds by @drmingdrmer in #14805
- feat: http handler txn support. by @youngsofun in #14802
- feat(planner): implement explain query in verbose mode by @leiysky in #14801
- feat: add new string view type [part 1] by @ariesdevil in #14662
- feat: upgrade arrow to version 50 by @sundy-li in #14784
- feat(query): introduce arrow-udf-js by @sundy-li in #14799
- feat: support left-related join spilling by @xudong963 in #14828
- feat: impl like selectivity compution by probability predication by @xudong963 in #14846
- feat: add vacuum table result table by @lichuang in #14830
- feat(query): read write inverted index by @b41sh in #14827
- feat: change tracking enabled table support replace into by @zhyass in #14831
- feat: support notification syntax with rpc, and support task error notification integration by @ZhiHanZ in #14845
- feat: support right-related join spilling by @xudong963 in #14853
- feat: add vacuum table dry run result table by @lichuang in #14851
- feat: add transaction and mutli-line SQL script support for task by @ZhiHanZ in #14865
- feat: add vacuum drop table dry run result table by @lichuang in #14870
- feat(query): support variant is_type functions by @b41sh in #14876
- feat(query): support queries queue by @zhang2014 in #14878
- feat(query): create drop inverted index by @b41sh in #14859
- feat(query): show grants support mget table/db name by table/db id. by @TCeason in #14862
Thoughtful Bug Fix 🔧
- fix: do compact first , then do sort for cluster table by @JackTan25 in #14707
- fix(query): filter response session settings with ScopeLevel::Session by @everpcpc in #14702
- fix: fix create or replace table as select bug by @lichuang in #14730
- fix: check if global memory is negative by @xudong963 in #14733
- fix: Don't return empty credentail if load failed by @Xuanwo in #14734
- fix: wrong result for CSV EMPTY_FIELD_AS = STRING. by @youngsofun in #14744
- fix: fix create or replace stage bug by @lichuang in #14745
- fix: force attachment to use EmptyFieldAs::FieldDefault. by @youngsofun in #14767
- fix: fix create or replace stage bug, only remove dir when success by @lichuang in #14766
- fix(planner): use correct alias for column reference in select clause by @leiysky in #14775
- fix(executor): queries pipeline executor schedule incorrectly by @dqhl76 in #14787
- fix: fix hash join spilling hang and add more tests by @xudong963 in #14794
- fix(ast): add missing #[drive(skip)] by @andylokandy in #14823
- fix: Transaction state is not changed correctly when returning error in commit statement by @SkyFan2002 in #14824
- fix(query): today, yesterday, tomorrow need consider tz by @TCeason in #14841
- fix: flaky sqllogic test by @zhyass in #14848
- fix: fix wrong results for left join spilling by @xudong963 in #14860
- fix: allow insert in multiline statment by @andylokandy in #14882
- fix: rewrite ifnull and nvl as coalesce by @PsiACE in #14877
- fix: fix incorrect schema in window expr in subquery with lazy_materialization by @ariesdevil in #14895
Code Refactor 🎉
- refactor: new impl for loading CSV. by @youngsofun in #14645
- refactor: add UdfError::Exist by @drmingdrmer in #14691
- refactor: remove kvapi::Key prefix const by @drmingdrmer in #14701
- refactor: add
Humanizer
traits for operator display by @leiysky in #14718 - refactor: schema_api::update_mask_policy by @drmingdrmer in #14724
- refactor: rename ShareAccountNameIdent::acount to tenant; remove unused serde derive by @drmingdrmer in #14722
- refactor: support decimal for quantile_cont by @ariesdevil in #14726
- refactor(query): refactor runtime part 4 by @zhang2014 in #14728
- refactor(query): migrate profile to common base by @zhang2014 in #14739
- refactor: refactor hash join spill probe side by @xudong963 in #14743
- refactor(storage): treat the standard stream as append only when only insertion by @zhyass in #14748
- refactor: Add RoleGrantee to provide meta-service key for RoleMgr by @drmingdrmer in #14752
- refactor: upgrade Openraft v0.9.0-alpha.5..v0.9.0-alpha.6 by @drmingdrmer in #14769
- refactor: hash join build spilling by @xudong963 in #14768
- refactor: Add RoleIdent to define kvapi::Key to access roles in meta-service by @drmingdrmer in #14772
- refactor: make IndexNameIdent.tenant a Tenant struct by @drmingdrmer in #14776
- refactor: refactor join spilling settings by @xudong963 in #14781
- refactor: add TenantUserIdent as meta-service key to access user data by @drmingdrmer in #14793
- refactor(executor): tracking memory for plan level by @zhang2014 in #14747
- refactor(ast): rename CreateOption::CreateIfNotExists(false) -> CreateOption::None by @andylokandy in #14795
- refactor(ast): refine crate structure by @andylokandy in #14803
- refactor: upgrade Openraft to v0.9.0-alpha.7 by @drmingdrmer in #14809
- refactor: tenant is not allowed to be empty by @drmingdrmer in #14815
- refactor(ast): add derive-visitor for expr and query by @andylokandy in #14814
- refactor(ast): add derive-visitor for Statement by @andylokandy in #14822
- refactor: remove backward compatibility for data version V0 by @drmingdrmer in #14818
- refactor: use relative expire time
ttl
to replaceexpire_at
, tolerate query-meta time drift by @drmingdrmer in #14807 - refactor: upgrade openraft v0.9.0-alpha.7 to v0.9.0-alpha.8 by @drmingdrmer in #14832
- refactor: enable feature flag openraft:storage-v2 by @drmingdrmer in #14838
- refactor: use NonEmptyString to access UDFMgr. by @drmingdrmer in #14844
- refactor: use NonEmptyString for creating StageMgr by @drmingdrmer in #14850
- refactor: add StageIdent and StageFileIdent by @drmingdrmer in #14863
- refactor: use KVPbApi to simplify StageMgr by @drmingdrmer in #14871
- refactor: use NonEmptyString to create Settings by @drmingdrmer in #14875
- refactor: enable openraft feature flag generic-snapshot-data by @drmingdrmer in #14881
- refactor: use SettingIdent to build string key to access settings by @drmingdrmer in #14883
- refactor(ast): migrate to derive-visitor by @andylokandy in #14829
- refactor(ast): remove databend-common-functions dependency by @andylokandy in #14893
Build/Testing/CI Infra Changes 🔌
- ci: add tests for transaction. by @youngsofun in #14826
- ci: fix clippy tests. by @youngsofun in #14855
Others 📒
- chore: add plan id for mutations by @JackTan25 in #14685
- chore: install sqlite3 for build-tool by @ariesdevil in #14695
- chore: install libsqlite3 for build-tool by @ariesdevil in #14696
- chore: minor refactor of
adjust_bloom_runtime_filter
by @xudong963 in #14697 - chore: extract common functions from metactl by @drmingdrmer in #14698
- chore(query): support custom query graceful shutdown timeout by @zhang2014 in #14712
- chore(query): improve new filter execution by @Dousir9 in #14715
- chore: add metrics for compact and recluster by @zhyass in #14717
- chore: static features for async-compression deps by @everpcpc in #14711
- chore: remove stream pruner by @zhyass in #14723
- chore(query): improve aggregate unary function by @sundy-li in #14729
- chore: polish error message. by @youngsofun in #14737
- chore: rename by @JackTan25 in #14740
- chore: refactor RoleMgr by @drmingdrmer in #14741
- chore: build-tool base for both arm64 & amd64 by @everpcpc in #14751
- chore: fix setup protoc aarch64 by @everpcpc in #14754
- chore: fix compile error by @SkyFan2002 in #14753
- chore: change build image to debian/alpine by @everpcpc in #14755
- chore(ci): fix LIBCLANG_PATH for alpine build-tool by @everpcpc in #14764
- chore(query): fix location ver by @sundy-li in #14762
- chore(query): add max_running_query_execute_time for http status by @zhang2014 in #14750
- chore(query): support string eq int by @sundy-li in #14759
- chore: add more transaction test by @SkyFan2002 in #14746
- chore(ci): change build runner with target by @everpcpc in #14761
- chore: fix build for cloud by @everpcpc in #14773
- chore(query): use create_time of process info for http status by @zhang2014 in #14777
- chore: show create table with location by @ariesdevil in #14778
- chore(query): add drop guard for impl drop trait by @zhang2014 in #14783
- chore(ci): fix target for release by @everpcpc in #14785
- chore(query): remove codes of storage/parquet2 by @sundy-li in #14305
- chore: patch z3-sys & add musl build to ci by @everpcpc in #14786
- chore: fix musl build dockerfile by @everpcpc in #14790
- chore: release add musl targets by @everpcpc in #14791
- chore(test): reduce the number of tables in large join test by @Dousir9 in #14797
- chore: add Date Type for runtime filter by @JackTan25 in #14780
- chore: rename update_star_columns by @JackTan25 in #14806
- chore(query): add unit test for mem stat by @zhang2014 in #14812
- chore(ci): include nohup.out in failure artifact by @everpcpc in #14820
- chore: remove unused package patch by @everpcpc in #14821
- chore(ci): fix upload/download sha256sums artifacts by @everpcpc in #14833
- chore(ci): fix release artifacts conflict by @everpcpc in #14834
- chore(ci): fix artifacts for benchmark by @everpcpc in #14837
- chore: tolerate inaccuracy of expire time for test by @drmingdrmer in #14836
- chore(query): modify new_agg_hashtable payload transmission method on singleton and support new_agg_hashtable on cluster by @Freejww in #14798
- chore: Enable stage table distributed by @JackTan25 in #14847
- chore: disable inlist to subquery under cluster by @xudong963 in #14864
- chore(query): enable fast return for cluster hash join by @Dousir9 in #14854
- chore: log txn id for multi stmt transaction by @SkyFan2002 in #14861
- chore: improve tolerance for expire test
test_kv_read_v1_on_follower
by @drmingdrmer in #14868 - chore: open join spilling by default by @xudong963 in #14843
- chore: increase tolerance for expiration test by @drmingdrmer in #14880