What's Changed

Exciting New Features ✨

• feat(query): implement ST_XMAX by @kkk25641463 in #15406
• feat(executor): record query profiling for standalone error query by @zhang2014 in #15425
• feat(query): add external_server_request_batch_rows setting by @sundy-li in #15440
• feat(function): Implement map_size function by @forsaken628 in #15444
• feat(query): add created_time column to processes table by @TCeason in #15448
• feat(function): Implement map_cat function by @shamb0 in #15348
• feat: support adjust query's priority by @dqhl76 in #15352
• feat(executor): record errors info into profiling log by @zhang2014 in #15472
• feat: optimize vacuum drop table, execute in parallel by @lichuang in #15478
• feat: http handler add endpoint /v1/login/. by @youngsofun in #15483
• feat(functions): add map_contains_key scalar function by @hanxuanliang in #15465
• feat(query): build image limit python 3.12.2 by @sundy-li in #15501
• feat(query): ensure git is installed in pyenv by @sundy-li in #15505
• feat(query): pyenv 3.12.2 by @sundy-li in #15512
• feat(query): support unset session setting by @TCeason in #15510
• feat: add version metrics for meta & query by @everpcpc in #15522
• feat: support explain and profile multi table insert statemennt by @SkyFan2002 in #15518
• feat(query): add query query_hash and query_parameterized_hash to query log by @TCeason in #15531
• feat: add meta key_num metrics by @lichuang in #15536
• feat(query): bump arrow-udf version to enhance types supported by JS UDFs by @maxjustus in #15529
• feat: resolve conflict when multi table insert commit by @SkyFan2002 in #15503
• feat(query): udf script support load from stage location by @sundy-li in #15542
• feat(metasrv): add metrics for snapshot key numbers by @everpcpc in #15545
• feat: select support consume stream by @zhyass in #15541
• feat(query): show users support display roles by @TCeason in #15544
• feat: remove XML input format. by @youngsofun in #15540
• feat(query): create table support add inverted index by @b41sh in #15547
• feat: fuzzy table data disk cache key reload by @dantengsky in #15566
• feat(query): variant type as map value support bloom filter by @b41sh in #15562
• feat(metasrv): add metrics for db_size by @everpcpc in #15595
• feat: support millisecond level task scheduling by @ZhiHanZ in #15599
• feat(query): show grants support where/limit option by @TCeason in #15607
• feat: otlp logging support http endpoint by @everpcpc in #15636
• feat(planner): support ExpressionScan and CacheScan by @Dousir9 in #15561
• feat(query): implement ST_YMAX/ST_XMIN/ST_YMIN by @kkk25641463 in #15594
• feat(query): add python udf script runtime by @sundy-li in #15494
• feat: compact and recluster add table lock by @zhyass in #15632
• feat(query): support disable vacuum temporary files after query by @zhang2014 in #15673
• feat(query): Role/UserInfo add field create_on and udpate_on by @TCeason in #15661
• feat: unload parquet with version as created_by. by @youngsofun in #15680
• feat(metrics): add process collector by @flaneur2020 in #15591
• feat: add meta-service config sled_max_cache_size_mb by @drmingdrmer in #15709
• feat(query): add async retry transform by @sundy-li in #15647
• feat: basic support input file format ORC. by @youngsofun in #15720
• feat: recluster unclustered blocks by @zhyass in #15623
• feat: add task_history function by @ZhiHanZ in #15731
• feat(query): show indexes add inverted indexes by @b41sh in #15733
• feat: Bump to opendal v0.47.0 rc by @Xuanwo in #15745
• feat: Add http config for all storage services by @Xuanwo in #15757
• feat(query): add inverted index size to block meta by @b41sh in #15752
• feat: add meta verifier binary by @lichuang in #15756
• feat(query): add shuffle before window plan by @sundy-li in #15726
• feat: parquet support option null_if by @youngsofun in #15776
• feat: rm CREDENTIALS as alias of CONNECTION. by @youngsofun in #15780
• feat(function): Implement map_delete function by @shamb0 in #15480
• feat(query): support try_to_timestmap(string, string), try_to_date(string, string) by @TCeason in #15775
• feat: add io delay chaos meta test scripts by @lichuang in #15769
• feat: support recursive cte by @xudong963 in #15675
• feat(ci): add logging test for query by @everpcpc in #15792
• feat: support dynamic enable of rust backtrace by @dqhl76 in #15749

Thoughtful Bug Fix 🔧

• fix(query): fix cse projection by @sundy-li in #15409
• fix(storage): change tracking with alias update row_version bug in merge into by @zhyass in #15412
• fix(executor): fix broken merge into source profiling by @zhang2014 in #15415
• fix(query): ensure all inputs pull all data that less than max partition count in new agg singleton by @Freejww in #15418
• fix(query): fix array_concat function domain panic by @b41sh in #15424
• fix: ExistsTable should has_result_set. by @youngsofun in #15431
• fix: max_files not work for transform on ndjson/csv/tsv. by @youngsofun in #15441
• fix(functions): improve array_prepend and array_append by @andylokandy in #15437
• fix(executor): fix queries executor cannot kill problem and reactivate kill test assertion by @dqhl76 in #15443
• fix(query): fix create table with cluster by get map type panic by @b41sh in #15475
• fix(query): RoleInfo from_pb should not err by @TCeason in #15496
• fix: fix group by cube with join panic assert by @lichuang in #15515
• fix: Enable opendal backtrace output to help debug by @Xuanwo in #15521
• fix: explain ast of multi table insert stmt by @dantengsky in #15527
• fix: Prevent premature snapshot commit on unfinished writes by @drmingdrmer in #15534
• fix(query): fix view flaky test with different dialect by @b41sh in #15533
• fix(query): fix qualify display by @sundy-li in #15537
• fix: FlightData must contain valid data header by @Xuanwo in #15559
• fix: http handler kill not working. by @youngsofun in #15558
• fix: Avoid duplicate temp file name for install-snapshot requests by @drmingdrmer in #15565
• fix(query): fix shrink_d256 for negative value by @sundy-li in #15574
• fix(query): fix bloom fitler using is_null and not function by @b41sh in #15577
• fix(query): fix filter with nested not expr return wrong result by @b41sh in #15580
• fix: fix normalize disjunctive filter by @xudong963 in #15587
• fix: consume stream inside explicit txn by @dantengsky in #15582
• fix: TxnManager with state AutoCommit should not be saved in http manager. by @youngsofun in #15602
• fix(query): fix subquery writer in udf by @sundy-li in #15610
• fix: analyze table error by @zhyass in #15614
• fix: update derive-vistor to unblock rustc upgrage by @andylokandy in #15625
• fix: allow http client retry when lost response with next_uri=/final. by @youngsofun in #15624
• fix: subquery correlated column is derived column by @xudong963 in #15629
• fix(query): unify optimize plan call in one place by @sundy-li in #15630
• fix: missing service label for otlp logs by @everpcpc in #15639
• fix(query): delete len check in mget_database/table_names_by_ids by @TCeason in #15640
• fix(udf): use Debug to display transport error on UDF server by @flaneur2020 in #15646
• fix(query): show grants also need to check user roles by @TCeason in #15650
• fix(query): fix current date/time functions return constant value in a statement. by @b41sh in #15659
• fix: merge into split by @dantengsky in #15644
• fix: make sure lock guard is dropped before next recluster by @zhyass in #15662
• fix(query): db owner role can access all tables under this db by @TCeason in #15634
• fix(cluster): fix complex plan serde in cluster actions by @zhang2014 in #15672
• fix: potential missing index creation in merge process by @dantengsky in #15674
• fix(query): fix result cache in empty result by @sundy-li in #15683
• fix: display expr considering associativity by @andylokandy in #15685
• fix: fix create table error but table id uncommttied by @lichuang in #15665
• fix: failed to select blocks for recluster by @zhyass in #15667
• fix: add unit test of concurrent commit table meta data by @lichuang in #15703
• fix(executor): fix lost pipeline on finished callback by @zhang2014 in #15693
• fix(executor): fix query log lost for http kill by @zhang2014 in #15715
• fix: fuse_snapshot only show the latest snapshot for external table by @dantengsky in #15721
• fix: fix union and lazy materialization by @xudong963 in #15727
• fix(ast): assert ast display and reparse by @andylokandy in #15666
• fix: Handle Duplicate Tombstones in Meta-Service List Operation by @drmingdrmer in #15736
• fix: rebuild missing bloom index while pruning by @dantengsky in #15738
• fix(storage): Fix a potential data loss while concurrent write by @Xuanwo in #15759
• fix: unload csv file too large. by @youngsofun in #15758
• fix: off by one mistake in auto compaction by @dantengsky in #15761
• fix(query): fix decimal cmp by @sundy-li in #15764
• fix: lazy materialization does not support set operation by @xudong963 in #15754
• fix: auto cast fail error in copy should show the related columns #15768 by @youngsofun in #15770
• fix: http handler handle panic when starting query. by @youngsofun in #15785
• fix(ci): meta chaos dockerfile path by @everpcpc in #15787
• fix: add settings enable_streaming_load, default 0. by @youngsofun in #15794
• fix(query): create ownership obejct only check current_role privilege by @TCeason in #15751

Code Refactor 🎉

• refactor: list stage use stream to avoid OOM. by @youngsofun in #15405
• refactor: Minimize lock duration during raft-log export by @drmingdrmer in #15417
• refactor: move storage/fuse/io/files.rs to storage/common/io/files.rs by @BohuTANG in #15428
• refactor: define CatalogIdIdent with TIdent by @drmingdrmer in #15429
• refactor: define CatalogIdToNameIdent with TIdent by @drmingdrmer in #15432
• refactor: set enable_experimental_rbac_check default 1 by @TCeason in #15436
• refactor(query): enable little vacuum after query finish by @zhang2014 in #15434
• refactor: remove MergeIntoSource operator by @xudong963 in #15435
• refactor: ScalarVisitor refactor replace_predicate_column by @TCeason in #15439
• refactor: use scalarvisitor replace predicate recursive by @TCeason in #15449
• refactor(query): remove useless settings by @zhang2014 in #15461
• refactor(query): add query id for query log by @zhang2014 in #15466
• refactor: Bump OpenDAL 0.46, arrow 51, tonic 0.11, reqwest 0.12, hyper 1, http 1 by @Xuanwo in #15442
• refactor: SessionContext doesn't require Arc by @drmingdrmer in #15491
• refactor: remove project operator by @xudong963 in #15479
• refactor(query): revert dev_setup by @sundy-li in #15516
• refactor: remove databend-common-meta-app from databend-common-ast by @andylokandy in #15509
• refactor: remove databend-common-base from databend-common-ast by @andylokandy in #15519
• refactor(query): allow udf server in insert source by @sundy-li in #15523
• refactor: remove redundant lifetime on fmt::Formatter by @andylokandy in #15528
• refactor: SessionContext::current_tenant should not be changed at runtime by @drmingdrmer in #15504
• refactor: migrate TSV input format to new framework. by @youngsofun in #15506
• refactor(cluster): refactor flight service actions by @zhang2014 in #15419
• refactor: Improve logging and snapshot management by @drmingdrmer in #15571
• refactor: Embed key number into snapshot id by @drmingdrmer in #15588
• refactor: Remove install_snapshot v0 meta-service API by @drmingdrmer in #15590
• refactor: Enhance Meta-Service Cluster Management by @drmingdrmer in #15601
• refactor: Decouple Snapshot Write and Commit Operations by @drmingdrmer in #15603
• refactor: polish memory usage when copy into parquet. by @youngsofun in #15596
• refactor(storage): bloom index filter use fold_with_domain instead fold by @b41sh in #15619
• refactor: refine cast reloperator error info by @xudong963 in #15648
• refactor: refactor merge into optimizer and fix unexpected distribution plan by @xudong963 in #15507
• refactor(query): no need to deserialize all userinfo when create user by @TCeason in #15671
• refactor: enable mem-profiling for meta-service by @drmingdrmer in #15705
• refactor: delete raft log one by one. by @drmingdrmer in #15723
• refactor: more explicit error confirmation for http query route error. by @youngsofun in #15621
• refactor: split None and Owner/Grantobject by @TCeason in #15739
• refactor(cluster): record query id for cluster mode log by @zhang2014 in #15609
• refactor: GrantEntry not store ownership privilege by @TCeason in #15730
• refactor: add workspace crates to Cargo.toml by @andylokandy in #15747
• refactor: remove unsafe impl Send+Sync for GlobalRegistry by @drmingdrmer in #15782
• refactor: remove meta-service V001 data version by @drmingdrmer in #15778
• refactor(ast): eliminate clone when displaying expr by @zhang2014 in #15773
• refactor: clustering_information support specify cluster keys by @zhyass in #15783
• refactor: add non-async version of mete-client API request_sync by @drmingdrmer in #15786

Build/Testing/CI Infra Changes 🔌

• ci: Bump version to 2024-02-08 (the same commit with 1.78) by @Xuanwo in #15455
• ci: tcph/tpcds use copy instead of streaming load for prepare data. by @youngsofun in #15681

Documentation 📔

• docs: delete repeat summary by @baerwang in #15744

Others 📒

• chore: Vendor parquet2 instead by @Xuanwo in #15422
• chore(query): refactor metrics by @sundy-li in #15421
• chore: remove unused file by @drmingdrmer in #15438
• chore(storage): allow append when change tracking was enabled by @zhyass in #15427
• chore: Bump to rust 2024-02-08 (also 1.78) by @Xuanwo in #15456
• chore: ignore stream column compare in block compact mutator by @zhyass in #15446
• chore: polish log for copy. by @youngsofun in #15463
• chore: improve show streams performance by @zhyass in #15460
• chore(ci): update deprecated actions by @everpcpc in #15470
• chore: change flood info log to debug by @BohuTANG in #15473
• chore: remove breakline for log by @BohuTANG in #15476
• chore: remove incorrect usage of Box::leak by @dantengsky in #15481
• chore: fix copy/spill log message by @BohuTANG in #15482
• chore: Upgrade opentelemetry and remove metrics crate by @Xuanwo in #15484
• chore: add KVPbApi::get_pb_stream() but it is not yet ready for production use by @drmingdrmer in #15471
• chore: log file_name name for short by @BohuTANG in #15487
• chore: Bump poem to 3.0 by @Xuanwo in #15486
• chore: fix some flood logs with batch progress by @BohuTANG in #15492
• chore(meta): tracking query id for meta log by @zhang2014 in #15497
• chore: Increase default io timeout for behavior changes by @Xuanwo in #15493
• chore(query): tracking query log for mysql handler by @zhang2014 in #15495
• chore: polish error message NDJSON. by @youngsofun in #15502
• chore(query): pyenv to 3.9 by @sundy-li in #15511
• chore: recluster disable sort spill by @zhyass in #15490
• chore: change load_file_metadata_expire_hours default from 24*7 to 12 hours by @BohuTANG in #15514
• chore(ci): use python3.12 as build tool base by @everpcpc in #15520
• chore: adjust elapsed time of various log messages by @dantengsky in #15517
• chore: Upgrade minitrace related crates to 0.6.5 by @Xuanwo in #15530
• chore: reduce snapshot building progress logging by @drmingdrmer in #15532
• chore: convert to arrow column remove ignore inside nullable by @b41sh in #15477
• chore: leave all un-committed data to purge/vacuum by @dantengsky in #15535
• chore: mask the config sensitive field for display by @BohuTANG in #15543
• chore: Remove tokio-console since we don't need it by @Xuanwo in #15546
• chore: Bump databend client and geozero to latest version by @Xuanwo in #15550
• chore: Bump rustls to 0.22 by @Xuanwo in #15552
• chore: remove unused micromarshal patch by @PsiACE in #15551
• chore(executor): set plain_statistics to false for spill time profiling statistics by @zhang2014 in #15555
• chore: change the short_sql from 64bytes to 10KB by @BohuTANG in #15554
• chore: support aborting table navigation by @dantengsky in #15549
• chore(query): add more heavy actions query type by @zhang2014 in #15556
• chore(storage): remove abort operation by @zhyass in #15557
• chore(query): all queryies should generate hash by @TCeason in #15560
• chore: Remove sentry entirely and bump databend client by @Xuanwo in #15563
• chore: Reduce the duplicated clap and gix by @Xuanwo in #15564
• chore(ast): unify all of {escape,unescape,quote,unquote}_{string/at_string/ident} by @andylokandy in #15548
• chore: short_sql from 10KB to 30KB by @BohuTANG in #15575
• chore: disk cache load refine and reduce the info flood logs by @BohuTANG in #15576
• chore: Bump jsonb version 0.4.0 by @b41sh in #15589
• chore: fix error capture of mulit statement transaction in mysql session by @SkyFan2002 in #15583
• chore: udpate doc for meta-service compat change by @drmingdrmer in #15593
• chore(deps): bump rustls from 0.22.2 to 0.22.4 by @dependabot in #15586
• chore: add more logs for vacuum temp files by @BohuTANG in #15597
• chore: Refine state machine v002 by @drmingdrmer in #15608
• chore(deps): bump gix from 0.62.0 to 0.63.0 by @dependabot in #15615
• chore: more commit logs by @dantengsky in #15617
• chore: prepare for SMV003: move shared code to separate dir by @drmingdrmer in #15622
• chore(query): add sort plan before window plan by @sundy-li in #15616
• chore(query): fix filter reorder check by @Dousir9 in #15613
• chore: reduce the short_sql to 128bytes by @BohuTANG in #15626
• chore: remove databend-common-{io, exception} from databend-common-ast by @andylokandy in #15618
• chore(query): add timeout for open telemetry logger by @zhang2014 in #15627
• chore: spill out short_sql to common/base/src/string.rs by @BohuTANG in #15633
• chore: add setting compact_max_block_selection by @zhyass in #15641
• chore(executor): add some time tracking log by @zhang2014 in #15651
• chore: minimize #[derive(skip)] by @andylokandy in #15654
• chore(planner): remove z3 by @Dousir9 in #15663
• chore: re-enable musl-g++ link for musl build image by @dantengsky in #15679
• chore(query): fix replace into from distributed select plan by @sundy-li in #15657
• chore: fix wrong print status by @zhyass in #15684
• chore(query): add assert on explain parser by @sundy-li in #15658
• chore(ci): add build for udf feature by @everpcpc in #15682
• chore(ci): fix release workflow by @everpcpc in #15687
• chore(ci): fix sqllogic test for release by @everpcpc in #15688
• chore(ci): fix pack for release by @everpcpc in #15689
• chore(ci): fix artifact upload/download for gcs by @everpcpc in #15691
• chore: add unit test of vacuum orphan table meta data by @lichuang in #15690
• chore(ci): fix checkout and version on release build by @everpcpc in #15692
• chore(ci): fix distribution for release by @everpcpc in #15695
• chore(ci): fix notify & publish release by @everpcpc in #15696
• chore(ci): fix release notify template by @everpcpc in #15697
• chore(query): add create query in admin tables api by @zhang2014 in #15694
• chore(ci): fix publish service image by @everpcpc in #15698
• chore(ci): fix run benchmark after release by @everpcpc in #15699
• chore(ci): ci-cloud build with python-udf by @everpcpc in #15701
• chore(ci): only provide debian based service image by @everpcpc in #15704
• chore(ci): fix download artifacts for docker_all_in_one by @everpcpc in #15708
• chore: more log for table history traversing by @dantengsky in #15710
• chore(executor): revert #15693 by @zhang2014 in #15711
• chore(ci): fix download artifacts for cloud build by @everpcpc in #15712
• chore(ci): fix dockerfile for cloud service image build by @everpcpc in #15713
• chore(ci): fix checking release status from jobs by @everpcpc in #15714
• chore(planner): fix merge predicates with different data types by @Dousir9 in #15725
• chore: add databend user to docker service images by @everpcpc in #15728
• chore(query): fix pipeline display by @sundy-li in #15729
• chore(ci): fix download artifacts for local benchmark by @everpcpc in #15735
• chore(ci): setup python3.12 for local benchmark by @everpcpc in #15737
• chore(ci): fix check release status by @everpcpc in #15748
• chore(ci): fix release notify icon by @everpcpc in #15750
• chore: output raft log index range for purge logging by @drmingdrmer in #15740
• chore: add more test case about create table fail by @lichuang in #15765
• chore: add error info when table name is misleading by @xudong963 in #15771
• chore: Pin opendal to released version by @Xuanwo in #15766
• chore: limit the count of set operation by @xudong963 in #15777
• chore: replace manually incr/decr with couter guard by @drmingdrmer in #15784
• chore(query): remove useless async keyword by @zhang2014 in #15781
• chore: add more alias name tests for recursive cte by @xudong963 in #15788
• chore: remove code about access share table into from meta service by @lichuang in #15793
• chore: remove meta chaos from release action by @lichuang in #15796
• chore(query): substr support function domain by @zhyass in #15795
• chore: update doc about meta compatibility by @drmingdrmer in #15799

New Contributors

• @forsaken628 made their first contribution in #15444
• @hanxuanliang made their first contribution in #15465
• @maxjustus made their first contribution in #15529
• @baerwang made their first contribution in #15744

Full Changelog: https://github.com/datafuselabs/databend/releases/tag/v1.2.530