Rust Polars 0.42.0

💥 Breaking changes

• Reject literal input in sort_by_exprs() (#17606)

🚀 Performance improvements

• Skip parquet page when unneeded (#18192)
• Improve binview extend/ifthenelse (#18164)
• Start on better Parquet delta decoding (#18049)
• Tune jemalloc to not create muzzy pages (#18148)
• Reduce default async thread count (#18142)
• Use single threaded algorithms if only 1 core given (#18101)
• Use Arc<Vec<_>> instead of Arc<[_]> for paths and hive partitions (#18066)
• SIMD View from FixedSizeBinary (#18059)
• Use bitmask to filter Parquet predicate-pushdown items (#17993)
• Zerocopy buffers for FixedSizeBinary to BinaryView cast (#18043)
• Integer fast path Parquet dict encoding (#18030)
• Speedup writing of Parquet primitive values (#18020)
• Remove temporary allocations in Parquet (#18013)
• Delay selection expansion (#18011)
• Optimize strings slices (#17996)
• Make .dt.weekday 20x faster (#17992)
• Shrink MemSliceInner enum (#17991)
• Push down slice with non-zero offset to Parquet (#17972)
• Reduce copy in MemSlice (#17983)
• Ensure metadata flags are maintained on vertical parallelization (#17804)
• Ensure only nodes that are not changed are cached in collapse optimizer (#17791)
• Use bitflags for OptState (#17788)
• Remove async directory auto-detection (#17779)
• Fix accidental quadratic horizontal concat (#17783)
• Batch parquet integer decoding (#17734)
• Use mmap-ed memory if possible in Parquet reader (#17725)
• Use bitflags for function options (#17723)
• Introduce MemReader to file buffer in Parquet reader (#17712)
• Better GC and push_view for binviews (#17627)
• Fix pathological perf issue in window-order-by (#17650)
• Cache path resolving of scan functions (#17616)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Rechunk before we go into grouped gathers (#17623)
• Cache schema resolve back to DSL (#17610)
• Add fastpath for when rounding by single constant durations (#17580)
• Improve parallelism in writing hive parquet (#17512)
• Support datetime in predicate during hive partition pruning (#17545)
• Batch nested embed parquet decoding (#17549)
• Batch nested Parquet decoding (#17542)
• Collect Parquet dictionary binary as view (#17475)
• Keep more parallelism when CSE plan cache hits (#17463)
• Batch parquet primitive decoding (#17462)
• Respect allow_threading in some more operators (#17450)
• Parallelize parquet metadata deserialization (#17399)

✨ Enhancements

• Create literals for datetime/date expressions (#18184)
• Create literals in 'datetime' expression (#18182)
• Add missing impl for Series (#18166)
• Raise on invalid 'is_between' and improve error message quality (#18147)
• Add boolean Parquet HybridRle encoding (#18022)
• Add nested SQL join support (#18006)
• Push down slice with non-zero offset to Parquet (#17972)
• Add support for binary size method to Expr and Series "bin" namespace (#17924)
• Add SQL interface support for PostgreSQL dollar-quoted string literals (#17940)
• Allow for parsing parquet file where the time zone is stored as lowercase "utc" (#17925)
• Expose binary_elementwise_into_string_amortized for plugin authors, recommend apply_into_string_amortized instead of apply_to_buffer (#17903)
• Decompress in CSV / NDJSON scan (#17841)
• Ensure unique names in HConcat (#17884)
• Support authentication with HuggingFace login (#17881)
• Support "BY NAME" qualifier for SQL "INTERSECT" and "EXCEPT" set ops (#17835)
• Raise informative error instead of panicking when passing invalid directives to to_string for Date dtype (#17670)
• Implement forward/backward fill for all types (#17861)
• Implement is_in operation on decimal type (#17832)
• Support hf:// in read_(csv|ipc|ndjson) functions (#17785)
• Allow literals in sort (#17780)
• Cloud support for NDJSON (#17717)
• Support API token for scanning hf:// (#17682)
• Raise error instead of panic in unsupported serde (#17679)
• Include file path option for NDJSON (#17681)
• Hugging Face path expansion (#17665)
• Add DSL validation for cloud eligible check (#17287)
• Raise informative error message if non-IntoExpr is passed by name in *Frame.group_by (#17654)
• Change API for writing partitioned Parquet to reduce code duplication (#17586)
• Cache schema resolve back to DSL (#17610)
• Expose returns_scalar to map_elements (#17613)
• Add option to include file path for Parquet, IPC, CSV scans (#17563)
• Support describe on decimal (#15092)
• Support datetime in predicate during hive partition pruning (#17545)
• Raise more informative error message for directories containing files with mixed extensions (#17480)
• Exclude empty files from directory/glob expansion (#17478)
• Add "future" versioning (#17421)
• Apply slice pushdown immediately to in-memory frames (#17459)
• Support writing hive partitioned parquet (#17324)
• Add right join support (#17441)
• Support hive partitioning in scan_ipc (#17434)

🐞 Bug fixes

• Fix struct shift and list builder (#18189)
• Don't load Parquet nested metadata (#18183)
• Throw bigidx error for Parquet row-count (#18154)
• Fix unpivot on empty df (#18179)
• Don't vertically parallelize cse contexts (#18177)
• Properly handle empty Parquet row groups with no dictionary (#18161)
• Struct outer nullabillity (#18156)
• Fix pyarrow predicate pushdown regression (#18145)
• Prevent unwanted supertype cast in 'search_sorted' (#18143)
• Parquet with filter=None (#18139)
• Don't raise when converting from pandas if index contains duplicate names when include_index=False (the default) (#18133)
• Don't remove leading whitespace in read_csv (#18131)
• Py-polars compilation with no features (#18129)
• String transform to_titlecase was too narrowly defined (#18122)
• Reading Parquet with Null dictionary page (#18112)
• Incorrect lazy CSV select(len()) for compressed files (#18067)
• Fix sink_ipc_cloud panicking with runtime error (#18091)
• Properly write Parquet for sliced lists (#18073)
• Panic reading multiple CSV files from cloud (#18056)
• Fix CloudWriter to use buffer before making requests (#18027)
• Fix typos and remove trailing whitespace (#18024)
• Handle cfg(feature) for shrink_dtype (#18038)
• Subtraction with overflow on negative slice offset in Parquet (#18036)
• Add nested SQL join support (#18006)
• Allow read_csv schema to take unparsable types (#17765)
• Multi-output column expressions in frame sort method (#17947)
• Fix Asof join by schema (#17988)
• Fix glob resolution for Hugging Face (#17958)
• Several parquet reader/writer regressions (#17941)
• Incorrect filter on categorical columns from parquet files (#17950)
• SQL COUNT(DISTINCT x) should not include NULL values (#17930)
• Scanning '%' from cloud (#17890)
• Respect glob=False for cloud reads (#17860)
• Properly write nest-nulled values in Parquet (#17845)
• Allow full-null Object series to be built (#17870)
• Fix from_arrow for struct type (#17839)
• Infer decimal scales on mixed scale input (#17840)
• Raise on unsupported fill strategy dtype (#17837)
• Properly write nested NullArray in Parquet (#17807)
• Check input type on list.to_struct (#17834)
• Fix right join schema (#17833)
• Non-compliant Parquet list element name (#17803)
• Correctly set should_broadcast flag in HStack CSE rewrite (#17784)
• Fix projection pusdhown of literals without names (#17778)
• Don't expand HTTP paths (#17774)
• Check funtion input len at expansion (#17763)
• Don't panic in invalid agg_groups (#17762)
• Raise empty struct (#17736)
• Fix GC logic in write_ipc (#17752)
• Panic in pl.concat_list and list.concat on empty inputs (#17742)
• Fix out nullability for structs coming from arrow (#17738)
• Percent encode for Hugging Face paths (#17718)
• Use bytemuck in slice reinterpret for Parquet ArrayChunks (#17700)
• Propagate struct outer nullability eagerly (#17697)
• Use ETag for HTTP file cache invalidation (#17684)
• Fix type inference failure caused by double transpose (#17663)
• Interpret %y consistently with Chrono in to_date/to_datetime/strptime (#17661)
• Fix explode invalid check (#17651)
• Tighten up error checking on join keys (#17517)
• Expand brackets in async glob expansion (#17630)
• Fix row index disappearing after projection pushdown in NDJSON (#17631)
• Fix struct -> enum is_in (#17622)
• Don't needlessly unwrap in pivot_schema (#17611)
• Reject literal input in sort_by_exprs() (#17606)
• Bitmap collect into safety (#17588)
• Method dt.truncate was sometimes returning incorrect results for pre-1970 datetimes (#17582)
• Defer path expansion until collect in file scan methods (#17532)
• Correct logic for descending sort of BooleanChunked (#17558)
• Don't unwrap send attempt to oneshot channel (#17566)
• Fix scanning from HTTP cloud paths (#17571)
• Properly implement struct (#17522)
• Add missing commas in python IR interchange (#17518)
• Fix predicate pushdown for .list.(get|gather) (#17511)
• Turn panic into error when serializing Object types (#17353)
• Fix struct expansion and raise on exclude (#17489)
• Fix decimal dyn float supertype (#17464)
• Don't rechunk on phys_repr (#17461)
• Harden alchemy session for old sqlalchemy versions (#17366)
• Fix swapping rename schema (#17458)
• Raise on oob decimal precision (#17445)
• Don't allow json inference method to be chunked/streaming (#17396)
• avoid panic when projecting solitary count into empty frame (#17393)
• Set literal nesting to 0 (#17392)
• Fix scanning cloud paths with spaces (#17379)
• Fix slice length no longer allowing None (#17372)
• Cull row index in scan if projection pushdown removes it (#17363)
• Fix typo in SchemaError exception message (#17350)

📖 Documentation

• Mention 'Array' in data types overview (#18060)
• Correct concat rechunk in user guide (#18080)
• Fix typo in title of Hugging Face docs page (#18097)
• Update pivot docstring for clarity (#18000)
• Fix aggregation guide discrepancies (#18003)
• Updating the join example input for rust for consistency with python example (#17898)
• Rename str.lengths to str.len_bytes in description text (#11577) (#17626)
• Create example for polars.Expr.bin.decode (#17508)
• Add right join in the user guide (#17608)
• Update SQL examples in README (#17568)
• Fix typo in Getting Started section of user guide (#17465)

📦 Build system

• Update Rust toolchain to nightly-2024-07-26 (#17891)

🛠️ Other improvements

• Consistently use PlRandomState, PlHashMap/Set (#18190)
• Remove unneeded growable (#18165)
• Allow non-scoped tasks to be spawned (#18163)
• Update Cargo.lock to fix build error on Linux (#18153)
• Remove Nth,Wildcard from ExprIR and make conversion falllible (#18115)
• Allow zip node to extend with nulls or broadcast inputs (#18077)
• Add @coastalwhite to parquet codeowners (#18032)
• Minor bump to comfy-table version (#18028)
• Add unit tests for str.contains_any and str.replace_many (#17961)
• Remove apply_generic, use unary_elementwise (#17902)
• Add general filters in Parquet (#17910)
• Fix broken typos command in make pre-commit for py-polars folder (#17897)
• Remove HybridRLE iter / batch nested parquet decoding (#17889)
• Deal with re-entrant expressions locally (#17885)
• Unify Parquet nested decoders (#17816)
• Add zip node to streaming engine (#17866)
• Fix simple projection in streaming engine (#17871)
• Make better use of typos configuration features (#17800)
• Remove complicated Parquet traits (#17767)
• Rename Unit to Plain in Parquet reader (#17751)
• Add missing feature flag for prefetching on arm (#17740)
• Add the capability to run new streaming engine on test suite (#17706)
• Update CODEOWNERS (#17707)
• Allow compute nodes in streaming engine to decide whether they want serial or parallel input/output (#17699)
• Name tests so they actually run (#17690)
• Add reduce ComputeNode in new streaming engine (#17389)
• Add DSL validation for cloud eligible check (#17287)
• Rename Pipe to Connector (#17655)
• Document and simplify MutableBinView::push_ignore_validity (#17645)
• Remove has_validity, use has_nulls (#17519)
• Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
• Move path logic to from utils to path_utils in polars-io (#17635)
• Fix struct gather (#17621)
• Back to StructChunked name (#17609)
• Re-enable struct related tests (#17597)
• Completely redo structure of Parquet decoder (#17589)
• Fix struct outer validity;fmt;is_in;cast;cmp (#17590)
• Make no-default-features compile (#17516)
• No longer blanket allow unused items in polars-stream (#17551)
• Smaller filter / valid in Parquet (#17569)
• Move path expansion utils to polars-io crate (#17521)
• Add streaming slice node to new streaming engine (#17451)
• Add ordered union to new streaming engine (#17359)
• Add automated check for PR title formatting (#17412)
• Remove transmute for object store path (#17395)
• Fix Python version resolver in release drafter (#17390)
• Remove dead code (#17378)

Thank you to all our contributors for making this release possible!
@5j9, @ByteNybbler, @EricTulowetzke, @JamesCE2001, @Julian-J-S, @KDruzhkin, @MarcoGorelli, @Object905, @SandroCasagrande, @Vincenthays, @alexander-beedie, @anergictcell, @arnabanimesh, @atigbadr, @brandon-b-miller, @brunobbaraujo, @cmdlineluser, @coastalwhite, @davanstrien, @deanm0000, @deepyaman, @delsner, @dependabot, @dependabot[bot], @diegoglozano, @eitsupi, @ember91, @flisky, @henryharbeck, @implicit-apparatus, @itamarst, @jonaylor89, @jparag, @knl, @kylebarron, @lukapeschke, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @phi-friday, @philss, @r-brink, @ragyabraham, @rcorty, @ritchie46, @rosstitmarsh, @ruihe774, @sherlockbeard, @stinodego, @szepeviktor, @tylerriccio33, @wangxiaoying and @wence-