v0.40.0

v0.40.0 (2024-09-02)

Highlights

• Targets now emit basic metrics. There's nothing to do on your end, they're enabled by default!
• The Faker class is available in stream maps context. Note that like the faker instance added in v0.35.0, this requires installing the faker library in the tap/target environment. For example, with tap-example[faker].

✨ New

• #2486 Emit target metrics
• #2567 A new schema_is_valid built-in tap test validates stream schemas against the JSON Schema specification
• #2598 Stream map expressions now have access to the Faker class, rather than just a faker instance
• #2549 Added a default user agent for REST and GraphQL taps

🐛 Fixes

• #2613 Mismatch between timezone-aware and naive datetimes in start date and bookmarks is now correctly handled

⚙️ Under the Hood

• #2628 Use context manager to read gzip batch files
• #2619 Default to UTC when parsing dates without a known timezone
• #2603 Backwards-compatible identifier quoting in fully qualified names
• #2601 Improved SQL identifier (de)normalization
• #2599 Remove pytest-durations dependency from testing and use native pytest option --durations
• #2597 Mark pagination classes with @override decorator
• #2596 Made auth_headers and auth_params of APIAuthenticatorBase simple instance attributes instead of decorated properties

📚 Documentation Improvements

• #2639 Documented versions where fake and Faker objects were added to the stream maps context
• #2629 Reference get_starting_timestamp in incremental replication guide
• #2604 Update project sample links
• #2595 Documented examples of stream glob expressions and property aliasing

📦 Packaging changes

• #2640 Remove upper constraint on faker extra

v0.39.1

v0.39.1 (2024-08-07)

• PyPI 🐍

🐛 Fixes

• #2589 Make sink assertion compatible with stream maps -- Thanks @JCotton1123!
• #2592 Fixed typos in --about plain text output
• #2580 Date fields are now properly serialized as ISO dates, i.e. "YYYY-MM-DD"
• #2583 Quote add-column-ddl with column starting with _ (underscore) based on engine -- Thanks @haleemur!
• #2582 DDL for adding a column now uses a SQLAlchemy-compiled clause to apply proper quoting -- Thanks @haleemur!
• #2418 Check replication method instead of key to determine if a SQL stream is sorted

⚙️ Under the Hood

• #2520 Remove unused dependencies pendulum and python-dateutil

📚 Documentation Improvements

• #2579 Documented support for packaging semantic type in contributing guide

v0.39.0

v0.39.0 (2024-07-27)

• PyPI 🐍

Highlights

• The json module can now be used within stream maps

  stream_maps:
    animals:
      id: id
      data: "json.dumps({'description': description, 'verified': verified, 'views': views, 'created_at': created_at})"
      __else__: null

• The crypto dependencies PyJWT and cryptography are now optional and can be installed by requesting the jwt extra, i.e. singer-sdk[jwt]. These are only currently required for JWT authentication, but if your package depends on any of these, make sure the dependency is explicit.

• The pendulum and python-dateutil dependencies are no longer used by this library and will be removed in a future release. If your package relies on any of them, make sure the dependency is explicit.

• We're working on supporting msgspec for more performant IO, so keep an eye out for that 😉.

✨ New

• #2432 Developers can now customize the default logging configuration for their taps/targets by adding default_logging.yml to their package
• #2531 The json module is now avaiable to stream maps -- Thanks @grigi!
• #2529 Stream sync context is now available to all instances methods as a Stream.context attribute

🐛 Fixes

• #2554 Use mapped stream aliases when handling ACTIVATE_VERSION messages in the base target class
• #2526 Moved up the supported Python versions in the Markdown output of --about

⚙️ Under the Hood

• #2564 Make SQLSink a generic with a SQLConnector type parameter
• #2540 Implement abstract serialize_message for Singer writers
• #2259 Centralize JSON SerDe into helper functions -- Thanks @BuzzCutNorman!
• #2525 Make PyJWT and cryptography dependencies optional
• #2528 Moved class-level attributes to the top in REST tap template
• #2132 Limit internal usage of pendulum

📚 Documentation Improvements

• #2557 Document that get_starting_timestamp requires setting a non-null replication_key
• #2556 Reference state partitioning in stream partitioning page
• #2536 Prepare for RTD addons migration
• #2535 Added more intersphinx links to Python and Faker docs
• #2530 Explained how the request URL is generated from url_base, path and the sync context
• #2527 Updated the footer
• #2506 Fixed a typo in the stream maps docs

v0.38.0

v0.38.0 (2024-06-17)

✨ New

• #2433 Tap developers can now disable HTTP redirects
• #2426 Added an optional GitHub workflow to publish to PyPI with trusted publishers

🐛 Fixes

• #2438 Null replication values are now handled when incrementing bookmarks
• #2431 Updated cookiecutter VSCode launch.json to use debugpy
• #2421 An error message is now logged every time schema validation fails for any record

⚙️ Under the Hood

• #2455 Use parent datetime.datetime class in type conforming checks
• #2453 Change to return type of utc_now from pendulum.DateTime to datetime.datetime

📚 Documentation Improvements

• #2449 Add a short guide on defining a configuration schema
• #2436 Documented how context fields are passed to a child stream
• #2435 Using an empty list for __key_properties__ to disable a stream primary keys is now recommended as an alternative to null

v0.37.0

v0.37.0 (2024-04-29)

Highlights

• You can reference your streams with glob patters in stream map configuration:

  stream_maps:
    # This will affect all streams with names starting with 'my_prefix_'
    "my_prefix_*":
      id: id
      first_name: first_name
      email:     # drop the PII field from RECORD and SCHEMA messages
      email_domain: email.split('@')[-1]
      email_hash: md5(config['hash_seed'] + email)
      __else__: null

✨ New

• #2389 JSON schema keyword allOf is now supported
• #1888 Added support for glob patterns in source stream names -- Thanks @DouweM!
• #2345 PropertiesList can now behave as an iterable -- Thanks @ReubenFrankel!

🐛 Fixes

• #2352 Removed unnecessary and problematic column caching -- Thanks @raulbonet!
• #2375 Added sensitive: true to password settings in templates
• #2301 Unmapped sub-fields in object-type fields are now no longer dropped when the field declares additionalProperties
• #2348 Added a condition to the No schema for record field warning -- Thanks @tobiascadee!
• #2342 Avoid failing if VSCode IDE config is not requested for target and mapper cookiecutter templates -- Thanks @ReubenFrankel!
• #2331 Allow importlib-resources >=6.3.2

⚙️ Under the Hood

• #2205 Added a jwt package extra, but the cryptography and jwt dependencies are still installed by default for now

📚 Documentation Improvements

• #2326 Documented BATCH as a default plugin capability -- Thanks @ReubenFrankel!

v0.36.1

v0.36.1 (2024-03-13)

🐛 Fixes

• #2310 Limited importlib-resources to < 6.2 due to breaking changes
• #2288 Bumped min joblib dependency to >=1.3.0 and replaced deprecated parallel_backend with parallel_config -- Thanks @BuzzCutNorman!
• #2281 The state_partition_context dictionary is now correctly interpolated in the error message when duplicate partitions/contexts are detected in the input state
• #2274 Test workflow now fails for any Python matrix job failure in cookiecutter template -- Thanks @ReubenFrankel!

v0.36.0

v0.36.0 (2024-02-26)

✨ New

• #2241 JSON schema keywords such as maxLength are now supported in StringType, IntegerType and NumberType JSON schema helpers
• #2263 Nested settings are now documented in the output of --about --format=markdown
• #2248 Targets now accept a batch_size_rows setting to configure how many rows are loaded in each record batch -- Thanks @BuzzCutNorman!

🐛 Fixes

• #2258 Database disconnects are now handled via SQLAlchemy pool_pre_ping parameter

⚙️ Under the Hood

• #2220 Deprecated singer_sdk.authenticators.BasicAuthenticator in favor of requests.auth.HTTPBasicAuth

v0.35.2

v0.35.2 (2024-02-19)

🐛 Fixes

• #2252 Null type is now correctly appended as {"type": "null"} to oneOf types

v0.35.1

v0.35.1 (2024-02-15)

🐛 Fixes

• #2243 Flattening level of record is now forced to match the provided flattened schema -- Thanks @joaopamaral!
• #2245 Instances of oneOf are now handled by null-appending logic
• #2242 Hard and soft deletes for handling ACTIVATE_VERSION messages now use the same WHERE clause -- Thanks @vitoravancini!
• #2232 Test workflow job now fails for unsupported Python versions in cookiecutter templates -- Thanks @ReubenFrankel!
• #2225 SQL columns that are non-nullable but not required (i.e. not part of a primary key) are now not included in the "required": [...] array of the discovered JSON schema

📚 Documentation Improvements

• #2239 Linked reference docs to source code
• #2231 Added an example implemetation of JSON schema validation that uses fastjsonschema
• #2219 Added reference docs for tap & target testing helpers

v0.35.0

v0.35.0 (2024-02-02)

Highlights

• The package now acts a proper pytest plugin, so you may need to remove pytest_plugins = ("singer_sdk.testing.pytest_plugin",) from your tests/conftest.py file for test fixtures to be loaded correctly.
• Dropped support for Python 3.7 (EOL 2023-06-27).
• Dropped direct dependency on pytz. If your package uses pytz in any capacity, you may need to add it as an explicit dependency to your project.
• Stream maps can now generate fake data! See the docs for more details.

✨ New

• #2208 Allow users to disable schema validation in targets
• #2170 Generate fake data with stream maps -- Thanks @ReubenFrankel!
• #937 Support validating configuration for any tap with a dynamic catalog
• #2144 Support fanning out parent record into multiple child contexts/syncs
• #1918 End RESTStream pagination if an empty page is received

🐛 Fixes

• #2203 Fix serialization of arbitrary objects (e.g. ObjectId from mongoDB) during flattening -- Thanks @dgawlowsky!
• #2200 Quote column names in INSERT statement
• #2195 Include empty schemas directory in REST tap cookiecutter
• #2187 Replace use of deprecated jsonschema._RefResolver with recommended referencing library
• #2184 Reduce amount of unnecessary whitespace in Singer output
• #2183 Ensure .vscode directory is included when requested in cookiecutters and avoid failing if it does not exist
• #2180 Limit supported Python versions in --about output to existing ones
• #2108 Log sink name when an unhandled error occurs during setup
• #2158 Fix pytest plugin declaration so it can be used without requiring defining pytest_plugins in conftest.py
• #2105 Default handling of ACTIVATE_VERSION messages to soft deletes and add new SQLConnector.delete_old_versions method

⚙️ Under the Hood

• #2189 Use functools.lru_cache instead of the stale memoization library (#1981)
• #2188 Remove unused logger parameter from private catalog helper functions
• #2143 Drop support for Python 3.7
• #2157 Remove pytz dependency and use datetime.timezone.utc instead of pytz.UTC where possible
• #2136 Create interface for schema validation in sinks, and implement it for python-jsonschema -- Thanks @BuzzCutNorman!
• #2130 Allow loading stream schemas from importlib.resources.abc.Traversable types

📚 Documentation Improvements

• #2204 Document supported package extras
• #2186 Call out minimum recommended cookiecutter version
• #2168 Explain Progress is not resumable if interrupted in docs FAQ
• #2140 Update auth caching example to use functools.cached_property