• revert tornado upgrade (#2504)
  Upgrading tornado unfortunately breaks older version of Python

Added

luigi:

• Add Data Revenue to the blogged list (#2472)
• Add default reviewers in CODEOWNERS (#2465)
• Optional TOML configs support (#2457)
• Add support for multiple requires and inherits arguments (#2475)
• Add a visiblity level for luigi.Parameters (#2278)
• Make logging of RPC retries configurable #2486
• Added a new event 'progress' (#2498)

luigi.contrib:

• Additions to provide support for the Load Sharing Facility (LSF) job scheduler (#2373)
• Added default port behaviour for Redshift (#2474)
• Add metadata columns to the RDBMS contrib (#2440)
• Use passed password when create a redis connection (#2489)

Changed

luigi:

• Update supplementary github files to improve repo organization and maintenance (#2463)
• Use task_id in Task.eq comparison (#2462)
• Replace luigi.Task by RunOnceTask in scheduler_visualisation_test (#2476)
• (Breaking change) Bump tornado milestone version (#2490)
  This changes requires Python version 2.7.9+ and 3.4+

luigi.contrib:

• S3 client refactor (#2482)
• Update moto to 1.x milestone version (#2471)

Fixed

luigi:

• Fix Scheduler.add_task to overwrite accepts_messages attribute. (#2469)
• Fix race condition (#2477)
• Fix attribute forwarding for tasks with dynamic dependencies (#2478)

luigi.contrib:

• Fix transfer config import (#2458)

Removed

luigi:

• Remove long-deprecated scheduler config variable alternatives (#2491)

Added

luigi:

• Add a configuration parameter to force multiprocessing (#2401)
• Add a configuration parameter to enable/disable the pause button (#2399)
• Send messages from scheduler to tasks (via "Send message" UI button) (#2426)
• Allow to inject a context manager around TaskProcess.run (via task_process_context configuration parameter) (#2449)

luigi.contrib:

• S3: use Boto3 for the S3Client (#2423, #2149)
• GCS: add method to push files using multiprocessing (#2376)
• HDFS: add get_merge to snakebite client (#2410)
• Redshift: add schema to DB if it doesn't exist (#2439)
• Redshift: add table constraints support (#2435)

Fixed

luigi:

• Allow long parameters in task history DB SQL result store (#2404)
• Fix MissingParameterException when generating execution summary (#2415)
• Fix luigid crash due to configuration file parsing (#2394)
• Allow explicit parsing of BoolParameters (via luigi.BoolParameter.parsing variable) (#2427)
• Make ChoiceParameter check if option is valid within .normalize (#2454)
• ...and a good deal of documentation fixes and similar.

luigi.contrib:

• BigQuery: fix bulk_complete failing when argument is a generator (#2441)
• Kubernetes: prevent KeyError in KubernetesJobTask (#2433)
• Kubernetes: don't set activeDeadlineSeconds by default (#2452)

Fixed

luigi

• append is vulnerable of xss #2391

Added

luigi:

• Add google-auth-httplib2 as dependency (#2384)
• Add new company user (#2388)
• Allow release of resources during task running. (#2346)

luigi.contrib:

• Add parameterized backoff limit (#2375)

Fixed

luigi:

• Typo in running luigi documentation (#2387)
• Skip running coverage on unreasonable files
• Pass kwargs to discovery.build() when instantiating GSCClient. (#2291)
• Removed message 'No Instance(s) Available.' in Windows when starting … #2294

Added

luigi:

• Added generated data files to .gitignore (#2367)

luigi.contrib:

• Add possibility to specify Redshift column compression (#2343)

Changed

luigi:

• Show status message button in worker tab when only progress is set (#2344)
• Verbose worker error logging (#2353)

luigi.contrib:

• Replace oauth2client by google-auth (#2361)

Fixed

luigi:

• Fix unicode formatting (#2339)

luigi.contrib:

• Fix contrib.docker_runner exit code check (#2356)

Added

luigi:

• A button to show errors for disabled tasks in the visualizer (#2266)
• OptionalParameter parameter class (#2295)

luigi.contrib:

• Support for the Docker Python SDK (#2158)

Changed

luigi:

• Change the logging status of prune messages from info to debug (#2254) - reduce repetitiveness of logging
• Speed up the visualizer so that a refresh doesn’t take minutes when number of tasks gets into the millions (#2239)
• Hide "re-enable" and "forgive failures" buttons on success (#2281)
• Convert dates to datetimes for DateHourParameter (#2285)
• Handle multiprocessing with request sessions (#2290) - Fixes bug with "RPCError: Received null response from remote scheduler"
• Replaced param_args with dynamic property with deprecation message
• Check task equality using param_kwargs instead of param_args

Removed

luigi.contrib:

• Copying of Avro field documentation by BigQueryLoadAvro (#2269) - said copying is done by BigQuery natively now

Fixed

luigi:

• Remove invalid entrypoint luigi.tools.migrate (#2257)
• Preserve filter on server on reload (#2273)
• Fix MRO on tasks using util.requires (#2307)

luigi.contrib:

• Fix Python 3 TypeError in contrib.hive.HiveTableTarget.exists (#2323)

Luigi 2.7.1 is mostly bug fixes and small feature additions.

• Standardize Redshift credential usage across Redshift tasks: #2068
• BigQueryLoadAvro now handles complex Avro types: #2224
• Support for user-specified number of paralleled scheduled processes:
  #2205
• ECS support for non-default cluster: #2045
• BigQuery ExtractTask support: #2134
• Support for autocommitting queries within Redshift and Postgres, allowing VACUUM statement execution: #2242
• High Availability (HA) support with WebHdfsClient using multiple namenodes: #2230
• Column mapping support for Redshift S3CopyToTable using the column list: #2245

There have also been some other feature additions, bugfixes, and docfixes. See all commits here.

Introducing Luigi 2.7.0!

• Removed deprecations! luigi.{postgres, s3} are removed, use luigi.contrib.{..} instead #2181
• As of the bullet above. We now believe all contrib and core modules are well seperated by either being under either luigi.xyx or luigi.contrib.xyz. This should simplify to move modules into seperate repositories, which is future work.
• Luigi finally has a progress bar! #2108
• Scheduling improvements / bug fixes: #2208 #2207
• Luigi can now run PySpark in cluster mode #2197
• The batch email notifier shouldn't stall any more on failures. #2177
• A global panic "stop scheduling button" #2145
• Fixed error emails from workers: #2191
• My personal favorite PR, as it removes many deprecation warnings when running the tests: #2130

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits
here.

Luigi 2.6.2 is also mainly about bug-fixes. Including many fixes in the scheduler web interface.

• One new feature is forgiving failures. #2091 (thanks @FxIII!)
• The annoying "is not of type" warnings are now easier to fix/trace, as the var name is also included in the log message #2107 (thanks @Lucas-C!)