use the xtrace flag to get more debugging output during bootstrap/runner#943
Merged
Conversation
anusha-r
pushed a commit
that referenced
this pull request
Sep 16, 2014
use the xtrace flag to get more debugging output during bootstrap/runner
coyotemarin
pushed a commit
to coyotemarin/mrjob
that referenced
this pull request
Aug 6, 2015
SO many bugfixes * jobs: * MRStep's constructor treats kwarg=None same as not setting it (Yelp#970) * parse_counters() and parse_output() are deprecated (Yelp#829) * self.mr is deprecated in favor of MRStep (Yelp#815) * runners: * All runners: * You can now set strict_protocols from mrjob.conf (Yelp#726) * new --no-strict-protocols command-line option * streaming output from closed runner shows a warning (Yelp#853) * EMR: * --check-input-paths and --no-check-input-paths options (Yelp#864) * skip (very slow) validation of s3 buckets if boto < 2.25.0 (Yelp#865) * Fix for max_hours_idle bug that was terminating job flows early (Yelp#932) * Job flows are visible to all IAM users by default (Yelp#922) * --emr-api-param allows users to pass additional parameters to boto's EMR API (Yelp#879) * unset paramaters with --no-emr-api-param * bootstrap_python_packages (deprecated) now works on 3.x EMR AMIs (Yelp#863) * Use TERMINATE_CLUSTER instead of deprecated TERMINATE_JOB_FLOW (Yelp#974) * updated EC2 instance type data for pooling (Yelp#995) * Hadoop: * exclude hadoop source jars when looking for streaming jar (Yelp#861) * Fixed mkdir_on_hdfs for Hadoop version 2.x (Yelp#923) * Fixed hadoop_bin on Windows (Yelp#843) * Local * bootstrap mrjob by default (Yelp#984) * Inline * fix for add_file_option() (Yelp#851) * cd to job's working directory before instantiating mrjob class (Yelp#988) * Use pytest to run tests (Yelp#898) * collect-emr-active-stats subcommand (Yelp#947) * Using xtrace flag to get more output during bootstrap (Yelp#943) * Fixed log printouts for command line tools (Yelp#901) * Fix to avoid interpreting windows paths as URIs (Yelp#880) * Better error message when ssh keyfile is missing (Yelp#858) * Update EMR tool ISO8601 parsing to be consistent with EMR runner (Yelp#869) * Dropped support for Python 2.5 (Yelp#713) * Dropped support for the 1.x EMR AMI series, which uses Python 2.5 Conflicts: CHANGES.txt tests/test_emr.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When running bootstrap commands, failures can be very mysterious. What command was actually run? What environment variables were properly interpolated? By adding the xtrace flag, the script will send to stderr the commands that are actually run. This should allow the user to debug the error.
This can be set by the user, but setting it as a default makes sense because the helpfulness of the output should outweigh the few extra statements logged.