Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@yongtang
Copy link
Member

@yongtang yongtang commented Jan 20, 2019

This PR will try to enhance the Travis CI builds. While Travis CI builds have been very helpful, there are several issues.

This PR has the intension of

  • Trigger Travis CI to run all unit tests (Bazel test //tensorflow_io/...) in tensorflow/tensorflow:custom-op, with python versions matching tensorflow ci build.
  • Trigger Travis CI to run all unit tests (Bazel test //tensorflow_io/...) in Ubuntu 16.04 and 18.04 docker images (not with Travis VM) with system python 2 and 3.
  • Trigger Travis CI to build whl files for python 2.7, 3.4, 3.5, 3.6 in tensorflow/tensorflow:custom-op, with python matching tensorflow ci build.
  • Setup integration/build test to test whl files in Ubuntu 16.04 and 18.04 with system python (added with stub test only now).

The following are the changes in this PR:

  • Move top level tf to third_party/tf
  • Add -D_GLIBCXX_USE_CXX11_ABI=0 to all third_party C++ libraries as Ubuntu 16.04 and 18.04 may have new C++ compilers
  • Add list(range()) in arrow_test.py as otherwise python 3 will not work.
  • Use Ubuntu 16.04 for base VM in Travis CI, and move all tests in docker containers to achieve consistency.

Signed-off-by: Yong Tang [email protected]

This fix moves top level tf to third_party/tf to reduced
the top level directories. The tf directory itself
is to configure the tensorflow headers to be used.

Signed-off-by: Yong Tang <[email protected]>
@yongtang yongtang force-pushed the travis branch 7 times, most recently from 5936012 to def9a9c Compare January 20, 2019 23:33
as Ubuntu 16.04 and 18.04 may have new C++ compilers

Signed-off-by: Yong Tang <[email protected]>
@yongtang yongtang force-pushed the travis branch 6 times, most recently from f81fd1c to 524a201 Compare January 21, 2019 02:36
and fix arrow build failure in Ubuntu 18.04

Signed-off-by: Yong Tang <[email protected]>
also, add Build test so that generated whl files could be tested.

Signed-off-by: Yong Tang <[email protected]>
@yongtang yongtang force-pushed the travis branch 2 times, most recently from ff1a6bf to 364c645 Compare January 21, 2019 04:26
@yongtang yongtang changed the title [WIP] Travis CI build enhancements Travis CI build enhancements Jan 21, 2019
@yongtang
Copy link
Member Author

/cc @terrytangyuan @dmitrievanthony to take a look.

Basic story: provide coverage for supported platforms.

When we provide a list of supported platforms, we are talking about several things:

  1. The supported platforms when user install from python tensorflow-io PyPI.org
  2. The supported platforms when user compile tensorflow-io from source.
  3. The supported platforms when user install from R (CRAN)

In detail:

  1. When user install tensorflow-io from PyPI.org, they could install with python 2.7, 3.4, 3.5, 3.6. Our position is that tensorflow-io support Linux platform that could run tensorflow. As such we have to build tensorflow-io exactly the same way as tensorflow for python 2.7/3.4/3.5/3.6. That is where customer-op tests comes from.
  2. When user install tensorflow-io from PyPI.org, they could install it on a platform different from custom-op docker image. Mostly likely user will install tensorflow-io/tensorflow with system default python on Ubuntu (16.04 and 18.04). So we have to test the binaries of *.whl files generated from customer-op could run on Ubuntu (16.04 and 18.04) with system-default python.
  3. When developer want to build tensorflow-io from source, they mostly likely will use a default Ubuntu 16.04/18.04 to compile (with system -default GCC version) so tests also have to be done on those platforms.
  4. For R support, users will install tensorflow-io from PyPI.org so the binary of .whl are actually build from custom-op. So we need to test the scenario of building whl from custom-op docker image, but runs on Ubuntu + R 3.5. (@terrytangyuan may have additional input on R)

@yongtang
Copy link
Member Author

All tests passes now. I think this PR is ready for review.

@terrytangyuan
Copy link
Member

@yongtang I changed the travis config for R to this recently:

io/.travis.yml

Lines 97 to 110 in e5dc0eb

- name: "Ubuntu 14.04 R 3.5"
dist: trusty
language: r
r:
- 3.5
cache:
packages: true
directories:
- $HOME/.cache/pip
before_install:
- bash -x ${TRAVIS_BUILD_DIR}/.travis/python.configure.sh
- pip install --upgrade --ignore-installed --user travis tensorflow-io pyarrow==0.11.1
- echo "options(repos = c(CRAN='http://cran.rstudio.com'))" >> ~/.Rprofile
- cd R-package

This includes the following:

  • language: r gives more R friendly setup and provides some built-in functionalities besides devtools::test() that are otherwise harder to set up ourselves.
  • echo "options(repos = c(CRAN='http://cran.rstudio.com'))" >> ~/.Rprofile configures R to use a specific CRAN mirror
  • --user travis is needed to get around some permission issues when language: r is specified
  • The cache section significantly speeds up the build

Could you incorporate the above mentioned items in the PR?

@yongtang
Copy link
Member Author

@terrytangyuan The PR made changes to CI so now R is tested inside docker containers with Ubuntu 16.04 and 18.04. This is actually different from Travis CI's Ubuntu 14.04 VM and R language spec.

The reason to test inside Ubuntu 16.04 and 18.04 containers is that users are more likely to use Ubuntu 16.04/18.04 than Travis CI's Ubuntu 14.04. (The Ubuntu 14.04 will be out of support in April). Also, Travis CI still hasn't been able to support Ubuntu 18.04 today.

With tests running inside containers some of the optimizations may not be applications. I managed to have the following two in pace in containers:

  • echo "options(repos = c(CRAN='http://cran.rstudio.com'))" >> ~/.Rprofile configures R to use a specific CRAN mirror
  • The directory cache of $HOME/.cache/pip (by mapping inside and outside of containers) which could speeds up the build

Please take a look and see if it helps.

@terrytangyuan
Copy link
Member

@yongtang Thanks. That makes sense and the changes look fine to me. I'll probably add the additional work that language: r does on Travis later besides devtools::test(). The R tests actually failed with:

Message: AttributeError: 'NoneType' object has no attribute 'video_dataset'
Detailed traceback: 
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 195, in make_one_shot_iterator
    _make_dataset.add_to_graph(ops.get_default_graph())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py", line 479, in add_to_graph
    self._create_definition_if_needed()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py", line 335, in _create_definition_if_needed
    self._create_definition_if_needed_impl()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py", line 344, in _create_definition_if_needed_impl
    self._capture_by_value, self._caller_device)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py", line 864, in func_graph_from_py_func
    outputs = func(*func_graph.inputs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 192, in _make_dataset
    return dataset._as_variant_tensor()  # pylint: disable=protected-access
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 2131, in _as_variant_tensor
    self._input_dataset._as_variant_tensor(),  # pylint: disable=protected-access
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_io/video/python/ops/video_dataset_ops.py", line 71, in _as_variant_tensor
    return video_ops.video_dataset(self._filenames)

Looks like the exit code of R -e 'devtools::test()' wasn't captured properly.

Signed-off-by: Yong Tang <[email protected]>

Add  -v $HOME/.cache/pip:/root/.cache/pip to cache pip in docker containers

Signed-off-by: Yong Tang <[email protected]>
@yongtang yongtang force-pushed the travis branch 2 times, most recently from 2d77507 to 052345b Compare January 22, 2019 17:14
…to capture errors

and add ffmpeg install to fix R build failure.

Signed-off-by: Yong Tang <[email protected]>
@yongtang
Copy link
Member Author

Thanks @terrytangyuan. I updated the PR and the failed R tests on ffmpeg have been fixed. Also changed the check code to R -e 'stopifnot(all(data.frame(devtools::test())$failed == 0L))' so that the error exit code could be captured. Please take a look.

Copy link
Member

@terrytangyuan terrytangyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Thanks for the efforts!

@terrytangyuan terrytangyuan merged commit 2d07df0 into tensorflow:master Jan 22, 2019
@yongtang yongtang deleted the travis branch January 22, 2019 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants