Worker crashes with 'DataObject is not finished'Β #13
Description
Rain worker crashes with thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18.
The deployment consisted of 1 local and 1 remote worker running on Exoscale infrastructure connecting from a remote client.
Worker log including stack backtrace:
ubuntu@test-0:~$ rain worker <server-ip>:7210 --listen=7211
π§ INFO 2018-03-26T10:22:40Z Starting Rain 0.1.1 as worker
π§ INFO 2018-03-26T10:22:40Z Resources: 1 cpus
π§ INFO 2018-03-26T10:22:40Z Working directory: "/tmp/rain-work/worker-test-0-1261"
π§ INFO 2018-03-26T10:22:40Z Server address <server-ip>:7210 was resolved as <server-ip>:7210
π§ INFO 2018-03-26T10:22:40Z Start listening on port=7211
π§ INFO 2018-03-26T10:22:40Z Connecting to server addr=<server-ip>:7210
π§ INFO 2018-03-26T10:22:40Z Connected to server; registering as worker
π§ INFO 2018-03-26T10:22:43Z New connection from <server-ip>:40118
π§ INFO 2018-03-26T10:23:33Z New connection from 127.0.0.1:54122
thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18
note: Run with `RUST_BACKTRACE=1` for a backtrace.
ubuntu@test-0:~$ RUST_BACKTRACE=1 rain worker 185.150.9.32:7210 --listen=7211
π§ INFO 2018-03-26T10:24:19Z Starting Rain 0.1.1 as worker
π§ INFO 2018-03-26T10:24:19Z Resources: 1 cpus
π§ INFO 2018-03-26T10:24:19Z Working directory: "/tmp/rain-work/worker-test-0-1317"
π§ INFO 2018-03-26T10:24:19Z Server address 185.150.9.32:7210 was resolved as 185.150.9.32:7210
π§ INFO 2018-03-26T10:24:19Z Start listening on port=7211
π§ INFO 2018-03-26T10:24:19Z Connecting to server addr=185.150.9.32:7210
π§ INFO 2018-03-26T10:24:19Z Connected to server; registering as worker
π§ INFO 2018-03-26T10:24:23Z New connection from 185.150.9.32:40124
π§ INFO 2018-03-26T10:24:24Z New connection from 127.0.0.1:54126
thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::sys_common::backtrace::print
at /checkout/src/libstd/sys_common/backtrace.rs:68
at /checkout/src/libstd/sys_common/backtrace.rs:57
2: std::panicking::default_hook::{{closure}}
at /checkout/src/libstd/panicking.rs:381
3: std::panicking::default_hook
at /checkout/src/libstd/panicking.rs:397
4: std::panicking::rust_panic_with_hook
at /checkout/src/libstd/panicking.rs:577
5: std::panicking::begin_panic
6: <librain::worker::rpc::datastore::DataStoreImpl as librain::datastore_capnp::data_store::Server>::create_reader
7: <librain::datastore_capnp::data_store::ServerDispatch<_T> as capnp::capability::Server>::dispatch_call
8: <futures::future::lazy::Lazy<F, R> as futures::future::Future>::poll
9: <capnp_rpc::attach::AttachFuture<F, T> as futures::future::Future>::poll
10: <futures::future::chain::Chain<A, B, C>>::poll
11: <futures::future::chain::Chain<A, B, C>>::poll
12: futures::task_impl::std::set
13: <capnp_rpc::forked_promise::ForkedPromise<F> as futures::future::Future>::poll
14: <futures::future::chain::Chain<A, B, C>>::poll
15: <futures::future::select::Select<A, B> as futures::future::Future>::poll
16: <futures::future::map::Map<A, F> as futures::future::Future>::poll
17: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
18: <futures::future::chain::Chain<A, B, C>>::poll
19: futures::task_impl::std::set
20: <futures::stream::futures_unordered::FuturesUnordered<T> as futures::stream::Stream>::poll
21: <capnp_rpc::task_set::TaskSet<T, E> as futures::future::Future>::poll
22: <futures::future::chain::Chain<A, B, C>>::poll
23: futures::task_impl::std::set
24: <futures::stream::futures_unordered::FuturesUnordered<T> as futures::stream::Stream>::poll
25: <capnp_rpc::task_set::TaskSet<T, E> as futures::future::Future>::poll
26: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
27: <futures::task_impl::Spawn<T>>::poll_future_notify
28: <std::thread::local::LocalKey<T>>::with
29: <tokio::executor::current_thread::scheduler::Scheduler<U>>::tick
30: <scoped_tls::ScopedKey<T>>::set
31: <std::thread::local::LocalKey<T>>::with
32: <std::thread::local::LocalKey<T>>::with
33: tokio_core::reactor::Core::poll
34: tokio_core::reactor::Core::turn
35: rain::main
36: std::rt::lang_start::{{closure}}
37: std::panicking::try::do_call
at /checkout/src/libstd/rt.rs:59
at /checkout/src/libstd/panicking.rs:480
38: __rust_maybe_catch_panic
at /checkout/src/libpanic_unwind/lib.rs:101
39: std::rt::lang_start_internal
at /checkout/src/libstd/panicking.rs:459
at /checkout/src/libstd/panic.rs:365
at /checkout/src/libstd/rt.rs:58
40: main
41: __libc_start_main
42: _start
Server log:
π§ INFO 2018-03-26T10:24:05Z Starting Rain 0.1.1 server at port 0.0.0.0:7210
π§ INFO 2018-03-26T10:24:05Z Start listening on address=0.0.0.0:7210
π§ INFO 2018-03-26T10:24:05Z Dashboard is running at http://test-1:8080/
π§ INFO 2018-03-26T10:24:05Z Lite dashboard is running at http://test-1:8080/lite/
π§ INFO 2018-03-26T10:24:16Z New connection from 127.0.0.1:54004
π§ INFO 2018-03-26T10:24:16Z Connection 127.0.0.1:54004 registered as worker 127.0.0.1:7211 with Resources { cpus: 1 }
π§ INFO 2018-03-26T10:24:20Z New connection from 185.150.8.101:42994
π§ INFO 2018-03-26T10:24:20Z Connection 185.150.8.101:42994 registered as worker 185.150.8.101:7211 with Resources { cpus: 1 }
π§ INFO 2018-03-26T10:24:24Z New connection from 195.113.113.226:55798
π§ INFO 2018-03-26T10:24:24Z Connection 195.113.113.226:55798 registered as client
π§ INFO 2018-03-26T10:24:24Z New task submission (11 tasks, 11 data objects) from client 195.113.113.226:55798
π§ INFO 2018-03-26T10:24:24Z New get_state request (0 tasks, 1 data objects) from client
π§ INFO 2018-03-26T10:24:24Z Client 195.113.113.226:55798 disconnected
π§ INFO 2018-03-26T10:24:25Z New connection from 195.113.113.226:55800
π§ INFO 2018-03-26T10:24:25Z Connection 195.113.113.226:55800 registered as client
π§ INFO 2018-03-26T10:24:25Z New task submission (11 tasks, 11 data objects) from client 195.113.113.226:55800
π§ ERROR 2018-03-26T10:24:26Z Connection to worker 185.150.8.101:7211 lost
thread 'main' panicked at 'not yet implemented', src/server/state.rs:94:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.