Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Edinburgh DataSync test #415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 23, 2021
Merged

Edinburgh DataSync test #415

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions notes/zrq/20210312-01-owncloud-sync.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
#
# <meta:header>
# <meta:licence>
# Copyright (c) 2021, ROE (http://www.roe.ac.uk/)
#
# This information is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This information is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# </meta:licence>
# </meta:header>
#
#zrq-notes-time
#zrq-notes-indent
#zrq-notes-crypto
#zrq-notes-ansible
#zrq-notes-osformat
#zrq-notes-zeppelin
#

Target:

Experiment with using the DataSync service at Edinburgh.

Edinburgh - DataSync
https://www.ed.ac.uk/information-services/computing/desktop-personal/datasync
https://www.ed.ac.uk/information-services/computing/desktop-personal/datasync/getting-started

Edinburgh - ownCloud Desktop Client for Linux
https://www.ed.ac.uk/information-services/computing/desktop-personal/datasync/desktop-clients/owncloud-linux-client

The Command Line Client
https://doc.owncloud.com/desktop/advanced_usage/command_line_client.html


Experiment with using the Edinburgh DataSync service is an ownCloud deployment.

ownCloud
https://owncloud.com/

ownCloud Desktop Client
https://owncloud.com/desktop-app/
https://github.com/owncloud/client

owncloudcmd
https://doc.owncloud.org/desktop/1.8/owncloudcmd.1.html

DataSync endpoints
https://datasync.ed.ac.uk/ocs/v1.php/cloud/capabilities

Result:

Integration with the DataSync service works.
Need somewhere to store the password.

Would work a lot better if Zeppelin used the user's uid and /home directory.

Shell interpreter for Apache Zeppelin
https://zeppelin.apache.org/docs/0.8.0/interpreter/shell.html
https://community.cloudera.com/t5/Community-Articles/How-to-enable-user-impersonation-for-SH-interpreter-in/ta-p/245627


# -----------------------------------------------------
# Login to our DataSync account.
#[user@fedora]

#
# https://www.ed.ac.uk/information-services/computing/desktop-personal/datasync/getting-started
# https://datasync.ed.ac.uk/index.php/apps/files/

#
# Create an application password for Aglais.
# https://datasync.ed.ac.uk/index.php/settings/personal?sectionid=security

#
# Keep a copy of the user name and password.
#

#
# Create a new folder called Aglais
#


# -----------------------------------------------------
# -----------------------------------------------------
# Install the ownCloud client.
# https://download.owncloud.com/desktop/ownCloud/stable/latest/linux/download/
#[fedora@zeppelin]

sudo rpm --import https://download.owncloud.com/desktop/ownCloud/stable/latest/linux/Fedora_33/repodata/repomd.xml.key
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://download.owncloud.com/desktop/ownCloud/stable/latest/linux/Fedora_33/owncloud.repo
sudo dnf install -y owncloud-client


# -----------------------------------------------------
# -----------------------------------------------------
# Login to Zeppelin and configure the client.
#[zeppelin]

%sh
cat > '/user/zrq/owncloud.env' << EOF
OWNCLOUD_URL=https://datasync.ed.ac.uk/remote.php/webdav/Aglais
OWNCLOUD_HOME=/user/zrq/owncloud
OWNCLOUD_USER=########
OWNCLOUD_PASS=####-####-####-####
EOF

%sh
source '/user/zrq/owncloud.env'
mkdir "$OWNCLOUD_HOME"

# -----------------------------------------------------
# Add some data to transfer.
#[zeppelin]

%sh
source '/user/zrq/owncloud.env'

mkdir "$OWNCLOUD_HOME/test1"
for file in $(ls /data/gaia/edr3/ | head)
do
cp "/data/gaia/edr3/${file}" \
"$OWNCLOUD_HOME/test1"
done

%sh
source '/user/zrq/owncloud.env'
ls -al "$OWNCLOUD_HOME/test1"

> -rw-r--r--. 1 fedora fedora 48614255 Mar 14 03:45 part-00000-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 48527951 Mar 14 03:45 part-00001-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 48623962 Mar 14 03:45 part-00002-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49694788 Mar 14 03:45 part-00003-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49737706 Mar 14 03:45 part-00004-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49650431 Mar 14 03:45 part-00005-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49323839 Mar 14 03:46 part-00006-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49561396 Mar 14 03:46 part-00007-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 49442127 Mar 14 03:46 part-00008-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet
> -rw-r--r--. 1 fedora fedora 50186730 Mar 14 03:46 part-00009-59b9273a-2ef1-4988-8778-e00f67e65264-c000.snappy.parquet


# -----------------------------------------------------
# Sync our local directory with the DataSync server.
#[zeppelin]

%sh
source '/user/zrq/owncloud.env'
owncloudcmd \
--silent \
--user "$OWNCLOUD_USER" \
--password "$OWNCLOUD_PASS" \
"$OWNCLOUD_HOME" \
"$OWNCLOUD_URL"