Codestin Search App

Release v0.14.0 is fresh from the oven.

This release is the result of some massive work in the xrootd package thanks to Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student.

While GSoC-2018 is now over, it's time to reflect on what Mikhail wrote:

an almost complete xrootd client, compatible with the C++ implementation;
xrd-fuse, a command to mount the contents of a remote XRootD server, locally;
the beginnings of an xrootd server.

Most notably, the client package allowed to:

to create the xrd-cp and xrd-ls commands that copy and list the contents of a remote XRootD server,
to seamlessly read ROOT files over XRootD.

The client package handles authentication with the unix and kerberos protocols.
Unfortunately, authentication via GSI couldn't be implemented because there were no publicly available specifications for that protocol, see xrootd/xrootd#757 for more details.

Here is the final report of this year's GSoC:

https://gist.github.com/EgorMatirov/303e4019606677d1838677a5a7145420

Thanks a lot Mikhail, hope we'll see you around :)

Another big new feature is the ability to write ROOT files, directly with go-hep/rootio.
This is still very much a work in progress, though, as only writing "empty" ROOT files or writing ROOT files with TObjStrings have been explicitly tested.
Next release should see explicit support for writing histograms and graphs.

Lastly, improvements on the build and continuous integration procedure have been applied during this release cycle:

added AppVeyor CI, for Windows:
added code coverage, with codecov.io:

Code coverage improvements

The following packages have been updated, with additional tests, to improve their code coverage:

brio, csvutil, csvutil/csvdriver,
fit, fmom,
heppdt, hepmc, hepevt,
hbook/ntup, hplot,
lcio, lhef,
rootio, sio,
xrootd.

Still some more work is needed to bring code coverage to a saner level (from ~55% to ~70-80%.)
Help more than welcome: it's "just" a matter of creating examples and tests.

hepevt

AsciiEncoder was renamed into simply Encoder
AsciiDecoder was renamed into simply Decoder

rootio

As noted above, it is now possible to create ROOT files.
The rootio package has a couple of examples:

Here is how you would create a ROOT file with one TObjString in it:

package main

import (
	"fmt"
	"log"

	"go-hep.org/x/hep/rootio"
)

func main() {
	w, err := rootio.Create(fname)
	if err != nil {
	    log.Fatal(err)
	}
	defer w.Close()

	var (
	    k   = "my-objstring"
	    v   = rootio.NewObjString("Hello World from Go-HEP!")
	)

	err = w.Put(k, v)
	if err != nil {
	    log.Fatal(err)
	}

	fmt.Printf("wkeys: %d\n", len(w.Keys()))
	
	err = w.Close()
	if err != nil {
	    log.Fatalf("could not close file: %v", err)
	}
}

It is quite possible the API may change as we gain experience with what it ought to look like in a Go world.
e.g. it is possible that rootio.Directory.Put would get clever enough to automatically translate a Go builtin, like string, in the ROOT equivalent, say TObjString, on the fly.
Still pondering on that...

Again, still a lot of work to do on the writing side of things:

support for TH1x,
support for TH2x,
support for TGraph, TGraphErrors, TGraphAsymErrors
support for TTree (this will take some time)
support for user-provided types.

xrootd

The xrootd/server package ships with support for the following XRootD requests:

open, read, close,
write, stat, truncate, sync, rename.

There is no authentication support (yet) on the server: DO NOT RUN THIS ON PUBLICLY ACCESSIBLE MACHINES :)

The xrootd package gained a new sub-command:

xrootd/cmd/xrd-srv

$> xrd-srv -h
xrd-srv serves data from a local filesystem over the XRootD protocol. 

Usage:

 $> xrd-srv [OPTIONS] <base-dir>

Example:

 $> xrd-srv /tmp
 $> xrd-srv -addr=0.0.0.0:1094 /tmp

Options:
  -addr string
    	listen to the provided address (default "0.0.0.0:1094")

AOB

Support for Go-1.6 and Go-1.7 has been dropped.
Please upgrade to the latest and finest Go version (1.11 is around the corner, with support for Go modules.)

Another interesting possible development avenue: exposing ROOT TTrees as Apache Arrow Arrays.
This would allow for a better interoperability with that ecosystem and the tools it provides for data science and data analysis in general.
Help wanted!

Release v0.13.0 is fresh from the oven.

This release ships with major improvements in the xrootd implementation and a few fixes in rootio.

rootio

leveraging the work that happened in xrootd, rootio is now able to read files over [x]root:

import (
    "go-hep.org/x/hep/rootio"
)

func foo() {
    f, err := rootio.Open("root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root")
    if err != nil { ... }
    defer f.Close()
}

all the root-xyz commands can now also leverage xrootd:

$> root-ls -t root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root
=== [root://ccxrootdgotest.in2p3.fr:9001/tmp/rootio/testdata/small-flat-tree.root] ===
version: 60806
TTree          tree                 my tree title (entries=100)
│ Int32        "Int32/I"            TBranch
│ Int64        "Int64/L"            TBranch
│ UInt32       "UInt32/i"           TBranch
│ UInt64       "UInt64/l"           TBranch
│ Float32      "Float32/F"          TBranch
│ Float64      "Float64/D"          TBranch
│ Str          "Str/C"              TBranch
│ ArrayInt32   "ArrayInt32[10]/I"   TBranch
│ ArrayInt64   "ArrayInt64[10]/L"   TBranch
│ ArrayUInt32  "ArrayInt32[10]/i"   TBranch
│ ArrayUInt64  "ArrayInt64[10]/l"   TBranch
│ ArrayFloat32 "ArrayFloat32[10]/F" TBranch
│ ArrayFloat64 "ArrayFloat64[10]/D" TBranch
│ N            "N/I"                TBranch
│ SliceInt32   "SliceInt32[N]/I"    TBranch
│ SliceInt64   "SliceInt64[N]/L"    TBranch
│ SliceUInt32  "SliceInt32[N]/i"    TBranch
│ SliceUInt64  "SliceInt64[N]/l"    TBranch
│ SliceFloat32 "SliceFloat32[N]/F"  TBranch
│ SliceFloat64 "SliceFloat64[N]/D"  TBranch

support for seeking (i.e.: event random access) has been added to scanners connected to chains of rootio.Trees,
rootio can now automatically generate streamers for std::vector<T> when a streamer for T exists,
rootio has been updated to v2 of pierrec/lz4 library to decode LZ4 compressed ROOT files.

xrootd

support for ping and protocol requests
support for dirlist, open, close and sync requests
support for read and write requests
support for rm, rmdir and truncate requests
support for stat, vstat, statx, mkdir, mv and chmod requests
support for signing requests
support for auth+unix request
introduction of the xrd-cp command to copy files from a remote xrootd server:

$> go doc go-hep.org/x/hep/xrootd/cmd/xrd-cp
Command xrd-cp copies files and directories from a remote xrootd server to
local storage.

Usage:

    $> xrd-cp [OPTIONS] <src-1> [<src-2> [...]] <dst>

Example:

    $> xrd-cp root://server.example.com/some/file1.txt .
    $> xrd-cp root://[email protected]/some/file1.txt .
    $> xrd-cp root://server.example.com/some/file1.txt foo.txt
    $> xrd-cp root://server.example.com/some/file1.txt - > foo.txt
    $> xrd-cp -r root://server.example.com/some/dir .
    $> xrd-cp -r root://server.example.com/some/dir outdir

Options:

    -r	copy directories recursively
    -v	enable verbose mode

introduction of the xrd-ls command to list the contents of directories on a remote xrootd server:

$> go doc go-hep.org/x/hep/xrootd/cmd/xrd-ls
Command xrd-ls lists directory contents on a remote xrootd server.

Usage:

    $> xrd-ls [OPTIONS] <dir-1> [<dir-2> [...]]

Example:

    $> xrd-ls root://server.example.com/some/dir
    $> xrd-ls -l root://server.example.com/some/dir
    $> xrd-ls -R root://server.example.com/some/dir
    $> xrd-ls -l -R root://server.example.com/some/dir

Options:

    -R	list subdirectories recursively
    -l	use a long listing format

a convenience xrootd/xrdfs.FileSystem interface has been introduced to model interacting with the remote xrootd server's filesystem, following the os package API
a convenience xrootd/xrdfs.File interface has been introduced to model interacting with the remote xrootd file, following the os.File API
a convenience xrootd/xrdio.File type, implementing various io.Xyz interfaces has been introduced as well.

Release v0.12.0 is fresh from the oven.

This release is first one to introduce preliminary support for vgo, the official Go way to handle versioning.
vgo is still in flux: the first Go version with experimental opt-in support should be Go 1.11 (to be released in August 2018.)
Obviously, on the Go-HEP side, adjustments will probably still be required as the user story solidifies and experience is accumulated.

Nonetheless, it is still an interesting new development!

geo/gdml

This release adds preliminary support for parsing Geometry Description Markup Language (GDML) files, a de facto standard for describing (detector) geometries.
The documentation for this new package is here: geo/gdml.

Help wanted and (gladly) accepted to get this package in a shape where it could be used for detailed detector studies!
This is tracked here:

https://github.com/go-hep/hep/projects/4

hplot

hplot was slightly updated to cope with an interesting development percolating from upstream gonum/plot, namely: the migration to a new PDF backend that allows to embed fonts inside the output PDF file.
No more PDFs that display weirdly on foreign computer. Yay!

This obviously means the resulting PDF files may be quite larger than with previous versions.
(You can't have your cake and eat it.)
You can use vgpdf.Canvas.EmbedFonts to get the old behaviour.

rootio

This release adds preliminary support for chaining multiple rootio.Trees into a logical view: the famed rootio.Chain.

Mohamed Amine El Gnaoui (a.k.a @maloft), our new summer student @LPC-Clermont, provided the initial implementation: thanks!
More tests and benchmarks improvements yet to come :)

Another noteworthy change: rootio/cmd/root-srv dropped its dependency against github.com/satori/go.uuid in favor of github.com/pborman/uuid.
The latter exposes a more stable API.

xrootd

This release adds yet another new package: xrootd.
This package will provide (eventually) a pure-Go implementation of an XRootD client as well as a server.

Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student has been already hard at work, providing support for:

the initial xrootd client,
the handshake with an XRootD-compliant server (C++ or otherwise), and
the protocol and login requests/responses.

The dirlist request/response is already in the pipe.

Release v0.11 is fresh from the oven.

This release drops official support for Go-1.7 and earlier, adds support for Go-1.10.

This allowed to migrate the fwk package away from golang.org/x/net/context.
fwk now directly uses the context.Context type from the standard library.

rootio

This release brings quite a few improvements in the rootio area:

add support for empty ROOT files,
add support for reading remote ROOT files over HTTP.
This is implemented in a quite naive way, using net/http.Get to
download the whole file under a temporary directory.
See go-hep/hep#142 for ideas on
how to improve this,
add support for TH1-v6,
add support for streamer-less TDirectoryFile,
add support for TBranchElements in cmd/root-dump,
add support for displaying TH1, TH2 and TGraph{,Error}s in cmd/root-dump,
add support for branches and leaves with names only differing by case in
cmd/root-gen-datareader.
Now, the struct type being generated contains fields whose names start
with ROOT_.
The original branch or leaf name is then appended after ROOT_, unmodified.
add support for TKeys with a large binary payload,
add preliminary support for creating new Go types at runtime directly from
a StreamerInfo value,
add more documentation about ROOT binary file format (TFile, TKey and a bit about
TStreamerInfo so far.)

hbook

fixed a bug in the YODA ASCII file parsing code that would choke on
analysis objects (histograms, scatters, ...) that contain whitespace.

AOB

LPC-Clermont has funded a 5 months internship
student, starting now.
Mohamed Amine El Gnaoui (@maloft) will:

implement reading all ROOT files created with ROOT-6,
implement writing ROOT files, in a ROOT-6 compatible way,
improve the read/write performances of Go-HEP to be on par with that of ROOT/C++ (using the builtin performance tools of the Go toolchain: pprof, execution tracer, ... and/or linux perf)
extend the test suite of Go-HEP for reading and writing ROOT files,
add benchmark tests for reading and writing ROOT files,
document the ROOT file format as understood by Go-HEP.

Welcome to Go-HEP, Amine!

Release v0.10 is fresh from the oven.

This release brings quite a few improvements in the rootio area:

support for ROOT files compressed with LZMA
support for ROOT files compressed with LZ4
support scanning ROOT trees with branches holding multiple leaves
support for ROOT trees holding bool, [X]bool and std::vector<bool>
support for ROOT files holding TBranch version 10
support for ROOT files holding TBranchElement version 8
support for ROOT files holding TH1 version 5
support for ROOT trees and branches produced with ROOT-6.12

The rootio/cmd/root-srv command dropped support for Go-1.6 and earlier.
Please upgrade to a newer Go version.

In other news, the csvutil/csvdriver and hbook/ntup/ntcsv gained the ability to read remote CSV files (over HTTP.)

AOB

Better support for TStreamerElements and thus the ability to read nested std::vector<std::vector<T>> data or the ability to run root-dump and root-diff on all ROOT files, has started.

Google Summer of Code has started and Go-HEP is part of it, thanks to the CERN-HSF umbrella.
Check out the 2 proposals:

Stay tuned.

Release v0.9 is fresh from the oven.

This is a quick and simple release fix with the following main features:

fix a broken link to the PDF LaTeX example (back from the multi- to mono-repo migration)
fix a compilation failure in rootio/cmd/root-srv originating from an API modification in a 3rd-party dependency (github.com/satori/go.uuid)

One interesting new feature is the introduction of a simple ROOT::TTree to CSV file conversion command: root2csv.

$> go get go-hep.org/x/hep/cmd/root2csv
$> root2csv --help
root2csv -h
Usage of root2csv:
  -f string
        path to input ROOT file name
  -o string
        path to output CSV file name (default "output.csv")
  -t string
        name of the tree to convert (default "tree")

$> go doc go-hep.org/x/hep/cmd/root2csv 
root2csv converts the content of a ROOT TTree to a CSV file.

    Usage of root2csv:
      -f string
                path to input ROOT file name
      -o string
                path to output CSV file name (default "output.csv")
      -t string
                name of the tree to convert (default "tree")

By default, root2csv will write out a CSV file with ';' as a column
delimiter. root2csv ignores the branches of the TTree that are not supported
by CSV:

    - slices/arrays
    - C++ objects

Example:

    $> root2csv -o out.csv -t tree -f testdata/small-flat-tree.root
    $> head out.csv
    ## Automatically generated from "testdata/small-flat-tree.root"
    Int32;Int64;UInt32;UInt64;Float32;Float64;Str;N
    0;0;0;0;0;0;evt-000;0
    1;1;1;1;1;1;evt-001;1
    2;2;2;2;2;2;evt-002;2
    3;3;3;3;3;3;evt-003;3
    4;4;4;4;4;4;evt-004;4
    5;5;5;5;5;5;evt-005;5
    6;6;6;6;6;6;evt-006;6
    7;7;7;7;7;7;evt-007;7

AOB

The next release cycle will see improvements in the rootio department (read and write.)

Stay tuned.

Release v0.8 is fresh from the oven.

No API changes in go-hep itself, just a few modifications to cater with changes in gonum/v1/gonum and gonum/v1/plot:

gonum/v1/gonum/stat/distuv distributions replaced the name of their rand.Rand source field from Source to Src (to align with the standard library)
gonum/v1/gonum/stat/... packages now use golang.org/x/exp/rand.Rand instead of math/rand.Rand

The biggest news (and where the work mostly happened between v0.7 and v0.8) is that there's now a Jupyter instance with a Go kernel installed with Go-HEP and Gonum libraries pre-installed.
This has been packaged up at go-hep/binder.
You can try it there:

fmom

fmom gained a new top-level function fmom.InvMass(p1, p2 P4) float64 :

// InvMass computes the invariant mass of two incoming 4-vectors p1 and p2.
func InvMass(p1, p2 P4) float64 {
       p := Add(p1, p2)
       return p.M()
}

hplot

hplot gained a new top-level function Show:

// Show displays the plot according to format, returning the raw bytes and
// an error, if any.
//
// If format is the empty string, then "png" is selected.
// The list of accepted format strings is the same one than from
// the gonum.org/v1/plot/vg/draw.NewFormattedCanvas function.
func Show(p *Plot, w, h vg.Length, format string) ([]byte, error) { ... }

Show is especially useful for the new support for Go in Jupyter notebooks (via the Neugram interpreter)

See https://github.com/go-hep/binder for more details and examples on using Go and Go-HEP in Jupyter.

David Blyth provided a new hplot.HInfoStdDev HInfoStyle value and made them bitset-like.
Thanks David!

rootio

cmd/root-dump can now recursively handle rootio.Directory and thus walk an entire ROOT file.
cmd/root-ls got the same handling of rootio.Directory.

Release v0.7 is fresh from the oven with a few breaking API changes.

cmd/yoda

The cmd/rio2yoda, cmd/root2yoda and cmd/yoda2rio commands now support gzipped YODA files.

The cmd/yoda2rio is also more lenient when dealing with YODA objects that are not yet supported on the hbook side (Counter, S1D, ...) (thanks Lukas Heinrich for the report.)

fastjet

Bastian added the ability to plot the Voronoi diagram from a set of 2D-points.

hbook

hbook/yodacnv was modified to support the more lenient approach with regard to unsupported (yet!) YODA objects.

hplot

hplot has seen the most user-facing work:

a new default style that is more in-line with current aesthetic standards (ie: matplotlib-like)
hplot.New uses this new default style and thus is now able to ensure it won't fail loading the fonts
hplot.New thus only returns *hplot.Plot, without an error value
hplot.NewH1D is also able to ensure no fonts-loading error will araise and thus only return a *hplot.H1D

So, where you were doing:

p, err := hplot.New()
if err != nil {
	log.Fatal(err)
}

h, err := hplot.NewH1D(h1d)
if err != nil {
	log.Fatal(err)
}

you can now just write:

p := hplot.New()
h := hplot.NewH1D(h1d)

pawgo

pawgo has been slightly updated to support plotting 2-dim histograms.
pawgo was also fixed to correctly handle YODA files converted to RIO (thanks Lukas Heinrich for the report.)

rootio

rootio gained 2 new commands:

rootio/cmd/root-diff: a command to print the differences between 2 ROOT files, including the content of their TTrees,
rootio/cmd/root-print: a command to print histograms contained in ROOT files into PDF, PNG, ... files.

rootio now also defines and exports the rootio.H1 and rootio.H2 interfaces that are implemented by the TH1x and TH2x (respectively) concrete types.

Release v0.6 is a small iteration (code-wise) but contains:

a software paper submitted to the Journal of OpenSource Software (JOSS):
a new WIP command, rootio/cmd/root-dump, to dump the content of ROOT files, including the entries of TTrees,
documentation fixes and updates in the rootio package,
still more work in the Delaunay and Voronoi area (thanks Bastian!)

One API change that is worth noting, is the retrieval of rootio.Keys from a rootio.Directory.
The d4823f0 changes:

type Directory interface {
    Get(namecycle string) (Object, bool)
}

to:

type Directory interface {
    Get(namecycle string) (Object, error)
}

this allows to more quickly bubble-up errors while reducing boilerplate code at the Directory.Get(...) call site:

diff --git a/cmd/root2npy/main.go b/cmd/root2npy/main.go
index c81e81d..e3827fa 100644
--- a/cmd/root2npy/main.go
+++ b/cmd/root2npy/main.go
@@ -159,9 +159,9 @@ func main() {
        }
        defer f.Close()
 
-       obj, ok := f.Get(*tname)
-       if !ok {
-               log.Fatalf("no object named %q in file %q", *tname, *fname)
+       obj, err := f.Get(*tname)
+       if err != nil {
+               log.Fatal(err)
        }

Release v0.5 has seen some interesting work.

General

Go-HEP has migrated all its Gonum imports to use "gonum.org/v1/gonum/{mat,stat,...}", instead of the old "github.com/gonum/{mat64,stat,...}"
Go-HEP has also migrated all its use of gonum/plot from "github.com/gonum/plot/..." to the new "gonum.org/v1/plot/..."
Go-1.6's support has been dropped from the Travis-CI matrix
Go-1.9's support has been added to the Travis-CI matrix

rootio

The rootio package has seen most of the work: performance improvements for reading ROOT files, a new root2npy command and the beginning of the work to provide write support.

The performances of rootio have been assessed for the ACAT conference.
The talk has all the details and can be read here.
In a nutshell, rootio is now "just" 2 times slower than C++ ROOT to read files, and uses less than 10 times the VMem of C++ ROOT.
More work to match the read speed of C++ ROOT is of course needed but we think we know how to recoup most of performance delta.

cmd/root2npy is a simple command to quickly convert a ROOT file containing a TTree into a NumPy array data file format.

cmd/root-srv has been updated to support https (thanks Michaël Ughetto).

cmd/root2yoda is a simple command to quickly convert a ROOT file containing histograms and profiles into a YODA file.

hbook & csvutil

csvutil/csvdriver has been improved to correctly handle CSV headers.
You need to tell the driver that the CSV file contains a header describing the columns of the CSV file, but once that's done, you can use the correct columns' names in your SQL queries.

hbook/ntup/ntcsv is a new package that leverages the new feature of csvdriver to easily create hbook/ntup.Ntuple values from a CSV file, with the correct columns' names.

fit

Initial support for automatically computing the Hessian matrix has been added.
If the user does not provide a Hess function, Gonum's diff/fd.Hessian will be used

fastjet

More of the GSoC work for the faster jet clustering has been integrated.
Thanks Bastian Wieck.

AOB

With release v0.5, Go-HEP gained its first external user: eicio, "an exploratory project testing ideas about forward-looking IO solutions to suit the needs of the Electron Ion Collider (EIC) community."
Welcome eicio!

Releases: go-hep/hep

Release v0.14.0

Code coverage improvements

hepevt

rootio

xrootd

AOB

Uh oh!

Release v0.13.0

rootio

xrootd

Uh oh!

Release v0.12.0

geo/gdml

hplot

rootio

xrootd

Uh oh!

Release v0.11

rootio

hbook

AOB

Uh oh!

Release v0.10

AOB

Uh oh!

Release v0.9

AOB

Uh oh!

Release v0.8

fmom

hplot

rootio

Uh oh!

Release v0.7

cmd/yoda

fastjet

hbook

hplot

pawgo

rootio

Uh oh!

Release v0.6

Uh oh!

Release v0.5

General

rootio

hbook & csvutil

fit

fastjet

AOB

Uh oh!