Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ds-horizon/datagen

Repository files navigation

datagen

Go Version License Go Report Card Join our Discord

datagen is a tool to generate coherent, synthetic data generation from models expressed in a simple, declarative DSL.

Watch the Demo

Watch the video

Salient features:

  • A declarative DSL for defining data models with Go-like syntax
  • High performance through transpilation to native Go code
  • Multiple output formats (CSV, JSON, XML, stdout)
  • Database integration with direct loading to MySQL
  • Model relationships via cross-references using self.datagen
  • Tag-based filtering for selective data generation
  • Built-in functions for common data items

Install

There are various ways of installing datagen.

Option 1: Install via Go

Check your $PATH, and choose a directory you would like to place the datagenc compiler in.

echo $PATH
/Users/username/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin

Say, you wish to place the binary in /opt/homebrew/bin;

export GOBIN=/opt/homebrew/bin
go install github.com/ds-horizon/datagen/cmd/datagenc@latest
Verify installation
datagenc --help

Option 2: Install from Source

Clone the repository
git clone github.com/ds-horizon/datagen
Build the compiler
make build-compiler

For permanent access on Mac/Unix, add the binary to your path, or add the current directory to your path:

echo 'export PATH=$PATH:$(pwd)' >> ~/.bashrc  # for bash
echo 'export PATH=$PATH:$(pwd)' >> ~/.zshrc   # for zsh

For permanent access on Windows, add to your shell profile:

echo '$env:PATH += ";C:\path\to\datagen"' >> $PROFILE

Now, source the rc files or fire up a new terminal window for the changes to take effect.

Verify installation
datagenc --help

Usage

You can launch datagen for trying it out with:

# Create a simple model file
cat > user.dg << 'EOF'
model user {
  metadata { count: 100 }
  fields {
    id() int
    name() string
  }
  gens {
    func id() { return iter + 1 }
    func name() { return Name() }
  }
}
EOF

# Generate data
datagenc gen user.dg -f csv -o ./output

this will generate a user.csv file in output directory with 100 user records.

More information

Contributing

Refer to CONTRIBUTING.md

License

MIT License, see LICENSE.

About

Generate coherent, synthetic data at scale

Resources

License

Contributing

Stars

Watchers

Forks

Contributors 7