Babelfish SDK contains the tools and libraries required to create a Babelfish driver for a programming language.
The Babelfish SDK has the following dependencies:
Make sure that you've correctly set your GOROOT and GOPATH environment variables.
Babelfish SDK gets installed using either Go:
$ go get -t -v github.com/bblfsh/sdk/v3/...
or make command:
$ make install
These commands will install bblfsh-sdk
program at $GOPATH/bin/
.
The SDK provides scaffolding templates for creating a new language driver.
These templates are converted to Go code that ends up in bblfsh-sdk
tool. Use make bindata
to update these templates:
$ make bindata
chmod -R go=r /home/user/bblfsh-sdk/etc/build; \
go run github.com/kevinburke/go-bindata/go-bindata/ \
-pkg build \
-modtime 1 \
-nocompress \
-prefix /home/user/bblfsh-sdk/etc/build \
-o assets/build/bindata.go \
/home/user/bblfsh-sdk/etc/build/...
go fmt ./assets/...
chmod -R go=r /home/user/bblfsh-sdk/etc/skeleton; \
go run github.com/kevinburke/go-bindata/go-bindata/ \
-pkg skeleton \
-modtime 1 \
-nocompress \
-prefix /home/user/bblfsh-sdk/etc/skeleton \
-o assets/skeleton/bindata.go \
/home/user/bblfsh-sdk/etc/skeleton/...
go fmt ./assets/...
You can validate this process has been properly done before submitting changes:
$ make validate-commit
If the code has not been properly generated, this command will show a diff of the changes that have not been processed and will end up with a message like:
generated bindata is out of sync
make: *** [Makefile:66: validate-commit] Error 2
Review the process if this happens.
On the other hand, If you need to regenerate proto and proteus files, you must run go generate
from protocol/ directory:
$ cd protocol/
$ go generate
It regenerates all proto and proteus files under protocol/ and uast/ directories.
Babelfish SDK helps both setting up the initial structure of a new driver and keeping that structure up to date.
Let's say we're creating a driver for mylang
. The first step is going to the location
where we want the repository for the driver to be bootstrapped:
$ cd $GOPATH/src/github.com/bblfsh
Now the driver should be bootstrapped with bblfsh-sdk
. This will create a git repository,
and some directories and files required by every driver. They will be overwritten if they
exist, like the README.md file in the example below.
$ bblfsh-sdk init mylang alpine
initializing driver "mylang", creating new manifest
creating file "manifest.toml"
creating file "Makefile"
creating file "driver/main.go"
creating file "driver/normalizer/normalizer.go"
creating file ".git/hooks/pre-commit"
creating file ".gitignore"
creating file ".travis.yml"
creating file "Dockerfile.build.tpl"
creating file "driver/normalizer/normalizer_test.go"
creating file "Dockerfile.tpl"
creating file "LICENSE"
managed file "README.md" has changed, overriding changes
$ git add -A
$ git commit -m 'initialize repository'
Note that this adds a pre-commit git
hook, which will verify
these files are up to date before every commit and will disallow commits if some
of the managed files are changed. You can by-pass this with git commit --no-verify
.
You can find the driver skeleton used here at etc/skeleton
.
Whenever the managed files are updated, drivers need to update them.
The update.go
script can be used to perform some of those updates in managed files.
For example, if the README template is updated,
running go run update.go
will overwrite it.
$ go run update.go
managed file "README.md" has changed, overriding changes
bblfsh-sdk
doesn't update the SDK itself. Check Gopkg.toml
for the target SDK version.
For further details of how to construct a language driver, take a look at Implementing the driver section in documentation.
In order to run test for a particular dirver, change to it's directory and run:
$ go run ./test.go
This will:
- compile a "test binary" that parses content of the
./fixtures
directory of the driver - create a docker image with all dependencies, native driver and a test binary
- run this test binary inside a Docker container, using that image
The test binary first parses all source files in ./fixtures
and generates a set of
*.native
AST files. Then the second set of test is run that uses *.native
files,
applies UAST annotations and normalizations to produce *.uast
and *.sem.uast
files.
If *.native
files were already generated, the second stage can be run on the host without
Docker container:
$ go test ./driver/...
Overall, SDK supports 4 different kind of tests for a driver:
- Native unit tests, parsing source files in
./fixtures
and writing*.native
. Runs only in Docker. - UAST transformation unit tests, using
*.native
and writing*.uast
files. Can be run both on host and in Docker. - Integration tests, using content of
./fixtures/_integration*
. Those are run in Docker usingbblfshd
. - Benchmarks, using content of
./fixtures/bench_*.native
. Can be run on the host or in Docker.
First two always run, benchmarks are only triggered by bblfsh-sdk test --bench
or go test -bench=. ./driver/...
.
GPLv3, see LICENSE