A little bit ago, a friend and I attended a RedHat event here in Montreal. Part way through one of the talks, an app was mentioned and demonstrated that had the “Oh! That's pretty close to what I was thinking”. In this instance, it was Apicurio. They let you design your API using a GUI builder which produces an OpenAPI spec file. I love this idea. We should all be using OpenAPI. We don't do it right away for this posting, but it will likely be in an update later on.
Once you've got your spec from Apicurio, you take said file and load it into another tool in the RedHat suite that produces code that can then be deployed to OpenShift, which is a pretty awesome orchastration / PaaS in my opinion. I have no affiliation with RedHat nor any of their products (that I'm aware of), I just really like this one (and CoreOS… which I may have mentioned previously).
Where we left off…
So, this is a Part 2. From Part 1, we've got our insanely simplified super (dope) dummy app. We can run and curl it. To back it, we have our MongoDB instance running as a Docker container. It's now time to rip that code apart and convert to templates. BUT… we don't want to destory our current project, as this is our target state. Let's be sure to back this up to compare against later. A simple mv -rf super{,.bkup} should do the trick here.
WARNING You need to review Part 1 prior to going forward. It may make sense if you skip it, but it's better to go back and see why we're evening going down this rabbit hole to begin with.
Ok, so we're not really going rip apart our super (dope) dummy app in this post. I'm going to cheat and do it for you, then point you to the repo where those templates live. What we will do here is have a look at the code uses said templates, because you need to get something out of this. Why else would you be here unless you're a friend of mine that I'm forcing to read this?
Now I'm going to have a little bit of fun on your behalf. My overall goal of this exercise is to treat each of these auto-generated services like a “worker”. Because I'm a nerd / geek/ etc, I thought it'd be a fantastic idea to call them replicants. Our manufacturer of these replicants… why the Tyrell Corporation, of course. So, create a new project directory mkdir tyrellcorp.
TDD
And now on to everyone's favorite tangent… Testing. I won't preach on about TDD, BDD, design by contract, Bob's new method for testing that the world should follow or you will all burn in DevOps hell. Testing is important. How you test can vary by context and crew. I'm only draggin you down this mild tangent of a soapbox because when I started writing Go code, I accidently developed in TDD. Yeah, accidentally.
I would write throwaway main() code with the // +build ignore tag at the top of the file, followed by a func main() { // call to func I want to test } that just ran the func I was building out, captured the results, and did a panic() on error or fmt.Printf("%+v”, …) on the data returned. The day I realized, this is just super hacky TDD, I started finding ways of cleaning it up and making it more formal. It's by no means perfect and I've got a long way to go before I'm writing any books for Penguin (call me). But… that's where we'll start for this one. Our informal but clean test that will be our blueprint to building out our templating functionality.
replicant_test.go
package tyrellcorp
import (
"encoding/json"
"io/ioutil"
"path/filepath"
"testing"
)
func testReplicant(t *testing.T) *Spec {
in, err := ioutil.ReadFile(filepath.Join("test_data", "sample_spec.json"))
if err != nil {
t.Fatal(err)
}
s := &Spec{}
if err := json.Unmarshal(in, &s); err != nil {
t.Fatal(err)
}
return s
}
func TestGenReplicant(t *testing.T) {
nexus := testReplicant(t)
if err := GenerateReplicant(nexus); err != nil {
t.Fatal(err)
}
// TODO:
// - include checks on generated content
}
And now we test…
$ go test -v -run TestGenReplicant
# github.com/elliottpolk/tyrellcorp [github.com/elliottpolk/tyrellcorp.test]
./replicant.go:3:30: undefined: Spec
./replicant_test.go:10:35: undefined: Spec
./replicant_test.go:16:8: undefined: Spec
FAIL github.com/elliottpolk/tyrellcorp [build failed]
Who screams at the screen and says “you idiot!” when you see the article doing something clearly wrong? Yeah, me neither. So yes, we need more code, because clearly there's way more code in that test file than we actually have available in this fresh project. Let's start with the Spec struct. And what have we been using to define things? Good ol’ protobufs. In our proto/ dir, that you're clearly creating now, we'll add:
record.proto
Because we were totally paying attention in Part 1.
syntax = "proto3";
package tyrellcorp;
import "google/protobuf/timestamp.proto";
option go_package = "github.com/elliottpolk/tyrellcorp;tyrellcorp";
option java_multiple_files = true;
option java_outer_classname = "RecordProto";
option java_package = "com.elliottpolk.tyrellcorp";
message RecordInfo {
google.protobuf.Timestamp created = 1;
string created_by = 2;
google.protobuf.Timestamp updated = 3;
string updated_by = 4;
enum Status {
draft = 0;
active = 1;
archived = 2;
invalid = 3;
}
Status status = 5;
}
field.proto
syntax = "proto3";
package tyrellcorp;
option go_package = "github.com/elliottpolk/tyrellcorp;tyrellcorp";
option java_multiple_files = true;
option java_outer_classname = "FieldProto";
option java_package = "com.elliottpolk.tyrellcorp";
message Field {
// the name of the field
string name = 1;
// the description of what the field is - will be output as a comment
string description = 2;
// the data type of the field
string type = 3;
// this is the "field number" used for protobufs
int32 sequence = 4;
// is this field a list
bool is_list = 5;
// is this a field used in identifying uniqueness
bool is_key = 6;
}
spec.proto
syntax = "proto3";
package tyrellcorp;
import "record.proto";
option go_package = "github.com/elliottpolk/tyrellcorp;tyrellcorp";
option java_multiple_files = true;
option java_outer_classname = "SpecProto";
option java_package = "com.elliottpolk.tyrellcorp";
message Spec {
// standard record values
tyrellcorp.RecordInfo record_info = 1;
// history of the stored record
repeated tyrellcorp.Spec history = 2;
// additional metadata of model
repeated string tags = 3;
// a friendly name of the spec
string name = 4;
// the package name - this will be prepended with `com.elliottpolk`
string package = 5;
// the git repository (including the user | group) the source code will live in
string repository = 6;
// the list of fields for the model
repeated tyrellcorp.Field fields = 7;
// flags to specify CRUD features that should be included
bool create = 8;
bool retrieve = 9;
bool update = 10;
bool delete = 11;
}
Time to run protoc like before:
# for loop to run the protoc command
$ for i in `ls proto`; \
do \
protoc \
-Iproto \
-I${GOPATH}/src \
-I${PWD}/proto \
-I${GOPATH}/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
--go_out=plugins=grpc,paths=source_relative:. \
--grpc-gateway_out=logtostderr=true,paths=source_relative,allow_delete_body=true:. \
"proto/${i}"; \
done
If we were to run our test again, we'd still get an error since our test data is missing, so let's add that. Be sure to create the test_data directory and get a copy of our wonderful sample_test.json from the accompanying repo. Generally, our sample file can contain anything we want, but I'll start with something that resembles our super (dope) dummy repo. This way, we have something to compare to when we're done.
After that is in place… Finally we get some happy results.
$ go test -v -run TestGenReplicant
=== RUN TestGenReplicant
--- PASS: TestGenReplicant (0.00s)
PASS
ok github.com/elliottpolk/tyrellcorp 0.003s
Generate
Now that we've got the ability to “test” something, we can start writing code to apply against the test. For this particular part in the project, the replicant.go file will be pretty sparse. We'll change that in the next iteration (i.e. Part 3).
replicant.go
package tyrellcorp
import (
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
func GenerateReplicant(spec *Spec) error {
log.Debugf("generating project for %s replicant", spec.Name)
if err := CreateProject(spec); err != nil {
return errors.Wrapf(err, "unable to generate the project directory for %s", spec.Name)
}
log.Debugf("generating project assets for %s replicant", spec.Name)
if err := GenerateAssets(spec); err != nil {
return errors.Wrapf(err, "unable to generate repository assets for %s", spec.Name)
}
return nil
}
Insert template {
Since the previous file was sparse, we'll let this one have some heft. It's where we pretend to do all the magic. In reality, we could take this to the next extreme and make it externally configurable, but I think being a little prescriptive at the moment is ok. For now, the goal is to get the idea working then optimize, extend, etc.
For the bulk of the work here (via the helper func parseTpl(…)), we'll rely on the text/template package that allows us to inject a data structure into the specified template. We'll use the Spec struct for most, the only exception being templates/project/version.tpl, of which we'll pass in an anonymous struct with the Version property. Partially because I like having a reason to use an anonymous struct, but also because it doesn't quite make sense to define a full type just for the one property.
project.go
package tyrellcorp
import (
"bytes"
fmt "fmt"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"strings"
"text/template"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
const (
// project directories
cmdDir = "cmd"
configDir = "config"
protoDir = "proto"
grpcDir = "grpc"
restDir = "rest"
// project assets
versionFile = ".version"
makefileFile = "Makefile"
licenseFile = "LICENSE"
readmeFile = "README.md"
recordProtoFile = "record.proto"
serviceProtoFile = "service.proto"
mainGoFile = "main.go"
serverGoFile = "server.go"
serviceGoFile = "service.go"
serviceTestGoFile = "service_test.go"
compositionGoFile = "composition.go"
errorGoFile = "error.go"
// project templates
versionTpl = "templates/project/version.tpl"
makefileTpl = "templates/project/makefile.tpl"
licenseTpl = "templates/project/license.tpl"
readmeTpl = "templates/project/readme.tpl"
modelProtoTpl = "templates/proto/model.proto.tpl"
recordProtoTpl = "templates/proto/record.proto.tpl"
serviceProtoTpl = "templates/proto/service.proto.tpl"
grpcServerTpl = "templates/go/grpc_server.go.tpl"
restServerTpl = "templates/go/rest_server.go.tpl"
compositionTpl = "templates/go/config_composition.go.tpl"
mainTpl = "templates/go/cmd_main.go.tpl"
modelCRUDTpl = "templates/go/model_crud.go.tpl"
modelTestTpl = "templates/go/model_test.go.tpl"
serviceTpl = "templates/go/service.go.tpl"
serviceTestTpl = "templates/go/service_test.go.tpl"
errorTpl = "templates/go/error.go.tpl"
)
func getDir(repo, pkg string) string {
return filepath.Join(os.Getenv("GOPATH"), "src", strings.ToLower(repo), strings.ToLower(pkg))
}
func CreateProject(spec *Spec) error {
dir := getDir(spec.Repository, spec.Package)
// generate main project directory
log.Debugf("generating working directory %s", dir)
if err := os.MkdirAll(dir, 0755); err != nil {
return errors.Wrapf(err, "unable to generate project directory %s", dir)
}
// generate sub directories
for _, d := range []string{cmdDir, configDir, protoDir, grpcDir, restDir} {
child := filepath.Join(dir, d)
log.Debugf("generating child director %s", child)
if err := os.MkdirAll(child, 0755); err != nil {
return errors.Wrapf(err, "unable to generate project subdirectory %s", child)
}
}
return nil
}
func parseTpl(f string, d interface{}) ([]byte, error) {
if _, err := os.Stat(f); err != nil {
return nil, errors.Wrapf(err, "required template file %s not found", f)
}
fm := template.FuncMap{
"ToLower": strings.ToLower,
"ToUpper": strings.ToUpper,
"Trim": strings.TrimSpace,
}
tpl, err := template.New(filepath.Base(f)).Funcs(fm).ParseFiles(f)
if err != nil {
return nil, errors.Wrapf(err, "unable to parse required template file %s", f)
}
buf := bytes.NewBuffer(make([]byte, 0))
if err := tpl.Execute(buf, d); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func GenerateAssets(spec *Spec) error {
dir := getDir(spec.Repository, spec.Package)
assets := []struct {
tpl string
data interface{}
path string
}{
{
makefileTpl,
spec,
filepath.Join(dir, makefileFile),
},
{
versionTpl,
struct{ Version string }{"1.0.0"},
filepath.Join(dir, versionFile),
},
{
recordProtoTpl,
spec,
filepath.Join(dir, protoDir, recordProtoFile),
},
{
modelProtoTpl,
spec,
filepath.Join(dir, protoDir, fmt.Sprintf("%s.proto", strings.ToLower(spec.Name))),
},
{
serviceProtoTpl,
spec,
filepath.Join(dir, protoDir, fmt.Sprintf("%s%s", strings.ToLower(spec.Name), serviceProtoFile)),
},
{
grpcServerTpl,
spec,
filepath.Join(dir, grpcDir, serverGoFile),
},
{
restServerTpl,
spec,
filepath.Join(dir, restDir, serverGoFile),
},
{
compositionTpl,
spec,
filepath.Join(dir, configDir, compositionGoFile),
},
{
mainTpl,
spec,
filepath.Join(dir, cmdDir, mainGoFile),
},
{
modelCRUDTpl,
spec,
filepath.Join(dir, fmt.Sprintf("%s.go", strings.ToLower(spec.Name))),
},
{
modelTestTpl,
spec,
filepath.Join(dir, fmt.Sprintf("%s_test.go", strings.ToLower(spec.Name))),
},
{
serviceTpl,
spec,
filepath.Join(dir, fmt.Sprintf("%s%s", strings.ToLower(spec.Name), serviceGoFile)),
},
{
serviceTestTpl,
spec,
filepath.Join(dir, serviceTestGoFile),
},
{
errorTpl,
spec,
filepath.Join(dir, errorGoFile),
},
}
for _, asset := range assets {
tpl, err := parseTpl(asset.tpl, asset.data)
if err != nil {
return errors.Wrapf(err, "unable to parse template file %s", asset.tpl)
}
if err := ioutil.WriteFile(asset.path, tpl, 0644); err != nil {
return errors.Wrapf(err, "unable to write %s to disk", asset.tpl)
}
}
log.Debugf("running protoc for %s", spec.Name)
// need to run the makefile in order to generate the additional go code from the .proto files
cmd := exec.Command("make", "proto")
cmd.Dir = dir
cmd.Stdout = os.Stdout
cmd.Stdout = os.Stdout
if err := cmd.Run(); err != nil {
return errors.Wrapf(err, "unable to process proto files for %s", spec.Name)
}
return nil
}
The actual bulk of the code is “setup", but we do have a bit to generate the main project directory along with all the subdirectories. This last bit just gives us the bare repository, in preparation for the template parsing.
As with test_data/sample_spec.json, for the templates, I'll refer you to the repo here since it doesn't make sense to embed each one directly here (plus, I promised I would share earlier). As you start digging through each of these .tpl files, you'll notice that they still resemble our original super (dope) dummy app files, only they have some funny looking additions (e.g. {{ .Name | ToLower | Trim }}). This is just the template syntax to ingest the incoming Spec struct or anon stuct in the case of version.tpl. So… assuming you've downloaded those templates from GitHub and added them to your project, we should be able to run our test and generate the new super (dope) dummy app.
$ go test -v -run TestGenReplicant
=== RUN TestGenReplicant
◉ running protoc commands...
--- PASS: TestGenReplicant (0.14s)
PASS
ok github.com/elliottpolk/tyrellcorp 0.140s
Ummm… where's the magic? It's the same thing we saw earlier, but a tad bit slower. What if we have a look one dir up. Your mileage ("kilometrage"?! “metridge"?! for metric?) may vary depending on your workspace, but you should see something like…
$ ls -arlt ..
drwxr-xr-x 1 me me 292 May 19 15:32 super.bkup
drwxr-xr-x 1 me me 306 May 21 23:17 super
If you do a diff on your previous repo in super.bkup/ vs. the newly generated super/, you should see minimal difference. Maybe some spacing differences (which can be sorted using go fmt from within super/), the gzipped FileDescriptorProto of the files like dope.pb.go, and some minor tweaks we snuck in. This is expected… plus, always read the code you've copy & pasted from the interwebs. <ominous_tone>ALWAYS.</ominous_tone>
So, only 2 more things to do. Run the test and run the code. NOTE: This assumes you're still running your MongoDB container.
$ go test -v
=== RUN TestCreate
--- PASS: TestCreate (0.68s)
=== RUN TestRetrieves
--- PASS: TestRetrieves (1.63s)
=== RUN TestUpdate
INFO[0002] attempting to update 1 records action_type=update user=fake_user
INFO[0002] updated 1 records action_type=update user=fake_user
--- PASS: TestUpdate (0.68s)
=== RUN TestDelete
INFO[0003] attempting to delete 3 records action_type=delete user=fake_user
INFO[0003] attempting to delete 1 records action_type=delete user=fake_user
INFO[0003] deleted 1 records action_type=delete user=fake_user
INFO[0003] attempting to delete 3 records action_type=delete user=fake_user
--- PASS: TestDelete (0.50s)
=== RUN TestServiceCreate
--- PASS: TestServiceCreate (0.45s)
=== RUN TestServiceRetrieve
--- PASS: TestServiceRetrieve (0.46s)
=== RUN TestServiceUpdate
INFO[0004] attempting to update 1 records action_type=update user=fake_user
INFO[0004] updated 1 records action_type=update user=fake_user
--- PASS: TestServiceUpdate (0.59s)
=== RUN TestServiceDelete
INFO[0005] attempting to delete 1 records action_type=delete user=fake_user
INFO[0005] deleted 1 records action_type=delete user=fake_user
--- PASS: TestServiceDelete (1.46s)
PASS
ok github.com/elliottpolk/super 6.463s
$ go run cmd/main.go
INFO[0000] starting HTTP/RESTful gateway...
INFO[0000] starting gRPC server...
Depending on what you've loaded into your super (dope) dummy app, running curl localhost:8080/api/v1/dopes should respond appropriately. In my case, it's empty, so I expectedly get {} as a result.
Conclusion
Phase 2 of my master plan is complete. We've gone from a nagging idea, to creating a super (dope) dummy app, to finally automatically generating said CRUD service. Unfortunately, this is only done via a test in this phase. But that's just for now. I'll be posting our third installment in this epic adventure of code shortly. That will give us the GUI we hinted at. I typically use VueJS. I have other toys I'd like to look into for UI, but… priorities.
If you cheated and skipped to the end only for the code, well… there ya go. It's on the branch blog.part2, so don't just checkout the master and start running everything.
K. Til next time.