SAP on Google Cloud: Creating the CI/CD Pipeline (pt. 3)
The image above shows the CI/CD flow we will implement. When the app developer commits the code, it will be picked up by a trigger in Cloud Build. It will then build, test and deploy our application. Let’s focus initially on a single environment pipeline since expanding it to multiple environments later is quite simple.
Demo app recap
Before we dive into the pipeline, let’s quickly recap what our application looks like:
- Frontend: web interface, allows the user access to the application.
- Backend: translation service
- DAL: the data access layer
- Database: SAP HANA — stores our data.
The diagram above shows how these components interact with each other. For more details, check out part 2 of this series.
Where do I even start?
Now you might be thinking: That is great! Where do we start? How do I pick the first microservice to become CI/CD’ed? (yes, I just invented a word… Poetic License FTW!!!)
When first starting a CI/CD pipeline for an existing application, it might be a challenge to know how to begin dissecting the beast. Look for the following traits in a service:
Fairly independent, meaning it doesn’t have to call a lot of other services within the same application (it’s ok to be a dependency for other services)
A service that has good test coverage for the unit, functional and integration tests (yes — they are all important)
Has a relatively low overall complexity
Take a peek at the diagram above again, which service do you think is the most adequate to be the first? Take a guess!
We are starting with the backend service since it’s quite isolated and well tested. When automating deployments, always try to improve your tests to ensure the reliability of your application — your future self will thank you.
Build Containers
The backend depends on the Google Translate API SDK for Golang; this means we would always need to install that SDK with a go get before can build. We also want to run go test and go vet as part of our test process.
Pipeline steps in Cloud Build run inside containers; this means we can simplify our build by creating a custom container with all the tools we need. Let’s go ahead and create a build container for the backend and store it in the container registry for later use within the pipeline.
Why bother, you ask? Eventually, you will want to run 10’s or even 100’s of builds a day — so those few seconds get magnified; this drives the needs to make builds as fast as possible. A build container allows us to skip the download and install step for build dependencies — saving time.
Here is what our backend build container Dockerfile looks like:
The Pipeline Configuration
Now that we have the environment, let’s check what the pipeline looks like:
- Run static tests using go vet
- Run unit tests go test
- Run functional tests (also go test)
- Build container (docker)
- Push container to the registry (docker)
- Deploy Container to the environment (gcloud)
- Run integration tests (go test)
The Cloud Build pipeline is configured via a YAML file called cloudbuild.yaml as follows
Of course, to implement the pipeline for other technologies the details will change; however, the overall flow will remain the same. For example, when testing the DAL layer using NodeJS the command will be npm test instead of go test; and so on.
Some important considerations here are:
- Each microservice has an individual pipeline (cloudbuild.yaml)
- Each microservice will generate its artefacts independently
- Each microservice should test the integration with its dependencies.
Remember this image Lucia explained in part 2?
Look at the cloudbuild.yaml closely and you will see a dir: translate on each step that points to the backend microservice and thus this will only run when we make changes in that folder. There are many ways of doing this so pick the version that best suits your release and development needs. Also, notice how the cloudbuild.yaml file sits inside the folder for its relevant microservice.
Notice that we configured the glob pattern to match only changes applied inside our microservice folder in the case of our backend it’s translate/**. This prevents this trigger from going off if we commit in other folders in the repo.
Also see how our regex is matching a branch named master; for non-master branches, you can check the “Invert Regex” checkbox; and finally the Cloud Build configuration file also points specifically for our microservice pipeline definition with translate/cloudbuild.yaml
By now you can pretty much visualize how you would create the pipeline for the other microservices too!
To run your pipeline commit to your repository in the correct folder, you also have the option to run it manually. The build logs can be viewed by going to the Console and looking at the build execution like so:
As a side note, this Dockerfile is not optimized and you may want to tweak it a bit before using.
Notice how we set up the container with the required tooling to connect to HANA via HDI. You can build this container in the same way we created our backend build container. This step is important because it essentially allows us to be ready to connect and deploy our db-things via HDI.
Another important difference is how the npm start command is configured. We added the exit flag, so it won’t wait for a human to press Ctrl+C without that the pipeline would hang and timeout.
