Category: Build

GitOps Primer

Originally posted on SogetiLabs:
https://labs.sogeti.com/gitops-primer/

With the rise of cloud, cloud-native and a lot of other buzzwords, we also see the rise of all kinds of Dev(fill here)Ops practices. While I’m an avid advocate of the DevOps mindset, all the buzz around pipelines sounds old-school to me.

What is GitOps?

GitOps evolved from DevOps. The specific state of deployment configuration is version-controlled (using Git for example). Changes to configuration can be managed using code review practices, and can be rolled back using version-controlling.

https://en.wikipedia.org/wiki/DevOps#GitOps

A quick look at Wikipedia brings you the above description. It is stuffed away somewhere in the DevOps article. Like with DevOps, there is a whole religion around what tool is the best to use, and people even claim to be ‘DevOps Engineers’ whatever that may be. In most cases they mean ‘cloud infrastructure’ because DevOps is a mindset, and not a function.

It doesn’t help that some tools like Azure DevOps are actually called this way, but hey, VSTS is also not really a snappy name IMHO.

To explain GitOps in my own words and in a single sentence: Instead of pushing your code and deploy, your system pulls the software from the version control system.

A nice one-liner by WeaveWorks is: GitOps is Continuous Delivery meets Cloud Native

https://www.weave.works/technologies/gitops/

GitOps versus DevOps Pipelines

So, it is the same as DevOps Pipelines right? No it is not. From a distance it can seem to be the same. In a way your code, if configured correctly, will be deployed automatically with CICD.

The fundamental difference is that your system (for example Kubernetes) and not your DevOps CICD pipeline, is in the lead. Remember, the settings of the k8s system itself also need a repository.

Let’s look at a typical CICD setup:

DevOps workflow with containers
Source: https://cloudblogs.microsoft.com/opensource/2018/11/27/tutorial-azure-devops-setup-cicd-pipeline-kubernetes-docker-helm/

As you can see in the example above, the Azure DevOps pipelines are in the lead. The Build step pulls in the code, the Release step creates the artifact and pushes it towards a registry, and also installs it on AKS. But this does not include the state of your cluster.

Let’s say your DevOps pipeline trigger failed, or someone made changes manually to the AKS applications or configuration. AKS has no idea where the code is located so it cannot check if there are any changes. This is where GitOps can be a solution. Let’s look at the following example:

GitOps CI/CD architecture
Source: https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-ci-cd

In the example above you can see that (in this case Flux) is used to pull the desired state towards your Kubernetes cluster. This pulling makes sure that you are always up-to-date. And it can also detect changes so if anything or anyone changes the configuration manually, this ‘drift’ will be resolved by the desired state pulling with Flux!

Yes, this complicates your setup, but you always know that your clusters are exactly how you described them in the desired state. Especially in situations with Kubernetes, deployment and configuration can be really complex, so these tools make sure you are in control of your systems.

Example of GitOps tools

So I need Kubernetes? No. In theory any system can use the GitOps setup, but for Kubernetes a lot of tools and so called operators are readily available.

In the previous paragraph, Flux is drawn in the picture. There are many more tools out there, but we see convergence on two major tools: Flux andArgo. The resources section at the end of the blog shows more tools and resources if you want to dive in the deep end of GitOps.

There is a nice writeup (sponsored by Red Hat) on deciding between Flux and Argo. Depending on your situation it can fall either way. GitOps on Kubernetes: Deciding Between Argo CD and Flux – The New Stack.

OpenGitOps

The emergence of GitOps led to the OpenGitOps project.

OpenGitOps is a set of open-source standards, best practices, and community-focused education to help organizations adopt a structured, standardized approach to implementing GitOps.

https://opengitops.dev/

The principles of GitOps are four-fold:

  1. Declarative
    A system managed by GitOps must have its desired state expressed declaratively.
  2. Versioned and Immutable
    Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
  3. Pulled Automatically
    Software agents automatically pull the desired state declarations from the source.
  4. Continuously Reconciled
    Software agents continuously observe actual system state and attempt to apply the desired state.

“The GitOps Working Group is a WG under the CNCF App Delivery SIG.

The focus of the GitOps WG is to clearly define a vendor-neutral, principle-led meaning of GitOps. This will establish a foundation for interoperability between tools, conformance, and certification. Lasting programs, documents, and code are planned to live within the OpenGitOps project.”

The main goal of the OpenGitOps CNCF Sandbox project is to define a vendor-neutral, principle-led meaning of GitOps. This will establish a foundation for interoperability between tools, conformance, and certification through lasting programs, documents, and code.

Concluding

GitOps has been around for several years, and continues to evolve. CNCF has OpenGitOps as Sandbox project and embraced Flux and Argo as Incubating projects. The shift in mindset can be difficult, as some people still try to embrace DevOps as a principle.

Watch this space for more information and an upcoming eBook on our take and practices surrounding GitOps!

Resources and further reading

OpenGitOpshttps://opengitops.dev/
GitOps eBookhttps://www.gitops.tech/
Weaveworks Guide to GitOpshttps://www.weave.works/technologies/gitops/
Harnesshttps://harness.io/blog/devops/what-is-gitops/
Fluxhttps://fluxcd.io/
Argo CDhttps://argo-cd.readthedocs.io/en/stable/
Jenkins Xhttps://jenkins-x.io/
CNCFhttps://www.cncf.io/

VSTS Extension – Tagging all resources within a Resource Group

In my current project I’ve stumbled upon a problem. We have many ARM templates and need to add tags to all resources. Doing this within templates can be cumbersome, so I’ve created a small VSTS extension that adds tags to all resources within a specific resource group.

The extension can be installed from https://marketplace.visualstudio.com/items?itemName=PeterRombouts.autotagging-extension

The working is pretty simple; enter the values for your subscription, and resourcegroup, and then add the tags.

screenshots-vsts-auto-tagging-1

The tag pairs should be entered, one pair per line, comma separated. The extension will add these tags to all resources in the given resource group, and will not do anything to the existing tags.

Remarks? Please do give feedback! The project is available on Github if you want to contribute!

ARM Outputs in VSTS pipelines

In my work as Cloud Solution Architect I often find teams struggling with the exact same problem. In this blogpost I will highlight one of those problems; passing on ARM output variables within a VSTS pipeline.

Many developers have Powershell or other scripts in place, that fix this problem, but I want to highlight an extension, which is freely available and does this for you, for free!

ARM Outputs by Kees Schollaart

 

This extension enables you to use the ARM Deployment outputs in your VSTS environment.

This step will use the last successful deployment within the selected resource group. If this deployment has outputs, all of them are copied to VSTS variables by the ARM Output key.

These outputs can then be used by default VSTS ways: $(same-key-as-in-arm-template)

Usually this task is ran directly after the ‘Azure Resource Group Deployment’ task.

So my pro-tip; Stop using your own scripts and implement this handy extension!

PS: On a sidenote, Kees is also a great guy, give him a tweet after you install this in your project 😉

Link to Marketplace: https://marketplace.visualstudio.com/items?itemName=keesschollaart.arm-outputs
Kees on Twitter: https://twitter.com/keesschollaart

Real-World Partitioning CosmosDb

Azure Cosmos DB is a globally distributed, multimodel database service designed to help you achieve fast, predictable performance. It scales seamlessly along with your application as it grows.

This blog explains exactly how it works and what to do in some scenarios. But in my work as Cloud Architect I sometimes need to dig deeper and figure out if my solution scales for a particular case.

For example, a ‘partition’ has a maximum size of 10GB, so it is really, really important to understand how this works, and how you can figure out your partition size over the time.

This blog will explain, using a recent real-world scenario, how you can determine your partition key and scaling.

Please note that your scenario can differ hugely, but there are some takeways in this blog that could help you when designing your solution.

Continue reading “Real-World Partitioning CosmosDb”

Debugging Azure Functions with LogicApps locally

Azure Functions are a great example of how serverless is conquering the cloud world we live in today. There are also many blogs in how-to-write-functions out there, of which the latest use the precompiled functions, in Visual Studio 2017.

Think of a scenario, where you want to test a LogicApp against your function, but that app obviously cannot connect to localhost. You have two options; You can deploy the Functions app to Azure and use remote debugging or you can use your own localhost, with the use of ngrok.

This blog will explain how to debug locally, when using other services in the cloud.
Continue reading “Debugging Azure Functions with LogicApps locally”

TechDays 2017 Presentation – Emotion and Face recognition using Cognitive Services

Slides of my presentation; Emotion and Face recognition using Cognitive Services.

Abstract:
In this session I will show how to train and use the Face API for recognition of people. Secondly I will show an example of how to determine emotions using the APIs. After this session attendees will have a basic understanding of setting up the APIs, and know how to use and implement the Face and Emotion APIs on the Cognitive Services Stack.

Deploy LogicApps using linked templates

Deploying LogicApps with VSTS is easy. You can create a new project using the steps defined in this post and you are all set with deployment settings in place.

But, when you add more and more LogicApps your JSON file will grow and you might be better off putting each LogicApp within its own JSON. Not to speak of the parameters running wild within your files…
This post will explain how to set this up, and what is needed in your VSTS builds to enable the deployment of all the linked apps.

Continue reading “Deploy LogicApps using linked templates”

Text Analytics – Dutch Support

Recently, Microsoft added Dutch language support for the Text Analysis API within the Cognitive Services stack. In this blogpost I will show how to determine the sentiment of a newsfeed in Dutch.

The importance of language support

Most development articles are written in English. Most documentation is written in English. I’m Dutch and even I am blogging in English!

With the rise of chatbots and other dialogue based UX apps, native language support becomes ever more important. Think of an insurance company wanting to let users talk to their chatbot, or a banking app that lets you choose what mortgage you want for your house. In these situations most companies and clients want to use their native language in communicating with the apps.

With the latest update, the Text Analysis API now partly supports my native language Dutch (along with several other ones).

Continue reading “Text Analytics – Dutch Support”

Emotion Recognition using WPF

A lot of examples of the Cognitive Services include UWP applications and samples. But sometimes you have to support older operating systems, that force you to fall back on techniques like WPF. The use of webcams in WPF is not included by default in the framework.

In this blog I will explain how to implement the Emotion API and build a WPF application capturing your webcam.

A sample of an application you can create shown in the following screenshot. This application shows the realtime spline chart of emotions, and the average bubble chart of the recorded session.

0appdemo

Continue reading “Emotion Recognition using WPF”