Recently I came across a few projects that had many Dependabot PRs open against them.

Best practice would be to check that after a version update, software behaves the same and there is no regression, but this means that after merging in each version bump, we need to:

1. Wait for pipeline to pass
2. If we don't have a pipeline with tests, do some manual checking
3. Revert if there is a problem

In an example where we have 5 packages that need updates, it would take 5 * the time it takes to merge a single one.
What if we could merge all 5 at the same time? 

```sh
git branch -r | grep 'origin/dependabot/' | while read -r branch ; do git cherry-pick $(git rev-parse $branch) ; done
```

This shell script parses `origin` for dependabot pull requests, and cherry picks all of them into your current branch.

This is a 500% increase! (in our example)

I put the gist [here](https://gist.github.com/yzAlvin/48d2079fa7dfcc031a6a86c7a8d0cdb0)

Faster Dependabot Updates

I wanted to convert a csv file to Markdown. Like my previous post about web scraping, I'd done it before in Python and it was trivial, so I decided to try it with Go.

The thing I liked most was parsing rows into a defined struct.

```go
type Blip struct {
    name        string
    ring        string
    quadrant    string
    isNew       string
    description string
}
```

Apparently in Python 3.7, dataclasses can be used to achieve a similar thing, but I wasn't aware until now.

In Go, there was also much more error handling than Python. I found it to be slightly annoying for my purposes, but perhaps it's actually a blessing in disguise.

My opinion:

* The csv packages in both are really similar
* I really like defining the shape of my rows in Go.
* I prefer Python's string interpolation over Go's string formatting.
* I don't really see a use for the error checking in Go.

This might be the case of me being a noob, not appreciating static typing and robust error handling, but for this use case I reckon Python would have done the same in less lines of code. Perhaps if I was working with more complex data I would prefer using Go.


Transforming csv Files with Go

Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports.

My first introduction to Dhall was in a team I was rotating through, where they used it to write their yaml files, such as Buildkite pipelines, Kubernetes definitions and AWS cloudformation templates.

I had previously worked with all of these, except I wrote them in yaml, and experienced literally every problem that Dhall solves. Indentation errors, invalid schema, incorrect types, small typos... Learning Dhall was challenging at first, but made me much more productive and quick when I got used to it.

It is SO beneficial to have the language server throw errors immediately, rather than write some yaml, check it in, wait however long for the pipeline only for it to fail, and repeat.

To talk to Dhall's description as 'JSON + functions + types + imports':

* JSON is json, you can still write regular json and it will compile to json/yaml fine.

* Functions are really nice because they help to DRY the config, and we can have dynamic templates depending on a parameter, which could be for example, the hosting environment.

* Types are great because it forces you to pass in the correct type. We can define our own types so types can require other types. No more indentation errors or missing fields!

* Imports are also really nice because it allows us to separate out our config into more workable files. For example, A cloudformation template could define 10 resources in one file, and end up as one huge file; Or we can define each resource in their own file and import them into a 'main' one, resulting in 11 more maintainable files.

I even enjoyed it so much I hosted a workshop for my graduate cohort to go through a demo.

The Dhall documentation is really amazing, and the examples on the [official website](https://dhall-lang.org/) aren't contrived at all. The only pain point I found is that dhall is not installed on most CI tools, so you will need to install it first. The team used `nix`, but I guess you could use `brew install dhall`.

These are the open source packages we used to define:

* [Buildkite Pipelines](https://github.com/myob-oss/buildkite.dhall)
* [Cloudformation](https://github.com/jcouyang/dhall-aws-cloudformation)
* [Kubernetes](https://github.com/dhall-lang/dhall-kubernetes)


Dhall is Cool

During the time I spent in phase 1 of the Future Maker's program I learnt a lot and took a bunch of notes about language design.

## Four Rules of Simple Design

If code follows these 4 rules, then usually it is pretty good code.

1. Passes Tests
  Code is functionally correct, and we can verify it is correct with comprehensive and fast automated tests.
2. KISS / Reveals Intent
  Code is simple enough that from an outsiders perspective, with minimal to no context, the implementation is clear and unambiguous.

  Readability of code is very important.

  More time is spent reading over code than actually coding, in larger projects it's important that others can understand your code and you can understand theirs.

  Comments are not a good way to reveal intent.

  Comments should be rarely used, if ever.

  Instead of comments, you could name things better, make code smaller, extract functionality.

  Naming should be consistent.

  Names should reference mainly domain concepts.
3. DRY / No Duplication
  No duplicated logic. Pull duplicate patterns into appropriate methods/classes. Polymorphism. "Duplicate code always represent a missing abstraction."
4. YAGNI / Fewest Elements
  If the first 3 rules are met, any other elements potentially YAGNI. Avoid adding functionality until it is necessary. This results in the fewest lines of code that still meets the requirements.

## Command Query Seperation

Command Query Separation is a principle of imperative programming.

Every method should either be a:

* **command** that performs an action
* **query** that returns data to the caller

but not both.

There are some exceptions, for example, the `pop` method of a stack datastructure both performs the action of removing the top element of the stack, but also returns the removed element.

## Object composition

Object composition is useful for when we want to share functionality between multiple classes, and inheritance is not a good solution.
To put it simply, composition contains classes of other classes that implement the desired functionality

Inheritance is when you design types around what they ARE

Composition is designing around what they DO

Composition is more maintainable than inheritance. It also helps to keep the size of the project smaller and (objectively) neat.

One way to remember when to use either is:
If there is a 'has a' relationship we could use composition
If there is a 'is a' relationship we could use inheritance

But like with everything else don't stick to rules without critical thinking.

Interfaces that are large should be split into smaller and more specific ones. Clients should only know about methods that are relevant.

Avoid object hierarchies.

Instead of creating a single parent class:

1. Create one more more smaller classes or interfaces that encapsulate the desired functionality
2. Include instances of these in any new classes we wish to share functionality with
3. Expose the desired functionality by using these instances

## Keeping Things Small // YAGNI

* Write code that fits in your head
* If a class needs lots of interfaces, simplify the class
* Don't make lines too long
* Think of the simplest way to do things
* Remove as many levels of indentation as feasible
* Classes and methods should be small components with clear responsibilities.
* High-level modules and low-level modules should depend on abstractions
* Abstractions should not depend on details - details should depend on abstractions

YAGNI is useful, because requirements may change in the future, so code that is not used could stay useless anyway or the feature it was intended for may never be necessary.

There would be more refactoring required later if there is some code that was not initially being used and is now being used to fit a new requirement.

There are costs to adding in the unnecessary code:

* Cost of delay - could have been working on another, necessary feature.
* Cost of build - time was spent on a currently useless feature
* Cost of repair - time spent needing to fix the code
* Cost of carry - unnecessary space used up by code, another thing to read

Language Design

Recently, we had an important API service suddenly stop working at work.

This service is supposed to send out a personalised link to everyone in the team, but it randomly stopped working, and stayed that way for 3 weeks. Here is how we fixed it.

There were many reasons why it took so long to be fixed:

1. Everybody that had once worked on it weren't part of the team anymore, so nobody really understood the codebase.
2. Nobody in the current team had the knowledge or confidence to troubleshoot and solve the issue. The service was written 2018 with the `netcoreapp2.1` framework, and most of the team was fairly new to C#, learning on the `.net5` framework.
3. There was other, more interesting work to do.

Eventually, it was becoming a bigger issue, as people weren't getting their personalised links anymore, we were losing observability. It was just too tedious and error-prone to have to manually customise the links for everyone.

At about week 2, four of us dedicated some time diving into the codebase and trying to troubleshoot the issue. We found that the service that wasn't working was dependent on another API, which was served with a Let's Encrypt certificate.

The [DST Root CA X3](https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816) expired around the day our service stopped working, and there was an issue with OpenSSL versions 1.0.0 through 1.0.2, where they will reject Let's Encrypt certificates, regardless of whether ISRG Root X1 is installed on the client. Lo and behold, [.NET Core 2.2 and earlier versions prefer OpenSSL 1.0.x over 1.1.x](https://docs.microsoft.com/en-us/dotnet/core/compatibility/cryptography#net-core-30-prefers-openssl-11x-to-openssl-10x).

Digging into this further, we ran our service in docker with different images from .Net Core 2.1 to .Net 5, and tried curling the external API.

| Runtime | curl result |
| ------- | ----------- |
| .Net Core 2.1 | SSL certificate problem: certificate has expired |
| .Net Core 2.2 | SSL certificate problem: certificate has expired |
| .Net Core 3.0 | 200 |
| .Net Core 3.1 | 200 |
| .Net 5.0 | 200 |

Great! So it seems like we just need to update to .Net Core 3.0 and our issues should be fixed.

Luckily for us, Microsoft has some pretty good [docs for migration ASP.NET Core versions](https://docs.microsoft.com/en-us/aspnet/core/migration/21-to-22). We just followed the docs, skipping over parts that we found weren't necessary. We made the decision to update to .Net 5 instead of sticking to .Net Core 3.0 so that hopefully when we eventually move on others will have less work to do, and have an easier time fixing anything.

What I learnt from this was: even if I don't know what's wrong or what to do, just volunteer and I'm bound to learn something in the process. The worst that could happen is the service stays down.


Migrating a Webapp from netcoreapp2.1

Recently I wrote a simple nodejs script to parse YAML front matter in Markdown files, and convert them into JSON. In the spirit of automation, I wanted to find a way to automate running the script.

I have used GitHub Actions a lot, but have never written my own, so I thought this might be a good time to learn. Luckily for me, [GitHub has great docs](https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action) so I didn't really run into many issues.

First, I just took my script and instead of writing to a file, just print out the results. I tried to see if it would run as an action. It did, but it couldn't find the directory I was pointing to. I had to use the `actions/checkout@v2` in a step before to get access to the files in my repo. I ran it again, this time it worked.

I changed the script to take input from the action, ran it again, and it worked.

I then tried to write to a file instead of printing the result. No luck, I couldn't figure it out, by this [GitHub Communtiy Post](https://github.community/t/can-github-actions-directly-edit-files-in-a-repository/17884/7) helped me figure it out. Instead of writing to the file system with node, I had to either use another action or run use git commands in my workflow. I chose the latter.

Before I could do that, instead of returning the result from my script, I had to send it to the action output. A bit of fiddling with shell commands later, It worked! This is the step I added:

```yaml
run: |
  echo '${{ steps.demo1.outputs.output }}' > 'res.json'
  git config user.email "actions@github.com"
  git config user.name "hehe"
  git add --all
  git commit -m "Update json" || echo "No changes to commit"
  git push
```

I did some additional configuration to add features to the action, so it was more generic and not only useful for my needs, and added a demo to the repo. I noticed at the top it was prompting me to publish the action so I did!

However I realised my documentation was awful, and it would probably be really hard to use it even if you wanted to, so I spent some time looking at other actions documentation and wrote my own. I also kept flip flopping between 'Front Matter to JSON' and 'Markdown Metadata to JSON', I decided on 'Front Matter to JSON' becauce you can have yaml front matter in files other than markdown, so I thought it'd be more general.

Please check out the action [here!](https://github.com/marketplace/actions/front-matter-to-json-action). I'm real keen to improve it so please do open issues, create PRs :smile:.


My First Github Action

I wanted to look back at all my [Codewars](https://www.codewars.com/) solutions, and perhaps create some kind of portfolio frontend for it, so I had a look at the [Codewars API](https://dev.codewars.com/). Unfortunately there was no endpoint to get the actual solution code, so I decided to try scrape it.

I had used BeautifulSoup in Python in the past, but didn't really like it. Maybe because I didn't read the documentation fully or because I didn't understand as much as I do now, but I decided I would try [Colly](http://go-colly.org/), in Go, a self-described 'Fast and Elegant Scraping Framework'.

To my surprise, it was really not too difficult, I was able to find everything I wanted to do easily in the documentation and the examples were very helpful.

I really liked defining a structure for the shape of what I wanted.

```go
type Kata struct {
    Kyu             string   `json:"kyu"`
    KataLink        string   `json:"kataLink"`
    KataTitle       string   `json:"kata"`
    LanguagesSolved []string `json:"languages"`
    Solutions       []string `json:"solutions"`
}
```

It made it real easy to extend the scraper bit by bit, and change the schema without getting confused.

I also really liked the functions a `collector` has, such as `OnHTML()` to match every HTML element by some jquery, rather than a looping over the elements, which is how I would have done it in Python. I also liked the functions to easily get the children elements.

Configuring the scraper was also much easier than with BeautifulSoup. I wanted to set a cookie and in Colly there's a function for it, whereas in BeautifulSoup I would have needed another library.

In the end, I had more issues with the structure of the HTML than scraping itself, but it was so easy I was able to modify the code to scrape a very different website with little code change.

I have built my Codewars solution scraper and you can see the [source code here](https://github.com/yzAlvin/Codewars-Solution-Scraper). I would still like to add more enhancements but that is separate to the scraping part.

I also built a Nestle website scraper for my side project, [source code here](https://github.com/yzAlvin/nestle-proj/tree/master/scraper). Again, I had more issues with the way the website was structured than scraping itself.

I would definitely recommend trying Colly for anyone looking to write any kind of crawler/scraper/spider.

Scraping in Go with Colly

I've stumbled across many tech blogs, and always found it interesting to read about someone's opinions/experiences. I particularly like [Project Nayuki](https://nayuki.io). I already write my notes in Markdown, so I figure it should be easy enough to turn them into blog posts.

Hopefully by writing these I am able to convey my thoughts clearer, and get better at explaining things. It might also be a great place for me solidify my knowledge, by writing it down and having something to refer to.

This place will probably become a dumping ground for whatever I want, but I'm hopeful someone will glean something from a post. I'll pretend this space is like CreedThoughts or Bob Loblaw's Law Blog 🤪.
