Heiko Carrasco

Adaptive Sampling using Stein's Discrepancy

Wed, 23 Feb 2022 19:08:04 -0500

Project Description

Investigation into the usage of different sampling methods based on Stein’s discrepancy in the context of Curriculum Learning.

Implemented both a gradient-based approach (Stein’s Variational Gradient Descent¹) and a direct minimization method (Stein Points²) in jax and evaluated them on different RL baselines.

Liu et al., Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm ↩︎
Chen et al., Stein Points ↩︎

Debugging SAC Implementations in ten Pull Requests (or less)

Sun, 23 Jan 2022 14:11:46 -0500

During the last semester I was working on a research project, where we combined Curriculum Learning with Domain Randomization to increase Sim-To-Real performance without lengthy data acquisition involving the targeted platform. Part of this work involved the open-source Reinforcement Learning library SimuRLacra, which was written by my supervisor and provided us with the necessary algorithms and environments on which we could implement and test our idea.

Unfortunately, the provided Soft Actor Critic (SAC) implementation was not working as well as it should (meaning not at all 😅). Debugging this led me down an interesting but also frustrating path, which I want to recount here together with their appropriate pull requests. So get ready for a deep dive into one of the weirdest (and longest) debugging sessions I ever had.

Step 1 (kinda): What Magic Numbers should I use?

Why only kinda? Well, the usual way to debug a problem in anything related to machine learning is to fiddle with the hyperparameters of the model until you see something. If you have ever worked with SAC you probably know this table, which shows a certain choice of them made by the creator of the algorithm himself¹.

Some potential Hyperparameters for SAC

While they form a good starting point, they are also not optimal to every application and especially not to every implementation of SAC. So usually it requires a lot of guesswork to find good hyperparameters to your problem (except when you are one of FANG, where computing time is free™️). I don’t think that this process is particularly interesting and since it also did not work I am skipping it and move on to the next step in the process:

Step 2: Is it you or is it me? - Finding a baseline

After trying this approach for a long time together with my advisor we decided that it is probably not the correct choice of the “magic numbers” but something else in the implementation or the environment. Since we had algorithms able to solve the environment (e.g. a working PPO implementation), it was probably the former of the two. Following this logic it seemed a good idea to test whether other implementations of SAC would perform similarly on the same task and environment.

The Quanser Qube platform which was used as the target environment

Some of the best implementations of a lot of common algorithms are provided by the stable-baselines project. Unfortunately, they require a specific interface to the environment such that their implementation can communicate and control without having to be adapted to every new environment. While this interface, the openai gym is fairly standardized, it still required some work to adapt to SimuRLacra’s specific implementation of the Quanser Qube².

So my first pull requests was the creation of an interface between SimuRLacra and gym.

After everything in place I was now able to use the implementation of SAC provided by stable-baselines with our environment and see how well it performs.

Step 3: Initialization is Key(-ming)

Running a few experiments with the stable-baselines implementation turned out to be eye-opening. Not only did it run much faster than the implementation provided by SimuRLacra, it also actually solved the task instead of stopping at a sub-optimal solution. To add further insult to injury it did all this without any changes to the default hyperparameters, which made the previous weeks look like a lot of wasted time on my end 😢. Looking at these results I decided to investigate further, trying to find out why there is such a mismatch between these two implementation.

One of the first avenues I took in debugging this problem were the neural network architectures of both implementations. SAC uses two neural networks called actor and critic, which are trained together. Their roles can be inferred from their names: While the actor proposes an action and the critic judges it and provides the feedback to the actor. Both are usually trained together ³ and there are a lot of interdependencies between them, which can be encoded into the architecture of the underlying networks⁴.

In my case I was looking into such a structural difference to explain the initial performance gap: Without any training, the stable-baselines implementation outperformed the one provided by SimuRLacra significantly. As it turned out however, both implementation did not differ that much in their architectural choices, those being size and activation functions. It took me some time and looking at a lot of gradients to find one major difference: Initialization.

From my perspective there is not that much talk about neural network initialization in the supervised and unsupervised space⁵, but even less in the area of reinforcement learning. This is quite curious, since a good initialization can make or break a good experiment run due to its influence on the initial exploration of the agent. However, many RL frameworks just defer to the standard initialization provided by the underlying library like pytorch or tensorflow and do not make any explicit changes to it. This was not the case for SimuRLacra however.

While SimuRLacra usually follows the conventions of pytorch, it did differ significantly in the case of initialization. Instead of just using the defaults, initialization was done explicitly, inspired by the “current” implementation of pytorch. But this code was not updated in some time and while pytorch moved on to use the Kaiming initialization ⁶, SimuRLacra just did not. I therefore adapted the current code to follow pytorchs path, leading to the second pull request (which technically came after the first one, but that is just the order I opened them in, not the order they were created).

And that had a huge impact! Not only in the case of SAC, but also for other algorithms like PPO the performance increased significantly.

Old vs new performance

While this moved the SimuRLacra implementation closer to the stable-baselines one, it did not bring it up to par in regards of later performance: Still the task was not being solved, but at least the intermediate performance looked promising.

Step 4: What to remember?

I want to write a blog post about the underlying project (Self-Paced Domain Randomization) at some point, so I won’t get into too much detail here regarding the intricacies of this specific application. Do understand the next step, however, I will give a short intro to Curriculum Learning.

To be continued

Haarnoja et al., Soft Actor-Critic Algorithms and Applications ↩︎
The gym interface is also not really documented and the missing type annotations make the implementation a little bit of guesswork and trial and error. ↩︎
But not directly updated together in the case of SAC ↩︎
For example you can have actor and critic share the same (learned) feature transformation ↩︎
That does not mean that initialization is a widely disregarded topic; far from it. I just want to point out that it is often glossed over in RL literature. ↩︎
He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification ↩︎

Self-Paced Domain Randomization

Wed, 01 Sep 2021 00:00:00 +0000

Project Description

Implemented a novel approach to Domain Randomization based on Curriculum Learning to bridge the Sim-2-Real gap. Evaluated on a real robotic platform (Quanser Qube). The final code was upstreamed to an open-source reinforcement learning library.

ldap2carddav

Mon, 05 Oct 2020 21:35:42 -0500

D120 Web Projects

Wed, 01 Jan 2020 21:41:52 -0500

During my time as a volunteer at the Computer Science Student Council at TU Darmstadt, I contributed to our web page projects. Highlights include:

Created an nametag creation and group assignment routine for the orientation week
Implemented an automated access code distribution system for the course evaluation

I also maintained these projects and made sure, that the automated dependency management and CI scripts were working and happy 😅.

Ansible Vaults with Gopass

Tue, 01 Oct 2019 12:30:13 +0200

Introduction Vault IDs

The main ansible commands ansible-vault and ansible-playbook already support the --vault-id parameter. It requires a compound of two things: the vault ID label (e.g. dev, prod, etc.) and the means of getting the password (e.g. prompt, a password file or an executable). The vault ID label can be specified while creating the vault or later by modifying the encrypted variant. In the following example the label is called vault-id-label. Notice the format version of 1.2, which you might need to change in your file too.

$ANSIBLE_VAULT;1.2;AES256;vault-id-label

During an ansible run this vault could be opened by adding the --vault-id vault-id-label@prompt parameter, which would prompt the user for the vault password. Changing prompt to vaultpass would let ansible load the password from the file vaultpass.

Password scripts

The third option would be to specify an executable, which will get called by ansible. There are two variants:

Normal scripts, which are called without any parameters and just return a password
Client scripts, which have to end in client and are called with the --vault-id option plus the label of the vault

In this example I will use a client script, since I have several vaults with different labels/passwords. The implementation is quite straightforward:

#!/usr/bin/env python

import sys
from subprocess import run, PIPE, STDOUT

VAULT_PASSWORD_LOCATION='pathToMyAnsibleVaultPasswords/'


def main():
 if len(sys.argv) != 3 or sys.argv[1] != "--vault-id":
 print('Necessary argument "--vault-id" missing')
 sys.exit(1)
 result = run(
 ["gopass", VAULT_PASSWORD_LOCATION + sys.argv[2]],
 text=True,
 stderr=STDOUT,
 stdout=PIPE,
 )
 if result.returncode != 0:
 print(result.stdout)
 sys.exit(result.returncode)
 else:
 sys.stdout.write(result.stdout)
 sys.exit(0)


if __name__ == "__main__":
 main()

Save this script as gopass-client.py next to your ansible.cfg (also make it executable) and use --vault-id vault-label@gopass-client in your ansible command and it will magically unlock your vault using gopass. Don’t forget to change the VAULT_PASSWORD_LOCATION to the path of your ansible passswords in your gopass store.

FB20 Software Development

Sat, 23 Feb 2019 22:04:15 -0500

During my time at the Computer Science department at the TU Darmstadt, I worked on several internal closed-source projects, such as:

kubernetes Ansible Roles
Hugo Website Themes
Evaluation of Matlab and Toolkit Usage at the department
User Management Website in Django
Created specialized Containers for CI usage
dfnclient

For references, please contact ISP CS TU Darmstadt

dfnclient

Tue, 01 Jan 2019 16:21:14 -0500

Project Description

Reverse-Engineered the Certificate API for the German Research Network, so SSL certificates can be requested with one command. Currently used by a number of German universities such as the TU Darmstadt, University of Heidelberg and TU Chemnitz.

Using Visual Studio Code with Okular and synctex

Mon, 24 Dec 2018 00:40:17 +0100

Visual Studio Code is a great editor, Okular a great PDF Viewer. To make them work together, a little bit of config is necessary though.

Update: Dan MacKinlay has written a nice introduction to Visual Studio Code as a LaTeX IDE. You can find his article here.

Recently I started to use Visual Studio Code more and more when I have to write LaTeX code. Together with the LaTeX Workshop Extension, creating beautiful documents is a breeze, since it has all the features of a good IDE. Except one… Synctex 🙄

The problem with pdf.js

If you use the built in pdf viewer, Visual Studio Code (or as I have recently taken to calling it VCS 😅) uses pdf js, a web based pdf viewer written by Mozilla. Though this is sufficient for some use cases, it breaks on several occasions like rendering a math formula. I therefore started using the external viewer function of the extension to mitigate such problems (namely Okular). This leads to another problem: How to use synctex with it?

Okular has a feature to jump to the correct location of a pdf, but there is one problem: Nobody ever bothered to write down the syntax for this command. So after a long Google search and several trial and error attempts low and behold the VCS config for okular and synctex. Just copy this into your user settings.

{
 "latex-workshop.view.pdf.external.command": {
 "command": "okular",
 "args": [
 "--unique",
 "%PDF%"
 ]
 },
 // If you want the external viewer to be default, uncomment the next line
 //"latex-workshop.view.pdf.viewer": "external",
 "latex-workshop.view.pdf.external.synctex": {
 "command": "okular",
 "args": [
 "--unique",
 "%PDF%#src:%LINE%%TEX%"
 ]
 }
}

Yubikey and systemd

Fri, 22 Jun 2018 23:33:47 +0200

Background

I love my Yubikey. Besides using it for local authentication it also contains my PGP key. By using the awesome pass password manager (or more accurately the go rewrite gopass) I do not have to write passwords into config files any more but just configure the correct call to pass and the rest is handled by the system.

In case of email this worked really well as long as I was using offlineimap. This program provides a background daemon which downloads all your mails to your drive. I was really happy with this setup, but as more and more accounts got added this was one of the biggest resource hogs on my system.

So I decided to switch to isync, which is a lightweight and really fast alternative to offlineimap. The problem is, that isync has to be called periodically since it just provides a simple sync command. That means my password is needed every X minutes. Usually this is no problem, since my Yubikey handles this for me. But sometimes I use my computer without it and this is a big problem.

Isync attempts to decrypt the gpg file containing my password and naturally asks for the key. Since it is saved on my Yubikey I get this handsome dialog:

Please insert card with serial number

I can manually disable the systemd timer I wrote to trigger the sync but this is cumbersome. So I searched for a way to automate this and found it:

Udev to the rescue

For some time now linux includes udev which is according to wikipedia

a device manager for the Linux kernel.

It is a complicated system and I only want to focus on the user facing part, called rules. These are simple text files which are evaluated by udev when hardware events trigger. A simple example is the addition of a usb keyboard which triggers a command to use the built in keyboard.

My very first udev rule

So I started to write my first udev rule. This is surprisingly easy, as a rule consists of a identifying part and a execution part. First I had to find something to identify my key. I decided to use the vendor and product id, which a USB device should usually have. They can easily be extracted by using the lsusb command. For my key the output was:

Bus 001 Device 030: ID 1050:0407 Yubico.com Yubikey 4 OTP+U2F+CCID

The format is: ID <vendor id>:<device id>.

With this in mind my udev rule looks like this:

SUBSYSTEM=="usb", ENV{ID_MODEL_ID}="0407", ENV{ID_VENDOR_ID}="1050", TAG+="systemd", SYMLINK="yubikey",
SYSTEMD_READY=0

Breaking this in its parts you get:

SUBSYSTEMS=="usb": limits to usb devices
ENV{ID_MODEL_ID}=="0407", ENV{ID_VENDOR_ID}=="1050",: limit to the specified usb prodcut and vendor id
SYMLINK="yubikey": create a device called “yubikey” in dev (/dev/yubikey)
TAG+="systemd": only devices with this tag are added to the systemd device system
SYSTEMD_READY=0: only show this device in the tree if it is actually plugged in

Saving this in a file called 85-yubikey.rules inside the /etc/udev/rules.d/ directory and calling sudo udevadm control --reload-rules activates the rule (the number in front of the rule just specifies the order; this rule gets loaded pretty late and after all rules with lower front numbers).

If you plug your device in now a new entry in /dev/ should appear, which is called yubico. If this does not happen, check the ids and maybe restart your system.

Telling systemd to look for the key

The rest is really simple. Systemd generates dev-devicename.device files for every udev device with the systemd tag. This device files can be used everywhere where a unit or target is required.

A quick look in the documentation suggests Requisite and After as the appropriate options:

[Unit]
...
Requisite=dev-yubikey.device
After=dev-yubikey.device

[Service]
...

This is basically the equivalent to the following meme 😉

And this is it. Now the service is only started when the key is inserted. This all took about half an hour and will save me a lot of time and nerves in the future.

Ansible Roles

Mon, 19 Jun 2017 21:54:18 -0500

Worked on several Ansible roles for both the CS department and student council. Some roles I wrote included:

BigBlueButton (Video Conferencing Software)
Bookstack (Wiki)
Mailman3 (Mailinglist Software)
Prometheus (Alerting and Monitoring Tool)
nextcloud (Document Storage)

Mon, 01 Jan 0001 00:00:00 +0000