Z3 is a Satisfiability Modulo Theories (SMT) solver made by Microsoft Research. It is cross-platform and is released under the MIT license. Z3 comes with a Python API that we will be using. Our goal is to encode the rules of the Binary Puzzle game in terms of mathematical equations that Z3 can comprehend. Once we have defined the rules of the game for Z3, we want to use it to solve any solvable Binary Puzzle for us or tell us if the puzzle is unsolvable.
I enjoy number puzzles such as Sudoku and Binary Puzzles. For some reason, I always end up solving more Binary Puzzles than I solve Sudokus. Binary Puzzles are more straightforward than Soduku and are thus playable in a shorter amount of time. A Binary Puzzle can be played online from various websites or via applications that are available for both Android and iOS. Look in the application store on your preferred platform, and you will most likely have numerous implementations of this uncomplicated puzzle available to you. The example puzzles I use in this article are taken from BinaryPuzzle.com, which is my preferred website for playing the game in a web browser.
Let us begin by having a closer look at the Binary Puzzle game before we begin implementing the solver in the Python programming language.
The Binary Puzzle game consists of an NxN two-dimensional game grid with some cells pre-filled with either zero or one. The rest of the cells remains empty for us to fill in with either a zero or a one. The difficulty of the game can be tuned by adding or removing pre-filled values in the initial game state.
The rules for the Binary Puzzle game are pretty simple: we must solve the puzzle using the following set of rules:
Each cell must contain either a zero or a one.
No more than two identical numbers are allowed immediately next to each other, both horizontally and vertically.
Each row and each column must contain an equal amount of zeros and ones.
Each row and each column must be unique.
An observation we can make from the third rule is that the smallest possible game grid is 2x2, and each NxN two-dimensional game grid must make use of an even N value. The 2x2 game grid is also the only size of a game grid where the second rule does not have any influence on the game, and the second rule is thus ignorable for this particularly sized game grid.
We begin with an easy 6x6 game grid with 14 pre-filled cells out a total of 36 cells. That is 38.9% of the game grid being pre-filled for us before we have even begun. This example game will hopefully allow us to build up some intuition about the game mechanics, and make it easier for us to understand the rules we need to implement using Python and Z3 later in the article.
The initially pre-filled cells are the only cells that remain immutable throughout the game while we try to discover the value of each of the empty cells in the game grid. The pre-filled cell values are set in bold typeface in all of the visualizations in this article to make sure we do not unintentionally change any of them.
The initial game grid looks as following:
We look for the pattern where two identical numbers exist immediately next to each other either horizontally or vertically in the game grid. Once we have identified one or more identical pairs in the game grid, we know that the cells before and after the pair cannot share the same value as the pair itself because of the second rule of the game. We update the game grid with the new values:
We continue the search for patterns in the updated game grid. We have created some new locations where two identical values are in a pair, which allows us to repeat the previous step.
We can also look for a new pattern, which is when we have a horizontal or vertical triplet, where the content of the first and last cells are identical, and the middle cell is empty. Since we know from the second rule of the game that no more than two identical values are allowed immediately next to each other, we can deduct that the content of the middle cell in the triplet must be the opposite of the first and last value of the triplet. The game grid now looks as follows:
We can now fill in the remaining three cells using a mixture of the second and the third rule of the game.
Now that the game grid is complete, and no empty cells remain, we can verify that the game state satisfies each of the four rules. Each cell contains either a zero or a one. No more than two identical values are next to each other neither horizontally nor vertically. Each row and each column have an identical amount of zeros and ones. Finally, each row and column are unique.
We have solved our first Binary Puzzle manually. We can now begin building a model of the game using Python and Z3.
The purpose of this article is to build a Python program that can solve arbitrary Binary Puzzles for us. We use the Z3 interface for Python to do “the hard labor” of this task, but we still need to describe the game rules to the Z3 solver before it can do anything useful for us.
Before we start defining the Z3 model of the game, we need to define the representation of the game grid in Python. We use the same initial game grid as used in the example game above. In Python, we encode the game grid as follows:
puzzle = [
[1, N, N, 0, N, N],
[N, N, 0, 0, N, 1],
[N, 0, 0, N, N, 1],
[N, N, N, N, N, N],
[0, 0, N, 1, N, N],
[N, 1, N, N, 0, 0],
]
We represent the game grid as a list of lists of integers and N
values in
Python. The N
value is defined as the Python value None
and is used
throughout this article to represent an empty cell. The task of the Z3 solver
will be to eliminate any N
values in the game grid and replace it with either
a zero or a one.
If we were to solve the puzzles without an engine like Z3, but using “pure” Python code, the naive approach would be to define several imperative steps that try to solve the game by eliminating the empty cells one by one.
The way Z3 works is by us adding “constraints” or “assertions” that will make it possible for its built-in solver to solve the domain-specific problem that we are describing using our constraints. In this case, the Binary Puzzle game. Once we have added all of the game rules encoded as constraints to the Z3 solver, we ask it to come with a possible solution for us. Z3 will try to find a solution where all constraints are satisfied or otherwise notify us of its inability to solve the given puzzle.
To implement the Binary Puzzle solver as “bug-free” as possible, we perform
some initial input validation of the input puzzle to ensure that it is
meaningful before we ask Z3 to try to do anything to it. We start by defining a
Python value representing the size of our game grid. We call this variable
size
, and we define it as follows:
size = len(puzzle)
We want to ensure that the input puzzle is non-empty:
if size == 0:
raise InvalidPuzzleError
We want to ensure that the game grid’s size
value is an even number in
accordance with the observation we made while going over the rules of the game:
if size % 2 != 0:
raise InvalidPuzzleError
We want to ensure that the NxN input puzzle has the correct dimensions, and
does not contain rows or columns of a different length than N. We verify this
by ensuring that each row is size
cells wide:
for row in puzzle:
if size != len(row):
raise InvalidPuzzleError
We want to ensure that each cell in the input puzzle contains either a zero,
one, or the None
value since no other values are allowed:
for row in puzzle:
for value in row:
if value not in {0, 1, N}:
raise InvalidPuzzleError
Now that we have validated the input puzzle to avoid the worst mistakes, we can start constructing the Z3 solver for the puzzles.
When we work with a constraint solver such as Z3, we do work with traditional programming concepts such as “variables,” but we do not assign values to them like we would in Python. Instead, we build up a set of equations that makes use of these variables, and then we ask Z3 to give us a result where all of the constraints are satisfied. If our input is impossible to solve because of violations of the game rules, Z3 will be unable to give us a solution, and the problem is considered unsatisfiable. However, if the problem is satisfiable, Z3 will have the correct value for each of our cells in the game grid.
The symbolic variables we define for Z3 has no structure, such as rows and columns. Instead, we later define the structure using the equations we add to the solver.
The first task we have to perform is to build a list of all possible x and y pairs we have in the game grid. We call these our “positions”:
positions = [(x, y) for x in range(size) for y in range(size)]
We can now create the symbolic variables used by Z3. Each symbolic variable
must have a name, which we in Python can represent as a string value. The
string value allows us to later identify the specific variable during debugging
if that becomes necessary. We create a Python dictionary of (x, y)
pairs as
key, and the symbolic Z3 integer as value for each cell in our game grid:
symbols = {(x, y): z3.Int("v{};{}".format(x, y)) for x, y in positions}
We have now defined a symbolic variable for each cell in the 6x6 game grid.
Each symbolic variable can now be looked up in our dictionary of symbols using
its x and y value as the key. We also named the symbolic variables “v0;0
”,
“v0;1
”, …, “v5;4
”, “v5;5
”, respectively. While we still have no
structure for the symbolic variables, we can visualize the symbolic variables
in the game grid in the way they will be used once we have build structure such
as “rows” and “columns”:
We do not have to inform the Z3 solver about the existence of each of the symbolic variables. Instead, the solver will learn about their existence as we use them in our constraints later in the article.
The dictionary of symbols allows us to build two Python lists representing each row and each column in the game grid as lists of symbolic variables. The added structure will make it easier to implement the rules of the game in the next steps. We create the rows and columns lists in Python:
rows = []
columns = []
for x in range(size):
row = [symbols[x, y] for y in range(size)]
rows.append(row)
for y in range(size):
column = [symbols[x, y] for x in range(size)]
columns.append(column)
To avoid unnecessary duplications in our source code, we also create a variable representing both the rows and the columns in the game grid as the rules of the game often apply to both:
rows_and_columns = rows + columns
We can now instantiate the Z3 solver which we will add the constraints of the game to:
solver = z3.Solver()
Since some cells are already pre-filled for us, we need to inform Z3 about the value of these cells. We do this by adding a constraint specifying the exact value of the given symbolic variable using the equality comparison operator in Python:
for x, y in positions:
value = puzzle[x][y]
if value is N:
continue
solver.add(symbols[x, y] == value)
An important detail to understand here is that even if we apply the equality comparison operator here, the Z3 variable overloads this operator. The operator overloading ensures that it is the expression we add to the solver and not the boolean result of Python comparing the symbolic variable with the content of the value variable for equality.
Notice how we explicitly ignore the empty cells in our puzzle since the goal is to have Z3 fill those out for us.
The first set of constraints directly related to the rules of the game will be coming from the first rule: all cells in the game grid must contain either a zero or a one. We add these constraints to all of the symbolic variables in the dictionary of symbols as follows:
for symbol in symbols.values():
solver.add(z3.Or([symbol == 0,
symbol == 1]))
An example of a violation we could make now would be if our input game grid contained a value such as two, which would be a violation of the set of constraints we have added to the solver.
The next constraints we add to the Z3 solver handles the third rule of the game
and ensures that each row and each column have the same amount of zeros and
ones. Instead of counting each zero and one in each row and column, we encode
these constraints as the sum of each row, and each column must be equal to the
size
divided by two:
for values in rows_and_columns:
solver.add(z3.Sum(values) == size // 2)
The constraints needed to check the uniqueness of each row and each column are slightly more complicated but required to implement the fourth rule of the game. For each row and column, we ensure that each other row or column does not contain the same values as the current row or column does. Remember that we pass the Z3 solver symbolic variables such that the Z3 solver will check the actual content of the variables when we execute the model. We implement these constraints in Python as follows:
for lines in [rows, columns]:
solver.add(z3.Not(z3.Or([z3.And([a == b for a, b in zip(line_a, line_b)])
for line_a in lines
for line_b in lines
if line_a != line_b])))
The final set of constraints we need to add to the Z3 solver are only necessary for all NxN game grids where N is greater than 2. These constraints implement the second rule of the game that says no more than two identical numbers are allowed immediately next to each other horizontally and vertically.
We model these constraints using a set of “sliding windows” of three cells in each window of the game grid: each triplet must not contain three identical values in it. We can visualize the sliding window algorithm of three cells as follows:
Implementing the sliding window constraints in Python looks as follows:
if size > 2:
for window in rows_and_columns:
for i in range(size - 2):
a, b, c = window[i:i + 3]
solver.add(z3.Not(z3.And([a == b,
b == c])))
Another approach we could have taken here is to check each window if the sum of the three symbolic variables is equal to 0 or 3. However, using equality checks for these constraints seemed more intuitive to the author at the time of writing this.
We have now implemented all the game rules as mathematical equations for Z3 to
be able to solve the puzzle, but first, we have to check the solver if the
current constraints are “satisfiable”. We use the solver’s check()
method to
achieve this:
if solver.check() != z3.sat:
raise UnsolvablePuzzleError
If the input puzzle contained a violation of some of the constraints, such as
containing two identical rows, then the call to check()
would fail, and we
would raise an exception.
Once we have run check()
successfully, we can fetch the model that Z3 has
created for the puzzle:
model = solver.model()
We can now query the model for the actual value of each of the symbolic variables stored in the dictionary of symbols. We build up a mapping between the cell positions, and the result of the evaluation of the symbolic variable:
result = {position: model.evaluate(symbol) for position, symbol in symbols.items()}
We can now compute the solution of the puzzle, and put it in a data structure equivalent to the input puzzle:
solved_puzzle = [[result[x, y].as_long() for y in range(size)] for x in range(size)]
If we visualize the solution from Z3, it will look as follows:
We have successfully programmed the Z3 solver such that it can solve the 6x6 game grid for us, but we implemented all of the game rules such that they will work for any NxN game grid with an even N value. We have specified the rules of the game as a set of mathematical equations instead of specifying each step Python needs to take to solve the puzzle.
It is much easier to write a validator for whether the game is correctly solved or not than it is to solve the game itself. However, we will skip the details of the validator implementation in this article.
Let us have a look at how the solver handles a more difficult input puzzle. We change the input puzzle to be a 14x14 game grid instead of the example 6x6 game grid. In the new puzzle, only 45 out of 196 cells (23.0%) are pre-filled for us, making this game much harder than the example game where 38.9% of the cells were pre-filled. The new game grid looks as follows:
The Z3 solver can solve this puzzle in around 2.5 seconds on the author’s 2.6 GHz Intel i7 desktop computer from 2016. The result seems to be correct. The solution looks as follows:
An interesting detail that is worth including here is what happens if we ask Z3 to solve an impossible puzzle. With the rules encoded as a set of mathematical equations, we could try to build an input puzzle that passes the initial input validation but would be unsatisfiable.
One of the most trivial puzzles we can construct that is unsatisfiable and passes the input validation is this 2x2 game grid for which no possible solution exists under the rules of the game:
This game grid will be a violation of the third rule of the game whereby each row, and each column, must contain the same number of zeros and ones if we try to solve it by filling in the two empty cells. Additionally, both rows of this game would be identical, which is a violation of the fourth rule of the game. Because of these violations, this puzzle will be unsatisfiable. Passing this puzzle to the solver will make our program throw an “Unsolvable Puzzle Error” exception.
Exploring Z3, together with the Python programming language, has been a fun learning exercise. I could see myself use Z3 to solve various real-world problems that I have historically relied on implementing manual solutions crafted by hand to solve. Changing my mindset from trying to solve the specific problem by hand over to modeling the problem in a declarative way is entertaining and something I wish I could make use of more often in my daily life as a programmer.
If you are interested in learning more about using Z3 together with the Python programming language, I suggest you take a look at the excellent Programming Z3 guide by Nikolaj Bjørner, Leonardo de Moura, Lev Nachmanson, and Christoph Wintersteiger from Microsoft Research.
The source code for the Binary Puzzle solver, we implemented in this article, is available from Github. The source code is published under the BSD 2-Clause license.
]]>We quickly changed the discussion to be about how the IRC clients should be able to verify that the SSL certificate, received from the server, is not a malicious certificate from someone doing MITM attacks. This was not the discussion I had hoped for, but nevertheless, it was an interesting discussion to participate in and made me spend a few days thinking about their concerns.
Sadly, as it is today, some IRC clients, including Irssi, only do full SSL
certificate validation as an opt-in option (via the -ssl_verify
option for
/connect
in Irssi’s case) rather than having it as an opt-out option, which
would be ideal. This is simply because people in the IRC community have
historically not wanted to spend money on certificates from the so called
“trusted” Certificate Authorities like we have seen on the web. Changing this
from opt-in to opt-out is something that I would like to see happen, but it is
not something that is going to be easy. We saw how many web sites got a “proper”
certificate after the Mozilla guys made it slightly harder to actually mark a
self-signed certificate as trusted. This was at first a very annoying move, but
these days we rarely see self signed certificates when we browse around the web.
A few days after the discussion on IRC, I was having dinner at Thomas‘s place and I mentioned the discussion with the Quakenet hackers. Thomas knows a lot about security, privacy and DNS, and he is an avid Quakenet user, so it appeared more than obvious to take the discussion with him and hear what his take to the problem was. His suggestion was to take a look at DNSSEC and DANE and see if that could be used as a possible solution.
Luckily for me, it was exactly what I was looking for.
A few days after the dinner conversation, I pushed a patch to Irssi’s source code repository that enabled support for DANE validation of SSL certificates.
Let’s have a look at how DANE works. This will hopefully give you enough knowledge to understand the basics of what is going on. I will document how to compile Irssi with DANE support enabled and test whether it works or not.
DANE is an acronym for “DNS-Based Authentication of Named Entities” and comes with a protocol named TLSA. DANE is an internet standard and you can read the full technical specification of DANE in RFC6698, but hopefully, this article will give you an introduction to get started using DANE for your IRC servers right away. The concepts are totally protocol agnostic so this will work for other protocols than IRC as well, but it does require modification to the client software to work.
DANE is a simple way of storing information about a certificate in the DNS system. Adding DNSSEC on top of the cake, gives you a very powerful way of validating certificates where the client relies on a trusted source (their ISP’s DNS server and DNSSEC) validating the information from the possibly eavesdropped IRC server.
DANE is implemented as a new DNS resource record named TLSA. You can see an example of such record here from our test IRC server linked to the IRCsource IRC network:
$ dig TLSA _6697._tcp.ircsource.baconsvin.org
; <<>> DiG 9.8.3-P1 <<>> TLSA _6697._tcp.ircsource.baconsvin.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38406
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 9
;; QUESTION SECTION:
;_6697._tcp.ircsource.baconsvin.org. IN TLSA
;; ANSWER SECTION:
_6697._tcp.ircsource.baconsvin.org. 3358 IN TLSA 3 0 1 9B954A014881108A9058DB80020909FFD8B4C44C6F41C8796B3A1EA4 3A444B94
;; AUTHORITY SECTION:
baconsvin.org. 50607 IN NS ns1.gratisdns.dk.
baconsvin.org. 50607 IN NS ns5.gratisdns.dk.
baconsvin.org. 50607 IN NS ns3.gratisdns.dk.
baconsvin.org. 50607 IN NS ns2.gratisdns.dk.
baconsvin.org. 50607 IN NS ns4.gratisdns.dk.
;; ADDITIONAL SECTION:
ns1.gratisdns.dk. 7417 IN A 109.238.48.13
ns1.gratisdns.dk. 36319 IN AAAA 2a02:9d0:3002:1::2
ns2.gratisdns.dk. 25447 IN A 185.10.10.53
ns3.gratisdns.dk. 31182 IN A 194.0.2.6
ns3.gratisdns.dk. 28269 IN AAAA 2001:678:5::6
ns4.gratisdns.dk. 31182 IN A 87.73.3.3
ns4.gratisdns.dk. 28269 IN AAAA 2a01:558:4000::3
ns5.gratisdns.dk. 25447 IN A 85.17.221.46
ns5.gratisdns.dk. 28269 IN AAAA 2001:6f8:3ad::1
;; Query time: 55 msec
;; SERVER: 89.233.43.71#53(89.233.43.71)
;; WHEN: Sat Aug 10 13:16:23 2013
;; MSG SIZE rcvd: 393
Note: If your version of dig
doesn’t recognize the TLSA
type, you can easily
replace it with TYPE52
like this: dig _6697._tcp.ircsource.baconsvin.org TYPE52
.
Notice how the port, 6697, and protocol, TCP, is part of the DNS query. This will be familiar for people who have worked with SRV DNS records.
The interesting part of the output is the answer section where you see the following:
3 0 1 9B954A014881108A9058DB80020909FFD8B4C44C6F41C8796B3A1EA4 3A444B94
What does all of this mean?
Let’s start out by looking at the format. The format for a TLSA reply is as following:
<certificate usage> <selector> <matching type> <certificate association data>
This means that our certificate usage field is 3
, our selector is 0
and our
matching type is 1
. The associated data is the string
"9B954A014881108A9058DB80020909FFD8B4C44C6F41C8796B3A1EA4 3A444B94"
.
It is important to understand the semantics of these fields, because they will dictate how and if the client is going to do further validation of the certificate once the client has received it from the IRC daemon.
Using 3 0 1
means that we are using a self-signed certificate and we will rely
on DANE for validating the certificate only (3); that we are using the full
certificate and not just the SubjectPublicKeyInfo part (0) and we will be using
a hexadecimal encoded SHA256 hash of the DER-encoded certificate (1).
To fully understand the various options available, I suggest you take a look at RFC 6698 section 2.1.
The first step you will have to take is to ensure that whoever runs your DNS servers supports both DNSSEC and TLSA records. In Denmark, a lot of users are using the free DNS hosting provider GratisDNS. GratisDNS supports both DNSSEC and TLSA records which makes setting this up a lot easier.
Sadly, GratisDNS’ interface is currently only available in Danish, so you might have to look for other solutions available online.
Once you have a DNS provider that supports DNSSEC and TLSA records, it is fairly easy to create the records. In our example, the following assumptions are made:
You already have an IRC daemon running with SSL enabled on port 6697 and you have verified that it actually works as expected.
Your certificate is self-signed, so you would like to rely on DANE support only for the validation. This means that the user will not see any self-signed certificate errors when connecting with certificate validation enabled.
We will create a record using a SHA-256 hash of the certificate data. Feel free to use something stronger, if you are more crypto paranoid than I am.
This means that our TLSA record will end up looking something similar to this:
_6697._tcp.irc.example.org TLSA 3 0 1 <SHA-256 hash of the certificate data>
This is basically going to be a description of the exact same setup that I am
using for ircsource.baconsvin.org
.
To find the SHA-256 value of your certificate, start by logging onto the server running the IRC daemon and find the directory that contains your certificate files. We are then going to find the SHA-256 value of the DER representation of our certificate:
$ openssl x509 -in ircsource.baconsvin.org.pem -outform DER | sha256sum
9b954a014881108a9058db80020909ffd8b4c44c6f41c8796b3a1ea43a444b94 -
This is the value we will be using in our final TLSA record, which now looks like the following:
_6697._tcp.irc.example.net TLSA 3 0 1 9b954a014881108a9058db80020909ffd8b4c44c6f41c8796b3a1ea43a444b94
Once you have added this record to your DNS zones, it is now time to actually test whether it works as expected.
This part is tested on FreeBSD 9.2-PRERELEASE. Hopefully, it works for other people as well. Feel free to report any issues you may experience.
Download the dnsval
tarball from its download
page. This is quite new software so
I haven’t run into many distributions that have packages available, so we
will assume that we have to compile it ourselves.
$ mkdir dane
$ cd dane
$ fetch http://www.dnssec-tools.org/download/dnsval-2.0.tar.gz
$ tar zxfv dnsval-2.0.tar.gz
$ cd dnsval-2.0
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
Next we will download the Irssi source code from the Git repository. We
start by cloning the repository into our newly created dane
directory:
$ cd dane
$ git clone git://git.irssi.org/irssi
$ cd irssi
We bootstrap the build system:
$ sh autogen.sh
We configure our test Irssi client:
$ CFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure --enable-dane --with-perl=no
Make sure that somewhere near the end of the output of the configure
script
contains:
Building with DANE support ....... : yes
Otherwise you should take a look at the config.log
file and look for places
where libval
is mentioned and figure out why it doesn’t find the library
correctly.
Compile Irssi:
$ make
Fire up your new Irssi client and give it a spin:
$ ./src/fe-text/irssi -!
Try to connect to our test server, ircsource.baconsvin.org
, using DANE:
/connect -ssl -ssl_verify ircsource.baconsvin.org 6697
If everything was done correctly, Irssi will now connect to the server, verify the signature of the certificate using TLSA and allow you to connect without seeing any self-signed certificate errors.
Here’s a list of IRC servers that supports DANE. If you are running a public IRC server and would like to see the server added here, feel free to drop me an email at ahf@irc6.net with information about the server.
IRCsource is a small network where people with a general interest in IRC hang out together to discuss and test various new concepts and ideas for IRC.
ircsource.baconsvin.org
(SSL ports: 6697 and 9999)I will do my best to maintain this list of servers supporting DANE in the future.
The next step for me is to start securing server-to-server links within the IRC networks with DANE. This will require some modifications to the IRC daemons themselves. I plan on looking into adding support for DANE in a personal feature branch of ircd-ratbox and some of its derivatives.
I am unable to say if DANE support is what the IRC community will be adopting. The IRC community is very conservative in general so time will have to tell.
If you believe you have found a bug in my code or have any troubles setting DANE
up for your own IRC server, I will be more than happy to help. Drop me an email
and I will take a look at it whenever I have time. Otherwise, feel free to poke
me on IRC. My nickname is ahf
and I am available on most of the “larger” IRC
networks (EFnet, Freenode, IRCnet and Quakenet).
All of this code will be available in the upcoming Irssi 0.8.16 release, but if you want to test it right away, my suggestion is to follow my guide from above and use Irssi directly from Git.
Hopefully, we will see other IRC client and server hackers implementing DANE support in the nearby future. If you like what you have read here, please help me making this happen by spreading the word about the possibilities available for enhancing the SSL support in IRC clients as well as other SSL based online services.
This is too easily implementable to be ignored.
I would like to thank Thomas Steen Ramussen for being the originator of the idea and setting up the initial DNS server for testing purpose; Peter Larsen for expeditiously implementing TLSA support for GratisDNS; the IRC6.net guys for late night discussions about DANE; Mickey Fischer for testing the Irssi patches on Gentoo Linux with various options enabled and disabled; the DNSSEC-Tools Project for creating the libraries used and finally the rest of the Irssi team for reviewing the patches and coming with recommendations for my code.
]]>I did a lightning talk about C++0x, where I showed how a standard C++03 application turns out in C++0x. The slides are available in PDF-format here.
Thanks to Ciaran McCreesh for giving some hints about the slides.
Jesper Louis Andersen and I did a 45 minutes talk about the source code management system known as Git. Instead of just giving the usual Git tutorial, we tried to talk about some of the concepts that Git makes use of which most people are not familiar with. These slides are also available in PDF-format here.
The conference itself was a pleasant experience and it was very nice to talk to friends from all over the country. The Exherbo and Irssi booth was crowded with people most of the time, so Bryan Østergaard and I didn’t even have to look after the booth as much as we had feared.
I’m looking forward to the Open Source Days conference in 2010. The next conference that I’ll be attending is FOSDEM at the beginning of February 2010. I’m already looking forward to seeing the other Exherbo and Irssi guys there.
]]>This post is mainly going to be a short summary of random stuff that has happened lately and especially about the upcoming Open Source Community Day conference, where I expect all of the Danish guys who are involved with FOSS to be present. Especially remember to pop by the Exherbo / Irssi booth and say hello; we won’t bite.
Firstly, all of my suggestions for talks have been accepted. Julia Lawall from DIKU is going to do a talk about bug hunting with Coccinelle. Bryan is going to do a talk about open source project management the right way (and I know it’s the right way having worked with Bryan on a couple of projects). And finally Jesper and I are going to do a talk about Git, the SCM-system that you are supposed to love and if you don’t, show up at our talk. I was also asked to do a lighting talk about the upcoming C++ standard (known as C++0x), which I accepted so it’s already going to be a much more stressful conference than the previous one was for me. But I’m still looking forward to seeing friends from near and far and discussing open source with them.
Secondly, I’m considering writing a couple of posts about C++0x in the near future since I clearly won’t be able to mention all of the interesting new features in a 15 minutes talk. My talk is mainly going to focus on the C++0x features that have already been implemented in GCC.
]]>Twelf ships with a Vim mode, but unfortunately they forgot to take account of different colour schemes and it only appears to be working well with the default Vim colour scheme. I decided to rewrite the syntax highlighting files for Twelf, so that they work with the colour scheme of your choice.
Releases are available for download from here, or you can grab the code directly from Github using:
$ git clone https://github.com/ahf/twelf-syntax.git
Patches encouraged! Check out the repository on Github.
]]>Peter Toft also suggested that people should let the coordination team know which talks they would like to hear about. Here are my suggestions:
Julia Lawall from DIKU about Coccinelle. Julia did perhaps the most interesting talk at Open Source Days 2008 about her research on semantic patches and how they are used. I would like to hear what has happened since then, especially regarding the focus on bug hunting using Coccinelle.
Bryan Østergaard about open source project management. Bryan is the leader of the Exherbo project and used to work on Gentoo before finally starting his own project. Bryan has written an interesting blogpost about how to manage an open source project the right way and getting more contributors and more useful contributions. It would be interesting to hear more about this as a talk at the conference.
Jesper Louis Andersen and I are working on a Git tutorial spread out over three workshops and we are planning to present them during the winter at SSLUG‘s location in Frederiksberg. We could do a one hour talk at Open Source Days, and I am sure a lot of people would find it interesting to hear.
A lightning talk about communities on IRC. This is not super important, but if someone found it interesting I would be more than willing to do it.
Looking forward to see you at the conference. Remember to follow the Zebrapig!
]]>Rest in Peace Ferris.
]]>Twelf is very different from any other programming language that I have ever played with. Twelf is an implementation of the logical framework – called LF – and it is used for logical programming and to formalise programming language theory.
From Twelf’s website:
Twelf is a language used to specify, implement, and prove properties of deductive systems such as programming languages and logics. Large research projects using Twelf include the TALT typed assembly language, a foundational proof-carrying-code system, and a type safety proof for Standard ML.
Twelf’s website is very useful and is, honestly, the only website available on the internet with a plethora of information about the language. You can find some decent tutorials if you look at some of the university websites out there (CMU especially has some very interesting slides about Twelf on their website).
I am going to give a short example on how to implement the factorial function in Twelf and hopefully this will enable you to understand Twelf a bit better than before you read this post.
There are tons of small examples on Twelf’s website, but here is another one. We are going to implement the factorial function in Twelf. The factorial function is, for functional programming languages, what “Hello World” is for imperative programming languages – the first function you will see when you open a book about the language.
The factorial function can be defined recursively as this:
$$ n! = \begin{cases} 1 & \text{if } n = 0 \cr n (n - 1)! & \text{if } n > 0 \end{cases} \qquad \forall n \in \mathbb{N} $$
That is easy to implement in most programming languages. In a functional programming language, like Haskell, it can be implemented with pattern-matching so:
factorial :: Int -> Int
factorial 0 = 1
factorial n | n > 0 = n * factorial (n - 1)
In particular, notice how much the pattern-matching method looks like the formula listed above.
It is nowhere near as easy to implement this in Twelf. The first issues we will encounter are:
Twelf has no knowledge of numbers. We really take that for granted in any modern programming language (for a very good reason: adding numbers to a language every time you have to write anything feels a bit like reinventing the wheel over and over again);
Twelf has no arithmetic operators at all. We need to define all of those and the numbers via the type-system of Twelf.
The first thing we need to define is the natural numbers ($\mathbb{N} = {0,1,2,\ldots}$). We are going to do that with Peano axioms which map well into the Twelf type-system. First we need to define a type for the natural numbers, and then two relations for it.
In Twelf we write:
nat : type.
nat/zero : nat.
nat/succ : nat -> nat.
This says: nat
is a type, nat/zero
is of the type nat
and is going to be the type that
represents the natural number zero, nat/succ
is the successor of the former natural number. This
way we have zero as nat/zero
, one as nat/succ nat/zero
, two as nat/succ (nat/succ nat/zero)
and so on. It is already beginning to get a bit disturbing, right?
We also need to be able to perform some basic arithmetic on the Peano numbers. We have to implement both addition and multiplication (since the latter requires the former) to be able to finally implement factorial in Twelf.
Addition on Peano numbers is defined recursively as:
$$ \eqalign{ a + 0 &= a \cr a + Succ(b) &= Succ(a + b) } $$
We need to convert this from the infix-notation above to something that fits into Twelf (Twelf is
able to do infix-operators via the %infix
-statement). Putting it together in Twelf will result in
this:
plus : nat -> nat -> nat -> type.
plus/zero : plus nat/zero B B.
plus/succ : plus (nat/succ A) B (nat/succ C) <- plus A B C.
Multiplication is defined using addition and this is also done recursively, like so:
$$ \eqalign{ a \cdot 0 &= 0 \cr a \cdot Succ(b) &= a + (a \cdot b) } $$
And in Twelf:
mult : nat -> nat -> nat -> type.
mult/zero : mult nat/zero B nat/zero.
mult/succ : mult (nat/succ A) B C' <- plus B C C' <- mult A B C.
The only operation that we are missing is to be able to do $a - 1$, which can easily be defined with:
pred : nat -> nat -> type.
pred/s : pred (nat/succ A) A.
This works for what we are going to use, but it won’t work for someone who decides to do a $0 - 1$, since our system does not have any knowledge of negative numbers. However, we are never going to reach this in our factorial function (since we have a case for $n = 0$) and we are simply going to ignore this issue for now.
The final relation that we have to define in Twelf is our factorial-relation, which can be defined in the form of the previous defined relations thusly:
factorial : nat -> nat -> type.
factorial/z : factorial nat/zero (nat/succ nat/zero).
factorial/s : factorial N R <- pred N N' <- factorial N' R' <- mult N R' R.
Twelf has a very nice declaration called %query
where we can query (Prolog-style) a relation for a
value. In most programming languages, we are used to functions being one-way. For instance, a
function takes an argument and returns a value. With Twelf you can do it the other way around and
ask for the input value for a given output value.
To query Twelf for the value of X in the equation $2 + 1 = X$, we write:
%query 1 1 plus (nat/succ (nat/succ nat/zero)) (nat/succ nat/zero) X.
And Twelf will reply with the answer:
---------- Solution 1 ----------
X = nat/succ (nat/succ (nat/succ nat/zero)).
One thing I discovered pretty quickly after I started playing around with Twelf, is that it becomes really tedious to work with Peano axioms like this. To make it a bit less annoying you can do the following:
0 : nat = nat/zero.
1 : nat = nat/succ 0.
2 : nat = nat/succ 1.
3 : nat = nat/succ 2.
And so on. This makes it possible to use “normal” numbers in your queries, but the answer will still be in the annoying format.
Getting the result of the factorial relation can be done in the same manner. With factorial we have two interesting cases. One where the input is zero and one where the input is larger than zero.
The output from Twelf with $n = 0$ and $n = 3$:
%query 1 1 factorial 0 X.
---------- Solution 1 ----------
X = nat/succ nat/zero.
%query 1 1 factorial 3 X.
---------- Solution 1 ----------
X = nat/succ (nat/succ (nat/succ (nat/succ (nat/succ (nat/succ nat/zero))))).
Both results are correct, but you will soon begin to really dislike the output-format – especially if you start trying with an $n$ value larger than $4$.
Twelf is fun to play with, but I do not have much use for it other than to be able to read the code. It is worth playing with if you want to try “a completely different programming language”.
We are going to use it in the Ex-SML project (a Moscow ML fork using LLVM as backend) to prove that the internal Lambda language is actually type-safe. Currently Moscow ML converts SML into the (currently) untyped Lambda language before it is turned into bytecode that runs on a virtual machine named Zinc. Our problem is that LLVM’s assembly language is typed, so we actually need to keep the type-information during every phase of the compilation.
]]>One of the important requirements for the blog engine is that I should be able to create, edit and remove posts via my favourite editor (Vim) and my favourite SCM (Git), and Jekyll is doing a very good job at this so far.
It was fairly trivial to setup and I was being lazy and decided to just use the style sheets from my personal website, so the only things left to do was to create a couple of templates and wait a couple of hours for Github to do the initial page generation.
Another very nice feature of Jekyll is that it supports code highlighting via the Python library named Pygments. I tried to get Ikiwiki to do something similar, but after messing around for 30 minutes I decided that it was a waste of time and gave up.
Jekyll should also be able convert inline LaTeX commands into graphics in your posts, but I have no idea if this feature is enabled on Github or not.
Thanks to Ali for the hint about Jekyll. Hopefully, I wont dislike this as much as I disliked Ikiwiki.
]]>New Features:
/reset
an alias for /set -default
./unset
an alias for /set -clear
.WHOX
reply (354 numeric) as a /who
reply.Bugfixes:
event_wallops()
.autolog_ignore_targets
logic to work correctly with manually opened log files.More information on irssi.org.
]]>Check out irssi.org for more information and remember
to read the NEWS
and ChangeLog
files.
New Features:
/set
.Window::get_history_lines()
TextBuffer
and TextBufferView
and low level api to add/remove lines, scripts should be fine using Window::print_after
and TextBufferView::remove_line
.print_after
method to Window perl object analogous to gui_printtext_after
but which also expands formats and forces a full line.Irssi::signal_register
./BAN
, /UNBAN
, /KICKBAN
, /KNOCKOUT
if channel is not synced. Requesting ban lists from an unsynced channel will ask them from the server, banning a user whose u@h irssi does not know will ban nick!*@*
and only bans irssi knows about can be removed./set autolog_ignore_targets
for cherry-picking targets that shouldn’t get logged.%FMT<string>
to %fmt%_<string>%_
, it will work fine in all irssi versions.scrollback_levelclear_levels
setting and add a ‘level’ option to sb levelclear
to specify a comma separated list of levels.__WARN__
handler for scripts (bug #427).Irssi::command_parse_options
function to parse options for a command./WINDOW SERVER
.TARGMAX
/MAXTARGETS
005 tokens.RPL_WHOISACTUALLY
(338 numeric) for both ratbox and ircu (bug #428).-idle
option of /notify
is gone./layout
save now makes window-channel bindings instantly effective (bug #35)./ping
without arguments does not send anymore a CTCP ping to a channel (bug #542).actlist_names
option to add active items names in ‘act’ statusbar item.word_completion_backward
command to scroll backwards in the completion list./bind
to print all the available commands.+I
lists-usermode
before -autosendcmd
(bug #548)./sb status
more accurate (higher).Irssi::Irc::Client
and signals to communicate with proxy clients to allow for scripting parts of the irssi-proxy.sb_search.pl
, a script for /SCROLLBACK SEARCH
Bugfixes:
/NOTIFY
list when nick is seen joining (bug #642)./JOIN -window
(bug #644).PERL_SYS_INIT3
)./script load
to reload it. (bug #525, patch by Lukas Mai)NETSPLIT_SERVER_REC
in signals for Perl./SCROLLBACK redraw
and /SET scrollback_save_formats
.$L
expando./script reset
.-channels
preference for ignore -replies
(bug #227)./set hilight_level
not taking effect immediately (bug #598)./LASTLOG
and buf.pl
.A thing that many users has been looking for is the ability to select various
targets (either channels or nicknames) that shouldn’t be logged via Irssi’s
autolog feature. This used to be impossible, but as of revision 5006 it’s
possible to use the autolog_ignore_targets
-setting.
If you want to disable logging for all #irssi
channels on every network to
which you are connected to:
/set autolog_ignore_targets #irssi
If you want to disable logging for only the #irssi
channel on EFNet:
/set autolog_ignore_targets EFNet/#irssi
Of course you can add multiple targets:
/set autolog_ignore_targets EFNet/#irssi Freenode/#exherbo ahf
]]>Highly inspired by Ciaran‘s Vim theme, Inkpot, I decided to create a theme for Mutt.
For rxvt-unicode users:
color attachment color30 color80
color header color10 color80 "^(From|Subject|cc|date|To|X-Spam-Level|User-Agent|X-Mailer):"
color signature color39 color80
color tree color26 color80
color message color26 color80
color status color85 color81
color normal color78 color80
color error color79 color64
color indicator color80 color73
color markers color26 color80
color index color64 color80 ~D
color index color64 color80 ~F
color index color30 color80 ~T
color tilde color80 color80
color body color10 color80 "(http|https|ftp|news|telnet|finger|irc)://[^ \">\t\r\n]*"
color body color10 color80 "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
color body color26 color80 "[;:=][-][)/(|]"
color quoted color52 color80
color quoted1 color22 color80
color quoted2 color71 color80
For xterm-256color users (submitted by Henrik Stuart):
color attachment color61 color232
color header color10 color232 "^(From|Subject|cc|date|To|X-Spam-Level|User-Agent|X-Mailer):"
color signature color63 color232
color tree color63 color232
color message color37 color232
color status color247 color235
color normal color229 color232
color error color231 color196
color indicator color232 color215
color markers color37 color232
color index color196 color232 ~D
color index color196 color232 ~F
color index color49 color232 ~T
color tilde color63 color232
color body color10 color232 "(http|https|ftp|news|telnet|finger|irc)://[^ \">\t\r\n]*"
color body color10 color232 "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
color body color37 color232 "[;:=][-][)/(|]"
color quoted color130 color232
color quoted1 color25 color232
color quoted2 color207 color232
]]>On my workstation and my IA64 server, not having framebuffer support together with Xen is tolerable, given that I never use anything but X on my workstation and I don’t have any console attached to my IA64 server; it’s remote SSH access only.
My laptop is a completely different story though. I mainly use my laptop when I’m out and I tend to not start X because I don’t have much use of it, so it would be really nice to have working framebuffer there.
Anyway, I’m not a fan of long posts. Here is my solution to get a working framebuffer with Xen:
title Gentoo Linux
root (hd0,0)
kernel /xen.gz vga=gfx-1024x768x32
module /kernel-2.6.21-dom0-selinux root=/dev/sda2 vga=0x317 selinux=1
Notice the vga
statement on the kernel
-line.
The vga
statement on the module
-line is probably redundant, but I haven’t
removed it yet to verify.