Sam Lewis

Investigating why reorgs happen in proof of stake Ethereum

2022-03-14T00:00:00+10:00

As part of my foray into learning about Ethereum, I started looking into the relatively new Proof of Stake supported Beacon Chain. The Beacon Chain is the main coordinating blockchain for Proof of Stake Ethereum and is currently running with the equivalent (in ETH) of ~$25 billion USD staked!

In doing so, I was surprised to see that there were regular reorganisations (or "reorgs") occurring on the Beacon Chain, so I decided to investigate to help learn more about the machinations of Ethereum and Proof of Stake. This article is a (slightly meandering) write up of my investigation into why reorgs happen somewhat regularly on the Beacon Chain. It also details some of my learnings along the way about ETH2 in general. If any of that is interest, stick around!

What Is a Reorg? Reorgs in a Proof of Work blockchain

As a brief bit of background, in a Proof of Work block chain, such as Bitcoin or the current Ethereum main chain, reorgs occur most commonly because miners are effectively racing each other to mine the next block. When two (or more) miners mine a block at similar times the chain forks:

These forks are resolved by the fork choice rule that instructs clients to follow the fork with the most mining difficulty behind it (may also be known as the "longest chain" rule). The idea being that eventually a single fork will "win". This is a very simple way of forming consensus but has been shown to work well in Bitcoin, Ethereum and many other Proof of Work blockchains.

In this way, the blocks on the "losing" fork are referred to as "reorg"ed blocks as anyone initially following this fork needs to reorganise their chain after they learn that the other fork has "won".

On the Proof of Work Ethereum blockchain reorgs happen quite regularly. You can see a list of these reorgs on the etherscan block explorer but, at a quick glance, they appear to be happening at a rate of around 5 per hour. The rate of reorgs is a function of the block time, which is why they occur more frequently on Ethereum than Bitcoin.

The reorg depth is how many blocks deep the fork was before the fork was resolved. Just about all the time it's 1 here, which likely occurs similar the scenario above but where one of the forks is chosen in the next block and all miners stick to that fork from then on.

If you're interested in learning more about reorgs in a proof of work blockchain, this article is informative.

Proof of Stake Reorgs

Proof of Stake works differently than Proof of Work in that in that validators (who are network participants that have staked ETH) are elected to create blocks, rather than needing to race to create a block. In Ethereum, a particular validator is chosen randomly to create a single block every 12 seconds.

Validators are rewarded with ETH for successfully creating blocks, so it is very much in the interest of a validator to have their proposed blocks included in the chain. The current reward for proposing a block that does get included is around 0.025 ETH at the time of writing (~$70 USD). There's a block proposed every 12 seconds and with currently around 308,000 validators, each validator will get to propose a block (on average, because it's randomised) every 42 days or so. So if you mess up and miss your slot it can be a while until you get another chance!

One reason a fork might occur is if a validator were to create and publish more than a single block when it was their turn to create a block. Validators are incentivized not to do this though, by the network "slashing" their stake if it can be proven that two blocks were created by the same validator for the same period. Looking at the list of slashings we can see that slashing occur much less often than reorgs (monthly instead of daily), so this is probably not usually the reason for the reorgs.

So then, assuming that proposers aren’t proposing multiple blocks and are attempting to act honestly, why are there still reorgs occurring?

A Super High Level Ethereum Proof Of Stake Overview

There's lots of better resources on eth2, but for the purposes of this article you should know that:

Ethereum validators are people that have "staked" a large amount of ETH to be able to participate in the validating/proposing/attesting process. To actually perform these processes, validators run software such as Lighthouse, Prysm or Teku.
Ethereum has the concept of "epochs" and "slots". Every epoch there are 32 slots and in each slot, a single validator is randomly chosen to create a block. Validators fill this block with transactions, and share it with the network.
Validators are also assigned to "attest" to one slot each epoch. As part of this attestation, the validator will select what it believes is the head of the chain. These attestations help the network form a consensus on what the state of the chain is.

Investigation

To investigate, I ran my own Lighthouse beacon node and used the standardized ETH2 HTTP API to listen to new blocks and attestations being announced, with the plan to collect enough data to be able to understand more about why these reorgs were occuring.

Over the ETH2 API, the block data is of the form:

{
  # the slot of the block
  "slot": "3332233", 
  # the hash of the block
  "block": "0x1dd45a53936a86a167e5e676dc52fef732290f45cd1ed861137ea4d7eba43646",
}

Whereas attestations are of the form:

{
  # an aggregation of other validators who are attesting the same thing in the slot
  "aggregation_bits": "0x00000000000000000000000400000000000080",
  "data": {
    # the slot of this attestation
    "slot": "3338073",
    # the committee this validator is attesting as part of
    "index": "9", 
    # the attester's vote for the head of the chain
    "beacon_block_root": "0x576513f60421f8aa0a4fa05bba137305cbe100cffac6afcf7c7822364b13fbff", 
    # (snipped some data around voting for "checkpoints")
  },
  # the signature of the attester - to prove that they really submitted this attestation
  "signature": "0xa62ba00daa2096ed24242d358f775b729fa291c6f08dafd26e663cb83b8e7884c432850bc769e436250fb60e072cc43f01d64784aaee935b4bb395d6f9b16ddb26fce05f1bf4043bb3989f6336f3f18ad72b98a79684abee4b73b4cfc7c185bd",
}

For around 20 hours I ran a simple script that basically collected all the blocks and attestations published to the network along with their arrival time to my computer.

messages = SSEClient('http://localhost:5052/eth/v1/events?topics=block&attestation')

blocks_log = open('blocks.json', 'a')
attestations_log = open("attestations.json", 'a')

for msg in messages:
    message = json.loads(msg.data)
    message['arrival'] = round(time.time() * 1000)

    if msg.event == 'block':
        blocks_log.write(json.dumps(message) + "\\n")
    elif msg.event == 'attestation':
        attestations_log.write(json.dumps(message) + "\\n")

This collected 5912 blocks and 3418459 attestations.

Finding Reorgs

With the logged block messages, we can find blocks that are not part of the main chain with a simple algorithm that starts at the most recently collected block and works backwards, using each blocks parent. Any blocks that don’t get visited will be the reorged blocks. Because we're not doing this live, we can do this while being sure that our most recently collected block isn't t a reorged block itself (by manually checking this). In the live system, beacon chain nodes don't get this luxury as they can't see into the future!

def get_block_info(block_hash: str, bn_api_url: str) -> dict:
    resp = requests.get(f"{bn_api_url}/eth/v1/beacon/headers/{block_hash}")
    resp.raise_for_status()
    return resp.json()

def find_reorged_blocks(blocks: list, bn_api_url: str):
    """
    Finds blocks that aren't part of the main chain.

    blocks: list of block objects, each with a 'slot' and a 'block' field
    bn_api_url: url of the beacon node api
    """
    block_map = {block["block"] : False for block in blocks}

    # Assume the most recent block we have is the head and is not a reorged block.
    last_block_on_chain, last_slot = blocks[-1]["block"], blocks[-1]["slot"]

    # The list of blocks might have some missing blocks (because of
    # interruptions in the data collection process) so we iterate until we're at
    # the last slot we know about
    min_slot = int(blocks[0]["slot"])
    while int(last_slot) >= min_slot:
        # mark this block as visited, it's on the chain!
        block_map[last_block_on_chain] = True

        block_info = get_block_info(last_block_on_chain, bn_api_url)
        parent = block_info["data"]["header"]["message"]["parent_root"]
        parent_slot = block_info["data"]["header"]["message"]["slot"]

        last_block_on_chain, last_slot = parent, parent_slot

    return [block for block in block_map if not block_map[block]]

Running this against my list of 5615 collected blocks I got the following blocks as blocks not on the main chain:

Slot	Hash	Notes
3345326	0xfc54ca25f7df62e9f3b24d8fb9fa6b21ec5bc2d312107ce29a48ba2d6b7188c3
3344246	0xa3b0a48f6cbd0a7ca7e6337e4b939a146e2e91fa7f275ebf75dcaf0630b95ee6
3344244	0x5862cb53788db14b989df390199d69aaebd694894183e152ccfae9bf0df283f6	Marked as skipped beaconscan
3343277	0xe2c00aabfc8d584d122518d32e58d8f317f6326205df9ed8b488486a971eef52
3340911	0x6a1adc86f899d66ac8b0cd2848b03112d5a25da3f9feb3e3a320585c4535c683	I never received this block but it's marked as reorged on beaconscan
3342234	0x5d97817910ad60452dba75b3cf92890324194ec65bb2c2a0ac484f970b3280ca	Marked as skipped on beaconscan
3340347	0x977491737cb0f60b6f5c1a21d5745162f9a649bc031f467a591a703613927e02
3340161	0x4cc45ee812f161b6eb478e292b9cc19f05f7e954d6d158dd5a503acf734448ee

Checking this against BeaconScan the list looks pretty good, with the exception of slot 3340911 which looks like it wasn't collected by my event listener. There were also a few slots that BeaconScan marked as skipped but I saw blocks for. These discrepancies are interesting, and potentially give a clues towards why these blocks were not included.

Finding information about reorged blocks was actually kind of tricky because beacon chain node implementations seem to prune reorged blocks out of their block storage. This make sense to do as once you're certain that a block isn't part of the main chain it is isn't useful to keep around. However, it does make this sort of investigation harder! For finding parents of these reorged blocks, I relied on block explorers like BeaconChain to find the data.

Slot 3340161

Starting the investigation by first examining the oldest reorged block: the block proposed at slot 3340161. This block has a parent of the block proposed at slot 3340160, however the block at 3340162 also has the same parent. From inspecting how this fork was resolved by looking at the state now, we can see that the end result of this fork was something like this, with the block at slot 3340161 being reorged:

My suspicion here is that 3340161 might have be sent "late" and wasn't received by the proposer of 3340162 in time. Looking at my log of blocks and their arrival time lends some credence to this theory:

{"slot": "3340158", "arrival": 1646905919766}
{"slot": "3340159", "arrival": 1646905932380}
{"slot": "3340160", "arrival": 1646905946542}
{"slot": "3340161", "arrival": 1646905964624}
{"slot": "3340162", "arrival": 1646905968938}
{"slot": "3340163", "arrival": 1646905981605}
{"slot": "3340164", "arrival": 1646905991401}

Which can be summarized as (reorged slot is bolded):

Slot	Time from last slot (ms)
3340159	12614
3340160	14162
3340161	18082
3340162	4314
3340163	12667
3340164	9796

Blocks are supposed to be produced every 12 seconds, so I think you'd expect the delay between blocks to be around 12 seconds. Indeed, looking at a histogram of the time between received blocks, we see a big spike around the 12s mark.

This is, of course, only showing my particular view of the network and because of the distributed nature of the system, other nodes might be seeing very different message arrival times. Still, it's all I have to work with so I'll use my view of the network as an approximation of what other nodes would have likely seen as well.

Given that, it does make some sense that the proposer of 3340162 has built their block on top of 3340160 instead of 3340161. A validator knows it has to meet slot deadlines (remembering a new block needs to be produced every 12 seconds), so will start building it on top of the last block it has received and verified. In this case, it seems like this block was 3340160.

Once we get to the next slot, 3340163, the validator chosen to produce a block a choice to make. The validator has had enough time to receive both 3340161 and 3340162 and needs to pick which block to build this block on top of. So why did it choose to build its block on top of 3340162 and not 3340161?

Looking at the ethereum 2 beacon chain specification (that is written in Python! Executable specifications are pretty cool) for the fork_choice get_head function gives a clue:

def get_head(store: Store) -> Root:
    # Get filtered block tree that only includes viable branches
    blocks = get_filtered_block_tree(store)
    # Execute the LMD-GHOST fork choice
    head = store.justified_checkpoint.root
    while True:
        children = [
            root for root in blocks.keys()
            if blocks[root].parent_root == head
        ]
        if len(children) == 0:
            return head
        # Sort by latest attesting balance with ties broken lexicographically
        # Ties broken by favoring block with lexicographically higher root
        head = max(children, key=lambda root: (get_latest_attesting_balance(store, root), root))

This code, as I understand it, will walk the chain until it comes to a fork. When it comes to a fork (like we have in the scenario above) it will choose the block that has the most attestation stake behind it.

Luckily we also collected attestations so we can try to verify this! But first some background on the attestation data that we collected: to decrease the network and processing load of thousands of individual attestations flying around the network, validators are divided up into committees to perform the attestations. Validators communicate their attestations to other validators within their committee, and select validators are selected to aggregate these attestations globally. These aggregations are stored in the aggregation_bits field of the aggregations messaged we received, so to find the total support for a particular head there's a little extra processing needed:

def calculate_head_attestation_votes(attestations):
    """
    Takes an array of attestation objects (as returned by the Beacon API) for a
    particular slot and returns a dictionary of head : number of votes
    """
    # Mapping of: head -> committee index -> votes
    attestation_aggregations_by_committee = defaultdict(lambda: defaultdict(int))

    # Multiple attestations might have the same aggregation bits set, so we need
    # to OR them together to avoid double counting
    for attestation in attestations:
        aggregate = int(attestation["aggregation_bits"], 16)
        # This is what this attestation is voting as the head of the chain
        head = attestation["data"]["beacon_block_root"]
        # The particular committee this aggregation is part of
        index = attestation["data"]["index"]
        attestation_aggregations_by_committee[head][index] |= aggregate

    attestation_head_votes = defaultdict(int)

    # sum the aggregations to get the total number of votes for each head each
    # bit in the aggregate represents a particular validator's vote
    for head, aggregations in attestation_aggregations_by_committee.items():
        for _, aggregate in aggregations.items():
            attestation_head_votes[head] += aggregate.bit_count()

    return attestation_head_votes

This code basically sums up the aggregations we've received so we can find the total support for each particular head. (I manually filtered out aggregations for individual slots with a little jq scripting: jq -c 'select(.data.slot=="<slot>")' < attestations.json > attestations_for_slot.json).

beacon_block_root here is what the validator is voting as the head of the chain at that point of time, so running this script to see how validators were attesting for the slots of interest.

First, looking at the attestations for the 3340161 slot:

Slot `beacon_block_root` vote	Count
3340160	5430

And then looking at the attestations for the 3340162 slot:

Slot `beacon_block_root` vote	Count
3340162	6384
3340160	36
3340161	7
3340122	2
3340156	2

These two tables are really quite informative because they show:

In the slot that the 3340161 block was produced, there was not a single attestation for that block. All attestations were for the block in the previous slot.
In the next slot, there are attestations for the 3340161 block, but there are a lot more for the block that was produced in the same slot.

Again the spec is illustrative in making sense of what might be going on here:

A validator should create and broadcast the attestation to the associated attestation subnet when either (a) the validator has received a valid block from the expected block proposer for the assigned slot or (b) 1 / INTERVALS_PER_SLOT of the slot has transpired (SECONDS_PER_SLOT / INTERVALS_PER_SLOT seconds after the start of slot) -- whichever comes first.

So, we can probably reasonably guess that:

In slot 3340161 it's likely no validators had received the 3340161 block, so all attested that the head of the chain was the 3340160 block. We can plot the arrival time of the block compared to the arrival time of the attestations for this slot, relative to the start of the slot to make extra sure (again with the caveat that this only represents my view of the distributed network, other participants likely saw a slightly different view):

The dashed line here represents when we received the block for this slot, whereas the red bars show a distribution of when we received attestations for this slot. INTERVALS_PER_SLOT is currently 3 so validators should wait only 4 seconds (12/3) before publishing an attestation if they haven't received the block from the proposer. There are, however, a considerable attestations received after a longer amount of time, which might be because of delays in me receiving the attestations or perhaps could mean that the validators aren't perfectly following the spec. It is curious that there's a big spike of attestations arrived after the (quite block) arrives but given that no validators seemed to attest to the late block, it seems likely that it might just be because of network delays.

In slot 3340162 we have a few different attestations for the head:

Attestations for 3340162: these are probably validators that received this block before SECONDS_PER_SLOT / INTERVALS_PER_SLOT . These attestations are essentially picking this block over the 3340161 block. At this stage, to these proposers, we can infer that both the 3340161 and 3340162 blocks have no attestations, as the fork choice rule says to pick the lexicographically higher root.
Attestations for 3340160: these are probably validators that didn't yet receive the 3340162 or 3340161 block so are voting for the last block they know about.
Attestations for 3340161: I think these are validators that are, at this point in time, following the "3340161 fork". I think this might be because they've received this block and haven't received 3340162 or perhaps because they have received attestations for 3340161.
Other attestations: Not really sure why these validators are voting for such old blocks as the head. Maybe they are experiencing network or processing issues and are really far behind?

Some other thoughts:

I think that the 3340161 block being late is likely the reason that the attestations aren't split at all for that block - with that block being late, all validators effectively have an extra slot time to receive the proceeding block so can all agree it is the up to date head. There's also probably an element of luck involved, the next block has some fairly "wrong" votes in it.
With ~312k current validators, there should be about 312k/32 = ~9700 attestations per slot. I'm not totally sure why I get a bit less than this amount, but I suppose it could be because my node couldn't keep up with the amount of attestations or perhaps it could mean that some validators were missed in the aggregated data.
It's a little curious that we see 7 validators following the "3340161 fork". I suspect (but really have no way of checking) that these validators received an attestations for this block before they made their attestations. Maybe this is because some small set of the network was able to receive this block without much delay.

Overall it's pretty neat that the protocol is able to resolve and move past the the problematic late block, essentially by skipping over it!

Slot 3340347

Lets then look at the next forked block, which was the block produced in slot 3340347. This fork looks slightly different to the last fork, in that the later block is the reorged block, not the earlier block:

Looking first at block arrival times:

Slot	Time from last slot (ms)
3340345	13304
3340346	22950
3340347	557
3340348	11481
3340349	13332

This is different to the previous case in that this block appears to have not actually been that slow to arrive, relative to the last block. The previous block was slow to arrive though!

Looking again at the attestations for the slots:

The 3340345 slot:

Slot `beacon_block_root` vote	Count
3340345	9667
3340344	2

The 3340346 slot:

Slot `beacon_block_root` vote	Count
3340345	7772

The 3340347 slot:

Slot `beacon_block_root` vote	Count
3340346	6063
3340345	4

The 3340348 slot:

Slot `beacon_block_root` vote	Count
3340348	4333
3340346	4

So the reconstructed timeline is:

3340345 is created on schedule and the vast majority of validators attest to this block
3340346 is created with some delays, no validators receive it in this block so all attest that 3340345 is the current head
3340347 is created, building off of 3340345, likely because the proposer didn't receive 3340346 in time. No attesters attest to 3340347 and instead all attest to 3340346.
3340348 builds off the head of the chain, which in this case is 3340346 as it has attestations whereas 3340347 does not.

This is a little weird to me, I don't completely understand why at the 3340347 slot there was not a single validator who voted for the block produced at 3340347. I initially thought that this might be due to the block arriving later than it should, so plotted the arrival time of the block relative to the attestations:

This time the block is not later than the 4 second SECONDS_PER_SLOT / INTERVALS_PER_SLOT deadline, and it looks like the validators have waited for the block before sending out attestations.

Unfortunately, I don't have a good handle on why the network didn't choose the 3340347 block when choosing the fork to follow. Reading through the specification, the fork choice appears to dictate that this block should be chosen and even incentived to be chosen, through the proposer_boost_root mechanism that 'boosts' blocks that arrive on time.

In my understanding, the only reasons for this block to not be chosen are either:

My view of the network was significantly different to what the rest of the network saw. Perhaps the 3340347 block was delayed for all other participants.
The 3340347 block did not pass the filter_block_tree check. I believe this could have been because it had a different finalized or justified checkpoint to what the rest of the network agrees upon.

Unfortunately, with the data I have, I'm not quite sure how to definitively prove or discover which of the two options caused this block to not included in the chain. The contents of this block is (as far as I can find) no longer available, due to beacon nodes pruning it from their history.

The rest of the reorged blocks in my log seem to have very similar symptoms to this block, in that:

The reorged block follows a delayed block
The reorged block creates fork to be created, most likely because they didn't receive the preceeding block in time
The network follows the earlier block instead of the later block

I think there most likely must be a possibility that I'm missing here as I don't think it's coincidence that these reorgs seem to happen under the same circumstances. It seems unlikely that each of these blocks were skipped by filter_block_tree or that my view of the network was considerably different to the rest of the network each time.

Conclusion

I learnt a lot about how attestation works on the beacon chain while investigating this, despite not completely solving the mystery of why exactly there are some reorged blocks.

To continue this investigation, maybe it would make sense to:

Have more collectors logging data, possibly on compute instances geographically distributed. This should allow me to get a better sense of what the rest of the network sees.
Log the contents of blocks, to better discover why nodes skip over them.

If anyone has any theories on what might've happened in the second reorged block (and the other, similar, reorged blocks), please let me know!

Update

Since posting this article the reason for this behavior was revealed thanks to a researcher the Ethereum foundation reaching out. It turns out, at the time this was written, Proposer Boost was not active on mainnet which was causing tiebreaks to be resolved through lexicographical ordering of the block hashes. Excerpt from the message I received from said researcher:

We've identified the reason for the reorg of block 3340347 - proposer boost is not yet active on mainnet. Proposer boost is a local fork choice change that does not affect consensus code (such as block validity rules or finality conditions), thus not requiring a soft/hard fork (forks have significant coordination costs across client teams & stakers). Clients are releasing it in their software at their own pace, and stakers can upgrade to software with this feature without a targeted timeline.

Without proposer boost, the fork choice tiebreaker is the block root (higher lexicographical ordering wins). The block roots were the following:

Block 3340346: 0xfaf0....

Block 3340347: 0x9774....

This particular reorg instance is explained by the lack of proposer boost, and the tiebreaker rule.

These "unfair" reorgs are probably a good rationale for having proposer boost turned on, so hopefully it is activated soon!

Creating a mesh VPN tool for fun and learning

2021-07-07T00:00:00+10:00

tldr: I created a mesh VPN networking tool named meshboi which you can checkout on my github! This post walks through the 'magic' behind mesh VPN tools, using meshboi as an example.

Mesh VPN tools, such as Nebula, Tailscale, Zerotier and Tinc, facilitate the creation of peer to peer (mesh) networks across the internet. These tools securely, performantly and transparently enable multiple distributed computers to communicate as though they were physically connected to the same network switch (layer 2 or 3, depending on the tool and configuration used). There’s lots of use cases for these tools, but some popular uses-cases are:

Allowing services on instances across multiple clouds to communicate
Allowing secure remote access to home networks for roaming users
Bridging “home lab” networks with cloud networks

These tools enable this communication to occur transparently to applications that are sending and receiving packets over the mesh, while also working around common barriers in networking over the internet, such as firewalls and NAT.

The mesh model is in contrast to traditional “star” VPN solutions, such as OpenVPN or StrongSwan. In a star VPN, traffic between peers needs to pass through a central VPN server, which increases latency and reduces scalability (the central server can only handle so much traffic). The image below shows a simplified high level view of the VPN types. A mesh VPN facilitates direct connections between peers, whereas a star requires connections to go through a central server.

It's worth making a quick side note on the use of "VPN" here. In this case, VPN is meant to mean a network of peers over the internet that can communicate amongst each other. This is different to “consumer” VPN products (such as NordVPN or Private Internet Access) that are used for privacy and/or to avoid geographic website blocks. For these VPN products, the VPN server is effectively used as a gateway for clients to connect to the broader internet through. Peers on these networks generally do not and cannot connect to each other. For the rest of this post I'll exclusively focus on the first kind of VPN, which lets peers communicate.

Introducing.. Meshboi!

There’s a bunch of interesting problems to solve in creating one of these tools, so I thought it’d be fun to create my own mesh VPN tool to learn more about how they work. While there’s a lot of complexity in creating a mesh VPN that is secure, performant and works across 100% of network configurations, creating something that isn’t as robust, for the purposes of fun and learning, is definitely doable.

The tool I created is named meshboi and if you want to jump straight to the source code, you can find it on my github. Meshboi is written in Go and was created purely for the purposes of fun and learning (so please use one of the other fantastic tools for running production things on!).

The below asciinema recording shows a simple demo of meshboi, showing how it enables communication between two physically separated computers, both behind NAT and firewalls, that are only connected via the internet.

Why is this hard?

There are a few core problems to solve when creating a tool like this.

Discoverability/Bootstrapping - if two peers want to talk to each other, how do they know how to find each other in order to form a link?
Transparency - The tool should allow applications to send their traffic to peers on the mesh without needing the applications to be aware of any of the subtleties of being on a mesh. The tool should abstract away the concept of the mesh VPN, such that the applications do not need to be modified and can behave as normal network applications.
Network Address Translation (NAT) - The majority of the internet is still IPv4 based, so there’s complexity in allowing communications between two computers that are both behind NAT. In some cases, one or even both can be behind multiple layers of NAT (CGNAT).
Firewalls - Between any two pairs of computers, there’s typically a lot of firewalls that typically (and probably rightfully!) act to stop unsolicited network traffic from entering networks. While some of these firewalls are configurable, and could be opened up, in some environments the firewalls are out of the control of the user (think corporate network or wifi hotspot in a cafe). Some ISPs might even block certain types of traffic.
Security - If a tool is exposing a local network to the internet, it should make sure that traffic is secured so that it cannot be read or tampered with by third parties. The model that most of these mesh VPN tools use is that traffic coming in from the internet is not trusted, but once it is in it is trusted.

I’ll go more into the details of each of these problems, and how meshboi (and mesh VPN tools in general) solve them.

Discoverability

When a new computer wants to join a mesh network, it needs a method of discovering other computers that are already part of the mesh. This discovery process allows the joining member of a mesh to find internet addresses of other members, so that it can initiate the communication channel.

In theory, this initial bootstrapping could be done manually, through some out of band communication, like email or instant-messaging but this is not so practical and is further complicated by dynamic IP addresses, NAT and firewalls (discussed later).

Instead, most mesh network tools solve this issue by having a centralized server that members of the mesh talk to in order to get the details of their peers. In Nebula this server is named a “Lighthouse”. Meshboi also uses this concept and calls it a “Rolodex''. The Rolodex should be hosted on a server with an IP address that does not change, so that it can always be accessed by peers wanting to join the network.

It’s important to note that unlike centralized VPNs, no real traffic passes through the Rolodex. However, some tools do allow the data to be proxied through this server as a last-ditch option when NAT or firewalls are particularly hard to overcome. In Meshboi’s case this is not possible and the Rolodex server is completely ignorant of any data moving around the network.

The meshboi Rolodex serves only as a method to bootstrap networks, with peers directly authenticating each other (using dTLS and PSK as explained further on) before any data is passed between them. Because of this, the Rolodex implementation supports multiple completely separate networks. The Rolodex server is basically a very unsophisticated in-memory key/value store with {networkName: list<IP addresses>}. I'm currently running a Rolodex server on a personal VPS (rolodex.samlewis.me) that anyone can use to bootstrap their own networks but it's also possible to run a separate instance yourself on a server that's directly accessible to the internet.

Transparency

For a mesh VPN tool to be truly useful it needs to be transparent to network applications that send and receive data over the mesh network. The tool’s usefulness would be very limited if the source code of each application needed to be modified in order to know how to communicate with peers over the mesh.

Mesh VPN tools solve the transparency issue using what is known as “tunneling” where the packet meant for one subnet is tunneled through a connection on another subnet (usually over the internet) and then injected into the subnet on the other side of the tunnel. On Linux systems, TUN/TAP devices facilitate tunneling in this way by providing virtualized software-only network interfaces.

Backreference has a nice explanation of TUNTAP devices:

"You can think of a tun/tap interface as a regular network interface that, when the kernel decides that the moment has come to send data "on the wire", instead sends data to some userspace program that is attached to the interface. In a similar fashion, the program can write to this [device], and the data .. will appear as input to the tun/tap interface.”

In this way, having a TUNTAP device with an assigned IP address on the mesh’s subnet allows applications wanting to communicate on that subnet to send data to the TUNTAP device. The data is then able to be read by the mesh VPN tool and forwarded over the internet to the corresponding peer.

In meshboi's case, the mesh network is a L3 network so TUN devices are used. Meshboi creates a TUN device at start up and assigns an IP address on the mesh's subnet (of the user's choosing) to it.

The diagram below shows, at a high level, how packets would flow between two mesh members running netcat (but this would look the same no matter which network applications were used - you can imagine NGINX running on one side and Firefox on the other, for example).

The details of what exactly the data being sent across the internet are further covered in the Security section below, but for now it's OK to imagine that it's a UDP packet that contains the original IPv4 packet. UDP is typically used for this tunneling use case as it does not suffer from the drawbacks of running TCP over TCP.

The following code snippet shows a (slightly modified for readability) snippet from the tun_router.go showing how this works for outbound packets.

for {
    // Read from the TUN device, this reads out a packet destined for another peer on the mesh
    n, err := tr.tun.Read(packet)
    header, err := ipv4.ParseHeader(packet[:n])

    // Find the VPN/Mesh IP that this packet is destined for
    vpnIP, ok := netaddr.FromStdIP(header.Dst)

    // See if we know of a peer with that Mesh IP address
    peerConn, ok := tr.store.GetByInsideIp(vpnIP)

    if !ok {
       // Drop the data if we aren't aware of any peers with that address
       log.Warn("Dropping data destined for ", vpnIP)
       continue
    }

    msg := make([]byte, n)
    copy(msg, packet[:n])

    // Queue the data to be sent to the corresponding peer
    // this will send the packet to their internet address
    // through the dTLS UDP tunnel
    peerConn.QueueData(msg)
}

The corresponding code for handling packets that have been sent by other peer members to us is handled in peer_conn.go:

b := make([]byte, bufSize)
for {
    // Read a packet that another peer has sent us over the interneet
    n, err := p.conn.Read(b)

    // Write the packet to the TUN device so applications awaiting data from the mesh receive it
    written, err := p.tun.Write(b[:n])
}

NAT & Firewalls

The majority of computers on the internet are behind both NAT and firewalls (sometimes multiple levels of both). Being behind NAT and firewalls adds difficulty in allowing external parties to talk into your network across the internet. For an average user this is typically good, as (unless you are hosting an internet accessible server on your LAN) you typically do not want random parties on the internet being able to initiate communications to computers/services running within your network.

However, this is a problem in establishing a mesh network. Members on the mesh need to be able to send arbitrary packets between each other. One way of solving this would be for each member to add port forwarding rules to their routers and open ports in their firewall(s). This is typically the appropriate solution to make a locally hosted server accessible on the internet and for a user with the technical know-how and access privileges this is doable (if kind of annoying). However this gets trickier and becomes a less ideal solution when you consider that users might be on a corporate network, using public wifi (for example at a cafe) or even behind Carrier Grade NAT.

Luckily there is another way! This method takes advantage of us knowing the peers that we want to talk to ahead of time (because the Rolodex has told us about them!).

Firstly, to explain the issues that NAT introduces in this context: NAT works by transparently translating source ports and IP addresses from a computer on a LAN to source ports and IP addresses from the internet facing side of its router. Typically, because this is done transparently, there is no need for anything except for the router to know what this mapping is. However, because we want to open a port and advertise what that port is to other peers it becomes necessary to know what the port is.

To solve this, peers on the mesh network reuse the same the source port that they used to communicate with the Rolodex in order to talk to their peers. This allows the Rolodex to store the IP address and port of each member that it sees from its perspective and send that to data to new peers. Assuming that the Rolodex is somewhere on the internet, this means that it has the “internet view” of each mesh member and can be the authority in sharing how to connect to each member.

The Rolodex server also serves as a way to ensure that the NAT mapping remains up to date. NAT devices will garbage collect mappings after an inactivity period (as otherwise eventually all the usable ports will be used up), so the Rolodex is used to send and receive heartbeat messages from peers to ensure that these remain valid for as long as the peer is running.

Great! But what about firewalls? As a very short intro, firewalls usually deny all incoming traffic unless it has been explicitly permitted. However, because of the way IP traffic works , for most users, firewalls need to be stateful. This basically means that the firewall allows incoming packets so long as it has seen corresponding outgoing packets. This is so that, for example, when you send a request to load www.google.com the firewall will allow the response from the Google web server. Without a stateful firewall, the firewall would need exceptions added for every website and IP address that a user expects to communicate with.

Meshboi (and most mesh networking tools) take advantage of this by having pairs of peers send each other packets in order to form a connection. As both sides have then "initiated" communications (in the view of their respective firewalls), the firewalls will then allow incoming packets from the corresponding peer.

For meshboi, as will be elaborated on in the next section, each pair of peers have one peer designated as the 'server' and one as the 'client'. The client will connect to the server immediately whereas the server only waits for the client to connect. For this reason, its only important for the server to send out a message to the client, to then enable the client to talk inward to it.

func (pc *PeerConnector) openFirewallToPeer(addr net.Addr) error {
    conn, err := pc.listenerDialer.Dial(addr)
    defer conn.Close()

    // .. snipped error handling ..

    // It doesn't really matter what is sent here - the important part is
    // something is sent. We're effectively telling any and all (stateful)
    // firewalls on our path to the peer to allow any future traffic that has
    // originated from that peer
    _, err = conn.Write([]byte("hello"))

    // .. snipped error handling ..

    return nil
}

I’d be remiss not to mention this excellent Tailscale blog post that goes into a lot more depth on the firewall/NAT problem and considers more than the basic NAT scenario that I considered here. It also has some very excellent diagrams! Worth the read if my explanation here isn't making sense or if you’re interested in the subject.

Security

As Mesh VPN tools tunnel traffic over the internet, they need some sort of security that affords protection against things like:

Data being readable by third parties
Ensuring that incoming packets are only accepted from trusted peers, not from malicious internet people
Data has integrity (people can’t mess with our messages without our knowledge)
Replay attacks

Tailscale uses Wireguard links for this security and Nebula uses the Noise Protocol Framework which is what Wireguard is built upon.

Not knowing much about this type of security (and admittedly, barely knowing much more now!), I initially naively considered encrypting each internet-bound packet transmitted by meshboi by using something like AES and a pre-shared key. However, while this would stop the data from being readable by third parties it wouldn't ensure protection against the other points above - attackers could still mess with the integrity of packets or could replay past packets.

That said, if we were certain that all the traffic over the mesh would already be in protocols that affords that sort of protection (such as HTTPS or SSH) then this sort of double protection would arguably be unnecessary. However, as meshboi should allow any sort of traffic to flow over its mesh it is better if it can guarantee some sort of security.

Because, as previously mentioned, traffic over the tunnel should be UDP, this somewhat limits the choice of protocols for implementing this security. To cut a long story short, for meshboi's case, I ended up using the not-super-popular UDP version of TLS: dTLS. To implement this I used the dTLS implementation in the Pion stack (which, as a complete aside, has to be just about the friendliest community I've ever encountered on the internet!).

Each link between peers in meshboi is implemented as a dTLS link. When peers learn of each other (via messages from the Rolodex) they will attempt to form this link using the pre-shared key they have been configured with. As dTLS requires the concept of a 'server' and 'client' for each connection, this is implemented somewhat arbitrarily in meshboi by using the peer with the 'largest' IP address to be the server:

func (pc *PeerConnector) AmServer(other netaddr.IPPort) bool {
    ipCompare := pc.myOutsideAddr.IP.Compare(other.IP)

    switch ipCompare {
    case -1:
       return  false
    case  0:
       if pc.myOutsideAddr.Port > other.Port {
           return  true
       } else if pc.myOutsideAddr.Port < other.Port {
           return  false
       }
    case  1:
        return  true
    }
}

As well as using dTLS to ensure that every peer that attempts to join the mesh is trusted, meshboi also makes use of the PSK 'hint' feature to transmit some out of band information between peers. As far as I understand, the ability to send hints is to facilitate dTLS connections where there's a human in the loop that might need a hint to remember the PSK. In Meshboi, this is somewhat abused to send the internal VPN address of the peer:

func getDtlsConfig(vpnIp netaddr.IP, psk []byte) *dtls.Config {
    return &dtls.Config{
        PSK: func(hint []byte) ([]byte, error) {
            return psk, nil
        },
        // We set the PSK identity hint as the IP address of this member in the
        // VPN as an quick and hacky way of signalling (out of band) who this
        // member is to other members we connect to. A more robust way of
        // achieving this would be to define an OOB messaging scheme to do this
        // with instead.
        PSKIdentityHint:      []byte(vpnIp.String()),
        CipherSuites:         []dtls.CipherSuiteID{dtls.TLS_PSK_WITH_AES_128_CCM_8},
        ExtendedMasterSecret: dtls.RequireExtendedMasterSecret,
    }
}

In this way, the messages sent across the dTLS link can be tunneled IPv4 packets that can be injected directly into the mesh network. A future improvement might be to change this so that the IPv4 messages are wrapped inside other messages with metadata, so that more out of band messages can be defined and sent between peers.

Performance

To test the performance of meshboi, I used iperf3 to check the achievable speed between a GCP e2-micro (2 vCPUs, 1 GB memory) Debian Google Cloud Platform instance and a Vultr $5/month instance (1 vCPU, 1 GB memory).

For comparison, I also tested Nebula, Tailscale and then opened up instances' firewalls to run the test without any tunneling/mesh VPN software at all. It's likely that the configuration could be improved for better performance with the other tools, but it's nice at least as a point of reference.

As these tools do networking and packet encryption/decryption in userspace, my benchmarking technique was to measure both the achievable throughput across a link and the CPU usage hit taken. To do this, I ran mpstat 1 to find the average non-idle CPU time while running iperf3 across the link. I ran the iperf3 server on the GCP instance and the client on the Vultr instance, and measured CPU only on the Vultr instance. My benchmarking technique is not very comprehensive and likely could be improved (open to hearing suggestions!) but I think gives somewhat of an indication of the difference between these tools.

Test	Non idle CPU time (%)	Average Bandwidth (Mbps)
No tunneling	2.74%	253 MBit/s
meshboi	75.47%	98.4 Mbit/s
Nebula	57.44%	222 Mbits/sec
Tailscale	72.02%	183 Mbit/s

As expected, meshboi is probably not the best choice compared to established tools! I'm sure there's lots of easy wins to be had here though, I suspect the choice of dTLS cipher could go a long way to speeding things up. Meshboi also copies data around somewhat unnecessarily which is also most likely slowing things down.

Conclusion

Hopefully this article gives a bit of an insight into how the magic of mesh VPN tools work. If you're interested in learning more then by all means feel free to dive into the meshboi source code, play around with meshboi or even submit a patch or two - there's likely boundless things to improve!

Otherwise, if you're after a production ready mesh VPN solution there's a lot of solutions out there that work much better than meshboi. In particular, solutions like Tailscale and Nebula can do things like:

Multi platform support (OSX, Windows)
Better authentication (cert based or SSO based)
Have better performance
General robustness
Have audited security

Forcing ethernet traffic onto physical interfaces

2020-04-28T00:00:00+10:00

When you're working with a device that has two ethernet ports, it can be sometimes useful for hardware benchmarking or testing purposes to do a loopback test where one port connects directly to the other.

By default, and somewhat annoyingly, Linux is smart about routing traffic between ethernet adapters on the same system, leading to results from iperf like this:

:::bash
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.70 GBytes  3.18 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  3.70 GBytes  3.16 Gbits/sec                  receiver

Which is impressive, but obviously means that no data is being put out through the physical interface.

Luckily network namespaces can be used to force traffic out across the physical medium. By placing each adapter into its own namespace, the namespaces can't see each other and so the traffic is forced out on the wire.

To do this:

Create a namespace for each adapter

ip netns add ns_eth0
ip netns add ns_eth1

Move the adapters into the namespaces

ip link set eth0 netns ns_eth0
ip link set eth1 netns ns_eth1

Assign ip addresses to each adapter

ip netns exec ns_eth0 ip addr add dev eth0 192.168.1.1/24
ip netns exec ns_eth1 ip addr add dev eth1 192.168.1.2/24

Bring the adapters up

ip netns exec ns_eth0 ip link set eth0 up
ip netns exec ns_eth1 ip link set eth1 up

Check that each adapter can ping the other

ip netns exec ns_eth0 ping 192.168.1.2
ip netns exec ns_eth1 ping 192.168.1.1

Run your tests, happy in the knowledge that the traffic is really going out on the physical interface!

ip netns exec ns_eth0 iperf3 -s
ip netns exec ns_eth1 iperf3 -c 192.168.1.1

A peek under Bitcoin's hood

2017-06-05T00:00:00+10:00

Bitcoin is really cool. Sure, there's arguments to be made about whether it's a useful technology, whether we're currently in a cryptocurrency bubble or if the governance problems that it's currently facing will ever be resolved.. But on a purely technical level, the mystical Satoshi Nakamoto created an impressive technology.

Unfortunately, while there's a lot of resources out there that give high level explanations of how Bitcoin works (one such resource I'd highly recommend is Anders Brownworth's fantastic blockchain visual 101 video), there isn't a whole heap of information at a lower level and, in my opinion, there's only so much you can properly grok if you're looking at the 10000 ft view.

As someone relatively new to the space, I found myself hungry to understand the mechanics of how Bitcoin works. Luckily, because Bitcoin is decentralised and peer to peer by its nature, anyone is able to develop a client that conforms to the protocol. In order to get a greater appreciation of how Bitcoin works, I decided to write my own small toy Bitcoin client that was able to publish a transaction to the Bitcoin blockchain.

This post walks through the process of creating a minimally viable Bitcoin client that can create a transaction and submit it to the Bitcoin peer to peer network so that it is included in the Blockchain. If you'd rather just read the raw code, feel free to check out my Github repo.

Address generation

In order to be part of the Bitcoin network, it's necessary to have an address from which you can send and receive funds. Bitcoin uses public key cryptography and an address is basically a hashed version of a public key that has been derived from a secret private key. Surprisingly, and unlike most public key cryptography, the public key is also kept secret until funds are sent from the address - but more on that later.

A quick aside on terminology: In Bitcoin, the term "wallet" is used by clients to mean a collection of addresses. There's no concept of wallets at a protocol level, only addresses.

Bitcoin uses elliptic curve public-key cryptography for its addresses. At an ultra high level, elliptic curve cryptography is used to generate a public key from a private key, in the same way RSA would but with a lower footprint. If you're interested in learning a bit about the mathematics behind how this works, Cloudflare's primer is a fantastic resource.

Starting with a 256 bit private key, the process of generating a Bitcoin address is shown below:

In Python, I use the ecsda library to do the heavy lifting for the elliptic curve cryptography. The following snippet gets a public key for the highly memorable (and highly insecure) private key 0xFEEDB0BDEADBEEF (front padded with enough zeros to make it 64 hex chars long, or 256 bits). You'd want a more secure method of generating private keys than this if you wanted to store any real value in an address!

As an amusing aside, I originally created an address using the private key 0xFACEBEEF and sent it 0.0005 BTC.. 1 month later and someone had stolen my 0.0005 BTC! I guess people must occasionally trawl through addresses with simple/common private keys. You really should use proper key derivation techniques!

from ecdsa import SECP256k1, SigningKey

def get_private_key(hex_string):
    return bytes.fromhex(hex_string.zfill(64)) # pad the hex string to the required 64 characters

def get_public_key(private_key):
    # this returns the concatenated x and y coordinates for the supplied private address
    # the prepended 04 is used to signify that it's uncompressed
    return (bytes.fromhex("04") + SigningKey.from_string(private_key, curve=SECP256k1).verifying_key.to_string())

private_key = get_private_key("FEEDB0BDEADBEEF")
public_key = get_public_key(private_key)

Running this code gets the private key (in hex) of

0000000000000000000000000000000000000000000000000feedb0bdeadbeef

And the public key (in hex) of

04d077e18fd45c031e0d256d75dfa8c3c21c589a861c4c33b99e64cf613113fcff9fc9d90a9d81346bcac64d3c01e6e0ef0828543edad73c0e257b845812cc8d28

The 0x04 that prepends the public key signifies that this is an uncompressed public key, meaning that the x and y coordinates from the ECDSA are simply concatenated. Because of the way ECSDA works, if you know the x value, the y value can only take two values, one even and one odd. Using this information, it is possible to express a public key using x and the polarity of y. This reduces the public key size from 65 bits to 33 bits and the key (and subsequent computed address) are referred to as compressed. For compressed public keys, the prepended value will be 0x02 or 0x03 depending on the polarity of y. Uncompressed public keys are most commonly used in Bitcoin, so that's what I'll use here too.

From here, to generate the Bitcoin address from the public key, the public key is sha256 hashed and then ripemd160 hashed. This double hashing provides an extra layer of security and a ripemd160 hash provides a 160 bit hash of sha256's 256 bit hash, shortening the length of the address. An interesting result of this is that it is possible for two different public keys to hash to the same address! However, with 2^160 different addresses, this isn't likely to happen any time soon.

import hashlib

def get_public_address(public_key):
    address = hashlib.sha256(public_key).digest()

    h = hashlib.new('ripemd160')
    h.update(address)
    address = h.digest()

    return address

public_address = get_public_address(public_key)

The code above generates a public address of c8db639c24f6dc026378225e40459ba8a9e54d1a. This is sometimes referred to as the hash 160 address.

As alluded to before, an interesting point is that both the conversion from private key to public key and the conversion from public key to public address are one way conversions. If you have an address, the only way to work backwards to find the associated public key is to solve a SHA256 hash. This is a little different to most public key cryptography, where your public key is published and your private key hidden. In this case, both public and private keys are hidden and the address (hashed public key) is published.

Public keys are hidden for good reason. Although it is normally infeasible to compute a private key from the corresponding public key, if the method of generating private keys has been compromised then having access to the public key makes it a lot easier to deduce the private key. In 2013, this infamously occurred for Android Bitcoin wallets. Android had a critical weakness generating random numbers, which opened a vector for attackers to find private keys from public keys. This is also why address reuse in Bitcoin is discouraged - to sign a transaction, you need to reveal your public key. If you don't reuse an address after sending a transaction from the address, you don't need worry about the public key of that address being exposed.

That standard way of expressing a Bitcoin address is to use the Base58Check encoding of it. This encoding is only a representation of an address (and so can be decoded/reversed). Base58Check generates addresses of the form 1661HxZpSy5jhcJ2k6av2dxuspa8aafDac. The Base58Check encoding provides a shorter address to express and also has an inbuilt checksum, that allows detection of mistyped address. In just about every Bitcoin client, the Base58Check encoding of your address is the address that you'll see. A Base58Check also includes a version number, which I'm setting to 0 in the code below - this represents that the address is a pubkey hash.

# 58 character alphabet used
BASE58_ALPHABET = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'

def base58_encode(version, public_address):
    """
    Gets a Base58Check string
    See https://en.bitcoin.it/wiki/base58Check_encoding
    """
    version = bytes.fromhex(version)
    checksum = hashlib.sha256(hashlib.sha256(version + public_address).digest()).digest()[:4]
    payload = version + public_address + checksum

    result = int.from_bytes(payload, byteorder="big")

    print(result)

    # count the leading 0s
    padding = len(payload) - len(payload.lstrip(b'\0'))
    encoded = []

    while result != 0:
        result, remainder = divmod(result, 58)
        encoded.append(BASE58_ALPHABET[remainder])

    return padding*"1" + "".join(encoded)[::-1]

bitcoin_address = base58_encode("00", public_address)

After all of that, starting with my private key of feedb0bdeadbeef (front padded with zeros), I've arrived with a Bitcoin address of 1KK2xni6gmTtdnSGRiuAf94jciFgRjDj7W!

With an address, it's now possible to get some Bitcoin! To get some Bitcoin into my address, I bought 0.0045 BTC (at the time of writing, around $11 USD) from btcmarkets using Australian dollars. Using btcmarket's trading portal, I transferred it to the above address, losing 0.0005 BTC to transaction fees in the process. You can see this transaction on the blockchain in transaction 95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7.

Connecting to the p2p network

Now that I have an address with some Bitcoin in it, things get more interesting. If I want to send that Bitcoin somewhere else, it's necessary to connect to the Bitcoin peer to peer network.

Bootstrapping

One of the sticking points I had when first learning about Bitcoin was, given the decentralised nature of the network, how do peers of the network find other peers? Without a centralised authority, how does a Bitcoin client know how to bootstrap and start talking to the rest of the network?

As it turns out, idealism submits to practicality and there is the slightest amount of centralisation in the initial peer discovery process. The principle way for a new client to find peers to connect to is to use a DNS lookup to any number of "DNS seed" servers that are maintained by members of the Bitcoin community.

It turns out DNS is well suited to this purpose of bootstrapping clients as the DNS protocol, which runs over UDP and is lightweight, is hard to DDoS. IRC was used as a previous bootstrapping method but was discontinued due to its weakness to DDoS attacks.

The seeds are hardcoded into Bitcoin core's source code but are subject to change by the core developers.

The Python code below connects to a DNS seed and arbitrarily chooses the first peer to connect to. Using the socket library, the code basically performs a nslookup and returns the ipv4 address of the first result on running a query against the seed node seed.bitcoin.sipa.be.

import socket

# use a dns request to a seed bitcoin DNS server to find a node
nodes = socket.getaddrinfo("seed.bitcoin.sipa.be", None)

# arbitrarily choose the first node
node = nodes[0][4][0]

After running this, the address returned was 208.67.251.126 which is a friendly peer that I can connect to!

Saying hi to my new peer friend

Bitcoin connections between peers are through TCP. Upon connecting to a peer, the beginning handshake message of the Bitcoin protocol is a Version message. Until peers swap Version messages, no other messages will be accepted.

Bitcoin protocol messages are well documented in the Bitcoin developer reference. Using the developer reference as guide, the version message can be constructed in Python as the snippet below shows. Most of the data is fairly uninteresting, administrative data which is used to establish the peer connection. If you're interested in more details than the attached comments provide, have a read of the developer reference.

version = 70014
services = 1 # not a full node, cant provide any data
timestamp = int(time.time())
addr_recvservices = 1
addr_recvipaddress = socket.inet_pton(socket.AF_INET6, "::ffff:127.0.0.1") #ip address of receiving node in big endian
addr_recvport = 8333
addr_transservices = 1
addr_transipaddress = socket.inet_pton(socket.AF_INET6, "::ffff:127.0.0.1")
addr_transport = 8333
nonce = 0
user_agentbytes = 0
start_height = 329167
relay = 0

Using Python's struct library the version payload data is packed into the right format, paying special attention to endianness and byte widths of the data. Packing the data into the right format is important, or else the receiving peer won't be able to understand the raw bytes that it receives.

payload = struct.pack("<I", version)
payload += struct.pack("<Q", services)
payload += struct.pack("<Q", timestamp)
payload += struct.pack("<Q", addr_recvservices)
payload += struct.pack("16s", addr_recvipaddress)
payload += struct.pack(">H", addr_recvport)
payload += struct.pack("<Q", addr_transservices)
payload += struct.pack("16s", addr_transipaddress)
payload += struct.pack(">H", addr_transport)
payload += struct.pack("<Q", nonce)
payload += struct.pack("<H", user_agentbytes)
payload += struct.pack("<I", start_height)

Again, the way in which this data should be packed is available in the developer reference. Finally, each payload transmitted on the Bitcoin network needs to be prepended with a header, that contains the length of the payload, a checksum and the type of message the payload is. The header also contains the magic constant 0xF9BEB4D9 which is set for all mainnet Bitcoin messages. The following function gets a Bitcoin message with header attached.

def get_bitcoin_message(message_type, payload):
    header = struct.pack(">L", 0xF9BEB4D9)
    header += struct.pack("12s", bytes(message_type, 'utf-8'))
    header += struct.pack("<L", len(payload))
    header += hashlib.sha256(hashlib.sha256(payload).digest()).digest()[:4]

    return header + payload

With the data packed into the right format, and the header attached, it can be sent off to our peer!

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((node, 8333))
s.send(get_bitcoin_message("version", payload))
print(s.recv(1024))

The Bitcoin protocol mandates that on receiving a version message, a peer should respond with a Verack acknowledgement message. Because I'm building a tiny "for fun" client, and because peers won't treat me differently if I don't, I'll disregard their version message and not send them the acknowledgement. Sending the Version message as I connect is enough to allow me to later send more meaningful messages.

Running the above snippet prints out the following. It certainly looks promising - "Satoshi" and "Verack" are good words to see in the dump out! If my version message had been malformed, the peer would not have responded at all.

b'\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00f\x00\x00\x00\xf8\xdd\x9aL\x7f\x11\x01\x00\r\x00\x00\x00\x00\x00\x00\x00\xddR1Y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\xce\x1d\xfc\xe9j\r\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\xb8>*\x88@I\x8e\x10/Satoshi:0.14.0/t)\x07\x00\x01\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2'

Bitcoin transactions

To transfer Bitcoin it's necessary to broadcast a transaction to the Bitcoin network.

Critically, the most important idea to understand is that the balance of a Bitcoin address is comprised solely by the number of "Unspent Transaction Outputs" (UTXO) that the address can spend. When Bob sends a Bitcoin to Alice, he's really just creating a UTXO that Alice (and only Alice) can later use to create another UTXO and send that Bitcoin on. The balance of a Bitcoin address is therefore defined by the amount of Bitcoin it is able to transfer to another address, rather than the amount of Bitcoin it has.

To emphasize, when someone says that they have X bitcoin, they're really saying that all of the UTXOs that they can spend sum to X bitcoin of value. The distinction is subtle but important, the balance of a Bitcoin address isn't recorded anywhere directly but rather can be found by summing the UTXOs that it can spend. When I came to this realisation it was definitely a "oh, that's how it works!" moment.

A side effect of this is that transaction outputs can either be unspent (UTXO) or completely spent. It isn't possible to only spend half of an output that someone has spent to you and then spend the rest at a later time. If you do want to spend a fraction of an output that you've received, you instead can send the fraction that you want to send while sending the rest back to yourself. A simplified version of this is shown in the diagram below.

When a transaction output is created, it is created with a locking condition that will allow someone in the future to spend it, through what are called transaction scripts. Most commonly, this locking condition is: "to spend this output, you need to prove that you own the private key corresponding to a particular public address". This is called a "Pay-to-Public-Key-Hash" script. However, through Bitcoin script other types of conditions are possible. For example, an output could created that could be spent by anyone that could solve a certain hash or a transaction could be created that anyone could spend.

Through Script, it's possible to create simple contract based transactions. Script is a basic stack based language with number of operations centred around checking equality of hashes and verifying signatures. Script is not Turing complete and does not have the ability to have any loops. The competing cryptocurrency Ethereum built on this to be able to have "smart contracts", which does have a Turing complete language. There's much debate about the utility, necessity and security of including a Turing complete language in cryptocurrencies but I'll leave that debate to others!

In standard terminology, a Bitcoin transaction is made up of inputs and outputs. An input is a UTXO (that is now being spent) and an output is a new UTXO. There can be multiple outputs for a single input but an input needs to be completely spent in a transaction. Any part of an input leftover is claimed by miners as a mining fee.

For my toy client I want to be able to send on the Bitcoin previously transferred from an exchange to my FEEDB0BDEADBEEF address. Using the same process as before, I generated another address using the private key (before padding of) BADCAFEFABC0FFEE. This generated the address 1QGNXLzGXhWTKF3HTSjuBMpQyUYFkWfgVC.

Creating a raw transaction

Creating a transaction is a matter of first packing a "raw transaction" and then signing the raw transaction. Again, the developer reference has a description of what goes into a transaction. What makes up a transaction is shown below but a few notes first:

Common Bitcoin parlance uses the terms signature script and pubkey script which I find a little confusing. The signature script is used to meet the conditions of the UTXO that we want to use in the transaction and the pubkey script is used to set the conditions that need to be met to spend the UTXO we are creating. A better name for the signature script might be a unlocking script and a better name for the pubkey script might be a locking script.
Bitcoin transaction values are specified in Satoshis. A Satoshi represents the smallest divisible part of a Bitcoin and represents one hundred millionth of a Bitcoin.

For simplicity, what is shown below is for a transaction for one output and one input. More complex transactions, with multiple inputs and outputs are possible to create in the same way.

Field	Description
Version	Transaction version (currently always 1)
Number of inputs	Number of inputs to spend
Transaction ID	Transaction from which to spend
Output number	Output of the transaction to spend
Signature script length	Length in bytes of the sig script
Signature script	Signature script in the Script language
Sequence number	Always 0xffffffff unless you wish to use a lock time
Number of outputs	Number of outputs to create
Value	Number of Satoshis to spend
Pubkey script length	Length in bytes of the pubkey script
Pubkey script	Pubkey script in the Script language
Lock time	Earliest time/block number that this transaction can be included in a block

Ignoring the signature script and pubkey script for now, it's quite easy to see what should go in the other fields of the raw transaction. To send the funds in my FEEDB0BDEADBEEF address to my BADCAFEFABC0FFEE address, I look at the transaction that was created by the exchange. This gives me:

The transaction id is 95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7.
The output that was sent to my address was the second output, output 1 (output numbers are 0 indexed).
Number of outputs is 1, as I want to send everything in FEEDB0BDEADBEEF to BADCAFEFABC0FFEE.
Value can be at most 400,000 Satoshis. It must be less than this to allow some fees. I'll allow 20,000 Satoshi to be taken as a fee so will set value to 380,000.
Lock time will be set to 0, this allows the transaction to be included at any time or block.

For the Pubkey script of our transaction, we use a Pay to Pubkey hash (or p2pk) script. The script ensures that only the person that holds the public key that hashes to the provided Bitcoin address is able to spend the created output and that a supplied signature has been generated by the person that holds the corresponding private key to the public key.

To unlock a transaction that has been locked by a p2pk script, the user provides their public key and a signature of the hash of the raw transaction. The public key is hashed and compared to the address that the script was created with and the signature is verified for the supplied public key. If the hash of the public key and the address are equal, and the signature is verified, the output can be spent.

In Bitcoin script operands, the p2pk script looks as follows.

OP_DUP
OP_HASH160
<Length of address in bytes>
<Bitcoin address>
OP_EQUALVERIFY
OP_CHECKSIG

Converting the operands to their values (these can be found on the wiki) and inputting the public address (before it has been Base58Check encoded) gives the following script in hex:

0x76
0xA9
0x14
0xFF33195EC053D6E58D5FD3CC67747D3E1C71B280
0x88
0xAC

The address is found using the earlier shown code for deriving an address from a private key, for the private key we're sending to, 0xBADCAFEFABC0FFEE.

Signing the transaction

There are two separate, but somewhat related, uses for the signature script in a (p2pk) transaction:

The script verifies (unlocks) the UTXO that we are are trying to spend, by providing our public key that hashes to the address that the UTXO has been sent
The script also signs the transaction that we are submitting to the network, such that nobody is able to modify the transaction without invalidating the signature

However, the raw transaction contains the signature script which should contain a signature of the raw transaction! This chicken and egg problem is solved by placing the Pubkey script of the UTXO we're using in the signature script slot prior to signing the raw transaction. As far as I could tell, there doesn't seem to be any good reason for using the Pubkey as the placeholder, it could really be any arbitrary data.

Before the raw transaction is hashed, it also needs to have a Hashtype value appended. The most common hashtype value is SIGHASH_ALL, which signs the whole structure such that no inputs or outputs can be modified. The linked wiki page lists other hash types, which can allow combinations of inputs and outputs to be modified after the transaction has been signed.

The below functions put together a python dictionary of raw transaction values.

def get_p2pkh_script(pub_key):
    """
    This is the standard 'pay to pubkey hash' script
    """
    # OP_DUP then OP_HASH160 then 20 bytes (pub address length)
    script = bytes.fromhex("76a914")

    # The address to pay to
    script += pub_key

    # OP_EQUALVERIFY then OP_CHECKSIG
    script += bytes.fromhex("88ac")

    return script

def get_raw_transaction(from_addr, to_addr, transaction_hash, output_index, satoshis_spend):
    """
    Gets a raw transaction for a one input to one output transaction
    """
    transaction = {}
    transaction["version"] = 1
    transaction["num_inputs"] = 1

    # transaction byte order should be reversed:
    # https://bitcoin.org/en/developer-reference#hash-byte-order
    transaction["transaction_hash"] = bytes.fromhex(transaction_hash)[::-1]
    transaction["output_index"] = output_index

    # temporarily make the signature script the old pubkey script
    # this will later be replaced. I'm assuming here that the previous
    # pubkey script was a p2pkh script here
    transaction["sig_script_length"] = 25
    transaction["sig_script"] = get_p2pkh_script(from_addr)

    transaction["sequence"] = 0xffffffff
    transaction["num_outputs"] = 1
    transaction["satoshis"] = satoshis_spend
    transaction["pubkey_length"] = 25
    transaction["pubkey_script"] = get_p2pkh_script(to_addr)
    transaction["lock_time"] = 0
    transaction["hash_code_type"] = 1

    return transaction

Calling the code with the following values creates the raw transaction that I'm interested in making.

private_key = address_utils.get_private_key("FEEDB0BDEADBEEF")
public_key = address_utils.get_public_key(private_key)
from_address = address_utils.get_public_address(public_key)
to_address = address_utils.get_public_address(address_utils.get_public_key(address_utils.get_private_key("BADCAFEFABC0FFEE")))

transaction_id = "95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7"  
satoshis = 380000
output_index = 1

raw = get_raw_transaction(from_address, to_address, transaction_id, output_index, satoshis)

It might be confusing to see that I'm using a private key to generate the to_address. This is really only done for convenience and to show how the to_address is found. If you were making a transaction to someone else, you'd ask them for their public address and transfer to that, you wouldn't need to know their private key.

In order to be able to sign, and eventually transmit the transaction to the network, the raw transaction needs to be packed appropriately. This is implemented in the get_packed_transaction function which I won't replicate here, as it's essentially just more struct packing code. If you're interested you can find it in the bitcoin_transaction_utils.py Python file in my Github repo.

This allows me to define a function that will produce the signature script. Once the signature script is generated, it should replace the placeholder signature script.

def get_transaction_signature(transaction, private_key):
    """
    Gets the sigscript of a raw transaction
    private_key should be in bytes form
    """
    packed_raw_transaction = get_packed_transaction(transaction)
    hash = hashlib.sha256(hashlib.sha256(packed_raw_transaction).digest()).digest()
    public_key = address_utils.get_public_key(private_key)
    key = SigningKey.from_string(private_key, curve=SECP256k1)
    signature = key.sign_digest(hash, sigencode=util.sigencode_der)
    signature += bytes.fromhex("01") #hash code type

    sigscript = struct.pack("<B", len(signature))
    sigscript += signature
    sigscript += struct.pack("<B", len(public_key))
    sigscript += public_key

    return sigscript

Essentially, the signature script is provided as an input to the pubkey script of the previous transaction I'm trying to use, so that I can prove I am allowed to spend the output that I'm now using as an input. The mechanics of how this works is shown below, which is taken from the Bitcoin wiki. Working from top to bottom, each row is a one iteration of the script. This is for a pay to pubkey hash pubkey script, which, as mentioned earlier is the most common script. It is also the script that both the transaction I'm creating and the transaction I'm redeeming use.

Stack	Script	Description
Empty	signature publicKey OP_DUP OP_HASH160 pubKeyHash OP_EQUALVERIFY OP_CHECKSIG	The signature and publicKey from the signature script are combined with the pubkey script.
signature publicKey	OP_DUP OP_HASH160 pubKeyHash OP_EQUALVERIFY OP_CHECKSIG	The signature and publicKey are added to the stack.
signature publicKey publicKey	OP_HASH160 pubKeyHash OP_EQUALVERIFY OP_CHECKSIG	The top stack item is duplicated by `OP_DUP`
signature publicKey pubHashA	pubKeyHash OP_EQUALVERIFY OP_CHECKSIG	Top stack item (publicKey) is hashed by `OP_HASH160`, pushing pubHashA to the stack.
signature publicKey pubHashA pubKeyHash	OP_EQUALVERIFY OP_CHECKSIG	pubKeyHash added to stack.
signature publicKey	OP_CHECKSIG	Equality is checked between pubHashA and pubKeyHash. Execution will halt if not equal.
True		The signature is checked to see if it is a valid signature of the hash of the transaction from the provided publicKey.

This script will fail if the provided public key doesn't hash to the public key hash in the script or if the provided signature doesn't match the provided public key. This ensures that only the person that holds the private key for the address in the pubkey script is able to spend the output.

You can see that here is the first time I have needed to provide my public key anywhere. Up until this point, only my public address has been published. It's necessary to provide the public key here as it is allows verification of the signature that the transaction has been signed with.

Using the get_transaction_signature function, we can now sign and pack our transaction ready for transmission! This involves replacing the placeholder signature script with the real signature script and removing the hash_code_type from the transaction as shown below.

signature = get_transaction_signature(raw, private_key )

raw["sig_script_length"] = len(signature)
raw["sig_script"] = signature
del raw["hash_code_type"]

transaction = get_packed_transaction(raw)

Publishing the transaction

With the transaction packed and signed, it's a matter of telling the network about it. Using a few functions previously defined in this article in bitcoin_p2p_message_utils.py, the below piece of code puts the Bitcoin message header on the transmission and transmits it to a peer. As mentioned earlier, it's first necessary to send a version message to the peer so that it accepts subsequent messages.

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((get_bitcoin_peer(), 8333))
s.send(get_bitcoin_message("version", get_version_payload())
s.send(get_bitcoin_message("tx", transaction)

Sending the transaction was the most annoying part of getting this to work. If I submitted a transaction that was incorrectly structured or signed, the peer often just dropped the connection or, in a slightly better case, sent cryptic error messages back. One such (very annoying) error message was "S value is unnecessarily high" which was caused by signing the transaction hash using the ECSDA encoding method of sigencode_der. Despite the signature being valid, apparently Bitcoin miners don't like ECSDA signatures formatted in such a way that allows spam in the network. The solution was to use the sigencode_der_canonize function which takes care to format the signatures in the other format. A simple, but extraordinarily hard to debug, issue!

In any case, I eventually got it to work and was very excited when I saw that my transaction made its way into the blockchain!! It was a great feeling of accomplishment knowing that my small, dinky, hand crafted transaction will now forever be a part of Bitcoin's ledger.

When I submitted the transaction, my transaction fee was actually quite low compared to the median (I used the bitcoin fees website to check) and as such it took around 5 hours for a miner to decide to include it in a block. I checked this by looking at the number of confirmations the transaction had - this is a measure of how many blocks deep the block with the transaction is in. At the time of writing this was at 190 confirmations.. meaning that after the block my transaction is in, there's another 190 blocks. This can be pretty safely considered confirmed, as it would take an impressive attack on the network to rewrite 190 blocks to remove my transaction.

Conclusion

I hope you've gained some appreciation of how Bitcoin works through reading this article, I know I certainly did during the months it took me to put all of this together! While most of the information presented here isn't too practicably applicable - you'd normally just use a client that does it all for you - I think having a greater understanding of how things work gives you a better appreciation of what's happening under the covers and gives makes you a more confident user of the technology.

If you'd like to peruse the code, or play around further with the toy examples, please check out my associated Github repo. There's a lot of room to explore further in the Bitcoin world, I've only really looked at a very common use case of Bitcoin. There's certainly room out there to do cooler feats than just transferring value between two addresses! I also didn't touch how mining, the process of adding transactions to the blockchain, works.. which is another rabbit hole all together.

If you've read this far you might have realised that the 380000 Satoshi (or 0.0038 BTC) that I transferred into 1QGNXLzGXhWTKF3HTSjuBMpQyUYFkWfgVC can, with enough smarts, be taken by anyone.. as the private key for the address exists within this article. I'm very interested to see how long it takes to be transferred away and hope that whoever takes it has the decency to do so using some of the techniques I've detailed here! It'd be pretty lame if you just loaded the private key into a wallet app to take it, but I guess I can't stop you! At the time of writing this amount is worth about $10 USD, but if Bitcoin "goes to the moon" who knows how much it might be worth! (Edit: And it's gone now! Taken by 1KgoPFVDNcx7H2VY9bB2dxxP9yNM2Nar1N after a few hours of posting this article. Well played!)

And just in case you're looking for an address to send Bitcoin to when you're playing around with this stuff, or if you think this post was valuable enough to warrant a tip - my address of 18uKa5c9S84tkN1ktuG568CR23vmeU7F5H is happy to take any small donations! Alternatively, if you want to yell at me about getting anything wrong, I'd love to hear it.

Further resources

If you found this article interesting, some further resources to check out:

Mastering Bitcoin is a book that explains the technical details of Bitcoins. I haven't read this completely, but on a skim it looks like it is a wealth of good information.
Ken Sheriff's blog article is a great source of information that covers a lot of the same topics as this article that I unfortunately only found this when I was nearly finished writing this article. If you didn't understand anything in this article, reading Ken's excellent post would be a good place to start.
Mentioned earlier, but Anders Brownworth's fantastic blockchain visual 101 video is an excellent top level view of how blockchain technologies work.
Unless you're a masochist for pain, I'd also recommend not doing everything from scratch unless you're interested in doing so for learning purposes. The pycoin library is a Python Bitcoin library that will save you a few headaches.
To also save yourself pain, it's probably advisable to use the Bitcoin testnet to play around with, rather than using the mainnet like I did. That said, it's more fun when the risk of your code being wrong is losing real money!
Lastly, it is probably worth repeating that the accompanying code for this article can be found in my Github repo.

Unit testing embedded code with cmocka

2016-09-26T00:00:00+10:00

Unit testing is a great software testing practice that gets a lot of love in web and application development but unfortunately isn't practiced as often in embedded/firmware development. This is a shame as the impacts of having a bug in firmware can be considerably more catastrophic than having a bug in a webapp. I've written this article in the hope of demonstrating some of the benefits that come when writing unit tests for embedded software and have also given a small toy example of how it's possible to unit test firmware using the cmocka framework.

Why unit test embedded software?

System level embedded software testing is notoriously difficult, mainly because embedded software runs on bare metal hardware. While you can never completely get away from testing embedded software on real hardware, testing the logic in the embedded software through unit testing is very valuable and gets you a lot of the way there. If you still need convincing, some of the things I really like about having unit tests for my embedded code:

Allows you to develop your application layer logic without the need for any hardware, saving lots of time
Gives you a greater appreciation and understanding of your hardware periphereals, as you need to understand their interfaces to write tests against them
Less 'wtf' bugs that are impossible to track down in development/production
Allows you to test your code against edge cases that otherwise hard to reproduce (eg: testing how your firmware reacts when your temperature sensor is used in negative temperatures)
Testable code tends to be cleaner, modular and reusable code
Takes some amount of pressure off your systems level testing
Stops regressions, particularly if you integrate the tests into your build

There are a lot of unit testing frameworks for C but for embedded testing, the framework I like best is cmocka. I've looked at others in the past, such as cmock and cutest but cmocka stands out as having all testing & mocking functionality I need while having minimal dependencies.

On caveat worth mentioning is that to properly use these frameworks, you'll most likely need to compiler your code with a different compiler to what your compile with for your hardware - both so that the framework can run and so that you can execute the tests on your workstation. However, this shouldn't present a major problem as long as you're not relying on non-standard or undefined compiler behaviour. Logic errors in your code will still be logic errors no matter what compiler you use to compile the code.

An example of unit testing C in an embedded context

To give a concrete example of how you could make use of cmocka in an embedded context, I'll show the process of unit testing code for a temperature sensor. I've chosen the TI TMP101 as the temperature sensor, which is a temperature sensor that works over I2C. The complete listing of code is checked in on my github, the below text gives a rundown of how it all comes together.

The first step as normal, is to write some code for the tmp101 sensor that can fetch a temperature. You can find the full code listing on my github. It's also reproduced below.

#include "tmp101.h"

static const float TMP_BIT_RESOLUTION = 0.0625;

float tmp101_get_temperature(void) 
{   
    // Need to set the TMP101 pointer register to point to the temp register
    uint8_t pointer_address = 0;
    i2c_transmit_blocking(TMP101_ADDRESS, 0, &pointer_address, 1);

    // The TMP101 stores 12 bit samples that are retrieved in two byte blocks
    uint8_t data[2];
    i2c_read_blocking(TMP101_ADDRESS, 0, &data[0], 2);

    // the 1st byte is bits 12 to 4 of the sample and the 2nd byte is bits 4 to 0
    // see page 16 of the TMP_101 datasheet
    uint16_t temperature_bits = (data[0] << 4) | (data[1] >> 4);

    // The 12 bit sample is represented using 2s complement, for simplicity 
    // (and because there's no 12 bit int representation), scale up the sample
    // to 16 bits and adjust the bit resolution when converting later
    int16_t temperature = temperature_bits << 4;

    // shift the sample back down and convert by the TMP_101 bit resolution
    return ((temperature / 16) * 0.0625f);
}

This function isn't the most complicated function but there's enough bit manipulation to make me nervous and unsure if it'll do exactly what I want all the time. Sure, we could run it on my hardware platform and see how it responds but that'd only prove that it works as whatever temperature it is now. Instead, lets write some unit tests for this code.

Conveniently, for this particular temperature sensor, TI are kind enough to provide a table of example temperatures with their corresponding 12 bit digital outputs. These prove a nice place to start in testing our get_temperature function.

In essence, we want to control the data that i2c_read_blocking returns to our tmp101_get_temperature function so that we can check against our function works as it should. A simple (but somewhat ugly) way to this might be to have the function look something like:

void i2c_read_blocking(uint8_t address, uint8_t offset, uint8_t* pData, uint8_t data_size)
{
    #ifdef TESTING
    return DUMMY_VALUE
    #endif

    //normal i2c logic here
}

But mixing test code with production code is ugly and an antipattern that should be avoided if possible. A much better way that allows us to both completely separate our testing and source code is to use the cmocka framework to help us (dynamically!) control what our i2c_read_blocking function does. The first step of using cmocka is to define mocks for the functions you want to mock out. The two mocked functions are shown below.

void __wrap_i2c_transmit_blocking(uint8_t address, uint8_t offset, uint8_t* data, uint8_t data_size)
{
    // allows the calling test to check if the supplied parameters are as expected
    check_expected(address);
    check_expected(offset);
}

void __wrap_i2c_read_blocking(uint8_t address, uint8_t offset, uint8_t* pData, uint8_t data_size) 
{
    // allow the calling test to specify the data it wants back
    // and copy it back out
    for(int i=0; i < data_size; i++) {
        pData[i] = mock_type(uint8_t);
    }
}

cmocka cleverly uses the linker to swap out the real function calls for the mocked ones. To allow for this, mocked out functions are prefixed with __wrap_. The linker is then provided with the arguments --wrap=i2c_read_blocking -Wl,--wrap=i2c_transmit_blocking which allows these functions to be mocked out. For a complete example of compiling/linking with cmocka, see the makefile in my example project.

With the mocks in place we can now write some tests! A complete look at the tests in my example project can be found on my github but, as an example, here's a single test:

static void test_negative_temperature(void **state)
{
    will_return(__wrap_i2c_read_blocking, 0b11100111);
    will_return(__wrap_i2c_read_blocking, 0b00000000);

    assert_true(tmp101_get_temperature() == -25);
}

The two calls to will_return set what the i2c_read_blocking function writes into the pData array. As per the datasheet, the 12 bit sample is returned over two bytes so these values correspond (from the above TI table) to -25 degrees celsius, which we check against. Much easier than having to stick the temperature sensor in the freezer!

Running all of the tests through a Makefile gives this cool print out:

Although this is just a simple example, it hopefully demonstrates the utility that having unit tests can provide. Having a suite of unit tests for a project gives developers a lot of confidence - I find unit tests especially useful in developing application layer protocols on top of lower level protocols.

If you're interested in poking around a little at the source for this example, see how it's all linked together an run as a test or even run it yourself, all code for this is avaliable on my github with instructions of how to run.

I hope you found this article interesting, if you have any question please don't hesitate to let me know!

Getting started with the xv6 operating system

2016-04-18T00:00:00+10:00

xv6 is a nifty operating system that is essentially a reimplementation of Sixth Edition Unix that, unlike most of the huge montholithic Linux or BSD distributions that you'd run on your workstation or server, is simple enough to be used as a learning tool. Though xv6 lacks in functionality that you'd expect from a modern operating system, much of the core unix concepts and structure remain the same. xv6 is also very lightweight, only taking a few seconds to compile and allows remote debugging. This all makes it great for understanding how modern operating systems hang together.

In fact, xv6 was developed by MIT as a teaching operating system for their 6.828 course. The course page for 6.828 is a great resource that MIT are kind enough to have publicly avaliable.

MIT also produce a fantastic xv6 reference book that provides commentary on the structure of the xv6 operating system. Additionally, the xv6 source actually self documents - the xv6 make file contains a target (make xv6.pdf) that generates a pdf of the source code, which amazingly comes in under 100 pages and 10k lines of source (including comments!). Not bad for an entire, functional, operating system.

xv6 is typically run virtualized using qemu on a Linux host machine. To get started with xv6, clone the xv6 repo:

git clone git://github.com/mit-pdos/xv6-public.git xv6

Then to start it as a virtual machine under qemu:

cd xv6
make qemu

If make fails, make sure you have your compiler toolchain installed. There's some good info on the 6.282 tools page. For me, on Fedora, I needed to install gcc with dnf install gcc.

Once you have your compiler toolchain working, qemu will pop up and you'll be presented with the xv6 shell.

From here, you'll be able to run a few simple commands like ls and cat and revel in the knowledge that you can understand what's happening behind the covers by reading through the source code for the shell - line 8500 in the source code listing.

The core while loop the shell executes is shown below.

 // Read and run input commands.
 while(getcmd(buf, sizeof(buf)) >= 0){
    if(buf[0] == ’c’ && buf[1] == ’d’ && buf[2] == ’ ’){
        // Clumsy but will have to do for now.
        // Chdir has no effect on the parent if run in the child.
        buf[strlen(buf)−1] = 0; // chop \n

        if(chdir(buf+3) < 0)
            printf(2, "cannot cd %s\n", buf+3);
        continue;
    }
    if(fork1() == 0)
        runcmd(parsecmd(buf));
    wait();
 }
 exit();

This loop gets a command from a user, checks if it's the cd command and attempts to change directory if it is, otherwise it forks, and runs the entered command in the child process. I personally love how simple and enjoyable the code is to read - check out the rest of the listing for sh.c in the xv6 source code pdf - it's only 400 lines of code and a delight to read.

This was only a brief peek and excitement builder for xv6 - there's so many other facets that are worth having a deeper dive into. Join next time when I'll step through and explore the xv6 bootstrap sequence!

Startup Catalyst 2015: I'm off to Silicon Valley!

2015-09-24T00:00:00+10:00

In just two short sleeps, I'll be jetting off to Silicon Valley as part of the Startup Catalyst program. The program was instigated by Australian entrepreneur/mentor/investor Steve Baxter (who is now a 'shark' on the Australian version of Shark Tank) and aims to expose a group of under 25s (affectionately referred to as 'youths') with technical backgrounds to the startup culture in Silicon Valley.

I'll be headed over with 19 other awesome youths from a variety of different backgrounds - spanning robotics to UI - and over the 10 days that we're in San Francisco we'll visit a whole bunch of tech companies (Google, Facebook and Twitter to name just a few!), have the chance to chat to startup founders and technical whizzes, attend tech meetups and even participate in a startup weekend. Sound like a lot for 10 days? It is! The schedule is completely packed with awesome events and as we were told by Steve and Colin (the project director and technology lead, respectively), they don’t want to kill us - just nearly kill us.

*The 2015 Catalyst group*

I'm only a recent convert to the startup world, having participated in my first 'Startup Weekend' (an event where you literally create a startup during the course of the weekend!) where I, and 3 others, created Drivevibe. I've been bit by the bug ever since, the Startup Mentality of 'just getting it done', is just so infectious and motivational.

I've always wanted to use my combination of skills, spanning my background in Electrical Engineering, current work in the Software development space and my interest in data analytics/machine learning to work on challenging problems and hopefully effect a positive change in the world. For this reason, having this chance to go to Silicon Valley, absorb the culture and make connections is so incredibly exciting.

I'm so excited to experience everything San Francisco has to offer with my cohort of amazingly talented youths. Even though the trip is only 10 days short, I'm sure the impacts of it will be very long lasting. Of course, none of this would be possible without the great help from the Startup Catalyst sponsors and supporters who have made this trip possible. A huge shout out to sponsors QUT Business School, Black Sheep Capital, Griffith University, Virgin Australia, Suncorp and the Queensland Government and supporters AWS, River City Labs, Uber, StartupAus and StartupQLD. A very special thank you to Steve Baxter for instigating and financially backing the program and Peter Laurie and Colin Kinner for making it all happen.

If you'll be in the San Francisco area while I'll be over there (28th of September to 7th of October) I'd love to shout you a coffee and geek out about whatever you’re passionate about! In the meantime, follow me on twitter to get all the hot goss from the trip!

Follow @awesamlewis

Using the MSP432 serial (eUSCI) modules

2015-05-23T00:00:00+10:00

As a follow up to my post on MSP432 printf functionality, this is a write-up of my notes on using the serial modules on the MSP432.

The MSP432 has two different types of serial modules; eUSCI_A modules and eUSCI_B modules. The eUSCI_A modules support both the UART and SPI protocols while the eUSCI_B modules support the SPI and I2C protocols. Using the MSP432 datasheet you can find which MSP432 pins correspond to the modules. The following picture, which is an excerpt from page 10 of the data sheet shows the pin mapping for the first eUSCI_A and eUSCI_B modules.

As each of the two modules support multiple protocols, you might not need to use all of the pins when you're using one of the modules. For example, if you wanted to use eUSCI_A0 for UART, you wouldn't need a 'slave transmit enable' pin or a 'clock signal input/output'. So you'd only be using P1.2 and P1.3. In fact, these are exactly the two pins you're using when you're communicating with your PC over UART as described in my previous post!

So now you know which pins to use, but how do you actually configure and use them? A word of warning first though: on the MSP432 launchpad, not all the eUSCI modules have pins exposed on the launchpad, so first check that the module you want has the relevant pins exposed on the launchpad. I've been playing around with a ESP8266 chip which talks over UART and have been using the eUSCI_A2 module which is on pins P3.2 and P3.3.

To configure the serial modules you have two choices; either read the family guide and toggle bits in all of the configuration registers manually or let TI do the heavy heavy lifting and use their driverlib. The driverlib is good but can be frustrating to use as the documentation provided is fairly minimal and you need to have an idea of what you're looking for first.

As an example of using the driverlib to configure a serial module, I'll step through the process of configuring the eUSCI_A2 module for UART with a baud rate of 9600 so that it can talk to my ESP8266 chip. The process should be similar for any serial chip that you want to interface with your MSP432 though!

First thing first, we want to put the pins in UART mode. The pins we're specifically interested in are P3.2 and P3.3, they correspond to the RX and TX pins respectively. To do this, we need to look at the datasheet again and find where the pin functions are defined. The following picture is an excerpt from page 104 of the datasheet.

An 'X' in the P3DIR column indicates a "don't care". So for both pins, it doesn't matter if we set them to an output or an input when configuring them for UART. For the P3SEL1 and P3SEL0 columns, we need to consult the family guide. The next picture is an excerpt from page 483 of the family guide.

So to use the UART functionality of these pins, we're using the 'primary IO module function'. Now we have all the information we need to set the pins up in UART mode! Because it doesn't matter whether the pins are in output or input mode, we're able to use the GPIO_setAsPeripheralModuleFunctionInputPin function or the GPIO_setAsPeripheralModuleFunctionOutputPin function. For no reason whatsoever, I'll use the output variant. The result is the following line of code:

/* Set pins 2 and 3 of port 3 to the primary module function (UART) */
GPIO_setAsPeripheralModuleFunctionInputPin(GPIO_PORT_P3, GPIO_PIN2 | GPIO_PIN3, GPIO_PRIMARY_MODULE_FUNCTION);

The next thing to do is to define a eUSCI config. This is basically from the TI provided examples. The only tricky bit is generating the BRDIV, UCxBRF and UCxBRS numbers which in this case are 78, 2 & 0. Luckily, once again TI have provided a tool to do this. This tool finds the optimal way of prescaling and modulating the clock source you're using to achieve your desired baud rate. More information about this process is on page 721 of the family guide.

Using the tool with a clock of 12MHz and baud rate of 9600 Hz, gives a clockPrescalar of 78, a firstModReg of 2, a secondModReg of 0 and oversampling turned on. Converting this into a config, we get the following:

const eUSCI_UART_Config uartConfig =
{
    EUSCI_A_UART_CLOCKSOURCE_SMCLK,          // SMCLK Clock Source
    78,                                      // BRDIV = 78
    2,                                       // UCxBRF = 2
    0,                                       // UCxBRS = 0
    EUSCI_A_UART_NO_PARITY,                  // No Parity
    EUSCI_A_UART_LSB_FIRST,                  // MSB First
    EUSCI_A_UART_ONE_STOP_BIT,               // One stop bit
    EUSCI_A_UART_MODE,                       // UART mode
    EUSCI_A_UART_OVERSAMPLING_BAUDRATE_GENERATION  // Oversampling
};

The parity, MSB/LSB first and the amount of stop bits will all be determined by the device or chip that you want to communicate with. Now, to use that config to initialise the module using driverlib:

UART_initModule(EUSCI_A2_MODULE, &uartConfig);

This sets all the relevant registers to achieve the configuration earlier specified. If you're really interesting in understanding what exactly this is doing, it's worth reading the family guide. Now that the module is intialised, it can finally be enabled on!

UART_enableModule(EUSCI_A2_MODULE);

From here, you can send a byte of data to the module with the following driverlib function call:

/* sends the 'g' character to the A2 module' */
UART_transmitData(EUSCI_A2_MODULE, 'g');

If you want to simplify the process of sending individual characters (or instead want to send strings/numbers etc), check out my post about a printf function for the MSP432.

The whole code would then end up looking like this.

#include "driverlib.h"

const eUSCI_UART_Config uartConfig =
{
    EUSCI_A_UART_CLOCKSOURCE_SMCLK,          // SMCLK Clock Source
    78,                                     // BRDIV = 78
    2,                                       // UCxBRF = 2
    0,                                       // UCxBRS = 0
    EUSCI_A_UART_NO_PARITY,                  // No Parity
    EUSCI_A_UART_LSB_FIRST,                  // MSB First
    EUSCI_A_UART_ONE_STOP_BIT,               // One stop bit
    EUSCI_A_UART_MODE,                       // UART mode
    EUSCI_A_UART_OVERSAMPLING_BAUDRATE_GENERATION  // Oversampling
};


int main(void)
{
    /* Halting WDT  */
    WDT_A_holdTimer();


    /* Configure pins P3.2 and P3.3 in UART mode.
     * Doesn't matter if they are set to inputs or outputs
     */
    GPIO_setAsPeripheralModuleFunctionInputPin(GPIO_PORT_P3,
                GPIO_PIN2 | GPIO_PIN3, GPIO_PRIMARY_MODULE_FUNCTION);

    /* Setting DCO (clock) to 12MHz */
    CS_setDCOCenteredFrequency(CS_DCO_FREQUENCY_12);

    /* Configuring UART Module */
    UART_initModule(EUSCI_A2_MODULE, &uartConfig);

    /* Enable UART module */
    UART_enableModule(EUSCI_A2_MODULE);

    while(1)
    {
        /* Send the 'g' character over UART */
        UART_transmitData(EUSCI_A2_MODULE, 'g');
    }
}

Hopefully that's helpful in understanding a little bit about configuring the serial modules on the MSP432. If you have a question or just want to yell at me for getting something completely wrong, don't hesitate to reach out.

If there's interest in learning more about the MSP432's serial, in particular setting up and using interrupts, let me know. :)

MSP432 serial printf

2015-05-17T00:00:00+10:00

While the MSP432, does seem to be an kickin' rad microcontroller, I've noticed a distinct lack of resources online to do the most basic microcontrolley stuff.

So this is a quick bit of code to get a serial/UART connection up and going from the MSP432 to your computer over the USB connection. Perfect for any debugging that you might want to do, or communication with other serial devices (for example the exciting ESP8266 which I'll cover in a later blog post).

The MSP432-printf library, which is largely from 43oh member OPossum (printf for the MSP430), lets you print to any serial module using standard printf syntax. For example: printf(EUSCI_A0_MODULE, "String one: %d", 1). The code for the library can be found on my GitHub.

An example of using this printf library with the MSP432 is shown below.

The code configures EUSCI_A0_MODULE to be used and then prints to it. EUSCI_A0_MODULE is, by default the application UART that you can connect to over USB from your PC. One thing to note is to make sure that EUSCI_A_UART_LSB_FIRST is set in the UART config, rather than EUSCI_A_UART_MSB_FIRST. According to wikipedia, MSB first serial communication is rarely used and it's much more common to use little endian or LSB first communications. If you use a program like PuTTY, it's likely that it'll interpret the serial stream as little endian.

To connect to the MSP432, I use PuTTY. On my PC, the MSP432 is on COM1. It's possible to configure and use serial modules other than EUSCI_A0_MODULE but this may involve some datasheet investigation!

AFL statistics SQL database

2015-02-21T00:00:00+10:00

The AFL stats database (and some relevant code) is on github!

Download the SQL(ite) stats database here: afl-stats.sqlite

The SQLite database has team statistics for each season and finals game for 1987-2014. In the earlier years not all stats are present as they were not all recorded. The database schema is shown below, which also shows the type of stats avaliable.

It's great for anyone nerdy who's wanting to do any sort of visualisation of stats, machine learning or just wanting to play around with the data easily. As an example of what can be quickly knocked up using Python in 10 minutes, here's the average amount of tackles in AFL games by year.

Some things I think would be interesting to examine, using the data:

How is the intensity of finals vs. normal season games captured in the statistics? Are there more tackles, is goal kicking less accurate, are there more or less free kicks etc etc?
Can historical stats be used to make a good tipping/betting system? How accurate would a machine learning algorithm based on the stats be?
How do attendances change when a team is on a hot or cold streak? Which teams have the most fair weather supporters?
Are there any teams that consistently are outliers in certain areas despite their position on the ladder? For example, are there any teams that consistently have great goal kicking accuracy regardless of their position on the ladder?
Which teams do best at home? Away from home? Are there any teams that are consistently bad/good in certain stadiums/states?

Would love to hear from you if you do anything interesting with it though. I'm sure there are lots of cool things that could be done.sda

Automatically tracking my thesis progress

2014-10-03T00:00:00+10:00

In my ongoing quest to procrastinate writing my thesis by working on marginally related work so that I still feel vaguely productive, I implemented a simple system to automatically keep track of how many words I've written in my thesis report.

Conveniently, I'm using LaTeX to write the thesis report. This allowed me to use the cool texcount script (included in most TeX distributions) to count the number of words in each of the .tex files in my thesis. The good thing about this script is it won't count the LaTeX markup towards my total word count.

Needing somewhere to store the word count, and because I've been looking for an excuse to use it, I used SparkFun's phant to create a quick and dirty data stream. The great thing about phant is once you've created the data stream, all that's needed to submit data to it is a single HTTP GET request which is doable in a single line of python. If you're that way inclined, you can see my data stream here.

Because I'm using git as version control, all that was then needed to tie everything together was a quick python script for the pre-commit git hook to count the words in each of my different sections, add them together and submit the total to my data stream on phant. This runs whenever I commit to git -- so whenever I've actually written something useful. Easy!

It was then simple to hook into the data stream with google charts. A chart with my progress is shown above, will be interesting to see how the line behaves (I'm expecting an exponential rise closer to the due date!).

Using C# to check if audio is playing in Windows

2014-08-05T00:00:00+10:00

Using the CSCore library (easily installed in Visual Studio with Install-Package CSCore) I was able to quickly knock up a prototype that let me check if audio was currently being played on my system.

The code below is heavily inspired by audio meter code in the EndPointTests file for CSCore.

using CSCore.CoreAudioAPI;
using System;

namespace AudioPlayingTest
{
    class AudioPlayChecker
    {
        // Gets the default device for the system
        public static MMDevice GetDefaultRenderDevice()
        {
            using (var enumerator = new MMDeviceEnumerator())
            {
                return enumerator.GetDefaultAudioEndpoint(DataFlow.Render, Role.Console);
            }
        }

        // Checks if audio is playing on a certain device
        public static bool IsAudioPlaying(MMDevice device)
        {
            using (var meter = AudioMeterInformation.FromDevice(device))
            {
                return meter.PeakValue > 0;
            }
        }

        static void Main(string[] args)
        {
            Console.WriteLine(IsAudioPlaying(GetDefaultRenderDevice()));
            Console.ReadLine(); //Just so the console window doesn't close
        }
    }
}

The code is hopefully self explanatory but basically gets the default system audio device, then uses the peak value from a audio meter to determine whether audio is playing.