Bitcoin core wallet.dat

Overview

In this tutorial, we will be taking a closer look at the data directory and files behind the Bitcoin core reference client. Having a better understanding of how this is managed allows us to overcome probing bitcoin's remote procedure call (RPC) and REST based interfaces for insights into the data maintained by the client.

Prerequisites

You will need access to a bitcoin node. We suggest executing against a node configured in mode so that we can have the freedom of playing with various scenarios without having to loose real money. You can however execute these against either the or configurations.

Note:
If you don't currently have access to a bitcoin development environment set up, dont' worry, we have your back! We've setup a web based mechanism which provisions your very own private session that includes these tools and comes preconfigured with a bitcoin node in mode. https://bitcoindev.network/bitcoin-cli-sandbox/
Alternatively, we have also provided a simple docker container configured in mode that you can install for testing purposes.

Getting started

Before we get started, let's have a look at the data directory of an existing running bitcoin core node.

Note
By default, bitcoind will manage files in the following locations.

Windows
Linux
Mac OS X

This default location can be overridden using the configuration parameter or by adding a parameter to the bitcoin.conf file.

A similar data directory is created for either the and configuration in sub directories assuming either of these have been configured to avoid conflicting with the files.

Only Only used in pre-0.8.0

• blktree/; block chain index (LevelDB); since pre-0.8, replaced by blocks/index/ in 0.8.0
• coins/; unspent transaction output database (LevelDB); since pre-0.8, replaced by chainstate/ in 0.8.0

Only used before 0.8.0

• blkindex.dat: block chain index database (BDB); replaced by {chainstate/,blocks/index/,blocks/rev000??.dat} in 0.8.0
• blk000?.dat: block data (custom, 2 GiB per file); replaced by blocks/blk000??.dat in 0.8.0

Only used before 0.7.0

• addr.dat: peer IP address database (BDB); replaced by peers.dat in 0.7.0

Files

As we can see, there are various files and directories which organise data behind our node, so let's take a closer look at each of these.

Some background on key store

For the purpose of this tutorial, we'll be having a closer look at the and directories and files.

We will be using LevelDB, a light-weight, single-purpose library for persistence with bindings to many platforms used by bitcoin core for storing this data.

By default, LevelDB stores entries lexicographically sorted by keys. The sorting is one of the main distinguishing features of LevelDB amongst similar embedded data storage libraries and comes in very useful for querying as we’ll see later.

A primer on leveldb

Before we look at these in more details, let's first familiarise ourselves with using nodejs.

1. Create a directory for hosting our code

1. Install the package.

1. Create a file called that contains the following code.

1. Run the script.

Great, you've just created your first level database!

A closer at the data behind leveldb

An interesting observation here will be checking the data directory created by our code.

Note
For more information on these files consult the LevelDB Documentation.

Here you should notice a similar structure as seen previously for our and directories.

Using the is great for developing applications, however, let's use a leveldb read–eval–print loop REPL utility called lev for exploring our data.

1. Install

1. Invoke our files created

1. Obtain a list of current keys stored in our database.

1. Obtain the value for the key

1. Add another key value pair to the database

1. Exit the interactive

Nice! Some additional commands we can use with include.

• GET
• PUT
• DEL
• LS - Get all the keys in the current range.
• START
• END
• LIMIT
• REVERSE - Reverse the records in the current range.

Looking at the data behind bitcoin core

Now that we've looked how level db works, let's take a closer look at our and directories.

Warning
It is recommended that you make a backup of your chaindata to avoid any accidental corruption..

Bitcoin core developer Pieter Wuille gives us a good explanation of these sections as follows.

Bitcoind since version 0.8 maintains two databases, the block index (in ) and the chainstate (in ). The block index maintains information for every block, and where it is stored on disk. The chain state maintains information about the resulting state of validation as a result of the currently best known chain.

Inside the block index, the used key/value pairs are:

• 'b' + 32-byte block hash -> block index record. Each record stores:
  • The block header.
  • The height.
  • The number of transactions.
  • To what extent this block is validated.
  • In which file, and where in that file, the block data is stored.
  • In which file, and where in that file, the undo data is stored.
• 'f' + 4-byte file number -> file information record. Each record stores:
  • The number of blocks stored in the block file with that number.
  • The size of the block file with that number ($DATADIR/blocks/blkNNNNN.dat).
  • The size of the undo file with that number ($DATADIR/blocks/revNNNNN.dat).
  • The lowest and highest height of blocks stored in the block file with that number.
  • The lowest and highest timestamp of blocks stored in the block file with that number.
• 'l' -> 4-byte file number: the last block file number used.
• 'R' -> 1-byte boolean ('1' if true): whether we're in the process of reindexing.
• 'F' + 1-byte flag name length + flag name string -> 1 byte boolean ('1' if true, '0' if false): various flags that can be on or off. Currently defined flags include:
  • 'txindex': Whether the transaction index is enabled.
• 't' + 32-byte transaction hash -> transaction index record. These are optional and only exist if 'txindex' is enabled (see above). Each record stores:
  • Which block file number the transaction is stored in.
  • Which offset into that file the block the transaction is part of is stored at.
  • The offset from the start of that block to the position where that transaction itself is stored.

Inside the chain state database, the following key/value pairs are stored:

• 'c' + 32-byte transaction hash -> unspent transaction output record for that transaction. These records are only present for transactions that have at least one unspent output left. Each record stores:
  • The version of the transaction.
  • Whether the transaction was a coinbase or not.
  • Which height block contains the transaction.
  • Which outputs of that transaction are unspent.
  • The scriptPubKey and amount for those unspent outputs.
• 'B' -> 32-byte block hash: the block hash up to which the database represents the unspent transaction outputs.
  Latest version of bitcoind(please add version compatibility) uses obfuscation of the value in key/value pair . So you need to XOR with the obfuscation key to get the real value.

Understanding the chainstate leveldb

Let's start by looking at the folder. The chainstate directory contains the state as of the latest block. In simplified terms, it stores every spendable coin, who owns it, and how much it's worth.

Note
Using this against your data appears to corrupt the file which requires restarting bitcoind with -reindex or -reindex-chainstate. It is suggested that you execute these against a backup of your bitcoin datadir.

1. LevelDB doesn't support concurrent access from multiple applications, so we'll first need to stop .

1. Make a backup of your chain data

1. Open the chainstate using the repl command.

1. Run the command.

Interesting, here we see a key called and another called . Some background on this can be found due to a pull request introduced into bitcoin core which helps overcome issues with Anti-Virus software from flagging bitcoin data as being hostile through intentionally adding virus signatures to the time chain. The obfuscation key is a 64-bit value identified by that should be XORed with each data value from the database.

Note
When setting the bitcoind field to or , we will notice the obfuscation key log entry from our file.

Writing a script for reading from the leveldb

Due to my experience with LevelDB's level library causing corruption to the database, I'd suggest making a backup of the data before executing any of these commands. I'll also be using a fresh copy of where we'll need to generate some blocks to get us going.

1. First, let's create a backup of our database.

1. Next, we create a javascript file that works based on details covered.

2. We can then run this against our backup database.

   Note
   The value for our should match that we saw earlier in our . In my local instance, this is which in the is prefixed with the value representing the ascii value for and is not reflected in the log output.

3. Start our bitcoind server, and check one of our previous blocks.

In the above example, we can see the utxo represented by its txid in little endian format leaded by a or in hex.

The value in this case is still obfuscated using the value of our keys value .

The reason for this is that the on disk storage files are often specially designed to be compact on disk, and not really intended to be easily usable by other applications (LevelDB doesn't support concurrent access from multiple applications anyway). There are several RPC methods for querying data from the databases (getblock, gettxoutsetinfo, gettxout) without needing direct access.

As you can see, only headers are stored inside this database. The actual blocks and transactions are stored in the block files, which are not databases, but just raw append-only files that contain the blocks in network format.

Decoding the values

To decode these values, using the obfuscation key.

1. Install to work with large numbers in javascript

1. Start a node in interactive mode

1. Use the big integer package to assign our previous and key value. You need to pop the character from this value and repeat it for the length of the value being decoded.

We now have the decoded version of our UTXO which can be decoded as per the instruction from here.

Personally identifiable data [v0.8 and above]

This section may be of use to you if you wish to send a friend the blockchain, avoiding them a hefty download.

• wallet.dat
  ** Contains addresses and transactions linked to them. Please be sure to make backups of this file. It contains the keys necessary for spending your bitcoins. You should not transfer this file to any third party or they may be able to access your bitcoins.
• db.log
  ** May contain information pertaining to your wallet. It may be safely deleted.
• debug.log
  ** May contain IP addresses and transaction ID's. It may be safely deleted.
• database/ folder
  ** This should only exist when bitcoin-qt is currently running. It contains information (BDB state) relating to your wallet.
• peers.dat
  ** Unknown whether this contains personally identifiable data. It may be safely deleted.

Other files and folders (blocks, blocks/index, chainstate) may be safely transferred/archived as they contain information pertaining only to the public blockchain.

Transferability

The database files in the "blocks" and "chainstate" directories are cross-platform, and can be copied between different installations. These files, known collectively as a node's "block database", represent all of the information downloaded by a node during the syncing process. In other words, if you copy installation A's block database into installation B, installation B will then have the same syncing percentage as installation A. This is usually ''far'' faster than doing the normal initial sync over again. However, when you copy someone's database in this way, you are trusting them '''absolutely'''. Bitcoin Core treats its block database files as 100% accurate and trustworthy, whereas during the normal initial sync it treats each block offered by a peer as invalid until proven otherwise. If an attacker is able to modify your block database files, then they can do all sorts of evil things which could cause you to lose bitcoins. Therefore, you should only copy block databases from Bitcoin installations under your personal control, and only over a secure connection.

Each node has a unique block database, and all of the files are highly connected. So if you copy just a few files from one installation's "blocks" or "chainstate" directories into another installation, this will almost certainly cause the second node to crash or get stuck at some random point in the future. If you want to copy a block database from one installation to another, you have to delete the old database and copy ''all'' of the files at once. Both nodes have to be shut down while copying.

Only the file with the highest number in the "blocks" directory is ever written to. The earlier files will never change. Also, when these blk*.dat files are accessed, they are usually accessed in a highly sequential manner. Therefore, it's possible to symlink the "blocks" directory or some subset of the blk*.dat files individually onto a magnetic storage drive without much loss in performance (see Splitting the data directory), and if two installations start out with identical block databases (due to the copying described previously), subsequent runs of rsync will be very efficient.

Conclusion

In this tutorial, we had a look at the files and directories behind how the bitcoin core reference client manages it's own data.

References