In this article I would like to start describing more specifically, with examples and explanations, everything that was in the article above. I don’t know about you, but it was always easier for me to learn from concrete examples. That’s why I offer you an experiment: we will create a cryptovite from scratch with the whole channel!
All the source code that we will describe will be at github. There you can also download node and run it on your computer: search the code, add something, make a fork and eventually create your own cryptographic software. Obviously, it will be a slightly shortened crypt, in order to save time, specifically for learning the basics, but in the future it can be added.
The first step of any cryptovoltaic is to define the data that will run within the network.
The data list is as follows:
We need to determine what the transaction is, what the block is and what the addresses look like and what format they have. All the work of the cryptovolta in general will depend on it.
For example, bitcoin transactions look like a sequence of the following parameters:
- vector array of inputs tx_in
- vector output array tx_out
- Variable Lock_time (i.e. lock by time)
As for the transaction version: this field is required for compatibility. It is possible that something will change in future versions of the transaction, as, for example, segwit (segregate witness) support has been added to the bitcoin. This has made it possible to make transactions with signatures and transactions without signatures, which has made it possible to store information about signatures. That is, the bulk of the data array that is stored in the bitcoin network. Thus, nodes can both store information about transactions and not, which significantly reduces the size of the blocklock, because signatures are about 70% of the data stored by nodes. Accordingly, this segwit has created the need for two transaction hashes. One hash, as in the original version of transactions, is generated from the entire transaction, while the second hash is generated from a transaction without signatures.
The signature is in the main transaction fields. The result is two hashes associated with one transaction. Therefore, knowing the version of the transaction, we can understand what we are dealing with. If the transaction does not contain a signature, we can go to the blockcheck to get the signatures for that transaction. This is a very brief and generalized description of segwit. Of course, there are still a lot of things there, but in the context of transaction naming it looks like this.
Transactions with and without segwit have different meaning in the version field. In addition, the software in the network (application of updates) directly related to changes in the structure of transactions – directly affects the value of the version field.
The input array determines from which unused outputs we will spend. In general, unused outputs (UTXO) is a topic for a separate article, but in brief: unused outputs are related to your coins that have not yet been spent. In order to spend them, you must specify the transaction by which you were sent the coins and the index in the list of outputs of the previous transaction.
That is, you need to specify which transaction (hash) and output (index) the coins were sent to you, and sign this data with your private key. The hash:index link is UTXO and communicates with the address to which the coins are going. Accordingly, all addresses on the network that have at least some coins – have such bindings in the UTXO index. Accordingly, each tx_in element looks like a transaction hash and an exit index in that transaction, where the coins you are going to spend – were received, as well as a digital signature from the transaction hash and the public key of the private with which the data was signed – to verify the signature.
Since the coins are sent to the public key (the address is ultimately a public key, though on the bitcoin network is a bit modified hash), the private key signature of the coin recipient’s private key ensures that the coins can actually be spent by that participant. The login array forms the amount we spend in this transaction.
Simply put, the output array is a list of recipients and amounts that we send. The list of recipients to whom we need to send, and the number of coins to be sent. It is from this array that the future UTXO is formed when the transaction enters the network.
It turns out that the transaction inputs and outputs form the input and output sums respectively, and the difference between these sums is the miner commission.
The last parameter described in the bitcoin protocol is lock_time, the so called lock variable. It indicates from which block this transaction can be unlocked and added to the common network or from which time. Everything depends on the value of this value. Now by default this value is set to 0 and not used, but it can be used and if it is less than 500 000 000, it will mean the height of the block, up to which this transaction is locked and cannot be added to the network. If the value is greater than this number, it means the time in unixtime format after which this transaction will be unlocked and added to the nearest block.
With the transactions clearly, let’s move on to the blocks. I will describe the blocks in the bitcoin network, and then we will move on to the implementation of our cryptographic currency.
The information about the block is divided into two parts: the block header and the block body. With the block body, everything is simple – it is a list of transactions that are included into the block. With the block header, everything is much more complicated. Okay, here we go. The fields of the block are:
- Previous block hash,
- So called merkle root, the root of the tree is merkle;
- timestamp, that is the time when this block was produced and added to the network;
- Bits field, which describes the complexity of creating this block in consensus;
Random number needed for mining and validation.
The version field is the same as the transaction field. From version to version in nodes the format of the block may change. Therefore, the version is necessary for further possibilities to change the network and the communication format. For example, official bitcoin clients signal that an unknown version of the block has arrived if something has changed in the network. The last time this was the case was when a bitcoin witness was introduced. And those people who didn’t have the client updated received a notification that they needed to update.
Prev Block hash
The reference to the previous block allows to understand at what height the block should be added to the current blockchain, and also to check if the added block is included in the blockchain. Whether its complexity is appropriate, whether there is a double waste relative to the previous blocks, the block time should be less than the previous one, and many other parameters that can be described in a separate article…
The so-called merkle root, or hash tree root, is a binary tree whose leaf tops contain hash transactions recorded in the block body. Internal vertices contain hashes of addition of values of child vertices. Thus, the root node contains all data set in this tree. It describes the sum of all transactions, and also allows you to get an imprint of all transactions in the block. Why is it necessary? This structure allows you to get information about whether any transaction is contained in a block without listing all the transactions. This is necessary for so-called narrow clients who do not download the full blockchain, and download headers without the body of the block. Therefore, when they receive a transaction they can check if it belongs to a block without downloading the entire blockchain. If there are four transactions, then to check one transaction for belonging to the merkle tree, you need to ask the full node for only two intermediate values of that tree. In addition, the root of the hash tree allows you to effectively describe all transactions with a single hash, which speeds up the work of the merkle tree. This eliminates the need to go through all the transactions, which optimizes the overall network.
The time field of the block is clear, there’s no point in stopping at it in detail. It obeys some processing and validation rules: it should not be less than the previous block, it should not be more than the existing time. True, with some limitations. Since it is a decentralized network, there are time corrections which allow (strangely!) to send a block from the past or from the future. But a very old block will not pass, and neither will a very new block. Since all the information in the header is linked, you will have to change the whole block and look for it again when the time changes.
There are two parameters left to discuss: bits – a parameter for mining, and nonce – a random number. I suggest to consider the method of forming the block by the miner, because it depends on this last two parameters.
Block formation, bits, nonce
In general, when the miner receives a job to add a new block, it already has a list of transactions from the mempole. All but the first one is a coinbase transaction that has no signatures and is only needed to send system information about the software, where to send the reward to the maintainer and so on. It is necessary, system and non-standard.
I deliberately missed an important part of Bitcoin Blockchain – the scripting language. Bitcoin initially contains the ability to create smart contracts. Inside this payment system there is a stack-oriented programming language, an example of which I will give below. All examples are described in a special language, which allows making payments more “smart”. But due to the fact that this language is rather complicated and error probability is rather high, this possibility was closed by developers. Some standard forms of this scripting language were used, which I will describe below. All other forms are prohibited, and all transactions with these forms will be rejected by the network and other participants. Therefore there is no point in describing this scripting language (at least in this article). But it is used in the transaction signature as well as in the recipient field, in the transaction output list this scripting language is also used. And there are examples of standard forms data:
OP_HASH160 OP_DATA_20 OP_EQUAL
OP_1 OP_DATA_65 <…> OP_DATA_65 OP_3 OP_CHECKMULTISIG
Bitcoin has been sorted out, now let’s move on to our cryptovite and compile data formats for it. It’s the culmination of our article.
I suggest we get rid of some fields.
But first, what I think should be left behind. First of all, I would leave the version as there may be changes in consensus in the future. For the nodes to know about another version of the block transaction, the version field is necessary.
Next, I would leave a list of transaction inputs, but I would change it: instead of the intricate bitcoin format, I would only leave four values:
- Input transaction hash;
- The exit index of the transaction from which the coins we are currently sending are received;
- Signature, which is associated with the address to which the coins were received;
- The public key to the private key to which we received the coins.
The array of transaction outputs remains unchanged, it is an amount and address. So, I changed the format in the transaction list and removed the Lock_time field because it is not used anyway. The block fields also remain unchanged because they are necessary. I suggest that you just use the public key as the address, thus simplifying the implementation. Besides, I will remove the script language with which everything is described in bitcoin.
Next, it is necessary to determine how this data will be converted to a hash, that is, how the hash from the block header and the hash from the transaction will be obtained. This is necessary in order to get the correct hashes from data sets sorted in different ways, for example. Of course, you could have used a hash from a json object, but if you change a couple of keys, the correctness of the json object will not change, but the hash can change completely. I suggest the following format: all data provided in it is converted to binary format, grouped together and hashed with sha256 algorithm.
It cannot be sent over the network separately, it can only come in a new unit.
So, the miner takes the information it already has about the existing transactions, adds the first transaction to the block body – coinbase transaction with its own data, generates merkleroot for the updated list of block transactions and generates the header of the block hash(version, prevblock, merkleroot, time,bits,nonce), where nonce is 0. After that it starts searching for the block by searching for the nonce value.
What is block search: the miner takes a byte sequence derived from the connection of all the above parameters and starts to pick a random nonce number until it finds a final hash that matches the conditions. A remark must be made here: the hash generation algorithm is, in fact, an algorithm tied to the law of large numbers and probability theory. The chance that a number will fall less than N – is multiplied by increasing N – this is what the mining algorithm is built on. The bits field controls the total N for the whole network – less than which there should be a hash.
The essence of the algorithm is that we take a list of the last n blocks of our blockchain and check the average time of block creation. If the block time is less than the calculated one (for a bitcoin it is 1 block in 10 minutes), we increase bits, thus making it more difficult to find a hash. If the time is longer than the calculated time, the complexity decreases proportionally to the time, thus making it easier to find a new hash and a new block for the miner (I will describe the process itself in a separate article, do not switch). Thus, the information for hash checking is written to the header block, from which this hash is generated. You don’t need a full blockcheck to check the block, you just need to know the hash and the bits field from which you can get the complexity and see if the complexity is suitable for this hash. Both hash and complexity are large numbers unsigned int 128. So, the complexity check is just a two-number comparison operation. Big numbers, but numbers. The block search is a hashing operation of our hash with a change in the last field, the field of a random number, nonce. The change occurs every millisecond (or more often, depends on the hash of the device), until the final hash fits the complexity. When this happens, the miner sends the block to the network, and all network members can check the block. This is the algorithm for compressed and summary mining. It turns out that this algorithm can be done with a normal pen on a notebook patch, and this does not require any computing tools, but then the speed of adding a new block to the network will, as you understand, much lower. Therefore computers are used!
The third block of data is addresses. For example, in a blockchain, addresses are just a public key. In a bitcoin blockchain, the address is an address encoded in the base58 hash from the public key (one of the options). This was done to maintain privacy and anonymity, but current realities allow us to determine the sources of the transaction. This measure is now considered redundant. In a more global sense, the address in bitcoin is scriptPubKey, a script for the input key, which can be quite diverse but uses only a couple of options. The base58 encoding was chosen for a reason. It allows you to select the entire address with a mouse click. Unlike the common base64, base58 does not contain any lowercase and some other characters that separate a mouse click into two parts. Bitcoin’s creator, Satoshi Nakomoto, envisioned this feature initially and made addresses more readable and convenient. The bitcoin address contains a checksum to ensure that the address is correct. In addition, it also contains a version of the address, which is very helpful in creating segwit addresses. As I’ve already written, segwit allows you to split a blockchain into two parts: with and without transaction signatures, but leave them bound. Addresses of a different version are used for a blockchain without a signature. Those who get bitcoins immediately understand which version and which protocol to use. This is the essence of the address version. Perhaps when you send or receive a bitcoin you have seen that there are addresses that start with a threesome. This is the version of the address.
Final transaction format
Final block format
Final data serialization format
Version + prev + merkle + time + bits + nonce + txlist
Version + tx_in + txout
Hash + index + sign + pubkey
Address + amount
Separately, it’s worth mentioning the data types. Uint[8-64] are numbers with different bits, for example uint8 – 1 byte, uint16 – 2 bytes, etc. Char[n] is a string of fixed length n.
Var_int – is a special field, which means a number of variable length. This field can store both uint8 and uint64 depending on the value. If the value is less than 0xfd, it is uint8, if it is less than 0xffffff – uint16, if it is less than 0xffffff – uint32, and it is more than uint64. Thus it’s a compact form of writing numbers in byte sequences, which is relevant for bitcoin protocol, where light clients with limited traffic can participate, for example, phones – where each byte is on the account.
Var_str is a string of variable length, it looks like count(var_int)+str(char). Where char length is specified in count parameter and can be arbitrary.
Besides, vector formats are var_int + type_data. I.e., the first step is specifying the number of elements of type, and then there is the enumeration of these elements.
All examples with working source code can be viewed at github:
Part one code:
Learn more about this code. We have created a chain module in our application which so far has only 2 block and tx submodules which are available in the whole application via app.chain.TX/app.chain.BLOCK. At the moment these submodules have only a couple of serialize and unserialize methods, which pack data into byte-string and unpack into json respectively.
Itog: within the framework of this article we have defined the format of communication of our nodes, the format of communication in the network, how it happens at bitcoin, what fields the block contains and what addresses consist of.