Let's work together!
Like my work? I'm happy to connect.
Python, Flask, SQLAlchemy
Exchange server validating signatures and bids, matching full/derived orders, executing transactions.
Centralized exchanges coordinate cryptocurrency trading on a large scale, often replicating the model set by a traditional stock exchange. Under this model, a centralized exchange will participate in the market by “clearing” trades through an order book, which lists open buy and sell orders. Here is information about a centralized cross-chain exchange implementation specifically servicing trades between Algorand and Ethereum.
Exchange Overview
Server Endpoints
Python-Flask is used to create a REST server with endpoints:
- 127.0.0.1/trade
- 127.0.0.1/order_book
- 127.0.0.1/address
Trade
The endpoint 127.0.0.1/trade provides a route which processes a POST request. The request should represent an order and must contain a JSON object including two fields, sig and payload, where payload contains fields as in the example below:
{
'sig': signature,
'payload': {
'sender_pk': public_key,
'receiver_pk': public_key,
'buy_currency': "Ethereum",
'sell_currency': "Algorand",
'buy_amount': 51,
'sell_amount': 257
'tx_id': "LR2CL4OXAMQ2EQV..."
}
}
The field sig should contain a valid signature on the JSONified dictionary payload. We check whether sig is a valid signature of json.dumps(payload) using the signature algorithm specified by the sell_currency field. The endpoint accepts signatures generated from both Ethereum and Algorand keys.
If the signature verifies, we check to see if an order is accompanied by a payment into the exchange's account. Then, the order is submitted to the Order database table, and we proceed to match and execute the order. If the signature does not verify, the orders are not inserted into the Order table. Instead, a record is inserted into the Log table, with the message field is set to be json.dumps(payload).
Order Book
The /order_book endpoint returns a list of all orders in the database (the Order table). The response is a list of orders formatted as JSON. Each order is a dict including the seven key fields referenced above (sender_pk, receiver_pk, buy_currency, sell_currency, buy_amount, sell_amount, and tx_id).
Address
We generate a key-pair for the exchange's address on both the Ethereum and Algorand blockchains. The endpoint /address accepts a (JSON formatted) request with the argument platform. /address returns a (JSON formatted) response with the exchange server's address, its public key on the specified platform (either 'Ethereum'; or 'Algorand').
Generating key-pairs
In order for the exchange to accept and transfer funds, the exchange needs an account on the platform blockchains. So, we generate key-pairs for the exchange on both the Ethereum and Algorand platforms.
Ethereum key-pairs
For Ethereum, we use:
w3.eth.account.enable_unaudited_hdwallet_features()
acct, mnemonic_secret = w3.eth.account.create_with_mnemonic()
Algorand Key-Pairs
For Algorand, you can generate accounts in many ways, e.g. through the generate_account function. We want to save the account information, so we use mnemonics which provide a convenient method for doing so:
from algosdk import mnemonic
mnemonic_secret = "YOUR MNEMONIC HERE"
sk = mnemonic.to_private_key(mnemonic_secret)
pk = mnemonic.to_public_key(mnemonic_secret)
Validating Orders
Validating Signatures
Buy/sell bids from users need to be signed by their creator, so that users cannot submit bids on behalf of others. The exchange server validates that each signature originates from the designated public key.
An account on most blockchain platforms is simply a key-pair for an Elliptic-Curve Signature algorithm. This means that it is possible to use the private key for a blockchain system to sign arbitrary messages, not just transactions. Anyone can verify that the signed message came from the owner of the specific blockchain account.
Signatures in Ethereum
Ethereum uses the ECDSA algorithm for signatures and account generation. We use the eth-account package to verify Ethereum signatures. The following code shows how to generate an account, sign a message, and verify the signature using eth-account:
import eth_account
eth_account.Account.enable_unaudited_hdwallet_features()
acct, mnemonic = eth_account.Account.create_with_mnemonic()
eth_pk = acct.address
eth_sk = acct.key
msg = "Sign this!"
eth_encoded_msg = eth_account.messages.encode_defunct(text=msg)
eth_sig_obj = eth_account.Account.sign_message(eth_encoded_msg,eth_sk)
print(eth_sig_obj.messageHash)
if eth_account.Account.recover_message(eth_encoded_msg,
signature=eth_sig_obj.signature.hex()
) == eth_pk:
print("Eth sig verifies!")
Signatures in Algorand
Algorand uses the EDD25519 signature algorithm for its accounts. We use the py-algorand-sdk library to generate accounts, sign messages and verify signatures on the Algorand platform.
import algosdk
msg = "Sign this!"
algo_sk, algo_pk = algosdk.account.generate_account()
algo_sig_str = algosdk.util.sign_bytes(msg.encode('utf-8'),algo_sk)
if algosdk.util.verify_bytes(msg.encode('utf-8'),
algo_sig_str,
algo_pk):
print("Algo sig verifies!")
Checking for Payment
When a user wants to trade using the exchange, they need to send a payment to the exchange. We check to see if the order is accompanied by a payment into the exchange's account.
When a user submits an order to the endpoint /trade the submission data should have a tx_id field, which specifies the transaction ID (sometimes called the transaction hash) of the transaction that deposited tokens to the exchange.
The tx_id should correspond to a transaction ID on the blockchain specified by sell_currency. In order to see if the order is valid, the exchange server checks that the specified transaction actually transferred sell_amount to the exchange's address. In particular, we check:
- The transaction specified by tx_id is a valid transaction on the platform specified by sell_currency.
- The amount of the transaction is sell_amount.
- The sender of the transaction is sender_pk.
- The receiver of the transaction is the exchange server (i.e., the key specified by the /address endpoint).
The exact method for checking the details of a transaction is different on the Ethereum and Algorand blockchains.
On Algorand, the V2 indexer has a method search_transactions that provides the desired functionality. On Ethereum, the get_transaction function provides a convenient way to search transactions by tx_id.
tx = w3.eth.get_transaction(eth_tx_id)
The Database
The database allows us to track and match currency exchange orders. It contains enough information to show all orders that have been submitted, and which were matched and filled.
Representing Orders
We represent orders as a dictionary with fields as in the example below. sender_pk is the public-key of the sender (on the platform specified by sell_currency), and receiver_pk is the key (also controlled by the order's originator) where tokens on the platform buy_currency will be sent if the order is filled.
{
'buy_currency': "Algorand",
'sell_currency': "Ethereum",
'buy_amount': 1245.00,
'sell_amount': 2342.31,
'sender_pk': 'AAAAC3NzbC1lZDI1NTr...',
'receiver_pk': '0xd1B77a920B0c50...',
'tx_id': "LR2CL4OXAMQ2EQV..."
}
Creating the Database
SQLAlchemy is used to handle the database operations. The database schema can be found in models.py. Running python3 models.py will create the file orders.db which contains the actual SQLITE3 database file. To interact with the database, the following headers are included:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from models import Base, Order
engine = create_engine('sqlite:///orders.db')
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
Matching Orders
When a new order comes in, the system tries to fill the order by matching it with an existing order (or orders) in the database. More specifically, we:
- Insert the order into the database.
- Make sure the user with a given pk exists in the User table.
- Insert the order into the Order table.
-
Check if there are any existing orders that match.
Each order can match at most one other.
Given new_order and existing_order, in order to match:
- existing_order.filled must be None
- existing_order.expiration must be None or a time in the future.
- existing_order.buy_currency = order.sell_currency
- existing_order.sell_currency = order.buy_currency
-
The implied exchange rate of the new order is
at least that of the existing order:
existing_order.sell_amount / existing_order.buy_amount >= order.buy_amount / order.sell_amount - The buy/sell amounts need not match exactly.
- If a match is found between order and existing_order, we:
- Add a record to the Matches table including the tx_ids of the two matching orders. The maker is the existing order, the taker is the new order.
- Set filled field to be the current timestamp on both orders.
- Set counterparty_id to be the tx_id of the other order.
-
If one of the orders is not completely filled
(the counterparty's sell_amount is
less than buy_amount), we:
- Create a new derivative order for the remaining balance.
- Set the new order's created_by field to the tx_id of its parent order.
- Set the new order's pk and platform as the same as its parent order.
- Set the sell_amount of the new order such that the implied exchange rate of the new order is at least that of the old order i.e., buy_amount/sell_amount on the new order is no bigger than on the order that created it.
- Then try to fill the new order (repeat the process from the beginning).
Note that this is a custodial exchange, so the exchange acts as a custodian for the sell_amount of sell_currency for every order. The exchange is responsible for transferring the buy_amount of buy_currency from its own reserves. Thus, the total discrepancy in the filled orders between the sell_amounts (paid to the exchange) and the buy_amounts (paid by the exchange) results in profit for the exchange.
Executing Transactions
The server uses a private Ethereum network, as well as a public Algorand testnet to execute trades on the Ethereum and Algorand blockchains.
Ethereum Transactions
We send transactions on the Ethereum blockchain using send_raw_transaction. In Ethereum, the computational complexity of a transaction is measured in gas, which we have to provide for the exchange. web3.eth provides a function to the estimate gas cost of any type of transaction. You can find the gas cost of basic operations in the Ethereum Yellow Paper (Appendix G). The total cost of the transaction is the gas price multiplied by the amount of gas used.
tx_dict = {
'nonce': w3.eth.get_transaction_count(sender_pk,"pending"),
'gasPrice':w3.eth.gas_price,
'gas': w3.eth.estimate_gas(
{
'from': sender_pk,
'to': receiver_pk,
'data': b'',
'amount': tx_amount
}
),
'to': receiver_pk,
'value': tx_amount,
'data':b''
}
signed_txn = w3.eth.account.sign_transaction(tx_dict, sender_sk)
tx_id = w3.eth.send_raw_transaction(signed_txn.rawTransaction)
Separating the gas price from the amount of gas used allows the gas price to change with the network congestion, while tying the amount of gas to the complexity of the transaction.
Ethereum Nonces
Since Ethereum uses the account balance model (rather than the UTXO model), every transaction needs a unique nonce to prevent replay attacks. In Ethereum, the nonces are sequential, and the nonce used for a transaction is essentially the number of transactions that originated from that address. The miners keep track of the number of transactions per address, and this can be accessed using the get_transaction_count function. Adding the keyword pending tells the function to also count the pending transactions. Unfortunately, this function may not include transactions that were recently submitted. Thus, if you use the following:
def send_eth(sender_pk,sender_sk,receiver_pk,tx_amount):
tx_dict = {
'nonce': w3.eth.get_transaction_count(sender_pk,"pending"),
'gasPrice':w3.eth.gas_price,
'gas': w3.eth.estimate_gas(
{
'from': sender_pk,
'to': receiver_pk,
'data': b'',
'amount': tx_amount
}
),
'to': receiver_pk,
'value': tx_amount,
'data':b'' }
signed_txn = w3.eth.account.sign_transaction(tx_dict, sender_sk)
tx_id = w3.eth.send_raw_transaction(signed_txn.rawTransaction)
return tx_id
for i in range(10):
send_eth(sender_pk,sender_sk,receiver_pk,tx_amount)
It will likely fail because get_transaction_count will return the same nonce multiple times. Instead, we keep track of the nonce locally.
def send_eth(sender_pk,sender_sk,receiver_pk,amounts):
starting_nonce = w3.eth.get_transaction_count(sender_pk,"pending")
tx_ids = []
for i,tx_amount in enumerate(amounts):
tx_dict = {
'nonce': starting_nonce+i, #Locally update nonce
'gasPrice':w3.eth.gas_price,
'gas': w3.eth.estimate_gas(
{
'from': sender_pk,
'to': receiver_pk,
'data': b'',
'amount': tx_amount
}
),
'to': receiver_pk,
'value': tx_amount,
'data':b''
}
signed_txn = w3.eth.account.sign_transaction(tx_dict, sender_sk)
tx_id = w3.eth.send_raw_transaction(signed_txn.rawTransaction)
tx_ids.append(tx_id)
return tx_ids
tx_amounts = [tx_amount for _ in range(10)]
send_eth(sender_pk,sender_sk,receiver_pk,tx_amounts)
Algorand Transactions
To create payments on the Algorand blockchain we:
- Use algo-sdk to create a Payment Transaction
- Use algo-sdk to sign the transaction with the secret key
- Use algo-sdk to send the signed transaction to the blockchain
Sending Algorand Transaction to the Testnet
Sending a transaction to the blockchain requires several parameters beyond the sender, receiver, and amount. These include:
- gen: genesis hash, this makes it clear which chain you're using.
- first and last: first valid round and last valid round, the transaction will be rejected if the current round number is not in this range.
- fee: transaction fee, how much the validator gets for validating the transaction.
In order to get reasonable values for these parameters, we use the suggested_params function.
from algosdk.future import transaction
params = acl.suggested_params()
unsigned_tx = transaction.PaymentTxn(
sender_address,
params,
receiver_address,
amount
)
Nonces in Algorand
Like Ethereum, Algorand uses the account balance model (rather than the UTXO model). Unlike Ethereum, Algorand does not use transaction nonces to prevent replay attacks. If you submit the same transaction twice, it will have exactly the same transaction id (the transaction id is essentially the hash of the transaction ), and the block producers will reject any transaction with a transaction id that has already been processed.
But there may be a case where you need the same transaction twice (ex. a recurring payment). Algorand signatures are deterministic (Algorand uses Ed25519 signatures unlike Bitcoin and Ethereum which use ECDSA), so if the transaction details are the same, the signatures will be the same, and thus the transaction ids will be the same as well.
To handle this, Algorand uses parameters first, first valid round, and last, last valid round, that specify a range of rounds (block heights) where the transaction could be included. By default, the first valid round is the current round, and the last valid round is 1,000 rounds later, but they can be set explicitly. You can also use them to bulk sign transactions that will only be valid at some future date.
from algosdk.future import transaction
params = acl.suggested_params()
tx1 = transaction.PaymentTxn(
sender_address,
params,
receiver_address,
amount
)
stx1 = txn1.sign(sender_sk)
acl.send_transaction(stx1)
#Second send fails
tx2 = transaction.PaymentTxn(
sender_address,
params,
receiver_address,
amount
)
stx2 = txn1.sign(sender_sk)
acl.send_transaction(stx2)
We can send the transaction if we modify the first or last valid round:
from algosdk.future import transaction
params = acl.suggested_params()
tx1 = transaction.PaymentTxn(
sender_address,
params,
receiver_address,
amount
)
stx1 = txn1.sign(sender_sk)
acl.send_transaction(stx1)
#Second send succeeds
params.first += 1
tx2 = transaction.PaymentTxn(
sender_address,
params,
receiver_address,
amount
)
stx2 = txn1.sign(sender_sk)
acl.send_transaction(stx2)
Recording payments from the exchange in the database
When a transaction is successfully executed, i.e., when the exchange sends tokens to two counterparties after matching an order, the exchange records the executed information in the database, in the transactions table (the table TX). This includes:
- platform: either 'Ethereum' or 'Algorand'.
- receiver_pk: the address of the payee, i.e., the recipient of the tokens.
- order_id: the id of the order (in the Order table) that generated the transaction.
- tx_id: the transaction id of the payment transaction from the exchange on the platform specified by platform.
Statistics
We can use database queries to calculate statistics about how much the exchange is responsible for paying, and thus how much profit it makes.
Gross deposits to the exchange
Since this is a custodial exchange, a user is expected to deposit the full sell_amount into the exchange's account when it initially places an order. Derived orders (created by an unfilled portion of an existing order) are not accompanied by another deposit from the user. Since derived orders must have a creator, we can get the total amount of currency taken in by the exchange with the following queries:
algo_total_in = sum(
[
order.sell_amount
for order in session.query(Order).filter(Order.creator == None).all()
if order.sell_currency == "Algorand"
]
)
eth_total_in = sum(
[
order.sell_amount
for order in session.query(Order).filter(Order.creator == None).all()
if order.sell_currency == "Ethereum"
]
)
Net deposits to the exchange
If the exchange were to shut down, it would have to refund the sell_amount to the users for every unfilled order. These liabilities can be calculated:
algo_unfilled_in = sum(
[
order.sell_amount
for order in session.query(Order).filter(Order.filled == None).all()
if order.sell_currency == "Algorand"
]
)
eth_unfilled_in = sum(
[
order.sell_amount
for order in session.query(Order).filter(Order.filled == None).all()
if order.sell_currency == "Ethereum"
]
)
Thus, the net deposits to the exchange can be calculated:
print( f"Eth in = {eth_total_in-eth_unfilled_in:.2f}" )
print( f"Algo in = {algo_total_in-algo_unfilled_in:.2f}" )
Payouts from the exchange
Whenever an order is filled, the exchange is responsible for sending payments to the counterparties. If the order is completely filled, the exchange must send buy_amount to each counterparty. If the order is only partially filled, the buyer must send buy_amount minus its derived order's buy_amount. Thus, we can calculate the net payments by the exchange:
eth_out = 0
algo_out = 0
#Get all filled orders
orders = session.query(Order).filter(Order.filled != "").all()
for order in orders:
if order.sell_currency == "Algorand":
eth_out += order.counterparty[0].buy_amount
if order.counterparty[0].child:
eth_out -= order.counterparty[0].child[0].buy_amount
if order.sell_currency == "Ethereum":
algo_out += order.counterparty[0].buy_amount
if order.counterparty[0].child:
algo_out -= order.counterparty[0].child[0].buy_amount
print( f"Eth out = {eth_out:.2f}" )
print( f"Algo out = {algo_out:.2f}" )
Net profits of the exchange
We can also calculate the net profits of the exchange by calculating how much the exchange took in, and subtracting how much it has to pay out:
eth_in = 0
eth_out = 0
algo_in = 0
algo_out = 0
orders = session.query(Order).filter(Order.filled != "").all()
for order in orders:
if order.sell_currency == "Algorand":
algo_in += order.sell_amount
eth_out += order.counterparty[0].buy_amount
if order.counterparty[0].child:
eth_out -= order.counterparty[0].child[0].buy_amount
if order.child:
algo_in -= order.child[0].sell_amount
if order.sell_currency == "Ethereum":
eth_in += order.sell_amount
algo_out += order.counterparty[0].buy_amount
if order.counterparty[0].child:
algo_out -= order.counterparty[0].child[0].buy_amount
if order.child:
eth_in -= order.child[0].sell_amount
print( f"Eth profits = {eth_in-eth_out:.2f}" )
print( f"Algo profits = {algo_in-algo_out:.2f}" )
See the code
Access to private GitHub repository can be provided upon request.