Introduction to trader botswith Python

Thomas Aglassinger

http://roskakori.at@Taglassinger

https://github.com/roskakori/talks/tree/master/pygraz/traderbot

Overview

● Outline a simple trading bot

● Example usages for several Python modules(most part of the standard library)

● (Probably) Python beginner friendly

Agenda

● Communication and configuration

● Trading

● Data analysis and debugging

● Testing

Limitations

● You won't get rich (but it's fun nevertheless)

● Code does not work anymore due API update

● Masterexchange is going to shut down soon

● Terms of service

Communication and Configuration

Overview

● Uses https://masterxchange.com/api.php (defunct after 2015-11-15)

● Communicate using HTTPS and JSON

● Public queries available to anyone (e.g. currentbids)

● Private queries requiring a personal tokenbound to you account (e.g. orders)

Query trades

● Query the last 500 trades for maidsafe coins:https://masterxchange.com/api/v2/trades.php?currency=maid

● Result: [{"tradeid":"31242","price":"0.00990000","amount":"0.15110800","date":"1446399439","market":"msc_btc"},{"tradeid":"31241","price":"0.00990000","amount":"0.09989200","date":"1446319270","market":"msc_btc"},{"tradeid":"31240","price":"0.00562223","amount":"0.03779028","date":"1446309947","market":"msc_btc"}, ...]

Print the top 3 trades

import jsonimport requests

def _without_utf8_bom(text): return text[3:] if text.startswith('\xef\xbb\xbf') else text

query = requests.get( 'https://masterxchange.com/api/v2/trades.php', headers={'User-Agent': 'demobot/0.1'}, params={'currency': 'maid'})print('query.status_code =', query.status_code)if query.status_code < 300: query_text = _without_utf8_bom(query.text) print('query_text = %r...' % query_text[:40])

trades = json.loads(query_text) print('trades =', trades[:3])

Print the top 3 trades - result

query.status_code = 200

query_text = '[{"tradeid":"31246","price":"0.00002603"'...

trades = [{'market': 'maid_btc', 'date': '1446500342', 'amount':'7000.00000000', 'price': '0.00002603', 'tradeid': '31246'}, {'market':'maid_btc', 'date': '1446489311', 'amount': '22000.00000000','price': '0.00002655', 'tradeid': '31244'}, {'market': 'maid_btc', 'date':'1446462486', 'amount': '1250.00000000', 'price': '0.00002655','tradeid': '31243'}]

Configuring the API key

● Private queries require an API key.

● Simple way to manage: configparser

● Store key in a config file

● Read it during startup

Example config file

[demobot]

api_key = ou1IurT4HQrFfN1ch...

Read the API key from the config

import configparser

config = configparser.ConfigParser()config.read('demobot.cfg')api_key = config.get('demobot', 'api_key')

Query your balances

https://masterxchange.com/api/v2/private/balances.php?APIkey=ou1IurT4H...

{"balances":{"total":{"btc":"9.16311816","msc":"34.63724456","maid":"43233.50000000"},"available":{"btc":7.16311816,"msc":26.63724456,"maid":42633.5}},"error_message":"","error_code":0}

Print your balances

query = requests.get( 'https://masterxchange.com/api/v2/private/balances.php', headers={'User-Agent': 'demobot/0.1'}, params={'APIkey': api_key})print('query.status_code =', query.status_code)if query.status_code < 300: query_text = _without_utf8_bom(query.text) print('query_text = %r...' % query_text[:40])

balances_result = json.loads(query_text) if balances_result['error_code'] == 0: balances = balances_result['balances'] print('balances =', balances)

Print your balances - result

query.status_code = 200

query_text = '{"balances":{"total":{"btc":0,"msc":0,"m'...

balances = {'total': {'xcpjenga': 0, 'colzoidy': 0,'coltdu': 0, 'xcpopcu': 0, 'colgauto': 0, ...}}

Masterexchange error handling

1.Http status < 300?

2.Query result error_code = 0?

3.Process actual data in query result

Wrap error handling in Exceptions

class BotError(Exception): pass

class HttpsConnection(object): ... def query(self, function_name, payload=None, use_api_key=True): function_url = 'https://masterxchange.com/api/v2/%s.php' % function_name actual_payload = {} if payload is None else dict(payload) if use_api_key: actual_payload['APIkey'] = self._api_key headers = {'User-Agent': 'demobot/0.1'} r = requests.get(function_url, headers=headers, params=actual_payload) if r.status_code >= 300: raise BotError( 'cannot query %s with %s: HTTP error code=%d' % (function_url, actual_payload, r.status_code)) result = json.loads( r.text[3:] if r.text.startswith('\xef\xbb\xbf') else r.text) if use_api_key and (result['error_code'] != 0): raise BotError( 'cannot query %s with %s: %s' % (function_url, actual_payload, result['error_message'])) return result

Trading

Processing monetary values

● Use decimal instead of floathttp://floating-point-gui.de/

● Python has a decimal module:https://docs.python.org/3/library/decimal.html

● json and configparser only support float→ convert after reading and before writing

● Bitcoin uses 8 digits after decimal separator

Use formatting for decimals

>>> from decimal import Decimal

>>> print(Decimal('0.00000001'))

1E-8

>>> print('%.8f' % Decimal('0.00000001'))

0.00000001

Modes of operation

● Advise: only suggest to sell or buy → user has to manually initiatetransactions

● Lower risk for “stupid” transactions

● Might miss opportunities due slow reaction time

● Helpful when trying out a hopefully improved trading algorithm

● Action: automatically sell and buy on market conditions deemedfavorable

● Can react quickly to changes

● Possibility for epic fail on buggy trading algorithms

● Recommendation: reduce risk (but also opportunities) by limitingamount traded per transaction and hour, stop loss limits etc.

Basic bot loop

1.Update own balances

2.Update open orders on the market

3.Apply trading algorithm and decide next action

4.Possibly buy or sell

5.Wait some time

6.Repeat

Trading algorithms

On the long run, nothing really works

Some simple trading algorithms

● Spread between 2 different but interchangeable items;e.g. Team Fortress 2's keys and earbuds:http://icrontic.com/article/tf2-black-market-explained

● Delayed correlation between two items; e.g. stocks forCoca Cola and Pepsi:http://www.investopedia.com/university/guide-pairs-trading/pairs-trading-correlation.asp

● Wait for slips from sellers, buy “unusually” cheap itemsand resell for “normal” price;article about such a bot (violating terms of service):http://diablo3story.blogspot.com.au/2014/07/a-diablo-3-story.html

Data analysis and Debugging

Tracking statistics

● Collect statistics in database

● To debug bot decisions

● To improve trading algorithm

● To monitor market conditions

Sqlite

● Robust and stable

● Efficient for single client use

● Easy to set up

● Included with Python:https://docs.python.org/3/library/sqlite3.html

● Rather creative type system

● “Real” instead of “decimal”

● “int” for timestamp instead of “datetime” type

● Type anarchy concerning comparison

● http://www.sqlite.org/datatype3.html

Create a statistics database

def _create_database(self, database_path): _log.info('connect to database %r', database_path) result = sqlite3.connect(database_path) with closing(result.cursor()) as cursor: cursor.execute(""" create table if not exists balances ( action char(4) not null, balance_time int not null, btc real not null, maid real not null, price_per_maid real not null, transferred int not null ) """) cursor.execute(""" create index if not exists idx_balance_time on balances (balance_time) """) result.commit() return result

Insert a statistics row

values_to_insert = ( action, int(time.time()), float(self.btc_balance), float(self.maid_balance), float(price_per_maid), int(maid_transferred),)with closing(self._database.cursor()) as cursor: cursor.execute(""" insert into balances ( action, balance_time, btc, maid, price_per_maid, transferred ) values (?, ?, ?, ?, ?, ?) """, values_to_insert)self._database.commit()

Logging

● More detailed tracking of trading decisions thandatabase

● But no easy structured analysis

● Use logging Modulehttps://docs.python.org/3/library/logging.html

● Use RotatingFileHandlerhttps://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler

Example logging configuration 1/2

# Logging configuration as described in

# <https://docs.python.org/3/howto/logging-cookbook.html>.

[loggers]

keys=root,demobot

[handlers]

keys=console,file

[formatters]

keys=default

[logger_root]

level=DEBUG

handlers=console,file

[logger_demobot]

level=DEBUG

handlers=console,file

qualname=demobot

propagate=0

Example logging configuration 2/2

[handler_console]

class=StreamHandler

level=INFO

formatter=default

args=(sys.stderr,)

[handler_file]

class=RotatingFileHandler

level=DEBUG

formatter=default

args=('/tmp/demobot.log', mode='a', maxBytes=1000000, backupCount=5, encoding='utf-8')

[formatter_default]

format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

datefmt=

Testing

Testing challenges

● Network communication is slow

● Many sites have limits on transactions persecond

● Testing actual orders requires money

Mock-up connections with scenarios

● Scenarios are simple text file containingexpected queries and JSON results

● The test case makes a decisions that results ina query, parses the scenarios JSON result, andmakes the next decision and query

● Scenarios can be maintained by domainexperts

Example scenario file# got maid, no open orders

# the bot should create a maid-order for 500maid and 0.10000001btc/maid

private/balances {

"balances": { "total":{"btc":0.9,"maid":500}, "available" {"btc":0,"maid":0} },

"error_message":"", "error_code":0

}

private/openedOrders {

"open_orders": [],

"error_message":"", "error_code":0

}

orderbook [

{"market":"maid_btc","type":"sell","amount":"120.00000000","price":"0.20000000","date_created":"1401077847"},

{"market":"maid_btc","type":"buy","amount":"270.00000000","price":"0.10000000","date_created":"1418566454"}

]

private/createOrder

Scenario implementation

● Bot constructor gets a connection

● Class HttpConnection → query() returns JSONfrom Masterexchange

● Class ScenarioConnection → query() checksthat function matches next line in scenario fileand if so returns next JSON from it

Audience feedback: try Gherkin!https://pypi.python.org/pypi/gherkin3

Summary

Summary

● Python has many useful libraries to quicklyimplement simple trading bots.

● You probably won't get rich.

● It's fun!