I’ve been following Bitcoin since 2011. I never really believed in it as a currency back then because I thought there was no way soverign nations would allow such a currency to circumvent their own. Additionally, I didn’t feel like it had any intrinsic value. Fast forward three years, Bitcoin hits $1000. People call it overvalued. They offer their expert layman advice and tell everyone they know not to buy Bitcoin. This, in my opinion, is when Bitcoin first made its presence known to software developers. This is when I saw the bots in action on MtGox. To be clear, not my bots, but other peoples bots. How do I know they are bots? Because I’m sure there aren’t thousands of people one-upping certain orders every second, 24 hours a day.
I didn’t think about it too much back then. I was only 17 years old at the time and didn’t have the aptitude for financial markets. As the years went by, my software skills grew. I started writing microservices, deploying SaaS products, Pokemon Go bots, and even worked on my own startup (not listed in my portfolio yet). I worked on several projects relating to artificial intelligence and machine learning. I worked in Silicon Valley, studied abroad and really got to discover what I wanted to do in life. Make cool things.
It is now May 2017. Bitcoin just broke $2000. They are many high volume cryptocurrency exchanges around. This time, I notice even more bots on these exchanges. That’s when it dawned on me….
This scene is from The Dark Knight Rises when Bane robs Gotham’s version of Wall Street. The day trader assumes he wants physical cash so he says “There’s no money for you to rob”. To which Bane responds: “Oh really? Then why are you people here”.
There would not be any bots if there wasn’t any money to be made.
I started up a few VPS servers physically near several exchanges servers and started scraping all the data I could. I initally started with Python, but found it to be too slow to keep up with the amount of data I was ingesting. So I moved on to NodeJS. NodeJS offered lots of performance, but for lots of headache as well. Finally, I decided I’d move to the most flexible and fastest language for this task - Golang. I rewrote all my services in Golang and started saving all the data to a PostgreSQL database. This unfortunately did not scale with the amount of data points I was collecting as PostgreSQL did not like consuming lots of data with one node. For various reasons, I decided to move to MongoDB. In order to migrate over to MongoDB completely, I needed to migrate my old data from PostgreSQL. That’s when I found out MongoDB did not like tens of millions of inserts. Since I wanted to avoid replication strategies at this stage of my project, I realized I had to find a better solution. I eventually moved to a Time Series Database called InfluxDB to handle all my scraping needs. As my primary index was time, it was able to handle everything I threw at it.
A few months went by and I’ve amassed a 40GB+ dataset. In the mean time, I’ve been learning about financial markets, technical analysis, day trading, and machine learning. I tried out hundreds of strategies based on financial technical indicators, risk assessment, and sentiment analysis. I wrote several backtesting tools and graphing tools to visualize and verify the strategies.
In fact, without spoiling too much, here’s a peek of my backtesting and visualization frontend I had created for this project. It interacted with my own custom REST API I created specifically to query and process both my raw and transformed data. The data plotted below is of one failed strategy.
My bot made 0.5% returns per week (Sept 2018) in USD after adjusting for volatility through the use of pattern matching, technical analysis, and machine learning. I continuously monitor its performance and signals through Grafana. The bot even tells me what it may do a few minutes ahead of time and what it has done on Slack.
I’m sure you’re curious to know what my setup looks like so I’ll give you a sneak peak. Note: I have not shown any of the graphs or slack notifications that actually give away my strategy or machine learning signals.
Slack notifications from two of my nine microservices that make up my trading pipeline.
Grafana Dashboard which displays indicators as well as latencies from the exchange to processing on my servers. Trade and Candle latencies are in seconds, pipeline durations are in nanoseconds.
