The Task
I was feeling pretty down after we had to close down a project we had been working on for months and wanted to take a small break from smart contracts to collect my thoughts about what we did wrong. Around the same time, I was digging around Github Copilot and OpenAIās GPT3. They released ChatGPT right around then, and - like most people - I was blown away by its abilities. The new Large Language Model was inpiring me to build something fun and useful to test its limits. But what?
Krenzx was following some Telegram groups that made trading calls in crypto markets and boasted about their success. Maybe we could make a bot that would execute those calls automatically and make some money?
As someone who had never traded before1, I saw this as the perfect opportunity.
The Test
There were 2 metrics to test the LLMs against:
- How much Google/StackOverflow do I need?
- How fast can I iterate on the product?
āHey, could you write me a Python script that takes in trading calls from a Telegram group and executes them on Binance?ā
The Good
- Assumptions: It made a few reasonable assumptions about (1) the contents of the trading call, (2) using a Telegram Bot and (3) using Spot on Binance.
- Fast Iteration: It churned out some mostly-usable lines of code that encompassed the entire workflow.
- Learning: It would also explain what it was doing. So even though I didnāt know anything about the Telegram or Binance APIs, I picked it up faster than I would otherwise2.
The Bad
- Bugs: The draft code had some bugs and I couldnāt run it directly. I had to ask ChatGPT to tweak some lines and tell it what errors the repl was throwing in order to fix them.
- Tweaking: I also gave it examples of chat messages the group sent and had it tweak the parser. The regex it wrote wasnāt perfect, but perhaps giving it more examples would have helped it write more generalizable code instead of overfitting to an example.
- Domain Knowledge: While ChatGPT greatly reduced my workload, I still had to be careful to ensure the code was bug-free and actually understand what I was doing. As a result, I ended up asking Krenzx and Madneutrino a lot about the futures concepts in order to understand the API, as I couldnāt fully trust the language model.3
-
Infinite Loop: There were also times when ChatGPT got caught up in loops or made incorrect assumptions about the APIs, requiring me to read the documentation and understand the nuances myself. For example, the Telegram API had multiple ways of doing things and the Binance API was total crap4. It also could not deal with multiple versions of SqlAlchemy, and could not code to SQLite spec.
Development and Deployment
- Coded and ran on ReplIt5. It was awesome and has come a long way since I last used it in 2018.
- But soon moved it to a DigitalOcean Droplet6 because Iād have to learn and make some ReplIt specific changes that I did not want to. Plus if I was ever moving this intro production, pretty sure Iād self host.
- Synced the calls from the Telegram channel to a local SQLite DB and stored logs for what the bot was doing on flat files (to debug - and there was a lot of debugging).
- Started off on the spot testnet but hit annoying limits. So we just YOLO-ed 1K of ACTUAL USDT. To our surprise, we did not lose money. Time spent so far was about a couple days of work, and I think we went at least 2x fast. As the code gets more complex returns you get from using LLM drops off.
Ape In
Encouraged by this initial success, we decided to try futures. This is when it all went downhill7. Both Copilot and ChatGPT were kind of useless and I ended up throwing away a lot of the Spot code, rewriting for the futures API and refactoring for maintainability8. To be fair, Copilot did act sometimes as a delightful auto-complete. This part of the project took 4-5 days of work, and mostly unaided by LLMs. Hereās a prime example of what the current version of ChatGPT/Codex would never be able to figure out: https://dev.binance.vision/t/how-to-implement-otoco-tp-sl-orders-using-api/1622/18
Future of the Project
Looking to the future, there were 2 potential directions for this side project:
- Build a BotAsAService. Anyone who wanted to copy trade a telegram channel could give it to us and weād run it for them for a flat daily fee. One interesting twist on this would be to integrate the LLM into the onboarding flow so it could talk to the user, write a parser, do some testing and deploy it to production all by itself (think @levelsIO style business for software consultancy, but where the typical onboarding personnel/tier3Dev would be replaced by an AI).
- Ramp up the risk and leverage by sending in a larger amount of money (say 10k USDT), and see where it takes us with a 10x leverage, take profits and stoplosses. While I was tempted, we ultimately didnāt have the motivation to risk such a large amount of money without proper backtesting (and I was too lazy to backtest properly). Similarly, we didnāt have the motivation to develop our own trading algorithm, as it felt like a PvP game that wasnāt adding much value to the world (spoiler alert: thatās the followup post!)
The money in either the directions did not motivate us, and we found alternative BotsAsAService solutions that seemed to have a reasonable UX. Just make sure youāre not using 3commas
But if youāre interested in tinkering with the bot yourself, or going ape shit with leverage, you can find the source code here.
Just be careful and donāt use more money than youāre willing to lose!
Future of Development
Looking back at the metrics we set earlier, we can safely say that LLMs reduced the amount of Googling/StackOverflow I needed by a dramatic amount at the start of the project. It also got me up and running fast with v1, but further iterations were much slower owing to (1) bugs and (2) its inability to access to internet.
Hereās someone on twitter saying the same thing, but on a thread:
I spent Christmas programming with ChatGPT.
ā Dan Shipper š§ (@danshipper) January 2, 2023
Some observations:
Both of those problems should be solvable by:
- Scaling up the AI so it can store more context about the entire codebase and relevant libraries. Copilotās underlying model is small in comparison to chatGPTās, but itās difficult to send the entire codebase to chatGPT to get better results.
- Hooking the AI up to pieces of infra it can use to simulate its own answer, test it, and fix bugs. This is whatās known as an āAgentic AIā and is a very active area of research. You can see a mind-blowing demo here: https://youtu.be/_3MBQm7GFIM?t=265
- Allowing the AI to search on the internet. This would have solved the versioning problems we saw with Binance, Telegram and SQLAlchemy. Lots of companies are trying to ātake on Googleā like this. You can check some of them out at You.com, PerplexityAI, Metaphor Systems.
Itās pretty clear itās going to change the nature of the job of a software engineer in a good way. Gone are the days of āgoogling stuff is 90%ā of my day, and āI will grind Leetcode until I pass interviewsā. Everyoneās productivity will 10x, when going from idea to execution. So what will matter is coming up with ideas, directing the AI superpower properly (almost like a smart, super-fast intern who doesnāt know what to do), and iterating on the end result till you are satisfied with your creation.
11 ways ChatGPT saves me hours of work every day, and why you'll never outcompete those who use AI effectively.
ā Santiago (@svpino) January 6, 2023
A list for those who write code:
1 of 16
Iām SO excited to see how this plays out.
-
seriously, Iāll struggle to explain the difference between margin and futuresĀ ā©
-
i.e. by reading docs, seeing and trying to replicate easy examplesĀ ā©
-
Yes, I tried asking chatGPT about the finance concepts but it wouldnāt go back and forth with examples, analogies, and step through specific scenarios like my friends would.Ā ā©
-
To be fair, even though ChatGPT made incorrect assumptions about the parameters and responses, Iād made the same assumptions before reading the docs. Or maybe it was a versioning problem yet again?Ā ā©
-
Because why not explore a shit tonne of things at the same time and whine when it doesnāt all work butter smooth - itās 2022 people! Iām an entitled af DevĀ ā©
-
Where I had to configure a minimal Ubuntu droplet with poetry+pyenv - BY HANDĀ ā©
-
apart from the fact that the futures testnet is SWEET in comparison to the rest of the clusterfuck Binance hasĀ ā©
-
e.g. onitoring, debugging, making future changes, db schema etc. ChatGPT also structures python projects in a very throwaway-script manner which is no bueno for productionĀ ā©
Leave a comment