Introduction

A colleague told me that he once worked on a project aimed at porting code from Python 2 to 3. It was a project for the whole team spanning several quarters. People burned out on it. Coincidentally, it was an open-source project that was also not very well covered by tests. Not pleasant.

Personal website

I initially created my personal website in WordPress, then rewrote it in Gatsby in JS. ChatGPT was becoming more popular, so I decided to try porting it to TS. I'll add that I didn't really know TS very well. It's not rocket science, but until then I hadn't written any code in TS. One could say that ChatGPT not only ported my codebase but also taught me TS and did coverage.

First attempts

The first attempts looked quite promising because JS and TS are fundamentally quite similar. Porting a single file was usually not a problem for ChatGPT. So I decided to go further and automate the porting.

Porter

Porter - this is my attempt at creating an automaton for porting an entire repository from programming language X to programming language Y. The source code is available on GitHub, but it's not a working thing. Theoretically, the task seemed simple.

First approach

for file in files:
    ported = prompt_gpt_to_port_file(file)
    file.write(ported)

Let's check if the code works

for file in files:
    ported = prompt_gpt_to_port_file(file)

    test_errors = test(ported)
    if test_errors:
        prompt_gpt_to_fix_errors(ported, test_errors)

    file.write(ported)

Let's check if the code compiles

for file in files:
    ported = prompt_gpt_to_port_file(file)

    compilation_errors = compile(ported)
    if compilation_errors:
        prompt_gpt_to_fix_errors(ported, compilation_errors)

    test_errors = test(ported)
    if test_errors:
        prompt_gpt_to_fix_errors(ported, test_errors)

    file.write(ported)

Sometimes one round of fixes is not enough

for file in files:
    ported = prompt_gpt_to_port_file(file)

    compilation_errors = compile(ported)
    while compilation_errors:
        prompt_gpt_to_fix_errors(ported, compilation_errors)

    test_errors = test(ported)
    while test_errors:
        prompt_gpt_to_fix_errors(ported, test_errors)

    file.write(ported)

Reflection

It seems that we added a lot of logic specific to a particular problem - it's not a general solution for porting from language X to Y. I can imagine that in many cases it's not as simple as just porting files one by one. Sometimes you have to create more files, make changes in some others, etc. However, it seems that ChatGPT should know that. After all, we can ask it how to port a codebase from language X to Y, and it will readily respond and outline a plan of action.

BabyAGI, AutoGPT

These projects are very ambitious, aiming for GPT to prompt itself in a loop, thus solving complex problems by planning and executing actions. When I started using them, it turned out that it wasn't that simple, and the devil is in the details:

It's expensive - GPT3.5 isn't super expensive per request, but the context is too short and doesn't perform as well as GPT4. Running one experiment with GPT4 cost a few dollars, so when I quickly calculated, I couldn't afford such experiments, especially considering it's uncertain if it would work in the end.
It takes a long time - GPT3.5 is fast but doesn't perform well, and GPT4 is much slower, so every test takes a long time.
Good prompting is difficult.
I didn't have access to the GPT-4 API, so I used a friend's API, but that has its drawbacks.

Langchain

The idea of this project is to define tools for GPT, such as:

Code validation tsc --noEmit
Code testing npm run test
Repository cloning git clone ...

This way, we make it easier for GPT not to have to figure everything out from scratch.

There was a minor success. I think that using one of these tools, I managed to make it so that after entering a command like:

python porter.py <link to the repo>

The script ported one file, made a commit, and opened a pull request.

Final solution

After several sleepless nights, ups and downs, I concluded that the problem was too big for me, or that the tools weren't powerful enough yet to write a porter. I decided that doing the task would be more valuable to me than creating a tool that was uncertain if it could be created. Therefore, I relied on the old tried-and-true design pattern of copy-paste. So, I copied each file to ChatGPT, pasted it into the IDE, ran tests, the compiler, and fed it back to ChatGPT if there were errors.

New features

A month ago, a Code Interpreter was added to ChatGPT. A cool thing is that you can throw a zip with code at it, and it can navigate and read files, etc. This made things easier.

Conclusion

Without ChatGPT, I still wouldn't know TS, and my codebase would be much weaker. Full automation of such complex tasks seems still out of reach, but things move so fast in this industry that maybe someone has already done it :p

Image credits

Photo by Esmonde Yong on Unsplash