AI Coding Assistant Comparison

As I already told I am currently testing AI assistants for coding.

There are so many different plugins available for PyCharm that I wanted to find out which ones bring the best results.
Although several of the plugins are based on GPT-3 there are still differences that make some more attractive than others.

I tested these plugins for the comparison:

– AI Coding Assistant
– Amazon Codewhisperer
– Ask Codi
– Bito
– ChatGPT
– Code4Me
– CodeGeeX
– Codeium
– Github Copilot
– Tabnine

To get the same conditions for all of the AIs I just grabbed 14 different coding exercises from PYnative and let all of the plugins complete the exercises.

Let´s first check the results, then I will tell a bit more about some of the different plugins.

Exercise 1

Write a program to create a recursive function to calculate the sum of numbers from 0 to 10.

Expected Output:

55

Results
AI Coding Assistant: Success
Amazon Codewhisperer: Success
Ask Codi: Success
Bito: Success
ChatGPT: Success
Code4Me: Test cancelled, no suggestions
CodeGeeX: Success
Codeium: Success
Github Copilot: Success
Tabnine: Success

Exercise 2

Write a program to create a function show_employee() using the following conditions.

It should accept the employee’s name and salary and display both.
If the salary is missing in the function call then assign default value 9000 to salary

Given:

showEmployee(“Ben”, 12000)
showEmployee(“Jessa”)

Expected output:

Name: Ben salary: 12000
Name: Jessa salary: 9000

Results
AI Coding Assistant: Success
Amazon Codewhisperer: Success
Ask Codi: Cancelled the test
Bito: Success
ChatGPT: Success
CodeGeeX: Success
Codeium: Fail
Github Copilot: Success
Tabnine: Success

Exercise 3

Write a program to create a function show_employee() using the following conditions.

It should accept the employee’s name and salary and display both.
If the salary is missing in the function call then assign default value 9000 to salary

Given:

showEmployee(“Ben”, 12000)
showEmployee(“Jessa”)

Expected output:

Name: Ben salary: 12000
Name: Jessa salary: 9000

Results
AI Coding Assistant: Success
Amazon Codewhisperer: Success
Bito: Success
ChatGPT: Success
CodeGeeX: Success
Codeium: Fail
Github Copilot: Success
Tabnine: Success

Exercise 4

Generate a Python list of all the even numbers between 4 to 30

Expected Output:

Results
AI Coding Assistant: Success, just didn´t print the result but the task itself got solved
Amazon Codewhisperer: Success
Bito: Success
ChatGPT: Success
CodeGeeX: Success, gave even 2 solutions
Codeium: Fail
Github Copilot: Success
Tabnine: Success

Exercise 5

Find the largest item from a given list
x =

Expected Output:

24

Results
AI Coding Assistant: Success
Amazon Codewhisperer: Success
Bito: Response was only the answer but the task itself was solved with it
ChatGPT: Success
CodeGeeX: Success, had a complicated solution ...