ChatGPT outperforms 85% of the 4m programmers assessed on Python on LinkedIn
The programmer examples
ChatGPT, a revolutionary new language model chatbot developed by OpenAI was released to the public last week. Within just a few days, it has become the talk of the town, amassing a million users within a short span of only 5 days. With its advanced natural language processing capabilities, ChatGPT is able to generate human-like text in real-time and revolutionise the way we use chatbots and search engines.
What is ChatGPT?
ChatGPT is a new chatbot developed by OpenAI. It is a variant of the GPT-3 model which was specially designed to generate human-like text in response to user inputs. ChatGPT is highly customizable and can be fine-tuned to perform specific tasks or to generate text in a particular style or on a specific topic. Some of its abilities include content creation, writing code, and translating languages instantaneously and with high accuracy, essentially helping us do a large portion of our work.
Can ChatGPT replace the workforce?
Given the huge jump in ability as compared to the previous GPT-2 model and the complexity of problems it can handle, one question begets among everyone – can ChatGPT replace our jobs? We decided to put that to the test using one of the most quantifiable test available – passing a coding assessment. The main objective of this test is to assess the level of programming of ChatGPT against humans.
Methodology
It is important to note however, that the test was conducted with a LinkedIn skills assessment, and there are many other more elaborated tools out there that can be used to assess one’s programming skill level. Given that it is an MCQ test as well, little to no critical thinking is involved in the test.
Limitations to the Assessment
We put ChatGPT to the test by putting it through a LinkedIn Python skills assessment. The assessment consists of 15 MCQ questions on Python language syntax and concepts, each with a time limit of 90 seconds. The assessment was conducted in real time with ChatGPT processing the prompts. That is to say, the prompts were given to ChatGPT while the assessment was ongoing, and we only proceeded after ChatGPT had provided an answer. Both the questions and the answer options were keyed into the ChatGPT’s chat box in its entirety and without changes, with the occasional prompt of specifying the language to be in Python 3.5.
Findings
Not only was ChatGPT able to complete all the questions in the assessment within the time limit and pass the assessment, but it was also able to achieve a score that beats 85% of the other 3.9 million programmers who took the assessment.
What does this mean for us and ChatGPT in our jobs?
ChatGPT has proven its ability to understand and write code at a basic level, and beat basic assessment tests that are used to assess a candidate’s programming skills. Given the emphasis of hard skills in the industry today and many companies hiring based on these criteria, it is imperative to assess the way we assess the suitability of a candidate. Assessing only hard skills is no longer the solution if an AI machine like ChatGPT can do better than 85% of 3.9 million people.
Going further, this new version of Open AI has improved so dramatically during the last couple of years that we are now faced with very urgent questions:
- Replacement of jobs by AI might come much faster than anticipated, and will impact white collar jobs, including programming or creative professions such as journalism, marketing or even artists.
- In 2017, Tim Cook said “If I were a French student and I were 10 years old, I think it would be more important for me to learn coding than English”. Is it really the case today ? Machines are on their ways to master hard skills at a much higher level than the average human, what are the skills we should focus on?
- And more generally, how do we make sure that in this new technological world, all have access to opportunities, and AI leads to fewer inequalities, not more?
We released the report on “New Skills in Financial Services” a few days before this research on ChatGPT. The conclusions of the report (focus on a broad set of skills rather than just hard skills, importance of adult education so that they do not fall behind) seem even more urgent now that a computer can do better than most people in high cognitive tasks.
You can explore in further depth OpenAI and its potential by starting our course on the topic: https://bit.ly/3FpgxaD
