Gemini AI solves CERMAT Tests with 43 points out of 50

Semi-automatic system vs. Mathematics entrance exams: How did Gemini AI fare on the CERMAT tests (Czech flavor of the S.A.T.) and where did it stumble? Take a look behind the scenes. What is the future of education and learning?

Czech version of this post too (Česká verze článku je zde.)

With a passion for discovery and experimentation, I embarked in my free time on the Kaggle platform on a series of semi-automatic experiments with CERMAT test sets for Grade 9 students (i.e., fifteen-year-olds) taking the JPZ (Joint Entrance Exams), a kind of Czech S.A.T. version. Thanks to this, the entire solution is open, transparent, and can serve didactic purposes or further development by anyone.

Launching the experiments might be complex for some. Therefore, the following summary of interesting findings and my reflections is for everyone. For those more knowledgeable, it will serve as a quick preview to decide whether to delve into them and discover further connections.

I used Google’s large language models Gemini, specifically their current main “workhorse“ Gemini Flash 2.0, and for the most challenging problems, the preliminary pre-release Gemini 2.5 Pro.

Most likely, current models of ChatGPT, Claude, and perhaps Llama 4 could be used just as well today. Maybe something from the DeepSeek series, maybe Mistral. It’s more about what you have readily available.

Both types I used are multimodal models. In addition to text input, they can natively process other types of inputs, which in this case were always images if the task included them as part of it.

Multimodal models have only been in the user mainstream for about the last six months. Image multimodality is, for example, the main new feature of this year’s release of Llama 4, the leading open-source model from Meta (Facebook), about whose introduction about 2 weeks ago I wrote here.

I will primarily discuss several interesting phenomena I encountered while trying to solve the problems automatically.

Then, I will try to consider what it all means.

And finally, there are more detailed instructions and links for those who want to try more.

The Pond

The Flash 2.0 model solved the pond problem correctly in a few seconds. Though, strictly speaking, the diagram might not be necessary; it’s more of an addition. However, the model was neither confused nor intimidated by the image.

Angles, Angles Everywhere

Understanding the image, on the other hand, is crucial for solving this angle problem. The presented result is from Gemini 2.5 Pro Experimental, which also regularly fails on the third sub-question of this task (it should be 140°). Flash 2.0 solves 0 to 2 sub-questions here, with the error occurring variably, perhaps depending on how it conceptually internalizes the situation at that moment.

Failure in All Geometric Constructions

I specifically modified the system for geometry construction tasks (problems 9 and 10). Different internal procedures are used, and I also deployed version 2.5 Pro especially for these tasks.

Unfortunately, it didn’t help.

However, the main difficulty is not with understanding and solving the task, but with final drawing.

The first sub-task here was trivial: construct the angle bisector of the larger angle formed by lines p and q.

At least in this question, it’s evident that the system correctly understood the geometric situation and proposed a construction procedure that would almost certainly lead to success. If, if it could draw like humans.

The system essentially always fails right from the preparatory reconstruction of the input assignment. Here, it at least managed to produce something somewhat similar. Subsequently, however, within its control procedures, it detects significant discrepancies and terminates its operation as futile.

Thus, the models can perceive reality, but their internal representation is not quantified, even when instructed to focus on the most accurate expression of the numerical parameters of the identified shapes.

It is, of course, good to understand that for the possibility of output on a display, which has the character of a raster, i.e., a coordinate system, we demand much more from the system than a human performs when drawing! Solving steps of the type of finding the intersections of two circles can mean solving systems of multiple quadratic equations. But if they are not correctly numerically parameterized from the beginning, it is truly hopeless.

I am strict in my evaluation. The result should be a drawing; there is no drawing, so there are no points. The fact that the model might know how to do it doesn’t count for us, here in the test.

External Interfaces from Models to the World via Users

The example is also interesting due to the used method of reverse function calling. The functions listed above, such as draw_line or draw_point, are not part of the system but were created by me to enable drawing on a prepared graphic grid.

Strictly speaking, the language model itself does not necessarily need to create images as part of itself, but it can use any tools that the user provides through their means, provided that the reverse textual calls are sufficient for their control.

And, of course, if it is capable of conceptually correctly and precisely controlling the given tools. Which it wasn’t here for the final stage of drawing on a raster.

However, it is not excluded in principle that a different interface could be prepared for the model. One that would, for example, be able to determine the parameters of curves from the input image with complete precision, and one that would accept instructions at the level of construction procedure steps. Creating such an interface would be relatively laborious, but it is possible in principle.

Similarly, the user’s computer can mediate the model’s contact with the world for any other functions as well. Most commonly, it is used for mediating internet searches. Thus, the language model does not have to be connected to a search service itself but can search through its user’s computer.

New standards for so-called agent interfaces, especially MCP, have been developing in the last few months. However, this is more of an intermediary layer whose purpose is to address the explosion of mutual connections between MxN systems by converting them to only M+N adjustments. The result will be significantly higher efficiency, but the principle of connecting the model with anything else exists independently.

Iterations and Sequences (Task 16)

Te final problem 16 presents a requirement for students for something “extra.“ The tasks are always so different that it’s impossible to simply prepare for them systematically in advance. The essence of the problem and the method of its solution must always be at least partially devised ad hoc while the end of the test time ticks away.

Problem 16 perfectly separated Flash 2.0 from Pro 2.5. The former consistently and completely failed on it in all sub-questions, while the latter always solved everything completely correctly in both tests (the example above is from it), albeit slowly.

Personally, I would note that I would tend to sketch out situations in various ways rather than describe them linearly, as is the case especially in 16.3. However, drawing is not a strong suit of common language models, even if the large ones; they probably prefer to write even for themselves.

Model Performance in Summary

Gemini Flash 2.0 showed partially irregular results. It was able to solve even identical questions correctly sometimes, incorrectly sometimes, and partially sometimes. Its best performance reached up to 40 points, the worst only 32. However, it only needed less than 50 seconds for this result.

Gemini Pro 2.5: I used various sub-versions of it, depending on what the API allowed at the time. It stumbled partially (but reproducibly) on tasks 7 and 11 in the first test, and 12 in the second. In addition, it always failed on construction geometry, for the reasons already mentioned above. Therefore, I provisionally gave up on tasks 9 and 10 and deducted 5-6 points. Nevertheless, it repeatedly managed to score 42-43 points out of 50 on the test. This is a result that could be sufficient for admission to a highly sought-after secondary school even in Prague, although not necessarily for the most sought-after ones. Depending on the 2.5 sub-version used, the test took 5-7 minutes.

All these mentioned results were produced using the original Czech version of the tests. The evaluation on the English rendition, though less thorough, indicates roughly the same performance, perhaps with Flash 2.0 being more inclined towards its upper range.

Tuning etc.

If anyone wanted to get involved, it’s possible to further improve the performance of the existing models! It might be necessary to have each task answered several times, and if the results differ, to have the entire situation examined in some other, more thorough way. The approaches and prompts I used were mainly designed for my rapid development. Thus, for example, they are only very minimally self-correcting. So, there is definitely room for increasing accuracy, although probably at the cost of longer runtime. And if someone had a lot of time, adding the implementation of a completely different interface for construction geometry would then very likely mean that with the 2.5 Pro models, it would be possible to get somewhere around 47-50 points, i.e., almost to perfection.

Everything I state above primarily concerns the mentioned Gemini models. It is likely that many other current large language models will behave similarly. However, it is not excluded that some will be even better, or worse. I do not mention them specifically because I have not tested them.

The task is openly developed and accessible to anyone for further development on Kaggle. Is it easy for you to quickly implement the same with OpenAI models? Take it, try it, and let me know the result :-).

How would Czech language tests turn out? Personally, I predict that the 47-50 mark of perfection can already be reached now. But I could be wrong.

What Does It Mean? Will Testing Become Obsolete?

The purpose of current testing is primarily to sort out excess demand, to rank applicants using at least some objective and fair criteria. This purpose will probably not disappear from one year to the next.

Tests somehow verify students’ abilities to think, to handle certain types of formal tasks. If people are to remain in control of technology, a sufficient number of individuals must know how it works. Therefore, it will still make sense for people to be present and sufficiently mentally trained.

Nevertheless, the situation we are heading towards, namely that this test, and many other tests, will be passed by AI systems at a level of practical perfection and faster, contains a certain paradox. If a machine is capable of automatic solving, a certain aspect of the raison d’être of having knowledge is lost.

I think many activities will change in their nature. They will become “AI aided,“ meaning they will still be performed, but AI will change the quality or quantity of their execution.

When calculators and computers replaced slide rules and tables, manual calculations, the situation was somewhat similar but mostly different. The replaced procedures were purely mechanistic and only in a narrow part of humanly used activities.

How to Integrate AI into Education?

Changes will also come to teaching and learning. AI already enables rapid penetration into new areas for an individual and into a significantly broader range of material than was possible even 2 years ago. We live in a pivotal era.

One early warning at the introduction: by transferring the execution of tasks to AI, there is, of course, a real risk of becoming dumber, or in more elegant wording, of suffering restricted cognitive engagement. At the core of all learning is a certain process, including an internal one. If everything is outsourced somewhere else outside the student, it bypasses them, including the desired outcome.

Children up to a certain age also need external organization and guidance.

Nevertheless, changes will come and will seem almost miraculous. First and foremost, the assistant will be capable of personalization towards the student.

Every child is different, has slightly different life experiences, sometimes didn’t pay attention or didn’t understand something. More professionally, each has a different prior. Much of learning essentially consists of grafting new knowledge onto an earlier foundation. However, it is assumed that everyone has roughly the same foundation, at least the curricula are designed that way.

However, assistants will allow even relatively young children to ask them “Why...?“ or “How...?“, “I don’t understand this and this, explain it to me in some other way...“ for as long as the children are interested. In essence, every child will gain their own private tutor. Historically, only the children of wealthy nobility had something like this. The problem of motivation, of someone to accompany them in this, will certainly remain.

It seems to me that one possibility would be to qualitatively divide education into two forms. In the first, some understandings or matters would still be taught so thoroughly that the student would recall them even if “awakened at midnight.“ Mastering at least a small multiplication table by heart is useful for life and as a prior for much else. Simply put, it is necessary to maintain and develop basic skills: reading comprehension, writing one’s own texts, basic mathematical operations, but also the ability to concentrate. For verifying such knowledge, in-class tests or written assignments will still be useful, but a renaissance of oral examinations is also expected.

However, the second form should, in my opinion, look completely different. I would undertake with children both occasional deep forays far beyond the currently taught field and, from what has been learned, create a rich coverage in breadth. If previously it was possible to have one or two examples for a topic, now there could be ten, twenty, diversely focused. As an example of a breadth-first approach, consider that my page on Kaggle, which deals with Czech tests, also has their complete English version, thus potentially for a worldwide audience. The translations are, of course, machine-generated, but they are sufficiently high quality and accurate. Something like this would have been completely beyond my capacity (time, economic) as a hobby activity even a year ago.

Greater concentration on the process rather than the result is recommended, such as project-based activities. The ability to correctly organize oneself and one’s surroundings for an activity is more important than precisely remembering what the outcome should be. And it will make sense to verify how the student arrived at the solution, what resources they used, how they critically evaluated the information, and how AI was integrated into the whole.

Finally, homework will lose its meaning!

Even with any use of AI, human responsibility for the result will not be lost. If even three out of 100 uses of AI are errors or inaccuracies, it is advisable to have one’s own precise ability to identify mistakes. If, for example, in mathematics education, the correct procedure was the traditional foundation of teaching and the final verification was considered a necessary but annoying relic, with the use of AI for practice, the importance of the latter will significantly increase. And verifying correctness will become generally important, not only in the math field.

AI will significantly help people with disabilities. It will be possible to control it, for example, by voice, and it will also be a powerful tool for them in bringing almost any information.

However, AI will also significantly help teachers. They will be able to use such tools to prepare materials. Not only texts but also the nature of visual, or even interactive, content. 3D models, simulations, or virtual reality will also become accessible to teachers, without them necessarily having to be experts in these areas beforehand.

Instead of the focus being on the subject matter itself, teachers will increasingly become mentors, guides, or facilitators of learning. Of course, it will be different for elementary school students and different for university students. In general, however, educators will focus more on supporting students, developing their knowledge and skills, and meaningful interactions. They will lose their knowledge monopoly and the authority derived from it, but these are already illusory today. Newly, they should have the position of educational specialists capable of influencing the flow of knowledge, skills, and values towards students in the ways mentioned above.

The goal is not to slightly increase productivity or efficiency. I believe that AI will become a multiplier. Of what? In the broadest sense, of children’s internal abilities. Of people. The essence is that people should remain the decisive actors.

Knowledge itself is devaluing before our eyes. But knowledge in action should remain valuable. But in what kind of action and in what overall executed action, with all its constituent parts? For both, people are still and absolutely necessary. Let’s hope they remain so.

–––––––––––––––

Did any of the ideas resonate with you? Do you have your own experiences with using AI in education or other insights into the future of testing? What tasks do you think current AI models would definitely still not be able to handle?

You can express your opinions on the topic in the attached discussion. I will try to occasionally respond if possible.

You can contact me by email via the link on ILAW.

–––––––––––––––

Recommendations for your launch on Kaggle

The Gemini Solving CERMAT Test project is implemented by me on the Kaggle platform. Think of it as a programming sandbox for everyone.

The main way to use such pages is by gradually running individual code cells (in the top menu bar, there is usually a bold right arrow, or a same bold right arrow next to the code cell on its left side) in sequence. The code in each cell always does something, you view the result below the cell, and then continue with the next cell. Even a trained monkey could probably manage this far. The trick is that if something really interests you, you can see exactly how it’s done. You can change the programmed execution commands, and only in the place where it’s needed, without having to program the entire rest of the page. If you hopelessly break the code, you can reload the page in its original state. You can be inspired, and copying is not technically prevented, while legally under the permissive Apache License 2.0.

Kaggle is currently probably the leading service on the market that provides the above services. They primarily have a didactic purpose of presentation to others. But you can also conduct your own completely private experiments there without needing your own computer for them. Another option is to share experimental code in smaller groups, i.e., not publicly. For each notebook-type page, you can also have a different technical environment, as needed. However, once you have a non-trivial amount of code, you will probably leave Kaggle.

If you have already done something on Kaggle, you can probably figure it out. For others, the main necessary technical steps are listed below.

However, it is entirely up to your discretion whether the information provided above in the article is already sufficient for you, or whether you want to try more. There is a lot of more, but it is up to you.

1. Kaggle Account. To run it, I think you need a Kaggle account. If you don’t have one, you need to create one by registering. Please note that not only your email but also your phone number is verified. However, not your credit card. The reason is probably that you are subsequently provided with a certain amount of free computing power and other resources. Within these, the code located on the Kaggle pages is run. Identity verification is therefore a defense against abuse.

2. Copy & Edit. In the upper right part of the project page, you should see a prominent „Copy & Edit“ button:

After clicking it, a complete copy of the page will be made into your own space on Kaggle, and it will open in edit mode.

Subsequently, you can really do anything with the code on the page. Any of your own experiments, without affecting the content of my original project page (or other users who also start with the original content).

3. GOOGLE_API_KEY. The core of the project’s code calls models from Google. From the code on the Kaggle page to Google’s web servers. Such calls are usually charged. However, manufacturers sometimes provide a certain capacity for people to try for free, within the so-called Free-Tier usage, which was also my chosen option and is probably yours too. Again, to know who is being provided with what, calling the service is usually tied to authentication. This is done using an API_KEY.

API_KEY, or “Application Programming Interface Key,“ today usually takes the form of an approximately 40-character string.

Simply imagine it as your login and a special password generated by the system together. If program code calls the service’s server with such a key, the server looks into its database and thus knows exactly who is calling it, and mainly who to bill for the traffic with the language model, if applicable. Or, in this case, not to bill monetarily but to check if the allowed frequency rate for the time period of use is not being exceeded.

Therefore, it is usually advisable to protect your key, not to publish its content anywhere, or even store it in unnecessary places. A certain advantage is that if you become unsure about the security of your key, you can invalidate it and have a new one issued without having to change other details in your account with the service provider.

So, for your Kaggle page to now run against the server, you must 1) obtain the key and 2) insert it into the page. The key will then be stored within your Kaggle account and your version of the Kaggle page. Neither I nor other users will know it, nor does anyone else need to know it!

3.1 "Google AI Studio“ Account: API KEY Generation

The easiest way to obtain an API_KEY is from your Google AI Studio account page. Creating an account here precedes the possibility of creating a key. Unfortunately, the page only allows you to sign in with an existing Google account.

The advantage, however, is that you don’t have to bother with managing a separate account. If you have a Google account, switching to the page is just a matter of choosing the account and clicking to agree to the terms of use. No further verification of your identity and no entering credit card details, etc., is required here.

The main page AI Studio itself provides an interface for interactive communication with language models of your choice, including options for tuning various parameters (e.g., Temperature). One of the more interesting ones is “Grounding with Google Search“; in this case, the called language model will use Google itself for searching to answer your queries. Thus, it can answer you even after the model’s cutoff date or in details that are not usually captured in the core of the models.

All other manufacturers have similar pages; it’s a version for developers with slightly more options.

On the Google AI Studio - API Keys page, in the upper right corner, you then have a large “Create API Key“ button. After clicking it, a smaller dialog appears, with a “Create API key in a new project“ button at the bottom. After clicking it and a notification about the generation, you will see a dialog like the one above (with the API KEY value displayed and a “Copy“ button). By clicking „Copy“, the key will be copied to your clipboard.

3.2 Inserting the API KEY into the Kaggle Page

On your copy of the page in Kaggle, you should now have a drop-down menu at the top. From it, expand the “Add-ons“ drop-down and click on “Secrets“:

A new panel will open on the right, and at the bottom of it, there is an „Add Secret“ button, which you click.

Now, the following dialog titled “Secrets“ will open at the top of the right sidebar.

In the “Label“ field, type exactly “GOOGLE_API_KEY“.

In the “Value“ field, place the cursor and invoke the Paste function. You will see your actual key value and not my asterisks.

Then, click the “Save“ button to save the value.

Finally, in the right sidebar, check if there is a line named GOOGLE_API_KEY that is to be checked:

If so, your page now has this „secret“ connected, i.e., when called in the fourth code cell from the top, the value will be loaded correctly:

...get_secret(“GOOGLE_API_KEY“)

and everything should work as it should. The secret will also no longer be displayed unnecessarily anywhere. It remains part of your account information on Kaggle. If you definitively finish using the page, you can also delete the secret.

––––––––

Application Control

It should be simple. Run the cells step by step with the bold right arrow; Steps 1-5 are one-time preparations, and in I-IV, the actual application logic is run.

1. Install the SDK. This is probably the only cell that occasionally causes problems. The purpose is to upload the specified SDK version to your environment for the cell. Always first place the cursor inside the code cell you want to run. Then, run the cell with the bold right arrow in the top menu bar. See the image:

As you can see here, the system is complaining about some conflicts, sometimes yes, sometimes no. However, the good news is that even such an error message does not necessarily matter.

But always wait until this spinning wheel

next to the cell stops spinning and disappears. This indicates that the cell has finished running, and you can proceed to the next one. Otherwise, the cells may not report anything.

2. Load the packages

It is important whether, after running the second cell, you manage to elicit this message from the system below it:

If the reported version is “GenAI version: 1.7.0“, the software packages have been loaded correctly, and you should have no further problems.

If there is no message, or version 0.8.0 is reported, it’s bad.

You can then try the three-dot icon in the upper right corner and, in the invoked menu, “Restart & Clear Cell Outputs“:

This will give you the option to run both top cells again.

If that doesn’t help, it’s advisable to stop the session by selecting “Stop session“, close all browser windows and the browser itself. This will deactivate the existing environment, and after waiting, say, an hour, you can try the whole procedure again, including the initial “Copy & Edit.“ There is then a good chance that Kaggle has recovered in the meantime. You already have the secret in your account; you just need to enable it in the checkbox for this new page.

3. Load Utility Functions. The cell loads supporting program functions and values. They are absolutely necessary but not central from the perspective of the demonstration.

4. Set up your API Key. This cell loads the API KEY that we have already inserted using the procedure above. It is also a necessary value, but now it’s done very quickly.

5. Automated retry. Used to set a mechanism if a query frequency limit is encountered to retry instead of hanging and repeating manually.

Only the initial cells 1 and 2 take on the order of tens of seconds, while 3-4-5 are instantaneous.

––

Options I – II – III – IV are also very fast, with the exception of waiting for the answer after cell IV when using 2.5 Pro models. These are so-called “thinking“ models that simply take their time.

I. Selection of the Test and its Language. Buttons for selecting the set will appear below the cell; there are two tests to choose from, but only one of them can be selected here a time, and only single language (English or Czech).

I chose the first test, then the language „English,“ and completed with „Confirm Selection.“

II. Loading the Selected Test in the Chosen Language. The cell does what it says. It will announce how many and what files it has processed.

III. Select Question(s) & Display Them. A list of tasks will appear below the cell. From it, you select the ones you want to solve. I only selected task „2“ (but it is possible to select multiple at the same time, it will just be less clear). After pressing „Confirm,“ the wording of the selected tasks, including any images, will be displayed below, exactly as they will subsequently be presented to the model.

So, take a look to check if it is the task that interests you.

If so, run the next cell.

IV. Let Gemini Answer. Contains the core of the entire experiment, the answering by language models.

It is also the most interesting code on the entire page at all. Gemini is called using exactly this code.

Below the cell, the exact wording of the question that Gemini will answer will first be displayed again, and then the wording of the answer.

If you’ve made it this far – voilà !

Want a different question? Change the selection below cell III and press “Confirm“, then run IV again.

Want a different test or a different language? Change the selections below cell I and press “Confirm Selection“, run II, III, IV again.