I can't give you a detailed, working solution. I can only give you some hints that might be useful to implement a working solution.

Have you implemented the additonal input fields using the Agerestriction plugin as example how to do it? If yes, you should already have an registration page with your additional questions and the answers given by the users when trying to create a new account would be evaluated in the plugin hook callback function you would have to have modified accordingly (using the workflow of the Agerestriction plugin as example).

This would result in the users either getting their account created when they provided the desired answers or they would be brought back to the registration page to be able to fix their mistakes.

Now if you don't want that they can change their answers as often as necessary it gets a bit more complicated. The Agerestriction plugin would not save any data to keep track of failed registration attempts. But if you want to limit the number of attempts you would have to keep track of failed attempts and therefore save data in the database (e.g. IP address used on failed attempts). The question is if you want to implement this from scratch on your own. One possibility could be to make use of the Spam Login Filter plugin blacklist. I can't tell you how exactly the Spam Login Filter plugin handles this in its code - you would have to study the code yourself. I just assume that there might be already some functions in this plugin that you could use (with the Spam Login Filter plugin installed and enabled) that would help to identify the IP address, add an IP address to the local blacklist and to check if an IP address is already blocked (though in this case the Spam Login Filter plugin would deny account creation already).

In the plugin hook function in your function you would add a counter variable that adds up correct answers. If the counter is above the threshold to allow account created, let the account get created. If the counter is below this threshold but still high enough to allow for a retry, forward to the register page. And if the counter is too low to allow a new try, identify and block the IP address (either by making use of functions offered by Spam Login Filter or by implementing the code of this plugin within your own plugin).