Large Language Module Cyber Security Multi-Choice-Questions Solver Agent [ MCQ-GPT-ROBOT ]

In this I will introduce a simple prompt AI engineering automated agent that we've developed for testing the performance of AI LLM on handling a large volume of cyber security multi-choice exam questions.

# Created:     2023/08/23
# version:     v_0.1.4
# Copyright:   Copyright (c) 2023 LiuYuancheng
# License:     MIT License

Program Design Purpose: Our objective is to develop a simple LLM-AI assistant program capable of efficiently automated processing batches of multi-choice cyber security questions sourced from various formats such as Markdown (md) , URL, HTML, Text (txt), Json and PDF. Leveraging Open-AI ( ChatGPT ), the program will provide answers to these questions, enabling researchers to assess the AI's accuracy in answering queries (such as the prompt engineering), its performance across diverse fields, and conduct comprehensive data analysis.

Important: the project is only developed for research purpose, please don't use in any exam or test.

Introduction

The MCQ-GPT-ROBOT is an automated AI assistant program to help the cyber security researchers to process batches of multi-choice cyber security questions by using LLM AI module. The program will try to convert different format of the questions source to standard question back format text file then get the answers. With different question prompt scenario setting, it can also applied to solve questions under other field. The program is designed to streamline three key functions:

• It parses multi-choice questions from various data sources, enabling the creation of standardized question bank files for further processing, such as training through data normalization.
• In cases where the question sources lack answers, the program leverages Open-AI to obtain the answers, either with or without scenario prompts.
• Additionally, if the question sources do provide answers, the program compares them with the AI-generated answers and calculates the AI's correctness rate.

To facilitate seamless communication with the Open-AI API, the program utilizes the LangChain framework. Furthermore, we provide both console interface and the web interface allow researchers to test their prompts and integrate the bot/function in their program.

Introduction of Program Workflow

The program employs a dual-threaded approach, with one thread dedicated to parsing MCQ source files and the other to solving MCQs, enabling parallel processing of security questions. The system workflow is depicted below:

The MCQ source parser module continuously loads all question source files/URLs specified in the configuration file, which can be selected from the command console interface or uploaded via the web interface. It converts these questions into a standardized format, adjusts the LLM's scenario prompts based on the question type, and then sends them to the question solver module to obtain the AI's solution. If the users have multiple OpenAI-API, they can also config multi-thread with several parser and question solver to increase the processing efficiency.

AI Solution Correctness Rate: Based on our test to applying on 500+ MCQ question, currently for different level difficulty cyber security question (such as CISCO-CCIE, Huawei Certified Network Associate exam, IBM Security QRadar certificate exam ...) , the AI can provide 60% to 80% correctness rate. For the correctness test, please refer to the reference section: AI Answer's Correctness rate for cyber security MCQ question test.

Program Design

As illustrated in the workflow diagram in the introduction section, the processing of MCQ data follows the data flow outlined below:

The program will provide three main modules to finish the steps:

Design of QuestionParser Module

The MCQ question data parser module ( QuestionParser ) conducts data mining and normalization. It extracts content from MCQ source files or URLs, utilizes AI to identify all MCQs, and generates standardized question bank data. Each question is formatted as follows:

Question:< Question string >
A. choice 1
B. choice 2
C. choice 3
D. choice 4

Design of McqDataManager Module

The MCQ data manager ( McqDataManager ) handles data storage and result archiving. It stores questions, AI-generated answers, and formats the results. Since multiple question sources are batch processed, the data manager logs progress (e.g., whether a source file's result has been archived). This ensures that if program execution is interrupted, users can resume without reprocessing the entire source.

Design of llmMcqSolver Module

The Large Language Model MCQ solver ( LLM-MCQSolver ) retrieves questions from the data manager, preloads MCQ scenario prompts to OpenAI, and calls the OpenAI API to obtain answers. It calculates the AI's correctness rate based on the settings. After processing, different formats of question sources are converted to the standardized question bank file format as depicted below:

The question bank file (data archiving) will be built as a text file which follow below format:

Question:< Question string >
A. choice 1
B. choice 2
C. choice 3
D. choice 4
Answer:<correct answer>
AiAns:<Answer gave by OpenAI>

Question:< Question string >
...
AI Answer compare (correct / total) : <correct number> / <total number>
Correctness rate: <>

Question Solution Prompt

Based on our test, the AI will provide a higher problem solving correctness rate if we load a problem solving scenario prompt to the AI before we pass the real MCQ question to llmMcqSolver. For example if we load a problem solving scenario to setup a simple Certified Ethical Hacker exam prompt, in the prompt give AI some background knowledge about the exam and the material for AI to refer to. (As shown the example below)

# Certified Ethical Hacker exam prompt here
CEH_SOL_PROMPT = """You are a helpful assistant who find the answer of the certified Ethical Hacker multi choice questions exam. 

Certified Ethical Hacker provides comprehensive training, handson learning labs, practice cyber ranges for engagement, certification assessments, cybercompetitions, and opportunities for continuous learning into one comprehensive program curated through our new learning framework: 1. Learn 2. Certify 3. Engage 4. Compete.

Please refer to the Certified Ethical Hacker latest(12th version STUDY GUIDE) training material http://eprints.binadarma.ac.id/1000/1/KEAMANAN%20SISTEM%20INFORMASI%20MATERI%201.pdf as the highest priority  guide to solve the questions, if there is any conflict between the command knowledge and the guiding material. 

Just give the correct choice's front indicator 
character or characters (if the question shows you need to choose more than one choice). 
Return choice indicator character in a in a comma separated list. 
"""

Program Setup

Development Environment : python 3.8.2 rc2

Additional Lib/Software Need :

1. OpenAI : https://github.com/openai/openai-python
2. langChain : https://python.langchain.com/docs/get_started/installation
3. Pylib need to install:

pip install unstructured
pip install pdf2image
pip install pdfminer
pip install pdfminer-six
pip install markdown
pip install --upgrade openai
pip install langchain

Program module files list

Program Usage

This section will introduce how to use the Cyber Security AI robot by console interface or the web interface and how to use the API to integrate the bot/function in your program.

Use MCQ-GPT-ROBOT Console Interface

Follow the below steps to use the MCQ-GPT-ROBOT via command interface. The user can use the config file to automated processing batches of multi-choice cyber security question files or use the manual mode to input the parameters.

Step1: Copy the cyber security MCQ source files

Copy the MCQ files ( *.html, *.txt, *.md, *.json , *.pdf ) you want to process and questionContents.json to the a folder in the src folder (such as the questionbank folder ). Add or append the files you want to process in the questionContents.json as below :

"name": "test_question_bank03",
"type": "url",
"src": "https://www.yeahhub.com/certified-ethical-hacker-v10-multiple-choice-questions-answers-part-9/"

The 3 input parameters detail is shown below:

• name : The MCQ question bank file name you want to archive (The result saving file's name).
• type: Mcq source file type ( current support type: html, url, json, md, pdf, txt).
• src: The source file name or URL process under the folder.

You can add multiple MCQ src in the list of the json config file.

Step2: Set the MCQ-GPT-ROBOT execution configuration file

Rename the configuration file template config_template.txt to config.txt and add you OpenAI-API key as below:

# This is the config file template for the module <mcqGptBot.py>
# Setup the parameter with below format (every line follow <key>:<val> format, the
# key cannot be changed):

# set openAI API key
API_KEY: <Yout own OpenAI API key>

# select the AI model apply to the mcq.
AI_MODEL:gpt-4.0-o

# folder name of the question source files, the source folder need to be in the 
# same folder of mcqGptBot.py.
QS_BANK_DIR:questionbank

# The json file which contents the source files information need to process in the 
# question source folder.
QS_CONT_JSON:questionContents.json

# Define the MCQ question AI prompt constant name in <mcqGptPromptRepo.py> will be used, 
# if not defined, will use the default one 'MCQ_TEMPLATE' in the <mcqGPTBotGlobal.py>
MCQ_PROMPT:MCQ_QA_PROMPT

# Define the MCQ solving scenario AI prompt constant name in <mcqGptPromptRepo.py> will 
# be used, if not defined, will use the default one 'SCE_TEMPLATE' in the <mcqGPTBotGlobal.py>
SCE_PROMPT:CCNP_SOL_PROMPT

Step 3: Run the MCQ-GPT-ROBOT to batch process all the MCQ sources

Run program:

python mcqGptBot.py

Auto process mode:

• Select the auto mode based on all parameters from config file by input 0 as shown below in the execution step1

An example to auto process a markdown MCQ file is shown below:

Manual Process mode:

• Step01: Select the auto mode based on all parameters from config file by input 1 as shown below in the execution step1.
• Step02: Select the process mode (get the answer or calculate the correctness rate)
• Step03: Select the MCQ solution prompt.
• Step4.1: Select the question bank source type.
• Step4.2: Input the question bank source file name
• Step4.3: Input the output result file name

An example to manual process a pdf MCQ file is shown below:

Step 4: Check the MCQ solving result or the AI correctness rate

The processed question will be saved in the text question bank file which same name as the name you set in the questionContents.json file. You can refer to the questionbank folder to check the detail. Example:

network-secuirty-quiz-questions-answers.pdf => test_question_bank03.txt

All the result will follow below format:

...
Question: What is the code written for?
A. Buffer Overflow
B. Encryption
C. Denial-of-service (DoS)
D. Bruteforce
Answer: A
AiAns:B
...
AI Answer compare (correct / total) : 4 / 6 Correctness rate : 0.67

Use MCQ-GPT-ROBOT Web Interface

Follow the below steps to use the MCQ-GPT-ROBOT via web interface.

Step 1: Set the MCQ-GPT-ROBOT execution configuration file

Set the configure file OpenAI key and module (same setting as the previous sectiohnUse MCQ-GPT-ROBOT Console Interface setp2 ), change the flask app parameters as shown below (Set the test mode flag TEST_MD to false, if set to true the web will not link to your OpenAI API)

#-----------------------------------------------------------------------------
# Init the Flask app parameters
TEST_MD:False
FLASK_SER_PORT:5000
FLASK_DEBUG_MD:False
FLASK_MULTI_TH:True

Step 2: Run the MCQ-GPT-ROBOT web host program

Run program:

python mcqGptApp.py

Based on the config file port, open browser and type in web interface URL : http://127.0.0.1:5000 , select the Security MCQ Solver from the left guide navigation bar. (As shown below)

Remark: if the log shows MCQ-Solver ready which means the program is connect to Open-AI server correctly.

Step 2.1 Set the robot function mode

Select the mode and press the button "Set MCQ-GPT-Bot mode " button as shown below:

Step 2.2 Upload the question source

Upload the source with different source type :

Step2.3 process the MCQ source

Press the "Start" button to press the MCQ source, when the source process finished, the result will be automated download.

Project GitHub Repo Link: https://github.com/LiuYuancheng/MCQ-GPT-Bot

Thanks for spending time to check the article detail, if you have any question and suggestion or find any program bug, please feel free to message me. Many thanks if you can give some comments and share any of the improvement advice so we can make our work better ~