Keywords: Feedback Generation(FG),Text to Text Transfer Transformer Model(T5)

Introduction:

Using the T5 model for feedback generation in our educational context represents a significant advancement in leveraging cutting-edge NLP techniques to streamline the feedback process. With a meticulously curated dataset comprising prompts, student responses, and corresponding feedback, we empower the T5 model to understand the intricacies of language and academic concepts, enabling it to generate insightful feedback tailored to each scenario. This comprehensive dataset, consisting of 256 prompts with 500 diverse responses for each, ensures that the T5 model receives ample training examples, enhancing its ability to generalize and produce high-quality feedback across various educational domains.

By harnessing the power of the T5 model, we aim to revolutionize the feedback loop between educators and learners, offering a scalable and efficient solution to provide personalized and constructive feedback. Through the automated generation of feedback based on prompts and student responses, we not only save time and effort but also ensure consistency and relevance in the feedback provided. This approach fosters a supportive learning environment where learners receive timely and actionable insights, ultimately enhancing their academic growth and success 

T5 Model:

T5, or Text-To-Text Transfer Transformer, represents a cutting-edge advancement in natural language processing developed by Google. This model, part of the Transformer family, is uniquely tailored for text generation tasks. T5 operates within a text-to-text framework, treating both inputs and outputs as textual strings, enabling it to tackle a wide array of NLP tasks. Its architecture comprises stacked encoder-decoder layers, each equipped with self-attention mechanisms to efficiently capture contextual relationships within input sequences. Despite variations in size, from small to large, T5's smaller variants maintain impressive capabilities in tasks like text generation, summarization, translation, and question answering, making them ideal for deployment in resource-constrained environments without sacrificing performance.

In your case, employing T5 small for feedback generation based on prompts and responses from your dataset is a promising strategy. By fine-tuning the model on your custom dataset, T5 can learn to generate high-quality feedback aligned with given prompts and responses. Leveraging T5's text-to-text framework, the model seamlessly transforms input prompts and responses into coherent and contextually relevant feedback. This approach streamlines feedback generation processes, ensuring consistency and efficiency across a vast volume of responses, thereby offering a potent solution for automating feedback provision.

 Figure 1 : Architecture of T5 model

Use cases and applications:

1. Customer Support and Service: T5 can generate automated responses to customer queries, improving response time and reducing the workload on support agents. It can understand diverse customer inquiries and provide relevant solutions or guidance, enhancing overall customer satisfaction.

2. Content Creation and Summarization: T5 can assist content creators by generating summaries of lengthy documents or articles. It can also aid in brainstorming ideas, generating headlines, or paraphrasing text, speeding up content creation processes while maintaining quality.

3. Language Translation: T5's ability to understand and generate text in multiple languages makes it valuable for translation tasks. It can translate text from one language to another while preserving the original meaning and context, facilitating communication across language barriers.

4. Educational Assistance: T5 can serve as a virtual tutor or educational assistant by generating explanations, answering questions, or providing feedback on assignments. It can customize learning materials based on students' needs and preferences, offering personalized learning experiences.

5. Legal and Compliance Documentation: T5 can assist legal professionals by generating legal documents, contracts, or compliance reports based on predefined templates and inputs. It can help streamline document preparation processes and ensure consistency and accuracy in legal documentation.

6. Medical and Healthcare Applications: T5 can support healthcare professionals by generating medical reports, summarizing patient data, or providing explanations of complex medical concepts. It can contribute to improving documentation efficiency and knowledge dissemination within the healthcare domain.

7. Data Analysis and Insights: T5 can assist in generating insights from large datasets by summarizing findings, generating reports, or explaining data trends. It can aid decision-making processes by providing actionable information in a concise and understandable format.

8. Creative Writing and Storytelling: T5 can inspire creative writing projects by generating story prompts, character descriptions, or plot summaries. It can serve as a tool for writers to overcome writer's block or explore new storytelling ideas.

Results:

                                   Figure 2: Average Loss for Data used for training(%)

The training loss trends illustrated in the figure 2 provide valuable insights into the training dynamics across different percentages of data utilization, each with a batch size of 4. Initially, we commenced our experimentation by training the model on a modest subset of the dataset, comprising 10% of the available data from each question, for a duration of 5 epochs.

As we incrementally increased the percentage of data utilized in training, stepping up by 10% increments until reaching 100%, we observed notable shifts in the loss trends. Notably, for training scenarios involving 100%, 90%, and 80% of the available data, we attained particularly accurate loss values indicative of the model's improved performance. This trend suggests that increasing the amount of training data enhances the model's ability to learn and generalize patterns effectively.

Figure 3: Comparsion of BLEU and BERT scores

The average BERT similarity score observed in our evaluation is 0.66, while the average BLEU similarity score is 0.0494. These scores provide insights into the performance of our T5 model for FG tasks.

BERT similarity score of 0.66 indicates a relatively high level of resemblance between the feedback generated by our model and the reference feedback provided. BERT embeddings capture semantic similarities between texts, allowing for a nuanced comparison that goes beyond surface-level lexical overlaps. A score of 0.66 suggests that our model successfully captures key semantic elements present in the reference feedback, leading to a strong degree of similarity.

On the other hand, the BLEU similarity score of 0.0494 is notably lower compared to the BERT similarity score. BLEU, being a metric primarily focused on n-gram overlap, tends to emphasize exact matches between generated and reference texts. The low BLEU similarity score indicates that while our model may produce feedback that shares some n-gram overlap with the reference feedback, it may not capture the full semantic richness or diversity present in the reference feedback.

Figure 4 ROUGE scores of proposed model for F1 score

                                   Figure 5 ROUGE scores of proposed model for Precision score

Figure 6 ROUGE scores of proposed model for Precision score

The research commenced by training a T5 model on the FG, followed by meticulous parameter tuning. The model's effectiveness was evaluated during the subsequent testing phase using ROUGE (1) scores, a widely adopted metric in feedback, as outlined in Table 2. ROUGE-1 assesses the overlap of unigrams between the generated and reference summaries, while ROUGE-2 considers bigrams. ROUGE-Measures longest matching sequence of words using LCS. The F1 score, precision and recall offer a balanced perspective on the model's performance, reflecting its overall effectiveness.

In Figure 4,5,6 we visually presented the ROUGE scores alongside average recall, precision, and F1 score. This visualization provided a comprehensive overview of the model's performance across different aspects of FG. By analyzing these metrics, we gained insights into the strengths and areas for improvement of the T5 model in FG.