CS_685_S22

Course: CS685_S22

Title: Special Topics in Computer Science

Subtitle: Natural Language Processing with Deep Neural Networks

Term: Spring 2022

Credit hours: 3

Meeting days/time/location: Tu, Th / 2:00 pm - 3:15 pm / Patterson Hall Room # 129

Instructor Information

Name: Muhammad Abu Bakar Siddique

Email: siddique@cs.uky.edu

Office building and room number: James F. Hardymon Building, Room # 233

Office hours: Tu, Th 3:30 pm - 5:00 pm

Course Description

Natural Language Processing (NLP) has become one of the most important technologies of the modern age. Applications of NLP are everywhere because people communicate almost everything in natural language data. It is the branch of artificial intelligence that enables computers to understand, process, and generate natural language. In recent years, deep neural networks have yielded results that were unimaginable only a few years ago on a wide range of demanding real-world NLP tasks. This course introduces students to cutting-edge research in NLP using deep learning. It will begin by introducing deep neural networks, and then demonstrate how they are applied to build NLP systems such as machine translation, summarization, conversational systems, question answering, stylized text generation, fake news detection, and information extraction. We will not only explore how to use high-level NLP libraries (e.g., Hugging Face) but also examine Lego-like building blocks of the pre-trained language models, such as Transformers, BERT, GPT-3.

Course Prerequisites

Instructor’s consent.

Required Materials

Readings will primarily be drawn from freely available resources, including:

• Dive into Deep Learning by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.

Associated Expenses

Nothing anticipated.

Activities Outside of Regular Class Meetings

Survey Paper presentations may be conducted outside of the regular class meetings, if needed.

Skill and Technology Requirements

Programing skills in python are strongly required. Access to either online or offline version of Latex is also required.

For technical assistance, contact ITS Customer Services 24/7 at 859-218-HELP (4357) for urgent needs. For non-urgent matters, choose the option that works best for you at https://techhelpcenter.uky.edu/gethelp

Student Learning Outcomes

After completing this course, the successful students will be able to:

• Analyze a complex computing problem and to apply principles of computing and other relevant disciplines to identify solutions. (SO1)
• Communicate effectively in a variety of professional contexts. (SO3)

• Describe the fundamental concepts of natural language processing and deep learning.
• Describe the state-of-the-art research in a wide range of topics related to NLP.
• Independently understand a research paper related to an NLP/DL topic, be able to appreciate the technical contributions, criticize the weaknesses, and (potentially) propose new technical ideas.
• Independently explore research papers related to an NLP topic, identify the key papers, summarize their contributions, be able to write a technical survey paper.

Course Details

Tentative Course Schedule

• Basics (1 week): Review of foundational mathematics, linear algebra, and probability for deep learning.
• Deep Learning Basics (2 weeks): MLP, Optimization algorithms, Convolutional Neural Networks.
• NLP (2 weeks): Recurrent Neural Networks, Attention, Transformers.
• Research Papers Presentations (8 weeks): One paper presentation per student.
• Survey Paper Presentation (2 weeks): One presentation per group.

Course Activities and Exams

• Paper Presentation + QA (40 points): Students will present a research paper and answer questions related to the paper. The students will be provided with list of papers to choose from.
• Survey Paper / Research Project (40 points): Students will survey (or implement a research idea) an NLP/DL topic and present their findings/conclusions in the form of a survey (or technical) paper and presentation. The initial seed papers on the topic will be provided by the instructor. This is a group activity; group size 2-4 preferably.

• Class Participation (20 points): Students are required to actively participate in class by asking questions, initiating discussions, answering questions, and so on. Class participation will also be evaluated by the technical summary submissions of the papers presented in the class.

Engagement Verification: In order to meet federal regulations, the instructor will monitor student participation in this class through class participations. Students whose attendance or participation cannot be determined one time during the first three weeks of the semester may be dropped from the course. If you will be missing a class period or will not be submitting some assignment during that period, it is your responsibility to notify the instructor, even if the absence or missed assignment is not excused under university rules.

Software to be used:

Python >= 3.7, HuggingFace Transformers >=4.1, sentencepiece >= 0.1.96, pytorch >= 1.10.0, scikit-learn >=1.0.2. I can provide an exhaustive list if needed.

Initial resource request if known(core hrs/storage,GPU,CPU):

Mainly, we will require GPU to train deep learning models.

Grading Scale

Sample grading scale for graduate students:

90 – 100% = A

80 – 89% = B

70 – 79% = C

Below 70%= E

Midterm Grades

Not applicable to this course.

Attendance Policy/Acceptable Documentation

Students need to notify the professor of absences prior to class when possible. Senate Rules 5.2.5.2

defines the following as acceptable reasons for excused absences: (a) serious illness, (b) death of family member, (c) University-related trips, (d) major religious holidays, (e) Interviews for full-time job opportunities, and (f) other circumstances found to fit “reasonable cause for nonattendance” by the professor.

Students anticipating an absence for a major religious holiday are responsible for notifying the instructor in writing of anticipated absences due to their observance of such holidays no later than the last day in the semester to add a class. Two weeks prior to the absence is reasonable but should not be given any later. Information regarding major religious holidays may be obtained through the Ombud (859-257-3737, https://www.uky.edu/ombud/absences-excused.

Assignment Policies

Assignment Submissions

Assignments will be submitted via Canvas. Detailed expectations will be provided with each assignment. Any assignment you turn in may be submitted to an electronic database to check for plagiarism.

Returning Assignments to Students

Grades will be posted on Canvas for each submitted assignment.

Late Assignments

I deduct 10% for each late day, up to a maximum of two days per assignment. After four days you will receive zero credit.

• Each late day corresponds to 24 hours or part thereof.
• I round up to the nearest day, so ten minutes late is equivalent to 23 hours late.
• This policy does not apply to presentations and class participations. No late presentations or missed class participations.

Please, also refer to the University Senate Rules at the following URL: https://www.uky.edu/universitysenate/rules-regulations.

Assignments Due during Prep Week

The course does not have a final exam. However, the survey paper submission and its presentations may be conducted outside of the regular class meetings in the prep week, if needed.

Academic Policy Statements

Please find Senate’s Academic Policy Statements on this URL: https://www.uky.edu/universitysenate/acadpolicy.

Academic Offenses (Cheating, Plagiarism, and Falsification or Misuse of Academic Records)

Per University policy, students shall not plagiarize, cheat, or falsify or misuse academic records. The minimum penalty for a first offense is a zero on the assignment on which the offense occurred. If the offense is considered severe or the student has other academic offenses on their record, more serious penalties, up to suspension from the University may be imposed. A plea of ignorance is not acceptable as a defense against the charge of academic dishonesty.

Plagiarism and cheating are serious breaches of academic conduct. Each student is advised to become familiar with the various forms of academic dishonesty as explained in the Code of Student Rights and Responsibilities. Complete information can be found at the following website: http://www.uky.edu/Ombud. It is important that you review this information as all ideas borrowed from others need to be properly credited.

Senate Rules 6.3.1 (see http://www.uky.edu/Faculty/Senate/ for the current set of Senate Rules) states that all academic work, written or otherwise, submitted by students to their instructors or other academic supervisors, is expected to be the result of their own thought, research, or self-expression. In cases where students feel unsure about a question of plagiarism involving their work, they are obliged to consult their instructors on the matter before submission.

When students submit work purporting to be their own, but which in any way borrows ideas, organization, wording, or content from another source without appropriate acknowledgment of the fact, the students are guilty of plagiarism.

Plagiarism includes reproducing someone else's work (including, but not limited to a published article, a book, a website, computer code, or a paper from a friend) without clear attribution. Plagiarism also includes the practice of employing or allowing another person to alter or revise the work, which a student submits as his/her own, whoever that other person may be. Students may discuss assignments among themselves or with an instructor or tutor, but when the actual work is done, it must be done by the student, and the student alone.

When a student's assignment involves research in outside sources or information, the student must carefully acknowledge exactly what, where and how he/she has employed them. If the words of someone else are used, the student must put quotation marks around the passage in question and add an appropriate indication of its origin. Making simple changes while leaving the organization, content, and phraseology intact is plagiaristic. However, nothing in these Rules shall apply to those ideas, which are so generally and freely circulated as to be a part of the public domain.

Moreover, please refer to the Rules Regarding Academic Offenses at the following URL: (https://www.uky.edu/universitysenate/ao.

Resources

Accommodations due to disability: If you have a documented disability that requires academic accommodations, please see me as soon as possible during scheduled office hours. In order to receive accommodations in this course, you must provide me with a Letter of Accommodation from the Disability Resource Center (DRC). The DRC coordinates campus disability services available to students with disabilities. It is located on the corner of Rose Street and Huguelet Drive in the Multidisciplinary Science Building, Suite 407. You can reach them via phone at (859) 257-2754 and via email at drc@uky.edu. Their web address is http://www.uky.edu/DisabilityResourceCenter.

Mental Health and Wellness: University life can be a demanding and stressful experience. Please know there are numerous resources offered by the counseling center to support student growth and assist students with mental health, academic and/ or other personal concerns that might interfere with academic performance or a sense of personal well-being while at UK.

The University of Kentucky Counseling Center provides same day walk-in crisis appointments (24-hour Crisis Consultation: 859-257-8701) for all students Monday thru Friday between 8 am to 4:30 pm. The Counseling Center (http://www.uky.edu/counselingcenter/) is in Frazee Hall (Room 106).

Moreover, please have a look at the following resources that might be useful:

UK’s Distance Learning Library Services, Tutoring and Coaching Resources, proctoring information, etc. (https://libraries.uky.edu/‌page.php?lweb_id=1020, https://www.uky.edu/studentacademicsupport/free-tutoring-and-coaching-resources)]

Diversity, Equity, and Inclusion

Syllabus Statement on Diversity, Equity, and Inclusion (DEI) can be found at this URL: https://www.uky.‌edu/universitysenate/syllabus-dei.

Student Resources

Please Visit the University Senate’s Resources Available to Students to access that list https://www.uky.edu/universitysenate/student-resources. 

Classroom Behavior Policies

No cellphones, no laptops, and be respectful during dialogue or discussion.

Mask mandate: It is imperative that we continue to follow Covid safety requirements including mask mandates inside buildings including in classrooms. If a student is not compliant, the instructor can ask them to put their mask on or leave the class to go get a mask.  The masks are available at the podium as well. Hopefully, it will not come to this but if a student refuses to comply, they can be asked to leave the classroom and building. Students who continue to be non-compliant may be reported to the Office of Student Conduct.

Course Recordings [if recorded]

The University of Kentucky Code of Student Conduct defines Invasion of Privacy as using electronic or other devices to make a photographic, audio, or video record of any person without their prior knowledge or consent when such a recording is likely to cause injury or distress.

Meetings of this course may be recorded. All video and audio recordings of lecturers and class meetings, provided by the instructors, are for educational use by students in this class only. They are available only through the Canvas shell for this course and are not to be copied, shared, or redistributed.

As addressed in the Code of Student Conduct, students are expected to follow appropriate university policies and maintain the security of the LinkBlue accounts used to access recorded class materials. Recordings may not be reproduced, shared with those not enrolled in the class, or uploaded to other online environments.

If the instructor or a University of Kentucky office plans any other uses for the recordings, beyond this class, students identifiable in the recordings will be notified to request consent prior to such use. In anticipation of such cases, students may be asked to complete an “authorization of use” form by a faculty member.

Video and audio recordings by students are not permitted during the class unless the student has received prior permission from the instructor. Any sharing, distribution, and or uploading of these recordings outside of the parameters of the class is prohibited. Students with specific recording accommodations approved by the Disability Resource Center should present their official documentation to the instructor.

Course Copyright

All original instructor-provided content for this course, which may include handouts, assignments, and lectures, is the intellectual property of the instructor(s). Students enrolled in the course this academic term may use the original instructor-provided content for their learning and completion of course requirements this term, but such content must not be reproduced or sold. Students enrolled in the course this academic term are hereby granted permission to use original instructor-provided content for reasonable educational and professional purposes extending beyond this course and term, such as studying for a comprehensive or qualifying examination in a degree program, preparing for a professional or certification examination, or to assist in fulfilling responsibilities at a job or internship; other uses of original instructor-provided content require written permission from the instructor(s) in advance.