Topic 1.4 Notes – AI-Based Cybersecurity Attacks

What AI-Augmented Cyberattacks Are

In this topic, AI is an attack amplifier. It usually does not invent a brand-new kind of attack. It makes an existing attack more convincing, faster, cheaper, or easier to scale.

A quick way to read any example is this:

• Objective: What does the adversary want?
• AI use: How is AI helping?
• Weakness: What trust gap or vulnerability is being exploited?
• Impact: What harm could happen?
• Mitigation: What defense fits best?

One reminder that shows up all over this topic: trust signals are not proof. A familiar voice, a realistic face, perfect grammar, or a confident chatbot answer can all be fake.

Main Ways Adversaries Use AI

There are six big uses you need to recognize:

• Synthetic impersonation means fake voice, image, or video made from real samples.
• AI-generated phishing means AI writes or improves the scam message.
• Prompt-based extraction means prompts are used to pull sensitive info from an LLM.
• False training information means bad public info is planted so future models may repeat it.
• AI-powered reconnaissance means AI gathers and connects public facts about a target.
• AI-enhanced coding and vulnerability discovery means AI helps write malicious code or find weaknesses.

Synthetic impersonation

AI can clone voices and generate fake images or video from public samples like interviews, livestreams, or social media posts. You’ll hear terms like deepfake and digital avatar.

Common impacts:

• Financial loss from fake urgent payment requests
• Sensitive info disclosure during phone or video scams
• Unauthorized changes to accounts or records
• Reputation damage
• Unauthorized access if a system trusts voice authentication alone

A classic test scenario is a fake executive asking for an urgent wire transfer, like the video-call scam shown here. The attack is still social engineering. AI just makes the impersonation believable.

AI-generated phishing

An LLM can write natural-sounding phishing in any language. That matters because bad grammar used to be a warning sign. Now a phishing email can sound polished and personal.

Keep these separate in your head:

• AI tool writes or improves the message
• Phishing is the social engineering method
• Goal is usually credentials, malware, money, or data

On MCQs, polished writing does not make a message safe.

Prompt-based extraction and false training information

Prompt-based extraction

Adversaries can craft prompts to make an LLM reveal sensitive information. That information may come from:

• User input like prompts, files, transcripts, code, or records
• Training data the model learned from

The main risk is confidentiality loss. Once sensitive data goes into an external AI tool, control over that data may be gone.

False training information

This is about changing what future models may say. The flow below shows how bad information can spread from a public post into model output.

1. False information is posted publicly.
2. Training data collects it.
3. The model learns from it.
4. The model repeats it.
5. A user trusts the output and makes a bad decision.

False training information propagation

That is different from extraction.

• Extraction pulls hidden or retained information out
• Training manipulation pushes false information in

AI-powered reconnaissance and coding help

Reconnaissance

Reconnaissance means gathering target information before or during an attack. AI can scan social media, job posts, websites, news, public records, and code repositories, then connect the pieces.

It can build profiles from:

• names and roles
• email patterns
• vendors and technologies
• schedules and projects
• family details
• voice or video samples

The danger is correlation at scale. Separate public facts become one useful target profile.

Coding and vulnerability discovery

AI can help adversaries write malware, modify code for malicious behavior, or search large code bases for weaknesses. It lowers time and skill barriers, even if the output is sometimes wrong.

A common example is AI flagging unsafe user input handling in a web app, which helps an adversary hunt for exploitable vulnerabilities.

How to Protect Against AI-Augmented Attacks

Independent identity checks

A shared secret is a private word or phrase known only to trusted people. If a “relative” or “boss” calls with an emergency request, that secret gives you identity evidence beyond voice or video. It must not be guessable from social media.

MFA matters when voice authentication is targeted. A cloned voice should not satisfy every requirement. The whole point is independent factors, not two versions of the same factor.

Controlling AI input and verifying AI output

Do not put sensitive information into AI tools, including:

• passwords or access tokens
• personal or financial data
• customer records
• private communications
• source code
• internal security details

Important AI output must be checked against reputable, stable, non-AI-based sources like official records, vendor documentation, government publications, or a previously verified communication channel. This matters most when the result affects money, access, privacy, security settings, or sensitive data.