- Microsoft confirms at least seven new on-device AI features for Windows 11 24H2.
- These features include Windows Recall, Windows Studio Effects, Live Captions, Super Resolution, Voice Clarity, Cocreator for the Paint app, and Restyle Image for the Photos app.
- This is in addition to various changes coming to Copilot for Windows 11.
Windows 11 24H2 (2024 Update) is set to introduce a slew of new features, with a significant focus on AI. In this guide, I will dive into these AI-centric features, highlighting their importance and how they enhance your experience.
In this new version of the operating system, you can expect at least seven new AI features that will run on-device, including Windows Recall, Windows Studio Effects, Live Captions (real-time translations), Super Resolution, Voice Clarity, Cocreator for the Paint app, and Restyle Image for the Photos app. (These features are exclusive to Copilot+ PCs.)
The new features are possible thanks to the new “Windows Copilot Runtime,” a system component on Windows 11 that integrates more than 40 AI models that run locally on the computer and provides the infrastructure to maintain them.
This also allows developers to integrate their apps with AI functionalities using the new APIs through the Windows Copilot Library.
In addition, the Windows 11 2024 Update will also roll out various significant changes and new capabilities for the Copilot chatbot AI.
Windows 11 AI features for version 24H2
Here’s an overview of the new features powered by AI.
Windows Recall
Windows Recall (or Recall) is a new feature that works as a photographic memory that runs in the background, taking snapshots of everything you do on the computer. Using several AI models operating locally on the device, the feature analyzes and makes every piece of content searchable using natural language, whether it is text or image.
The Recall feature is an exclusive feature for Copilot+ PCs, as it requires the latest Qualcomm Snapdragon X series processor with NPU, 16GB of RAM, and 256GB of SSD.
You will have the choice to turn the feature on or off from the out-of-box experience (OOBE), but will be able to enable or disable Recall manually from the Settings app or Group Policy.
Windows Studio Effects
Windows Studio Effects (or Studio Effects) is a combination of features that uses AI to enhance video calls and audio quality using a Neural Processing Unit (NPU).
Currently, there are four main effects, including automatic framing, background blur, eye contact, voice focus, portrait light, creative filters, and eye contact teleprompter.
- Automatic framing: Keeps you centered in the camera view, even if you move around.
- Background blur: Blurs the background behind you during calls, providing more privacy or hiding clutter.
- Eye contact: Since you might be looking at your screen during a call, this feature makes it appear that you’re maintaining eye contact with other people.
- Voice Focus: This effect reduces background noise from your microphone, making your voice clearer during calls.
It’s important to note that Studio Effects have been part of Windows 11 for some time, but starting with version 24H2, Microsoft plans to make these features more broadly available on Copilot+ PCs.
Live Captions with translation
Although Windows 11 has had the Live Captions feature for a while, it’s now being updated with live translations. Live Captions is a feature designed to help everyone, including people with hearing issues, to understand audio through automatic transcriptions.
As part of the AI integration in the operating system, the feature can also translate conversions from a video or voice-only call to your native language, including videos you watch from YouTube or other sources. Technically, the feature is now capable of translating any audio that passes through the computer in real-time.
The audio can be live or pre-recorded from any app or video. The feature can translate over 40 languages into English even without an internet connection, as it requires an NPU to work.
Voice Clarity
On Windows 11, Voice Clarity is an AI-powered feature that removes background noise, cancels echoes, and reduces real-time reverberation on video or voice-only calls or when recording audio.
In addition, this feature will also benefit PC games with online communication. Apps can toggle “Deep Noise Suppression” for voice-only or generic audio content. Voice Clarity ensures a clear voice in online meetings and smoother online communication.
Voice Clarity works automatically on supported applications with supported hardware that includes NPU. You won’t find an option that says turn on Voice Clarity.
Auto Super Resolution
Auto Super Resolution (SR) is a feature that uses AI to increase game resolution and visual quality, providing a smoother, more efficient experience that scales across hardware, including better frame rates. In other words, Super Resolution uses AI to upscale games and improves frame rates and image quality.
The feature is similar to Nvidia’s DLSS, AMD’s FidelityFX Super Resolution (FSR), and Intel’s Xe Super Sampling (XeSS), which also uses AI to upscale games.
As part of this feature, Microsoft also created the “DirectSR” API in partnership with Nvidia, AMD, and Intel to integrate SR into the next generation of games.
Microsoft will exclusively offer the feature for Copilot+ PCs with the latest Qualcomm Snapdragon X Elite processors, and it will only be available for a select number of games.
Cocreator for Paint
In the Paint app, Microsoft is also adding a new Cocreator feature that scans your drawing instead of creating an AI image for you and helps you create a new artwork, which you can refine, edit, and evolve with your ideas.
This feature uses a diffusion-based algorithm to produce a high-quality image with minimum effort.
Cocreator even includes a setting to make the drawing more literal or more expressive.
Cocreator is an AI feature exclusive for Copilot+ PCs, but it shouldn’t be confused with “Image Creator,” which is also an AI feature that uses the cloud to convert a text prompt into an image.
Restyle Image for Photos
In the Photos app, as part of Windows 11 24H2 and exclusive to the Copilot+ PCs, Microsoft is adding a new “Restyle Image” that uses AI with the device’s NPU to change the style of a photo using image generation and photo editing techniques.
Using the feature, you can select a picture, and you can then transform the image with new styles, similar to using filters on your phone. You can even use a text prompt to create a new background.
The Photos app is also getting the same “Image Creator” feature found in the Paint app, which allows you to create AI images with a single text prompt. This feature isn’t exclusive to Copilot+ PCs because it doesn’t require an NPU since the process happens in the cloud with Copilot using the DALL-E language model.
Copilot for Windows 11
Although Copilot has been available on Windows 11 for some time, starting with version 24H2, Microsoft will be making some changes. For example, in this new version, Copilot becomes a traditional app that you can pin or unpin from the Taskbar. You can also resize and move it around like any other app.
The app is actually the web version of Copilot installed as a web app on Windows 11. It works as before, but the difference is that it can no longer change system settings. Also, the interface has been updated with a new design similar to the OpenAI ChatGPT experience on the web.
Furthermore, Microsoft plans to update Copilot soon with OpenAI GPT-4o (or GPT-4 Omni), which is a new powerful language model that understands and processes information across different formats, such as text, images, and even audio.
Copilot is also coming to the Settings app. However, you won’t get a button to summon the chatbot. Instead, you will find “Copilot suggestions” within the settings. For example, if you’re in the System section, you may see a collection of suggestions to fix audio problems, turn on the energy-saving feature, or cast your screen to a wireless display.
On File Explorer, Microsoft plans to bring Copilot to help you get things done in fewer steps. For example, if you are in the Gallery tab, you can hover over a picture and clicking the Copilot button will bring up different actions you can apply to the image file.
Furthermore, Copilot is also coming to Notifications. When you receive a toast notification, Copilot will appear with a suggestion to perform in that specific case. For example, if you’re getting an email with a PowerPoint attachment, the chatbot may suggest summarising the content.
On Windows 11, this also means that you will be able to share your screen with Copilot to get assistance with anything you’re doing.
It’s also important to point out that developers will now start integrating AI into their apps, meaning that starting with the next version of the operating system, we will start seeing more apps with AI capabilities.
Wrapping things up
The operating system will receive a new feature update with some of the new features starting on June 18 for Copilot+ PCs, while the final version of the feature update is scheduled to roll out for everyone sometime in September or October. Once the new version is available to everyone, Microsoft will roll out the remaining features and changes for Copilot+ PCs.
Copilot+ PCs are a new category of computers with AI capabilities. The first wave of devices will feature Qualcomm Snapdragon X series processors with integrated NPUs to handle virtually any AI feature and provide a true-all-day battery life. These devices will come with a minimum of 16GB of memory and 256GB of storage. In the future, it’s expected that Intel and AMD will release processors with NPUs to handle these new experiences.
Some of these new Copilot+ PCs include the Surface Pro (11th Edition), Surface Laptop (7th Edition), Acer Swift 14 AI, HP Omnibook X, HP EliteBook Ultra G1q, Lenovo Yoga Slim 7x, Samsung Galaxy Book4 Edge series, ASUS Vivobook S 15, and Dell laptops like the XPS 13, Inspiron 14, Inspiron 14 Plus, Latitude 5455 and Latitude 7455.