Ever since the groundbreaking launch of ChatGPT, the tech industry has eagerly anticipated a standalone AI chatbot that operates locally without reliance on the cloud. Nvidia has now introduced Chat with RTX, a local AI chatbot that enables users to leverage an AI model to sift through their offline data.
This guide will walk you through the setup and usage of Chat with RTX. Please note that this is a demonstration version, so expect some glitches during your interaction with the tool. Nonetheless, this could pave the way for more local AI chatbots and similar tools in the future.
Downloading Chat with RTX
The initial step involves downloading and configuring Chat with RTX, which is slightly more intricate than one might assume. While the process primarily entails running an installer, it is prone to failures, necessitating the fulfillment of specific minimum system requirements.
You will require an RTX 40-series or 30-series GPU with a minimum of 8GB of VRAM, along with 16GB of system RAM, 100GB of disk space, and Windows 11.
Step 1: Obtain the Chat with RTX installer from Nvidia’s official website. The compressed folder size is 35GB, so the download process may take some time.
Step 2: Upon completion of the download, right-click on the folder and choose Extract all.
Unpacking the Chat with RTX Installation Folder
Step 3: Within the folder, you will discover several files and folders. Opt for setup.exe and proceed through the installation process.
Step 4: Before commencing the installation, the installer will prompt you to designate a storage location for Chat with RTX. Ensure that the chosen location has a minimum of 100GB of available disk space, as Chat with RTX downloads the AI models to this location.
Installing Chat with RTX
Step 5: The installation process may take up to 45 minutes to finalize. Do not be alarmed if there are brief periods of apparent inactivity. It might also cause your PC to slow down, particularly while configuring the AI models, so it is advisable to step away momentarily until the installation completes.
Step 6: In the event of installation failure, simply rerun the installer and select the same data storage location as before. The installer will resume from where it paused.
Step 7: Upon successful installation, you will find a shortcut to Chat with RTX on your desktop, and the application will open in a browser window.
Utilizing Chat with RTX with Your Data
The primary appeal of Chat with RTX lies in its compatibility with user-generated data. It utilizes a technique known as retrieval-augmented generation (RAG) to browse through documents and provide answers based on their content. Rather than addressing any query, Chat with RTX excels at responding to specific questions related to a particular dataset.
While Nvidia includes sample data for trial purposes, unlocking the full potential of Chat with RTX necessitates adding your own dataset.
Step 1: Create a dedicated folder to store your dataset. Remember the folder’s location as you will need to direct Chat with RTX to this location. Presently, Chat with RTX supports .txt, .pdf, and .doc file formats.
Step 2: Launch Chat with RTX and click on the pen icon within the Dataset section.
Step 3: Navigate to the folder containing your data and select it.
Step 4: Within Chat with RTX, click on the refresh icon in the Dataset section. This action will regenerate the model based on the new dataset. It is advisable to refresh the model each time you introduce new data to the folder or switch to a different dataset.
Step 5: With your data incorporated, choose the AI model you wish to utilize from the selection available. Chat with RTX offers Llama 2 and Mistral, with the latter being the default option. While experimenting with both is encouraged, Mistral is recommended for newcomers.
Step 6: Subsequently, you can commence posing questions. Nvidia emphasizes that Chat with RTX does not consider context from previous interactions, meaning each response is independent. Specific queries typically yield more accurate results than general ones. Additionally, Nvidia acknowledges that Chat with RTX may occasionally reference incorrect data while formulating responses, so this aspect should be kept in mind.
Step 7: In case Chat with RTX encounters issues and restarting does not resolve them, Nvidia suggests deleting the preferences.json file to rectify the situation. This file is located at C:UsersAppDataLocalNVIDIAChatWithRTXRAGtrt-llm-rag-windows-mainconfigpreferences.json.
Utilizing Chat with RTX with YouTube
Apart from leveraging your personal data, Chat with RTX can also be employed with YouTube videos. The AI model relies on the transcripts of YouTube videos, which imposes certain inherent limitations.
The AI model cannot perceive anything beyond what is transcribed in the video. For instance, inquiring about visual aspects not covered in the transcript is not feasible. Furthermore, the accuracy of YouTube transcripts may vary, potentially affecting the responses provided.
Step 1: Open Chat with RTX and within the Dataset section, select the dropdown menu and opt for YouTube.
Step 2: Paste a link to a YouTube video or playlist in the designated field. Adjacent to this field, you will find an option to specify the maximum number of transcripts to download.
Step 3: Click on the download button beside this field and await the completion of the transcript download process. Once finished, click on the refresh button.
Step 4: Following the transcript generation, you can engage in conversations akin to those with your personal data. Specific questions tend to elicit more precise responses, and when discussing multiple videos, Chat with RTX might misinterpret references if the queries are overly general.
Step 5: Should you wish to switch to a new set of videos, manual deletion of the old transcripts is necessary. A button to access an Explorer window is provided next to the refresh button for this purpose. Navigate to this location to delete the transcripts if you intend to discuss different videos.
