Geneious Prime runs in a Java Virtual Machine. When this JVM starts, it will be allocated a certain amount of RAM and the program can use less than that but never more. An appropriate amount of RAM based on your total RAM will automatically be allocated initially. To change the amount of memory allocated, go to Tools→Preferences, and in the General tab increase the Max memory available to Geneious Prime setting. On Windows you need to run Geneious Prime as an administrator to change this setting (to do this, right-click on the Geneious icon and choose “Run as administrator”).
You should never allocate the total memory of your computer to Geneious, as you need to leave some RAM available for your operating system. As a general rule of thumb, on a 64-bit machine with up to 8GB of RAM in total you can allocate half the RAM to Geneious. On machines with more than 8GB RAM you should leave 3-4GB spare for your operating system and allocate the rest to Geneious.
To see how much memory Geneious Prime is using, check the Memory Usage Bar under the Sources panel. You may need to turn this on under Tools → Preferences → Appearance and Behavior. Clicking this bar will force a garbage collection and free up memory within the JVM.